Instance Pools API 2.0

The Instance Pools API allows you to create, edit, delete and list instance pools.

An instance pool reduces cluster start and auto-scaling times by maintaining a set of idle, ready-to-use cloud instances. When a cluster attached to a pool needs an instance, it first attempts to allocate one of the pool’s idle instances. If the pool has no idle instances, it expands by allocating a new instance from the instance provider in order to accommodate the cluster’s request. When a cluster releases an instance, it returns to the pool and is free for another cluster to use. Only clusters attached to a pool can use that pool’s idle instances.

Databricks does not charge DBUs while instances are idle in the pool. Instance provider billing does apply. See pricing page.

Requirements

Important

To access Databricks REST APIs, you must authenticate.

Create

Endpoint HTTP Method
2.0/instance-pools/create POST

Create an instance pool. Use the returned instance_pool_id to query the status of the instance pool, which includes the number of instances currently allocated by the instance pool. If you provide the min_idle_instances parameter, instances are provisioned in the background and are ready to use once the idle_count in the InstancePoolStats equals the requested minimum.

Note

Databricks may not be able to acquire some of the requested idle instances due to instance provider limitations or transient network issues. Clusters can still attach to the instance pool, but may not start as quickly.

Example

curl --netrc -X POST \
https://1234567890123456.7.gcp.databricks.com/api/2.0/instance-pools/create \
--data @create-instance-pool.json

create-instance-pool.json:

{
  "instance_pool_name": "my-pool",
  "node_type_id": "n1-highmem-4",
  "min_idle_instances": 10,
  "custom_tags": [
    {
      "key": "my-key",
      "value": "my-value"
    }
  ]
}
{ "instance_pool_id": "1234-567890-fetch12-pool-A3BcdEFg" }

Request structure

Field Name Type Description
instance_pool_name STRING The name of the instance pool. This is required for create and edit operations. It must be unique, non-empty, and less than 100 characters.
min_idle_instances INT32 The minimum number of idle instances maintained by the pool. This is in addition to any instances in use by active clusters.
max_capacity INT32 The maximum number of instances the pool can contain, including both idle instances and ones in use by clusters. Once the maximum capacity is reached, you cannot create new clusters from the pool and existing clusters cannot autoscale up until some instances are made idle in the pool via cluster termination or down-scaling.
node_type_id STRING The node type for the instances in the pool. All clusters attached to the pool inherit this node type and the pool’s idle instances are allocated based on this type. You can retrieve a list of available node types by using the List node types API call.
custom_tags An array of ClusterTag

Additional tags for instance pool resources. Databricks tags all pool resources with these tags in addition to default_tags.

Databricks allows at most 43 custom tags.

Response structure

Field Name Type Description
instance_pool_id STRING The ID of the created instance pool.

Edit

Endpoint HTTP Method
2.0/instance-pools/edit POST

Edit an instance pool. This modifies the configuration of an existing instance pool.

Example

curl --netrc -X POST \
https://1234567890123456.7.gcp.databricks.com/api/2.0/instance-pools/edit \
--data @edit-instance-pool.json

edit-instance-pool.json:

{
  "instance_pool_id": "1234-567890-fetch12-pool-A3BcdEFg",
  "instance_pool_name": "my-edited-pool",
  "node_type_id": "n1-highmem-4",
  "min_idle_instances": 5,
  "max_capacity": 200
}
{}

Request structure

Field Name Type Description
instance_pool_id STRING The ID of the instance pool to edit. This field is required.
instance_pool_name STRING The name of the instance pool. This is required for create and edit operations. It must be unique, non-empty, and less than 100 characters.
min_idle_instances INT32 The minimum number of idle instances maintained by the pool. This is in addition to any instances in use by active clusters.
max_capacity INT32 The maximum number of instances the pool can contain, including both idle instances and ones in use by clusters. Once the maximum capacity is reached, you cannot create new clusters from the pool and existing clusters cannot autoscale up until some instances are made idle in the pool via cluster termination or down-scaling.
node_type_id STRING The node type for the instances in the pool. All clusters attached to the pool inherit this node type and the pool’s idle instances are allocated based on this type. You can retrieve a list of available node types by using the List node types API call.

Delete

Endpoint HTTP Method
2.0/instance-pools/delete POST

Delete an instance pool. This permanently deletes the instance pool. The idle instances in the pool are terminated asynchronously. New clusters cannot attach to the pool. Running clusters attached to the pool continue to run but cannot autoscale up. Terminated clusters attached to the pool will fail to start until they are edited to no longer use the pool.

Example

curl --netrc -X POST \
https://1234567890123456.7.gcp.databricks.com/api/2.0/instance-pools/delete \
--data '{ "instance_pool_id": "1234-567890-fetch12-pool-A3BcdEFg" }'
{}

Request structure

Field Name Type Description
instance_pool_id STRING The ID of the instance pool to delete.

Get

Endpoint HTTP Method
2.0/instance-pools/get GET

Retrieve the information for an instance pool given its identifier.

Example

curl --netrc -X GET \
https://1234567890123456.7.gcp.databricks.com/api/2.0/instance-pools/get \
--data '{ "instance_pool_id": "1234-567890-fetch12-pool-A3BcdEFg" }'
{
  "instance_pool_name": "my-pool",
  "min_idle_instances": 10,
  "max_capacity": 200,
  "node_type_id": "n1-highmem-4",
  "custom_tags": {
    "my-key": "my-value"
  },
  "enable_elastic_disk": false,
  "instance_pool_id": "0504-170131-filth3-pool-wwKOBw2o",
  "default_tags": {
    "Vendor": "Databricks",
    "DatabricksInstancePoolCreatorId": "1096092131808652",
    "DatabricksInstancePoolId": "0504-170131-filth3-pool-wwKOBw2o"
  },
  "state": "ACTIVE",
  "stats": {}
}

Request structure

Field Name Type Description
instance_pool_id STRING The instance pool about which to retrieve information.

Response structure

Field Name Type Description
instance_pool_name STRING The name of the instance pool. This is required for create and edit operations. It must be unique, non-empty, and less than 100 characters.
min_idle_instances INT32 The minimum number of idle instances maintained by the pool. This is in addition to any instances in use by active clusters.
max_capacity INT32 The maximum number of instances the pool can contain, including both idle instances and ones in use by clusters. Once the maximum capacity is reached, you cannot create new clusters from the pool and existing clusters cannot autoscale up until some instances are made idle in the pool via cluster termination or down-scaling.
node_type_id STRING The node type for the instances in the pool. All clusters attached to the pool inherit this node type and the pool’s idle instances are allocated based on this type. You can retrieve a list of available node types by using the List node types API call.
custom_tags An array of ClusterTag  
instance_pool_id STRING The canonical unique identifier for the instance pool.
default_tags An array of ClusterTag

Tags that are added by Databricks regardless of any custom_tags, including:

  • Vendor: Databricks
  • DatabricksInstancePoolCreatorId: <create_user_id>
  • DatabricksInstancePoolId: <instance_pool_id>
state InstancePoolState Current state of the instance pool.
stats InstancePoolStats Statistics about the usage of the instance pool.

List

Endpoint HTTP Method
2.0/instance-pools/list GET

List information for all instance pools.

Example

curl --netrc -X GET \
https://1234567890123456.7.gcp.databricks.com/api/2.0/instance-pools/list
{
  "instance_pools": [
    {
      "instance_pool_name": "my-pool",
      "min_idle_instances": 10,
      "max_capacity": 200,
      "node_type_id": "n1-highmem-4",
      "enable_elastic_disk": false,
      "instance_pool_id": "0504-170131-filth3-pool-wwKOBw2o",
      "default_tags": {
        "Vendor": "Databricks",
        "DatabricksInstancePoolCreatorId": "1096092131808652",
        "DatabricksInstancePoolId": "0504-170131-filth3-pool-wwKOBw2o"
      },
      "state": "ACTIVE",
      "stats": {}
    },
    ...
  ]
}

Response structure

Field Name Type Description
instance_pools An array of InstancePoolAndStats A list of instance pools with their statistics included.

Data structures

InstancePoolState

The state of an instance pool. The current allowable state transitions are:

  • ACTIVE -> DELETED
Name Description
ACTIVE Indicates an instance pool is active. Clusters can attach to it.
DELETED Indicates the instance pool has been deleted and is no longer accessible.

InstancePoolStats

Statistics about the usage of the instance pool.

Field Name Type Description
used_count INT32 Number of active instances that are in use by a cluster.
idle_count INT32 Number of active instances that are not in use by a cluster.
pending_used_count INT32 Number of pending instances that are assigned to a cluster.
pending_idle_count INT32 Number of pending instances that are not assigned to a cluster.

InstancePoolAndStats

Field Name Type Description
instance_pool_name STRING The name of the instance pool. This is required for create and edit operations. It must be unique, non-empty, and less than 100 characters.
min_idle_instances INT32 The minimum number of idle instances maintained by the pool. This is in addition to any instances in use by active clusters.
max_capacity INT32 The maximum number of instances the pool can contain, including both idle instances and ones in use by clusters. Once the maximum capacity is reached, you cannot create new clusters from the pool and existing clusters cannot autoscale up until some instances are made idle in the pool via cluster termination or down-scaling.
node_type_id STRING The node type for the instances in the pool. All clusters attached to the pool inherit this node type and the pool’s idle instances are allocated based on this type. You can retrieve a list of available node types by using the List node types API call.
custom_tags An array of ClusterTag  
enable_elastic_disk BOOL (Not supported in this release) Autoscaling Local Storage: when enabled, the instances in the pool dynamically acquire additional disk space when they are running low on disk space.
instance_pool_id STRING The canonical unique identifier for the instance pool.
default_tags An array of ClusterTag

Tags that are added by Databricks regardless of any custom_tags, including:

  • Vendor: Databricks
  • DatabricksInstancePoolCreatorId: <create_user_id>
  • DatabricksInstancePoolId: <instance_pool_id>
state InstancePoolState Current state of the instance pool.
stats InstancePoolStats Statistics about the usage of the instance pool.