Create a cluster

There are two types of clusters:

• All-Purpose clusters can be shared by multiple users. These are typically used to run notebooks. All-Purpose clusters remain active until you terminate them.
• Job clusters run a job. You create a job cluster when you create a job. Such clusters are terminated automatically after the job is completed.

This article describes how to create an all-purpose cluster. To learn how to create job clusters, see Create a job.

You can also create a cluster using the Clusters API 2.0.

Note

You must have permission to create a cluster. See Configure cluster creation permission.

Use the Create button

The easiest way to create a new cluster is to use the Create button:

1. Click Create in the sidebar and select Cluster from the menu. The Create Cluster page appears.

2. Name and configure the cluster.

   There are many cluster configuration options, which are described in detail in cluster configuration.

   

3. Click the Create Cluster button.

   The cluster Configuration tab displays a spinning progress indicator while the cluster is in a pending state. When the cluster has started and is ready to use, the progress spinner turns into a green circle with a check mark. This indicates that cluster is in the running state, and you can now attach notebooks and start running commands and queries.

Use the cluster UI

1. Click Compute in the sidebar.

2. Click the Create Cluster button.

   

3. Follow steps 2 and 3 in Use the Create button.

Terraform integration

You can manage clusters in a fully automated setup using Databricks Terraform provider and databricks_cluster:

data "databricks_node_type" "smallest" {
  local_disk = true
}

data "databricks_spark_version" "latest_lts" {
  long_term_support = true
}

resource "databricks_cluster" "shared_autoscaling" {
  cluster_name            = "Shared Autoscaling"
  spark_version           = data.databricks_spark_version.latest_lts.id
  node_type_id            = data.databricks_node_type.smallest.id
  autotermination_minutes = 20
  autoscale {
    min_workers = 1
    max_workers = 50
  }
}