This article describes how to manage Databricks clusters, including displaying, editing, starting, terminating, deleting, controlling access, and monitoring performance and logs.
To display the clusters in your workspace, click Compute in the sidebar.
The Compute page displays clusters in two tabs: All-purpose clusters and Job clusters.
At the left side are two columns indicating if the cluster has been pinned and the status of the cluster:
Starting , Terminating
At the far right of the right side of the All-purpose clusters tab is an icon you can use to terminate the cluster.
The All-purpose clusters tab shows the numbers of notebooks attached to the cluster.
30 days after a cluster is terminated, it is permanently deleted. To keep an all-purpose cluster configuration even after a cluster has been terminated for more than 30 days, an administrator can pin the cluster. Up to 100 clusters can be pinned.
You can pin a cluster from the cluster list or the cluster detail page:
To pin or unpin a cluster, click the pin icon to the left of the cluster name.
To pin or unpin a cluster, click the pin icon to the right of the cluster name.
You can also invoke the Pin API endpoint to programmatically pin a cluster.
Sometimes it can be helpful to view your cluster configuration as JSON. This is especially useful when you want to create similar clusters using the Clusters API 2.0. When you view an existing cluster, simply go to the Configuration tab, click JSON in the top right of the tab, copy the JSON, and paste it into your API call. JSON view is ready-only.
You edit a cluster configuration from the cluster detail page. To display the cluster detail page, click the cluster name on the Compute page.
You can also invoke the Edit API endpoint to programmatically edit the cluster.
Notebooks and jobs that were attached to the cluster remain attached after editing.
Libraries installed on the cluster remain installed after editing.
If you edit any attribute of a running cluster (except for the cluster size and permissions), you must restart it. This can disrupt users who are currently using the cluster.
You can edit only running or terminated clusters. You can, however, update permissions for clusters that are not in those states on the cluster details page.
For detailed information about cluster configuration properties you can edit, see Configure clusters.
You can create a new cluster by cloning an existing cluster.
From the cluster list, click the three-button menu and select Clone from the drop down.
From the cluster detail page, click and select Clone from the drop down.
The cluster creation form is opened prepopulated with the cluster configuration. The following attributes from the existing cluster are not included in the clone:
Cluster access control within the Admin Console allows admins and delegated users to give fine-grained cluster access to other users. There are two types of cluster access control:
Cluster creation permission: Admins can choose which users are allowed to create clusters.
Cluster-level permissions: A user who has the Can manage permission for a cluster can configure whether other users can attach to, restart, resize, and manage that cluster from the cluster list or the cluster details page.
From the cluster list, click the kebab menu (/_static/images/clusters/cluster-3-buttons.png) and select Edit Permissions.
From the cluster detail page, click and select Permissions.
To learn how to configure cluster access control and cluster-level permissions, see Cluster access control.
Apart from creating a new cluster, you can also start a previously terminated cluster. This lets you re-create a previously terminated cluster with its original configuration.
You can start a cluster from the cluster list, the cluster detail page, or a notebook.
To start a cluster from the cluster list, click the arrow:
To start a cluster from the cluster detail page, click Start:
To start a cluster from a notebook, click the Connect dropdown above the notebook. There you can select which cluster to attach to your notebook.
You can also invoke the Start API endpoint to programmatically start a cluster.
Databricks identifies a cluster with a unique cluster ID. When you start a terminated cluster, Databricks re-creates the cluster with the same ID, automatically installs all the libraries, and re-attaches the notebooks.
When a job assigned to an existing terminated cluster is scheduled to run or you connect to a terminated cluster from a JDBC/ODBC interface, the cluster is automatically restarted. See Create a job and JDBC connect.
Cluster autostart allows you to configure clusters to autoterminate without requiring manual intervention to restart the clusters for scheduled jobs. Furthermore, you can schedule cluster initialization by scheduling a job to run on a terminated cluster.
If your cluster was created in Databricks platform version 2.70 or earlier, there is no autostart: jobs scheduled to run on terminated clusters will fail.
To save cluster resources, you can terminate a cluster. A terminated cluster cannot run notebooks or jobs, but its configuration is stored so that it can be reused (or—in the case of some types of jobs—autostarted) at a later time. You can manually terminate a cluster or configure the cluster to automatically terminate after a specified period of inactivity. Databricks records information whenever a cluster is terminated. When the number of terminated clusters exceeds 150, the oldest clusters are deleted.
Unless a cluster is pinned, 30 days after the cluster is terminated, it is automatically and permanently deleted.
Terminated clusters appear in the cluster list with a gray circle at the left of the cluster name.
When you run a job on a New Job Cluster (which is usually recommended), the cluster terminates and is unavailable for restarting when the job is complete. On the other hand, if you schedule a job to run on an Existing All-Purpose Cluster that has been terminated, that cluster will autostart.
You can manually terminate a cluster from the cluster list or the cluster detail page.
To terminate a cluster from the cluster list, click the square:
To terminate a cluster from the cluster detail page, click Terminate:
You can also set auto termination for a cluster. During cluster creation, you can specify an inactivity period in minutes after which you want the cluster to terminate. If the difference between the current time and the last command run on the cluster is more than the inactivity period specified, Databricks automatically terminates that cluster.
A cluster is considered inactive when all commands on the cluster, including Spark jobs, Structured Streaming, and JDBC calls, have finished executing.
Clusters do not report activity resulting from the use of DStreams. This means that an autoterminating cluster may be terminated while it is running DStreams. Turn off auto termination for clusters running DStreams or consider using Structured Streaming.
The auto termination feature monitors only Spark jobs, not user-defined local processes. Therefore, if all Spark jobs have completed, a cluster may be terminated even if local processes are running.
Idle clusters continue to accumulate DBU and cloud instance charges during the inactivity period before termination.
You configure automatic termination in the Auto Termination field in the Autopilot Options box on the cluster creation page:
You can opt out of auto termination by clearing the Auto Termination checkbox or by specifying an inactivity period of
Auto termination is best supported in the latest Spark versions. Older Spark versions have known limitations which can result in inaccurate reporting of cluster activity. For example, clusters running JDBC, R, or streaming commands can report a stale activity time that leads to premature cluster termination. Please upgrade to the most recent Spark version to benefit from bug fixes and improvements to auto termination.
Deleting a cluster terminates the cluster and removes its configuration.
You cannot undo this action.
You cannot delete a pinned cluster. In order to delete a pinned cluster, it must first be unpinned by an administrator.
From the cluster list, click the three-button menu and select Delete from the drop down.
From the cluster detail page, click and select Delete from the drop down.
You can also invoke the Permanent delete API endpoint to programmatically delete a cluster.
When you restart a cluster, it gets the latest images for the compute resource containers and the VM hosts. It is particularly important to schedule regular restarts for long-running clusters, which are often used for some applications such as processing streaming data.
It is your responsibility to restart all compute resources regularly to keep the image up-to-date with the latest image version.
If you enable the compliance security profile for your account or your workspace, long-running clusters are automatically restarted after 25 days. Databricks recommends that admins restart clusters before they run for 25 days and do so during a scheduled maintenance window. This reduces the risk of an auto-restart disrupting a scheduled job.
You can restart a cluster in multiple ways:
Use the UI to restart a cluster from the cluster detail page. To display the cluster detail page, click the cluster name on the Compute page. Click Restart.
Use the Clusters API to restart a cluster.
Use the script that Databricks provides that determines how long your clusters have run, and optionally restarts them if they exceed a specified number of days since they were started
Run a script that determines how many days your clusters have been running, and optionally restart them
If you are a workspace admin, you can run a script that determines how long each of your clusters has been running, and optionally restart them if they are older than a specified number of days. Databricks provides this script as a notebook.
The first lines of the script define configuration parameters:
min_age_output: The maximum number of days that a cluster can run. Default is 1.
True, the script restarts clusters with age greater than the number of days specified by
min_age_output. The default is
False, which identifies the long running clusters but does not restart them.
REPLACE_WITH_KEYwith a secret scope and key name. For more details of setting up the secrets, see the notebook.
If you set
True, the script automatically restarts eligible clusters, which can cause active jobs to fail and reset open notebooks. To reduce the risk of disrupting your workspace’s business critical jobs, plan a scheduled maintenance window and be sure to notify workspace users.
You can view detailed information about Spark jobs in the Spark UI, which you can access from the Spark UI tab on the cluster details page.
You can get details about active and terminated clusters.
If you restart a terminated cluster, the Spark UI displays information for the restarted cluster, not the historical information for the terminated cluster.
Databricks provides three kinds of logging of cluster-related activity:
Cluster event logs, which capture cluster lifecycle events, like creation, termination, configuration edits, and so on.
Apache Spark driver and worker logs, which you can use for debugging.
Cluster init-script logs, valuable for debugging init scripts.
This section discusses cluster event logs and driver and worker logs. For details about init-script logs, see Init script logs.
The cluster event log displays important cluster lifecycle events that are triggered manually by user actions or automatically by Databricks. Such events affect the operation of a cluster as a whole and the jobs running in the cluster.
For supported event types, see the REST API ClusterEventType data structure.
Events are stored for 60 days, which is comparable to other data retention times in Databricks.
Click Compute in the sidebar.
Click a cluster name.
Click the Event Log tab.
To filter the events, click the in the Filter by Event Type… field and select one or more event type checkboxes.
Use Select all to make it easier to filter by excluding particular event types.
The direct print and log statements from your notebooks, jobs, and libraries go to the Spark driver logs. These logs have three outputs:
You can access these files from the Driver logs tab on the cluster details page. Click the name of a log file to download it.
To view Spark worker logs, you can use the Spark UI. You can also configure a log delivery location for the cluster. Both worker and cluster logs are delivered to the location you specify.
You can install Datadog agents on cluster nodes to send Datadog metrics to your Datadog account.
You can install Datadog agents on cluster nodes to send Datadog metrics to your Datadog account. The following notebook demonstrates how to install a Datadog agent on a cluster using a cluster-scoped init script.
To install the Datadog agent on all clusters, use a global init script after testing the cluster-scoped init script.