Manage cluster policies
Preview
This feature is in Public Preview.
A cluster policy limits the ability to configure clusters based on a set of rules. The policy rules limit the attributes or attribute values available for cluster creation. Cluster policies have ACLs that limit their use to specific users and groups.
Cluster policies let you:
Limit users to create clusters with prescribed settings.
Simplify the user interface and enable more users to create their own clusters (by fixing and hiding some values).
Control cost by limiting per cluster maximum cost (by setting limits on attributes whose values contribute to hourly price).
For an introduction to cluster policies and configuration recommendations, view the Databricks cluster policies video:
Cluster policy permissions limit which policies a user can select in the Policy drop-down when the user creates a cluster:
A user who has cluster create permission can select the Unrestricted policy and create fully-configurable clusters.
A user who has both cluster create permission and access to cluster policies can select the Unrestricted policy and policies they have access to.
A user that has access to only cluster policies, can select the policies they have access to.
Note
If no policies have been created in the workspace, the Policy drop-down does not display.
Only admin users can create, edit, and delete policies. Admin users also have access to all policies.
This article focuses on managing policies using the UI. You can also use the Cluster Policies API 2.0 to manage policies.
Requirements
Cluster policies require the Databricks Premium Plan.
Enforcement rules
You can express the following types of constraints in policy rules:
Fixed value with disabled control element
Fixed value with control hidden in the UI (value is visible in the JSON view)
Attribute value limited to a set of values (either allow list or block list)
Attribute value matching a given regex
Numeric attribute limited to a certain range
Default value used by the UI with control enabled
Managed cluster attributes
Cluster policies support all cluster attributes controlled with the Clusters API 2.0. The specific type of restrictions supported may vary per field (based on their type and relation to the cluster form UI elements).
In addition, cluster policies support the following synthetic attributes:
A “max DBU-hour” metric, which is the maximum DBUs a cluster can use on an hourly basis. This metric is a direct way to control cost at the individual cluster level.
A limit on the source that creates the cluster: Jobs service (job clusters), Clusters UI, Clusters REST API (all-purpose clusters).
Unmanaged cluster attributes
The following cluster attributes cannot be restricted in a cluster policy:
Libraries, which are handled by Libraries API 2.0. A workaround is to use a custom container or an init script.
Number of clusters created per user (either total nor simultaneously). The scope of a policy is a single cluster, so there is no knowledge of the clusters created by a user.
Cluster permissions (ACLs), which is handled by a separate API.
Define a cluster policy
You define a cluster policy in a JSON policy definition, which you add when you create the cluster policy.
Create a cluster policy
You create a cluster policy using the cluster policies UI or the Cluster Policies API 2.0. To create a cluster policy using the UI:
Click
Compute in the sidebar.
Click the Cluster Policies tab.
Click the Create Policy button.
Name the policy. Policy names are case insensitive.
In the Definition tab, paste a policy definition.
Click Create.
Clone an existing cluster policy
You can create a cluster policy by cloning an existing policy. To clone a cluster policy using the UI:
Click
Compute in the sidebar.
Click the Cluster Policies tab.
Click the name of the policy you want to clone.
Click Clone.
In the next page, all fields are pre-populated with values from the existing policy. Change the values of the fields that you want to modify, then click Create.
Manage cluster policy permissions
By definition admins have permission to all policies. You can manage cluster policy permissions using the cluster policies UI or the Cluster Policy Permissions API.
Add a cluster policy permission
To add a cluster policy permission using the UI:
Click
Compute in the sidebar.
Click the Cluster Policies tab.
Click the Permissions tab.
In the Name column, select a principal.
In the Permission column, select a permission:
Click Add.
Edit a cluster policy using the UI
You edit a cluster policy using the cluster policies UI or the Cluster Policies API 2.0. To edit a cluster policy using the UI:
Click
Compute in the sidebar.
Click the Cluster Policies tab.
Click a policy name.
Click Edit.
In the Definition tab, edit the policy definition.
Click Update.
Delete a cluster policy using the UI
You delete a cluster policy using the cluster policies UI or the Cluster Policies API 2.0. To delete a cluster policy using the UI:
Click
Compute in the sidebar.
Click the Cluster Policies tab.
Click a policy name.
Click Delete.
Click Delete to confirm.
Cluster policy definitions
A cluster policy definition is a collection of individual policy definitions expressed in JSON.
In this section:
Policy definitions
A policy definition is a map between a path string defining an attribute and a limit type. There can only be one limitation per attribute. A path is specific to the type of resource and reflects the resource creation API attribute name. If the resource creation uses nested attributes, the path concatenates the nested attribute names using dots. Attributes that aren’t defined in the policy definition are unlimited when you create a cluster using the policy.
interface Policy {
[path: string]: PolicyElement
}
Policy elements
A policy element specifies one of the supported limit types on a given attribute and optionally a default value. You can specify a default value without defining a limit on the attribute in the policy.
type PolicyElement = FixedPolicy | ForbiddenPolicy | (LimitingPolicyBase & LimitingPolicy);
type LimitingPolicy = AllowlistPolicy | BlocklistPolicy | RegexPolicy | RangePolicy | UnlimitedPolicy;
This section describes the policy types:
Fixed policy
Limit the value to the specified value. For attribute values other than numeric and boolean, the value of the attribute must be represented by or convertible to a string.
Optionally the attribute can be hidden in the UI when the hidden
flag is present and set to true
. A fixed policy cannot specify a default value.
interface FixedPolicy {
type: "fixed";
value: string | number | boolean;
hidden?: boolean;
}
Forbidden policy
For an optional attribute, prevent use of the attribute.
interface ForbiddenPolicy {
type: "forbidden";
}
Limiting policies: common fields
In a limiting policy you can specify two additional fields:
defaultValue
- the value that populates the cluster creation form in the UI.isOptional
- a limiting policy on an attribute makes it required. To make the attribute optional, set theisOptional
field to true.
interface LimitedPolicyBase {
defaultValue?: string | number | boolean;
isOptional?: boolean;
}
Allow list policy
A list of allowed values.
interface AllowlistPolicy {
type: "allowlist";
values: (string | number | boolean)[];
}
Block list policy
The list of disallowed values. Since the values must be exact matches, this policy may not work as expected when the attribute is lenient in how the value is represented (for example allowing leading and trailing spaces).
interface BlocklistPolicy {
type: "blocklist";
values: (string | number | boolean)[];
}
Regex policy
Limits the value to the ones matching the regex. For safety, when matching the regex is always anchored to the beginning and end of the string value.
interface RegexPolicy {
type: "regex";
pattern: string;
}
Range policy
Limits the value to the range specified by the minValue
and maxValue
attributes. The value must be a decimal number.
The numeric limits must be representable as a double floating point value. To indicate lack of a specific limit, you can omit one of minValue
, maxValue
.
interface RangePolicy {
type: "range";
minValue?: number;
maxValue?: number;
}
Unlimited policy
Does not define value limits. You can use this policy type to make attributes required or to set the default value in the UI.
interface UnlimitedPolicy {
type: "unlimited";
}
Example
To require adding the COST_BUCKET
tag:
{
"custom_tags.COST_BUCKET": { "type": "unlimited" }
}
To set default a value for a Spark configuration variable, but also allow omitting (removing) it:
{
"spark_conf.spark.my.conf": { "type": "unlimited", "isOptional": true, "defaultValue": "my_value" }
}
Cluster policy attribute paths
The following table lists the supported cluster policy attribute paths.
Cluster policy virtual attribute paths
Attribute path |
Type |
Description |
---|---|---|
|
number |
Calculated attribute representing (maximum, in case of autoscaling clusters) DBU cost of the cluster including the driver node. For use with range limitation. |
|
string |
Represents the type of cluster that can be created:
Allow or block specified types of clusters to be created from the policy. If the |
Array attributes
You can specify policies for array attributes in two ways:
Generic limitations for all array elements. These limitations use the
*
wildcard symbol in the policy path.Specific limitations for an array element at a specific index. These limitation use a number in the path.
Cluster policy examples
In this section:
General cluster policy
A general purpose cluster policy meant to guide users and restrict some functionality, while requiring tags, restricting the maximum number of instances, and enforcing timeout.
Simple medium-sized policy
Allows users to create a medium-sized cluster with minimal configuration. The only required field at creation time is cluster name; the rest is fixed and hidden.
Job-only policy
Allows users to create job clusters and run jobs using the cluster. Users cannot create an all-purpose cluster using this policy.
High Concurrency passthrough policy
Allows users to create clusters in High Concurrency mode with passthrough enabled by default. This way, users don’t need to set the appropriate Spark parameters manually.
{
"spark_conf.spark.databricks.passthrough.enabled": {
"type": "fixed",
"value": "true"
},
"spark_conf.spark.databricks.repl.allowedLanguages": {
"type": "fixed",
"value": "python,sql"
},
"spark_conf.spark.databricks.cluster.profile": {
"type": "fixed",
"value": "serverless"
},
"spark_conf.spark.databricks.pyspark.enableProcessIsolation": {
"type": "fixed",
"value": "true"
},
"custom_tags.ResourceClass": {
"type": "fixed",
"value": "Serverless"
}
}
External metastore policy
Allows users to create a cluster with an admin-defined metastore already attached. This is useful to allow users to create their own clusters without requiring additional configuration.
{
"spark_conf.spark.hadoop.javax.jdo.option.ConnectionURL": {
"type": "fixed",
"value": "jdbc:sqlserver://<jdbc-url>"
},
"spark_conf.spark.hadoop.javax.jdo.option.ConnectionDriverName": {
"type": "fixed",
"value": "com.microsoft.sqlserver.jdbc.SQLServerDriver"
},
"spark_conf.spark.databricks.delta.preview.enabled": {
"type": "fixed",
"value": "true"
},
"spark_conf.spark.hadoop.javax.jdo.option.ConnectionUserName": {
"type": "fixed",
"value": "<metastore-user>"
},
"spark_conf.spark.hadoop.javax.jdo.option.ConnectionPassword": {
"type": "fixed",
"value": "<metastore-password>"
}
}