Monitor usage using cluster and pool tags

To monitor cost and accurately attribute Databricks usage to your organization’s business units and teams (for chargebacks, for example), you can tag clusters and pools.

The Databricks billable usage graphs in the account console can aggregate usage by individual tags. The billable usage CSV reports downloaded from the same page also include default and custom tags. Tags also propagate to GKE and GCE labels.

Tagged objects and resources

You can add custom tags for the following objects managed by Databricks:

Object	Tagging interface (UI)	Tagging interface (API)
Pool	Pool UI in the Databricks workspace	Instance Pool API
Cluster	Cluster UI in the Databricks workspace	Clusters API

Databricks adds the following default tags to all pools and clusters:

Pool tag key name	Value
`Vendor`	Constant value: `Databricks`
`DatabricksInstancePoolCreatorId`	Databricks internal ID of the user who created the pool
`DatabricksInstancePoolId`	Databricks internal ID of the pool

Cluster tag key name	Value
`Vendor`	Constant value: `Databricks`
`ClusterId`	Databricks internal ID of the cluster
`ClusterName`	Name of the cluster
`Creator`	Username (email address) of the user who created the cluster

On job clusters, Databricks also applies the following default tags:

Cluster tag key name	Value
`RunName`	Job name
`JobId`	Job ID

On resources used by Databricks SQL, Databricks also applies the following default tag:

Cluster tag key name	Value
`SqlWarehouseId`	Databricks internal ID of the SQL warehouse

Limitations

Tag keys and values can contain only characters from the ISO 8859-1 (latin1) set. Tags containing other characters are ignored.
If you change tag keys names or values, these changes apply only after cluster restart or pool expansion.
The maximum custom number of tags that can propagate to GCE labels is 54.
The maximum length for GCE label keys and values is 63 characters.
Label propagation can be delayed due to GCE API rate limits for the project. You can resolve this by increasing the GCE API rate limits for the Google Cloud project.

How tags propagate to usage reports

Cluster and pool tags propagate in three different ways that you can use for aggregating costs:

Tags in DBU reports: Tags propagate to DBU usage reports in the downloaded reports.
GKE (Kubernetes) labels for each pod: Tags propagate to labels on GKE pods. This allows you to use GKE usage metering to attribute costs for all Databricks compute resources.
GCE labels for each VM and its persistent disks: Tags propagate to labels on GCE resources such as VM and its persistent disks. This allows you to use GCE usage metering to attribute costs, which is more accurate than GKE labels for aggregation of Google Cloud costs for all Databricks compute resources. The tag’s keys and values are transformed to conform with GCE label format limits.

How tags propagate for clusters created from pools

Tags propagate to node instances differently depending on whether or not a cluster was created from a pool.

If a cluster is not created from a pool, its tags propagate as expected to node instances.
If a cluster is created from a pool, its instances inherit both the pool tags and the cluster tags. The pool’s tags are used directly for VM usage data only for the idle VMs.
If there is a tag name conflict, Databricks default tags take precedence over custom tags, and pool tags take precedence over cluster tags.

GCE label limits

GKE labels can directly use the Databricks tag keys and values.

For GCE labels, there are limitations:

Keys and values must consist only of lower case letters, numeric characters, underscores, and dashes.
Maximum length for GCE label keys and values is 63 characters.
Maximum number of tags that can propagate to GCE labels is 54.

To conform to GCE format rules, tags are transformed before becoming GCE label keys and values. If there are duplicates after transformation, the pair of keys and values that appear later (lower) in tag definitions are the ones that persist.

The following table compares GKE and GCE default cluster tags.

GKE label key	GKE label value	GCE label key	GCE label value
`Vendor`	`Databricks`	`vendor`	`databricks`
`ClusterId`	Databricks ID	`clusterId`	Databricks ID
`ClusterName`	Customer-defined name	`clustername`	Customer-defined name in lower case. Characters are removed if they are not letters, numbers, underscores or dashes. For example, `My Cluster` becomes `mycluster`.
`Creator`	Creator user’s email address with `@` replaced with `_at_`. For example `user123_at_databricks.com`.	`creator`	Creator user’s email address with `@` replaced with `_at_`. Letters are lower cased. Characters are removed if they are not letters, numbers, underscores or dashes. For example, `X+Y@databricks.com` becomes `xy_at_databricks.com`.

The following table compares GKE and GCE default instance pool tags:

GKE label key	GKE label value	GCE label key	GCE label value
`DatabricksInstancePoolId`	Databricks ID	`databricksinstancepoolid`	Databricks ID
`DatabricksInstancePoolCreatorId`	Databricks ID	`databricksinstancepoolcreatorid`	Databricks ID

The following table compares GKE and GCE for all other tags (custom tags):

GKE label key	GKE label value	GCE label key	GCE label value
Customer-defined key	Customer-defined value	Customer-defined key in lower case. Characters are removed if they are not letters, numbers, underscores or dashes. For example, `My Key` becomes `mykey`.	Customer-defined value in lower case. Characters are removed if they are not letters, numbers, underscores or dashes. For example, `My.Val` becomes `myval`.