Customer-managed keys for encryption

This article provides an overview of customer-managed keys for encryption.

Note

This feature requires the Premium plan.

To configure customer-managed keys for encryption, see Configure customer-managed keys for encryption.

Customer-managed keys for encryption overview

Some services and data support adding a customer-managed key to help protect and control access to encrypted data. You can use the key management service in your cloud to maintain a customer-managed encryption key.

Databricks has two customer-managed key use cases that involve different types of data and locations:

  • Managed services: Data in the Databricks control plane (notebooks, secrets, and Databricks SQL query data).

  • Workspace storage: The two workspace GCS buckets and the GCE Persistent Disk volumes of compute resources.

To configure customer-managed keys for workspace storage, see Configure customer-managed keys for encryption.

Customer-managed keys for managed services

Managed services data in the Databricks control plane is encrypted at rest. You can add a customer-managed key for managed services to help protect and control access to the following types of encrypted data:

To configure customer-managed keys for managed services, see Configure customer-managed keys for encryption.

Customer-managed keys for workspace storage

You can add a customer-managed key for workspace storage to protect and control access to the following types of encrypted data:

  • Your workspace’s two GCS buckets: If you add a workspace storage encryption key, Databricks encrypts the data on the two GCS buckets associated with the Google Cloud project that you specified when you created your workspace. These are sometimes called the workspace’s root GCS buckets. One bucket contains your workspace’s DBFS root, which includes the FileStore area, MLflow Models, and Delta Live Table data in your DBFS root (not DBFS mounts). Another bucket includes your workspace’s system data, which includes job results, Databricks SQL results, notebook revisions, and some other workspace data.

  • Your cluster’s GCE persistent disks: Your workspace storage encryption key is used to encrypt the GCE persistent disks of Databricks Runtime cluster nodes and other compute resources in the compute plane.

Compare customer-managed keys use cases

The following table lists which customer-managed key features are used for which types of data.

Type of data

Location

Which customer-managed key feature to use

Notebook source and metadata

Control plane

Managed services

Personal access tokens (PAT) or other credentials used for Git integration with Databricks Repos

Control plane

Managed services

Secrets stored by the secret manager APIs

Control plane

Managed services

Databricks SQL queries and query history

Control plane

Managed services

The remote GCE Persistent Disk volumes for Databricks Runtime cluster nodes and other compute resources.

Compute plane in your Google Cloud account.

Workspace storage

Customer-accessible DBFS root data

Your workspace’s DBFS root in your workspace’s two GCS buckets buckets Google project. This also includes the FileStore area.

Workspace storage

Job results

Workspace’s two GCS buckets in your Google Cloud account

Workspace storage

Databricks SQL query results

Workspace’s two GCS buckets in your Google Cloud account

Workspace storage

MLflow Models

Workspace’s two GCS buckets in your Google Cloud account

Workspace storage

Delta Live Table

If you use a DBFS path in your DBFS root, this is stored in the workspace’s two GCS buckets in your Google Cloud account. This does not apply to DBFS paths that represent mount points to other data sources.

Workspace storage

Interactive notebook results

By default, when you run a notebook interactively (rather than as a job) results are stored in the control plane for performance with some large results stored in your workspace’s two GCS buckets in your Google Cloud account. You can choose to configure Databricks to store all interactive notebook results in your Google Cloud account.

For partial results in the control plane, use a customer-managed key for managed services. For results in the two GCS buckets, which you can configure for all result storage, use a customer-managed key for workspace storage.