Secret management

Sometimes accessing data requires that you authenticate to external data sources through JDBC. Instead of directly entering your credentials into a notebook, you can use Databricks secrets to store your credentials and reference them in notebooks and jobs. This article provides an overview of Databricks secrets.

Secrets overview

To configure and use secrets you:

  1. Create a secret scope. A secret scope is collection of secrets identified by a name.

  2. Add secrets to the scope

  3. Assign permissions on the secret scope.

  4. Access secrets using Databricks Utilities, see Secrets utility (dbutils.secrets).

For an end-to-end example of how to use secrets in your workflows, see Tutorial: Create and use a Databricks secret. To use a secret in a Spark configuration property or environment variable, see Use a secret in a Spark configuration property or environment variable.

Warning

Administrators, secret creators, and users granted permission can read Databricks secrets. While Databricks makes an effort to redact secret values that might be displayed in notebooks, it is not possible to prevent such users from reading secrets. See Secret redaction.

Manage secret scopes

A secret scope is collection of secrets identified by a name. Databricks recommends aligning secret scopes to roles or applications rather than individuals.

Secret scopes are stored in an encrypted database owned and managed by Databricks.

After creating a secret scope, you can assign permissions to grant users access to read, write, and manage scret scopes.

Create a Databricks-backed secret scope

This section describes how to create a secret scope using the What is the Databricks CLI? (version 0.205 and above). You can also use the Secrets API.

Secret scope names:

  • Must be unique within a workspace.

  • Must consist of alphanumeric characters, dashes, underscores, @, and periods, and can not exceed 128 characters.

  • Are case insensitive.

Secret scope names are considered non-sensitive and are readable by all users in the workspace.

To create a scope using the Databricks CLI:

databricks secrets create-scope <scope-name>

By default, scopes are created with MANAGE permission for the user who created the scope. After you have created a Databricks-backed secret scope, you can add secrets to it.

List secret scopes

To list the existing scopes in a workspace using the CLI:

databricks secrets list-scopes

You can also list secret scopes using the Secrets API.

Delete a secret scope

Deleting a secret scope deletes all secrets and ACLs applied to the scope. To delete a scope using the CLI, run the following:

databricks secrets delete-scope <scope-name>

You can also delete a secret scope using the Secrets API.

Manage secrets

A secret is a key-value pair that stores sensitive material using a key name that is unique within a secret scope.

This section describes how to create a secret scope using the What is the Databricks CLI? (version 0.205 and above). You can also use the Secrets API. Secret names are case insensitive.

Create a secret

This section describes how to create a secrets using the What is the Databricks CLI? (version 0.205 and above). You can also use the Secrets API. Secret names are case insensitive.

When you create a secret in a Databricks-backed scope, you can specify the secret value in one of three ways:

  • Specify the value as a string using the –string-value flag.

  • Input the secret when prompted interactively (single-line secrets).

  • Pass the secret using standard input (multi-line secrets).

For example:

databricks secrets put-secret --json '{
  "scope": "<scope-name>",
  "key": "<key-name>",
  "string_value": "<secret>"
}'

If you are creating a multi-line secret, you can pass the secret using standard input. For example:

(cat << EOF
this
is
a
multi
line
secret
EOF
) | databricks secrets put-secret <secret_scope> <secret_key>

List secrets

To list secrets in a given scope:

databricks secrets list-secrets <scope-name>

The response displays metadata information about the secrets, such as the secrets’ key names. You use the Secrets utility (dbutils.secrets) in a notebook or job to list this metadata. For example:

dbutils.secrets.list('my-scope')

Read a secret

You create secrets using the REST API or CLI, but you must use the Secrets utility (dbutils.secrets) in a notebook or job to read a secret.

Delete a secret

To delete a secret from a scope with the Databricks CLI:

databricks secrets delete-secret <scope-name> <key-name>

You can also use the Secrets API.

Manage secret scope permissions

By default, the user that creates the secret scopes is granted the MANAGE permission. This allows the scope creator to read secrets in the scope, write secrets to the scope, and manage permissions on the scope.

This section describes how to manage secret access control using the What is the Databricks CLI? (version 0.205 and above). You can also use the Secrets API. For secret permission levels, see Secret ACLs

Grant a user permissions on a secret scope

To grant a user permissions on a secret scope using the Databricks CLI:

databricks secrets put-acl <scope-name> <principal> <permission>

Making a put request for a principal that already has an applied permission overwrites the existing permission level.

The principal field specifies an existing Databricks principal. A user is specified using their email address, a service principal using its applicationId value, and a group using its group name. For more information, see Principal.

View secret scope permissions

To view all secret scope permissions for a given secret scope:

databricks secrets list-acls <scope-name>

To get the secret scope permissions applied to a principal for a given secret scope:

databricks secrets get-acl <scope-name> <principal>

If no ACL exists for the given principal and scope, this request fails.

Delete a secret scope permission

To delete a secret scope permission applied to a principal for a given secret scope:

databricks secrets delete-acl <scope-name> <principal>

Secret redaction

Storing credentials as Databricks secrets makes it easy to protect your credentials when you run notebooks and jobs. However, it is easy to accidentally print a secret to standard output buffers or display the value during variable assignment.

To prevent this, Databricks redacts all secret values that are read using dbutils.secrets.get(). When displayed in notebook cell output, the secret values are replaced with [REDACTED].

For example, if you set a variable to a secret value using dbutils.secrets.get() and then print that variable, that variable is replaced with [REDACTED].

Warning

Secret redaction for notebook cell output applies only to literals. The secret redaction functionality does not prevent deliberate and arbitrary transformations of a secret literal. To ensure the proper control of secrets, you should use access control lists to limit permissions to run commands. This prevents unauthorized access to shared notebook contexts.