Databricks client unified authentication
Databricks client unified authentication centralizes setting up and automating authentication to Databricks. It enables you to configure Databricks authentication once and then use that configuration across multiple Databricks tools and SDKs without further authentication configuration changes.
Participating tools and SDKs
Participating Databricks tools and SDKs include:
The Databricks CLI
All participating tools and SDKs accept special environment variables and Databricks configuration profiles for authentication. The Databricks Terraform provider and the Databricks SDKs for Python, Java, and Go also accept direct configuration of authentication settings within code. For details, see Developer tools for the tool’s or SDK’s documentation.
Default methods for client unified authentication
Whenever a tool or SDK must authenticate to Databricks, it tries the following types of authentication in the following order by default. When the tool or SDK succeeds with the type of authentication it tries, it stops trying to authenticate with the remaining authentication types. To force an SDK to authenticate with a specific authentication type, set the Config
API’s Databricks authentication type field.
Authenticate access to Databricks with a service principal using OAuth (OAuth M2M)
Authenticate access to Databricks with a user account using OAuth (OAuth U2M)
For each authentication type that the participating tool or SDK tries, the tool or SDK tries to find authentication credentials in the following locations, in the following order. When the tool or SDK succeeds in finding authentication credentials that can be used, the tool or SDK stops trying to find authentication credentials in the remaining locations.
Credential-related
Config
API fields (for SDKs). To setConfig
fields, see the SDK’s reference documentation.Credential-related environment variables.
Credential-related fields in the
DEFAULT
configuration profile within the.databrickscfg
file. To set configuration profile fields, see (/dev-tools/auth/config-profiles.md).Any related authentication credentials that are cached by the Google Cloud CLI. See Google Cloud ID authentication.
To provide maximum portability for your code, Databricks recommends that you create a custom configuration profile within the .databrickscfg
file, add the belowrequired fields below for your target Databricks authentication type to the custom configuration profile, and then set the DATABRICKS_CONFIG_PROFILE
environment variable to the name of the custom configuration profile.
Environment variables and fields for client unified authentication
The following tables list the names and descriptions of the supported environment variables and fields for Databricks client unified authentication. In the following tables:
Environment variable, where applicable, is the name of the environment variable.
.databrickscfg
field, where applicable, is the name of the field within a Databricks configuration profiles file or Databricks Terraform configuration. To set.databrickscfg
fields, see Databricks configuration profiles.Terraform field, where applicable, is the name of the field within a Databricks Terraform configuration. To set Databricks Terraform fields, see Authentication in the Databricks Terraform provider documentation.
Config
field is the name of the field within theConfig
API for the specified SDK.
General host, token, and account ID environment variables and fields
Common name |
Description |
Environment variable |
|
|
---|---|---|---|---|
Databricks host |
(String) The Databricks host URL for either the Databricks workspace endpoint or the Databricks accounts endpoint. |
|
|
|
Databricks token |
(String) The Databricks personal access token. |
|
|
|
Databricks account ID |
(String) The Databricks account ID for the
Databricks account endpoint. Only has effect
when the Databricks host is also set to
|
|
|
|
Google Cloud-specific environment variables and fields
Common name |
Description |
Environment variable |
|
|
---|---|---|---|---|
Client ID |
(String) The Databricks service principal’s client ID. |
|
|
|
Client secret |
(String) The Databricks service principal’s client secret. |
|
|
|
Google Cloud service account |
(String) The Google Cloud service account’s e-mail address. |
|
|
|
Google Cloud credentials |
(String) The local path to the Google Cloud service account key file, or the contents of the service account key file in JSON format. |
|
|
|
.databrickscfg-specific environment variables and fields
Use these environment variables or fields to specify non-default settings for .databrickscfg
. See also Databricks configuration profiles.
Common name |
Description |
Environment variable |
Terraform field |
|
---|---|---|---|---|
|
(String) A non-default path to the
|
|
|
|
|
(String) The default named profile to
use, other than |
|
|
|
Authentication type field
Use this environment variable or field to force an SDK to use a specific type of Databricks authentication.
Common name |
Description |
Terraform field |
|
---|---|---|---|
Databricks authentication type |
(String) When multiple authentication attributes are available in the environment, use the authentication type specified by this argument. |
|
|
Supported Databricks authentication type field values include:
oauth-m2m
: Set this value if you are using a Databricks service principal for M2M authentication with OAuth 2.0. For more details, see Authenticate access to Databricks with a service principal using OAuth (OAuth M2M).pat
: Set this value if you are using Databricks personal access tokens. For more details, see Databricks personal access token authentication.databricks-cli
: Set this value if you are using the Databricks CLI with OAuth 2.0. For more details, see Authenticate access to Databricks with a user account using OAuth (OAuth U2M).
google-id
: Set this value if you are authenticating access using a Google ID. For more details, see Google Cloud ID authentication.