Google Cloud credentials authentication
Google Cloud credentials authentication uses Google Cloud service account credentials to authenticate the target Google Cloud service account.
To prepare to use Google Cloud credentials authentication, see Set up and use Google Cloud credentials authentication.
To configure Google Cloud credentials authentication with Databricks, you must set the following associated environment variables, .databrickscfg
fields, Terraform fields, or Config
fields:
The Databricks host.
For account operations, specify
https://accounts.gcp.databricks.com
.For workspace operations, specify the workspace URL, for example
https://1234567890123456.7.gcp.databricks.com
.
For account operations, the Databricks account ID.
The Google Cloud credentials, specified as the local path to the private key portion of the Google-managed key pair for the Google Cloud service account, or the contents of this private key portion in JSON format.
To perform Google Cloud credentials authentication with Databricks, integrate the following within your code, based on the participating tool or SDK:
To use environment variables for a specific Databricks authentication type with a tool or SDK, see Authenticate access to Databricks resources or the tool’s or SDK’s documentation. See also Environment variables and fields for client unified authentication and the Default methods for client unified authentication.
For account-level operations, set the following environment variables:
DATABRICKS_HOST
, set to the value of your Databricks account console URL,https://accounts.gcp.databricks.com
.DATABRICKS_ACCOUNT_ID
GOOGLE_CREDENTIALS
For workspace-level operations, set the following environment variables:
DATABRICKS_HOST
, set to the value of your Databricks workspace URL, for examplehttps://1234567890123456.7.gcp.databricks.com
.GOOGLE_CREDENTIALS
Create or identify a Databricks configuration profile with the following fields in your .databrickscfg
file. If you create the profile, replace the placeholders with the appropriate values. To use the profile with a tool or SDK, see Authenticate access to Databricks resources or the tool’s or SDK’s documentation. See also Environment variables and fields for client unified authentication and the Default methods for client unified authentication.
For account-level operations, set the following values in your .databrickscfg
file. In this case, the Databricks account console URL is https://accounts.gcp.databricks.com
:
[<some-unique-configuration-profile-name>]
host = <account-console-url>
account_id = <account-id>
google_credentials = <path-to-google-service-account-credentials-file>
For workspace-level operations, set the following values in your .databrickscfg
file. In this case, the host is the Databricks workspace URL, for example https://1234567890123456.7.gcp.databricks.com
:
[<some-unique-configuration-profile-name>]
host = <workspace-url>
google_credentials = <path-to-google-service-account-credentials-file>
For the Databricks CLI, do one of the following:
Set the environment variables as specified in this article’s “Environment” section.
Set the values in your
.databrickscfg
file as specified in this article’s “Profile” section.
Environment variables always take precedence over values in your .databrickscfg
file.
See also Google Cloud credentials authentication.
Note
Google Cloud credentials authentication is supported on the following Databricks Connect versions:
For Python, Databricks Connect for Databricks Runtime 13.3 LTS and above. The Databricks SDK for Python 0.14.0 or above must also be installed.
For Scala, Databricks Connect for Databricks Runtime 13.3 LTS and above. The Databricks SDK for Java 0.14.0 or above must also be installed.
For Databricks Connect, you can do one of the following:
Set the values in your
.databrickscfg
file for Databricks workspace-level operations as specified in this article’s “Profile” section. Also set thecluster_id
environment variable in your profile to your workspace instance URL, for examplehttps://1234567890123456.7.gcp.databricks.com
.Set the environment variables for Databricks workspace-level operations as specified in this article’s “Environment” section. Also set the
DATABRICKS_CLUSTER_ID
environment variable to your workspace instance URL, for examplehttps://1234567890123456.7.gcp.databricks.com
.
Values in your .databrickscfg
file always take precedence over environment variables.
To initialize the Databricks Connect client with these environment variables or values in your .databrickscfg
file, see one of the following:
For Python, see Configure connection properties for Python.
For Scala, see Configure connection properties for Scala.
For the Databricks extension for Visual Studio Code, do the following:
Set the values in your
.databrickscfg
file for Databricks workspace-level operations as specified in this article’s “Profile” section.In the Configuration pane of the Databricks extension for Visual Studio Code, click Configure Databricks.
In the Command Palette, for Databricks Host, enter your workspace URL, for example
https://1234567890123456.7.gcp.databricks.com
, and then pressEnter
.In the Command Palette, select your target profile’s name in the list for your URL.
For more details, see Authentication setup for the Databricks extension for Visual Studio Code.
For account-level operations, for default authentication:
provider "databricks" {
alias = "accounts"
}
For direct configuration (replace the retrieve
placeholders with your own implementation to retrieve the values from the console or some other configuration store, such as HashiCorp Vault. See also Vault Provider). In this case, the Databricks account console URL is https://accounts.gcp.databricks.com
:
provider "databricks" {
alias = "accounts"
host = <retrieve-account-console-url>
account_id = <retrieve-account-id>
google_credentials = <retrieve-google-credentials>
}
For workspace-level operations, for default authentication:
provider "databricks" {
alias = "workspace"
}
For direct configuration (replace the retrieve
placeholders with your own implementation to retrieve the values from the console or some other configuration store, such as HashiCorp Vault. See also Vault Provider). In this case, the host is the Databricks workspace URL, for example https://1234567890123456.7.gcp.databricks.com
:
provider "databricks" {
alias = "workspace"
host = <retrieve-workspace-url>
google_credentials = <retrieve-google-credentials>
}
For more information about authenticating with the Databricks Terraform provider, see Authentication.
For account-level operations, use default authentication:
from databricks.sdk import AccountClient
a = AccountClient()
# ...
For direct configuration, replace the retrieve
placeholders with your implementation to retrieve the values from the console or some other configuration store, such as Google Cloud Secret Manager. In this case, the Databricks account console URL is https://accounts.gcp.databricks.com
:
from databricks.sdk import AccountClient
a = AccountClient(
host = retrieve_account_console_url(),
account_id = retrieve_account_id(),
google_credentials = retrieve_google_credentials()
)
# ...
For workspace-level operations, use default authentication:
from databricks.sdk import WorkspaceClient
w = WorkspaceClient()
# ...
For direct configuration, replace the retrieve
placeholders with your implementation to retrieve the values from the console or some other configuration store, such as Google Cloud Secret Manager. In this case, the host is the Databricks workspace URL, for example https://1234567890123456.7.gcp.databricks.com
:
from databricks.sdk import WorkspaceClient
w = WorkspaceClient(
host = retrieve_workspace_url(),
google_credentials = retrieve_google_credentials()
)
# ...
For more information about authenticating with Databricks Python tools and SDKs that implement Databricks client unified authentication, see:
For account-level operations, use default authentication:
import com.databricks.sdk.AccountClient;
// ...
AccountClient a = new AccountClient();
// ...
For direct configuration, replace the retrieve
placeholders with your implementation to retrieve the values from the console or some other configuration store, such as Google Cloud Secret Manager. In this case, the Databricks account console URL is https://accounts.gcp.databricks.com
:
import com.databricks.sdk.AccountClient;
import com.databricks.sdk.core.DatabricksConfig;
// ...
DatabricksConfig cfg = new DatabricksConfig()
.setHost(retrieveAccountConsoleUrl())
.setAccountId(retrieveAccountId())
.setGoogleCredentials(retrieveGoogleCredentials());
AccountClient a = new AccountClient(cfg);
// ...
For workspace-level operations, for default authentication:
import com.databricks.sdk.WorkspaceClient;
// ...
WorkspaceClient w = new WorkspaceClient();
// ...
For direct configuration, replace the retrieve
placeholders with your implementation to retrieve the values from the console or some other configuration store, such as Google Cloud Secret Manager. In this case, the host is the Databricks workspace URL, for example https://1234567890123456.7.gcp.databricks.com
:
import com.databricks.sdk.WorkspaceClient;
import com.databricks.sdk.core.DatabricksConfig;
// ...
DatabricksConfig cfg = new DatabricksConfig()
.setHost(retrieveWorkspaceUrl())
.setGoogleCredentials(retrieveGoogleCredentials());
WorkspaceClient w = new WorkspaceClient(cfg);
// ...
For more information about authenticating with Databricks’ Python tools and SDKs that implement Databricks client unified authentication, see:
For account-level operations, for default authentication:
import (
"github.com/databricks/databricks-sdk-go"
)
// ...
a := databricks.Must(databricks.NewAccountClient())
// ...
For direct configuration (replace the retrieve
placeholders with your own implementation to retrieve the values from the console or some other configuration store, such as Google Cloud Secret Manager). In this case, the Databricks account console URL is https://accounts.gcp.databricks.com
:
import (
"github.com/databricks/databricks-sdk-go"
)
// ...
a := databricks.Must(databricks.NewAccountClient(&databricks.Config{
Host: retrieveAccountConsoleUrl(),
AccountId: retrieveAccountId(),
GoogleCredentials: retrieveGoogleCredentials(),
}))
// ...
For workspace-level operations, for default authentication:
import (
"github.com/databricks/databricks-sdk-go"
)
// ...
w := databricks.Must(databricks.NewWorkspaceClient())
// ...
For direct configuration (replace the retrieve
placeholders with your own implementation to retrieve the values from the console or some other configuration store, such as Google Cloud Secret Manager). In this case, the host is the Databricks workspace URL, for example https://1234567890123456.7.gcp.databricks.com
:
import (
"github.com/databricks/databricks-sdk-go"
)
// ...
w := databricks.Must(databricks.NewWorkspaceClient(&databricks.Config{
Host: retrieveWorkspaceUrl(),
GoogleCredentials: retrieveGoogleCredentials(),
}))
// ...
For more information about authenticating with Databricks tools and SDKs that use Go and that implement Databricks client unified authentication, see Authenticate the Databricks SDK for Go with your Databricks account or workspace.