Use the Databricks connector to connect to another Databricks workspace

This article provides syntax examples of using the Databricks connector to connect to another Databricks workspace. This connector leverages the Databricks JDBC driver, which is included in Databricks Runtime 13.3 LTS and above.

Important

For most data sharing operations, Databricks recommends Delta Sharing. See What is Delta Sharing?. You may also prefer Lakehouse Federation for managing queries on data in other Databricks workspaces. See What is Lakehouse Federation?.

Connecting to another Databricks workspace

The Databricks Spark connector allows you to connect to compute resources configured in another Databricks workspace and return results to your current Databricks workspace. You must have access to active compute on both workspaces for queries to succeed.

The JDBC driver is registered for jdbc:databricks:// URLs. You must configure and use a personal access token that grants you permissions on the workspace resources being accessed remotely. See the Token management API.

Note

If you have a Databricks JDBC library attached to your cluster, the library version attached your cluster is used instead of the version included in Databricks Runtime.

Read data from another Databricks workspace

You can specify the format databricks to use the Databricks Spark connector when you’re reading data, as in the following example:

df = (spark.read
  .format("databricks")
  .option("host", "<workspace-id>.<random-number>.gcp.databricks.com")
  .option("httpPath", "/sql/1.0/warehouses/<warehouse-id>")
  .option("personalAccessToken", "<auth-token>")
  .option("dbtable", "<table-name>")
  .load()
)

Create an external table against another Databricks workspace

You can register an external table in a Databricks workspace linked to a separate Databricks workspace.

The following example demonstrates this syntax, using the secret function to get credentials stored with Databricks secrets:

Note

For more on Databricks secrets, see secret function.

CREATE TABLE databricks_external_table
USING databricks
OPTIONS (
  host '<workspace-id>.<random-number>.gcp.databricks.com',
  httpPath '/sql/1.0/warehouses/<warehouse-id>',
  personalAccessToken secret('<scope>', '<token>'),
  dbtable '<table-name>'
);