Create an external location to connect cloud storage to Databricks
This article describes how to configure an external location in Unity Catalog to connect cloud storage to Databricks.
External locations associate Unity Catalog storage credentials with cloud object storage containers. External locations are used to define managed storage locations for catalogs and schemas, and to define locations for external tables and external volumes.
You can create an external location that references storage in a Google Cloud Storage (GCS) or Cloudflare R2 bucket.
You can create an external location using Catalog Explorer, the Databricks CLI, or SQL commands in a notebook or Databricks SQL query.
Note
When you define a volume, cloud URI access to data under the volume path is governed by the permissions of the volume.
Before you begin
Prerequisites:
You must create the Google Cloud Storage (GCS) or Cloudflare R2 bucket that you want to use as an external location before you create the external location object in Databricks.
You must have a storage credential defined in Databricks that gives access to the cloud storage location path. See Create a storage credential for connecting to Google Cloud Storage and Create a storage credential for connecting to Cloudflare R2.
Permissions requirements:
You must have the
CREATE EXTERNAL LOCATION
privilege on both the metastore and the storage credential referenced in the external location. Metastore admins haveCREATE EXTERNAL LOCATION
on the metastore by default.
Create an external location using Catalog Explorer
You can create an external location manually using Catalog Explorer.
Permissions and prerequisites: see Before you begin.
To create the external location:
Click Catalog to open Catalog Explorer.
Click the + Add button and select Add an external location.
Enter an External location name.
Optionally copy the bucket path from an existing mount point (GCS buckets only).
If you aren’t copying from an existing mount point, use the URL field to enter the bucket path that you want to use as the external location.
For example,
gs://mybucket/<path>
orr2://mybucket@my-account-id.r2.cloudflarestorage.com/<path>
.Select the storage credential that grants access to the external location.
(Optional) If you want users to have read-only access to the external location, click Advanced Options and select Read only. For more information, see Mark an external location as read-only.
Click Create.
Grant permission to use the external location.
For anyone to use the external location you must grant permissions:
To use the external location to add a managed storage location to metastore, catalog, or schema, grant the
CREATE MANAGED LOCATION
privilege.To create external tables or volumes, grant
CREATE EXTERNAL TABLE
orCREATE EXTERNAL VOLUME
.
To use Catalog Explorer to grant permissions:
Click the external location name to open the details pane.
On the Permissions tab, click Grant.
On the Grant on
<external location>
dialog, select users, groups, or service principals in Principals field, and select the privilege you want to grant.Click Grant.
Create an external location using SQL
To create an external location using SQL, run the following command in a notebook or the SQL query editor. Replace the placeholder values.
Permissions and prerequisites: see Before you begin.
<location-name>
: A name for the external location. Iflocation_name
includes special characters, such as hyphens (-
), it must be surrounded by backticks (` `
). See Names.
<bucket-path>
: The path in your cloud tenant that this external location grants access to. For example,gs://mybucket
orr2://mybucket@my-account-id.r2.cloudflarestorage.com
.<storage-credential-name>
: The name of the storage credential that authorizes reading from and writing to the bucket. If the storage credential name includes special characters, such as hyphens (-
), it must be surrounded by backticks (` `
).
CREATE EXTERNAL LOCATION [IF NOT EXISTS] `<location-name>`
URL '<bucket-path>'
WITH ([STORAGE] CREDENTIAL `<storage-credential-name>`)
[COMMENT '<comment-string>'];
Next steps
Grant other users permission to use external locations. See Manage external locations.
Define managed storage locations using external locations. See Specify a managed storage location in Unity Catalog.
Define external tables using external locations. See Create an external table.
Define external volumes using external locations. See Create and work with volumes.