Data access configuration

This article describes the data access configurations performed by Databricks administrators for all SQL warehouses (formerly SQL endpoints) using the UI.

To configure all SQL warehouses using the REST API, see Global SQL Warehouses API.

Important

Changing these settings restarts all running SQL warehouses.

For a general overview of how to enable access to data, see Access control.

Requirements

You must be a Databricks administrator to configure settings for all SQL warehouses.

Note

You can also edit the Data Access Configuration textbox entries directly.

Important

To set a configuration property to the value of a secret without exposing the secret value to Spark, set the value to {{secrets/<secret-scope>/<secret-name>}}. Replace <secret-scope> with the secret scope and <secret-name> with the secret name. The value must start with {{secrets/ and end with }}. For more information about this syntax, see Syntax for referencing secrets in a Spark configuration property or environment variable.

Configure a Google Cloud service account

To configure all warehouses to use a Google Cloud service account when accessing Google Cloud Storage (GCS):

  1. In Google Cloud Platform, create a service account that has permissions on the underlying Google Cloud Platform services required to access your Google Cloud Storage assets.

  2. Navigate to the admin console.

  3. Click the SQL Warehouse Settings tab.

  4. In the Google Service Account field, enter the email address of the service account whose identity will be used to launch all SQL warehouses.

    All queries running on these warehouses will have access to underlying Google Cloud Platform services scoped to the permissions granted to this service account in Google Cloud Platform.

  5. Click Save.

Configure data access properties for SQL warehouses

  1. Click your username in the top bar of the workspace and select Admin Console from the drop down.

  2. Click the SQL Warehouse Settings tab.

  3. In the Data Access Configuration textbox, specify key-value pairs containing metastore properties.

    Important

    To set a Spark configuration property to the value of a secret without exposing the secret value to Spark, set the value to {{secrets/<secret-scope>/<secret-name>}}. Replace <secret-scope> with the secret scope and <secret-name> with the secret name. The value must start with {{secrets/ and end with }}. For more information about this syntax, see Syntax for referencing secrets in a Spark configuration property or environment variable.

  4. Click Save.

You can also configure data access properties with the Databricks Terraform provider and databricks_sql_global_config.

Supported properties

The following properties are supported for SQL warehouses. For an entry that ends with *, all properties within that prefix are supported. For example, spark.sql.hive.metastore.* indicates that both spark.sql.hive.metastore.jars and spark.sql.hive.metastore.version are supported, as well as any other properties that start with spark.sql.hive.metastore.

For properties whose values contain sensitive information, you can store the sensitive information in a secret and set the property’s value to the secret name using the following syntax: secrets/<secret-scope>/<secret-name>.

  • spark.databricks.hive.metastore.glueCatalog.enabled

  • spark.sql.hive.metastore.*

  • spark.sql.warehouse.dir

  • spark.hadoop.datanucleus.*

  • spark.hadoop.fs.*

  • spark.hadoop.hive.*

  • spark.hadoop.javax.jdo.option.*

  • spark.hive.*

For more information about how to set these properties, see External Hive metastore.