Update permissions for GCE compute deployment

In an effort to speed up compute startup times, Databricks will begin deploying compute resources on Google Compute Engine (GCE) instead of GKE.

To launch compute resources on GCE, you must grant Databricks a new set of permissions. For a full list of these updated permissions and their purposes, see Required permissions.

Grant permissions for GCE migration

To update the permissions, you must be a Databricks account admin who has permissions on the Google Cloud project where Databricks is deployed. There are two ways to grant the new permissions:

  • For account owners who have the required permissions on the Google Cloud project:

    1. Navigate to the account console.

    2. In the banner at the top of the account console, click the Update permissions button.

    3. Confirm you have sufficient privileges on the workspace projects then click Update permissions.

  • For accounts that use a service account with required permissions to manage workspace provisioning, verify that you are authenticated to the Databricks API and have obtained an ACCESS_TOKEN and AUTH_TOKEN. Then, make the following API call:

    curl --location --request PATCH ‘https://accounts.gcp.databricks.com/api/2.0/accounts/<account-id>/migrateToComputeOnGce’ \ --header ‘X-Databricks-GCP-SA-Access-Token: ’$ACCESS_TOKEN \ --header ‘Authorization: Bearer ‘$AUTH_TOKEN

Updates needed for customer-managed VPCs

If your account uses a customer-managed VPC to deploy Databricks workspaces, the permission update attempts to add a firewall rule to your VPC automatically. If this fails due to lack of privileges, you must manually add the following firewall rule to your VPC. This firewall rule permits traffic between Databricks-managed VMs within your VPC. The rule does not permit ingress from outside the VPC.

The required rule should be:

  • Rule name: databricks-{WORKSPACE_ID}-ingress

  • Direction: Ingress

  • Priority: 1000

  • Targets: Network tag: databricks-{WORKSPACE_ID}

  • Source filter: IPv4 range: primary CIDR range of subnet

  • Protocols and ports: Allow all

For more information about Shared VPCs, see Configure a customer-managed VPC.

Try compute on GCE

After you have made the permission updates, you can test whether your workspace is launching compute in GCE.

Create a new all-purpose or job compute resource and add the following key-value pair to the custom tags fields:

key: x-databricks-nextgen-cluster value: true

After the compute resource starts up, the resource should include a GCE label next to its name.