Review and set Google Cloud resource quotas for a workspace

Google Cloud enforces quotas on resource usage for projects to protect the community of Google Cloud users by preventing unforeseen spikes in usage. When you create a Databricks workspace, you specify a Google Cloud project, which Databricks uses to create new resources such as virtual machine instances for clusters.

Before you create a workspace, review the Google Cloud resource quotas that Databricks requires on the Google Cloud project that you plan to use with your workspace to run clusters. You may need to request quota increases.

Note that not all projects have the same initial quotas. Initial quotas can vary based on your Google Cloud account type, the age of your Google Cloud account, and other factors. As you increasingly use Google Cloud, quotas for initial or existing projects in your account might increase accordingly. If you expect a notable upcoming increase in usage, you can proactively request quota adjustments.

Important

You may need to raise some quotas on your project for Databricks to work as expected. For example, if a project exceeds its CPU quota, launching a Databricks cluster fails.

Google Cloud resource quotas that Databricks requires

The required quotas are grouped by the relevant Google API service. Note that there are different values for minimum functionality and the recommendation for running at scale. For cells that have Default in the Required minimum column, the Google Cloud default is acceptable for initial use and testing.

Compute Engine API

Quotas for Compute Engine API Required minimum Recommended for running at scale Field name
CPUs 60 2500 compute.googleapis.com/cpus
Routes Default 300 compute.googleapis.com/routes
Subnetworks Default 275 compute.googleapis.com/subnetworks
Regional in-use address Default 500 compute.googleapis.com/regional_in_use_addresses
Instance group managers Default 500 compute.googleapis.com/instance_group_managers
Instance groups Default 500 compute.googleapis.com/instance_groups
Disks total storage Default 50 TB compute.googleapis.com/disks_total_storage
N2 family CPUs 50 300 compute.googleapis.com/n2_cpus
SSD total storage 7.5 TB 50 TB compute.googleapis.com/ssd_total_storage

Cloud Monitoring API

Quotas for Cloud Monitoring API Required minimum quota Recommended quotas for running at scale Field name
Time series ingestion requests per minute Default 6000 monitoring.googleapis.com/ingestion_requests

Identity and Access Management (IAM) API

Quotas for IAM API Required minimum quota Recommended quotas for running at scale Field name
Service account count Default 100 iam.googleapis.com/quota/service-account-count

Review quotas and request quota increases

  1. Go to the Quotas page in the Cloud Console.

  2. In the top navigation’s project picker, select the Google project that you plan to use with your workspace to run clusters.

    Marketplace listing project picker
  3. For each each of the Databricks project quotas, confirm the current quotas for your Google Cloud region.

    1. For each quota, type either the quota name or its field name into the filter field. Find the row for the quota for the Google Cloud region in which you deploy your workspace. In the filter editor, you can add a region filter to ensure that you are reviewing only the quotas for your desired region. For example, you can search for the CPUs quota for a specific region:

      One quota and filtered by region
    2. To request quota increases, click the checkbox to the left of the quota, and click the blue text at the top of the page that says EDIT QUOTAS. If you do not see that text, you may not have the right permissions or account type. Contact Google Support and ask how to request quota increases for your account.

    3. In the panel that appears to the right, increase the quota values according to the guidance in Google Cloud resource quotas that Databricks requires for each quota. Depending on the values you set, your user account, and the Google project, increasing quotas may require approval from your service provider. If you request quota increases, you’ll need to wait for email confirmation for each request before creating a workspace.

      Request raise quota
    4. Repeat this process for all quotas in Google Cloud resource quotas that Databricks requires.

    5. Wait for email confirmation of any quota increases that you requested.

    Important

    If you change any quotas, wait 15 minutes after changes or email confirmation of quota increases before you create a workspace.

Related Google documentation articles: