Review and set Google Cloud resource quotas for a workspace

Google Cloud enforces quotas on resource usage for projects to protect the community of Google Cloud users by preventing unforeseen spikes in usage. When you create a Databricks workspace, you specify a Google Cloud project, which Databricks uses to create new resources such as virtual machine instances for clusters.

Before you create a workspace, review the Google Cloud resource quotas that Databricks requires on the Google Cloud project that you plan to use with your workspace to run clusters. You may need to request quota increases.

Note that not all projects have the same initial quotas. Initial quotas can vary based on your Google Cloud account type, the age of your Google Cloud account, and other factors. As you increasingly use Google Cloud, quotas for initial or existing projects in your account might increase accordingly. If you expect a notable upcoming increase in usage, you can proactively request quota adjustments.

Important

You may need to raise some quotas on your project for Databricks to work as expected. For example, if a project exceeds its CPU quota, launching a Databricks cluster fails.

Google Cloud resource quotas that Databricks requires

The required quotas are grouped by the relevant Google API service. Note that there are different values for minimum functionality and the recommendation for running at scale. For cells that have Default in the Required minimum column, the Google Cloud default is acceptable for initial use and testing.

Compute Engine API

Quotas for Compute Engine API	Required minimum	Recommended for running at scale	Field name
CPUs	Default	2400	compute.googleapis.com/cpus
Routes	Default	300	compute.googleapis.com/routes
Subnetworks	Default	275	compute.googleapis.com/subnetworks
In-use IP addresses	Default	500	compute.googleapis.com/regional_in_use_addresses
Managed instance groups	Default	500	compute.googleapis.com/instance_group_managers
Instance groups	Default	500	compute.googleapis.com/instance_groups
Persistent Disk Standard (GB)	Default	50 TB	compute.googleapis.com/cpus
N2 CPUs	Default	300	compute.googleapis.com/cpus
Persistent Disk SSD (GB)	Default	50 TB	compute.googleapis.com/disks_total_storage
Local SSD (GB)	Default	50 TB	compute.googleapis.com/local_ssd_total_storage

Databricks SQL quotas for Compute Engine API

SQL warehouses won’t start if you do not provision the required amount of CPU and storage resources. If needed, you can increase the resource quotas to support your use of SQL warehouses. See Review and increase quotas. For information about workspace cost, see cost per workspace.

Databricks SQL quotas for Compute Engine API	Required minimum	Recommended for running at scale
N2 CPUs	100	500
Persistent Disk SSD (GB)	5 TB	30 TB
Local SSD (GB)	15 TB	100 TB

Cloud Monitoring API

Quotas for Cloud Monitoring	Required minimum quota	Recommended quotas for running at scale
Time series ingestion requests per minute	Default	6000

Identity and Access Management (IAM) API

Quotas for IAM	Required minimum quota	Recommended quotas for running at scale
Service account count	Default	100

Review quotas and request quota increases

Go to the Quotas page in the Cloud Console.
In the top navigation’s project picker, select the Google Cloud project that you plan to use with your workspace to run clusters.
For each each of the Databricks project quotas, confirm the current quotas for your Google Cloud region.
1. For each quota, type either the quota name or its field name into the filter field. Find the row for the quota for the Google Cloud region in which you deploy your workspace. In the filter editor, you can add a region filter to ensure that you are reviewing only the quotas for your desired region. For example, you can search for the CPUs quota for a specific region:
2. To request quota increases, click the checkbox to the left of the quota, and click the blue text at the top of the page that says EDIT QUOTAS. If you do not see that text, you may not have the right permissions or account type. Contact Google Support and ask how to request quota increases for your account.
3. In the panel that appears to the right, increase the quota values according to the guidance in Google Cloud resource quotas that Databricks requires for each quota. Depending on the values you set, your user account, and the Google Cloud project, increasing quotas may require approval from your service provider. If you request quota increases, you’ll need to wait for email confirmation for each request before creating a workspace.
4. Repeat this process for all quotas in Google Cloud resource quotas that Databricks requires.
5. Wait for email confirmation of any quota increases that you requested.
Important

If you change any quotas, wait 15 minutes after changes or email confirmation of quota increases before you create a workspace.