Databricks architecture overview

This article provides a high-level overview of Databricks architecture, including its enterprise architecture, in combination with Google Cloud.

High-level architecture

Databricks operates out of a control plane and a compute plane.

• The control plane includes the backend services that Databricks manages in your Databricks account. The web application is in the control plane.

• The compute plane is where your data is processed. There are two types of compute planes depending on the compute that you are using.

  • For serverless compute, the serverless compute resources run in a serverless compute plane in your Databricks account.

  • For classic Databricks compute, the compute resources are in your Google Cloud resources in what is called the classic compute plane. This refers to the network in your Google Cloud resources and its resources.

The following diagram describes the overall Databricks architecture.

Serverless compute plane

In the serverless compute plane, Databricks compute resources run in a compute layer within your Databricks account. Databricks creates a serverless compute plane in the same Google Cloud region as your workspace’s classic compute plane.

To protect customer data within the serverless compute plane, serverless compute runs within a network boundary for the workspace, with various layers of security to isolate different Databricks customer workspaces and additional network controls between clusters of the same customer.

Classic compute plane

In the classic compute plane, Databricks compute resources run in your Google Cloud account. New compute resources are created within each workspace’s virtual network in the customer’s Google Cloud account.

A classic compute plane has natural isolation because it runs in each customer’s own Google Cloud account. To learn more about networking in the classic compute plane, see Classic compute plane networking.

For regional support, see Databricks clouds and regions.