What is Lakehouse Federation

Preview

This article introduces Lakehouse Federation, the query federation platform that enables you to use Databricks to run queries against multiple external data sources. It also describes how to set up Lakehouse Federation connections and create foreign catalogs in your Unity Catalog metastore.

What is Lakehouse Federation?

Lakehouse Federation is the query federation platform for Databricks. The term query federation describes a collection of features that enable users and systems to run queries against multiple data sources without needing to migrate all data to a unified system.

Databricks uses Unity Catalog to manage query federation. You configure read-only connections to popular database solutions using drivers that are included on Pro SQL Warehouses, Serverless SQL Warehouses, and Databricks Runtime clusters. Unity Catalog’s data governance and data lineage tools ensure that data access is managed and audited for all federated queries made by the users in your Databricks workspaces.

Why use Lakehouse Federation?

The lakehouse emphasizes central storage of data to reduce data redundancy and isolation. Your organization may have numerous data systems in production, and you might want to query data in connected systems for a number of reasons:

Ad hoc reporting.
Proof-of-concept work.
The exploratory phase of new ETL pipelines or reports.
Supporting workloads during incremental migration.

In each of these scenarios, query federation gets you to insights faster, because you can query the data in place and avoid complex and time-consuming ETL processing.

Lakehouse Federation is meant for use cases when:

You don’t want to ingest data into Databricks.
You want your queries to take advantage of compute in the external database system.
You want the advantages of Unity Catalog interfaces and data governance, including fine-grained access control, data lineage, and search.

Overview of Lakehouse Federation setup

To make a dataset available for read-only querying using Lakehouse Federation, you create the following:

A connection, a securable object in Unity Catalog that specifies a path and credentials for accessing an external database system.
A foreign catalog, a securable object in Unity Catalog that mirrors a database in an external data system, enabling you to perform read-only queries on that data system in your Databricks workspace, managing access using Unity Catalog.

Supported data sources

Lakehouse Federation supports connections to the following database types:

Connection requirements

Workspace requirements:

Workspace enabled for Unity Catalog.

Compute requirements:

Network connectivity from your Databricks Runtime cluster or SQL warehouse to the target database systems. See Networking recommendations for Lakehouse Federation.
Databricks clusters must use Databricks Runtime 13.1 or above and shared or single-user access mode.
SQL warehouses must be Pro or Serverless.

Permissions required:

To create a connection, you must be a metastore admin or a user with the CREATE CONNECTION privilege on the Unity Catalog metastore attached to the workspace.
To create a foreign catalog, you must have the CREATE CATALOG permission on the metastore and be either the owner of the connection or have the CREATE FOREIGN CATALOG privilege on the connection.

Additional permission requirements are specified in each task-based section that follows.

Create a connection

A connection specifies a path and credentials for accessing an external database system. To create a connection, you can use Catalog Explorer or the CREATE CONNECTION SQL command in a Databricks notebook or the Databricks SQL query editor.

Permissions required: Metastore admin or user with the CREATE CONNECTION privilege.

In your Databricks workspace, click Catalog.
In the left pane, expand the External Data menu and select Connections.
Click Create connection.
Enter a user-friendly Connection name.
Select the Connection type (database provider, like MySQL or PostgreSQL).
Enter the connection properties (such as host information, path, and access credentials).

Each connection type requires different connection information. See the article for your connection type, listed in the table of contents to the left.
(Optional) Click Test connection to confirm that it works.
(Optional) Add a comment.
Click Create.

Run the following command in a notebook or the Databricks SQL query editor. This example is for connections to a PostgreSQL database. The options differ by connection type. See the article for your connection type, listed in the table of contents to the left.

CREATE CONNECTION <connection-name> TYPE postgresql
OPTIONS (
  host '<hostname>',
  port '<port>',
  user '<user>',
  password '<password>'
);

We recommend that you use Databricks secrets instead of plaintext strings for sensitive values like credentials. For example:

CREATE CONNECTION <connection-name> TYPE postgresql
OPTIONS (
  host '<hostname>',
  port '<port>',
  user secret ('<secret-scope>','<secret-key-user>'),
  password secret ('<secret-scope>','<secret-key-password>')
)

For information about setting up secrets, see Secret management.

For information about managing existing connections, see Manage connections for Lakehouse Federation.

Create a foreign catalog

A foreign catalog mirrors a database in an external data system so that you can query and manage access to data in that database using Databricks and Unity Catalog. To create a foreign catalog, you use a connection to the data source that has already been defined.

To create a foreign catalog, can use Catalog Explorer or the CREATE FOREIGN CATALOG SQL command in a Databricks notebook or the Databricks SQL query editor.

Permissions required: CREATE CATALOG permission on the metastore and either ownership of the connection or the CREATE FOREIGN CATALOG privilege on the connection.

In your Databricks workspace, click Catalog.
Click the Create Catalog button.
On the Create a new catalog dialog, enter a name for the catalog and select a Type of Foreign.
Select the Connection that provides access to the database that you want to mirror as a Unity Catalog catalog.
Enter the name of the Database that you want to mirror as a catalog.

Requirements differ depending on the data source:
- MySQL uses a two-layer namespace and therefore does not require a database name.
- For connections to a catalog in another Databricks workspace, enter the Databricks Catalog name instead of a database name.
Click Create.

Run the following SQL command in a notebook or Databricks SQL editor. Items in brackets are optional. Replace the placeholder values:
- <catalog-name>: Name for the catalog in Databricks.
- <connection-name>: The connection object that specifies the data source, path, and access credentials.
- <database-name>: Name of the database you want to mirror as a catalog in Databricks. Not required for MySQL, which uses a two-layer namespace.
- <external-catalog-name>: Databricks-to-Databricks only: Name of the catalog in the external Databricks workspace that you are mirroring. See Create a foreign catalog.
```
CREATE FOREIGN CATALOG [IF NOT EXISTS] <catalog-name> USING CONNECTION <connection-name>
OPTIONS (database '<database-name>');
```

For information about managing and working with foreign catalogs, see Manage and work with foreign catalogs.

Limitations

Queries are read-only.
Throttling of connections is determined using the Databricks SQL concurrent query limit. There is no limit across warehouses per connection. See Queueing and autoscaling for pro and classic SQL warehouses.
Tables and schemas with names that are invalid in Unity Catalog are not supported and are ignored by Unity Catalog upon creation of a foreign catalog. For example, table names are converted to lowercase in Unity Catalog, which means that lookups must use lowercase names. See the list of naming rules and limitations in Unity Catalog limitations.

For each foreign table referenced, Databricks schedules a subquery in the remote system to return a subset of data from that table and then returns the result to one Databricks executor task over a single stream.
Single-user access mode is only available for users that own the connection.