Shallow clone for Unity Catalog managed tables

Important

In Databricks Runtime 13.1 and above, shallow clone support for Unity Catalog is in Public Preview.

You can use shallow clone to create new Unity Catalog managed tables from existing Unity Catalog managed tables. Shallow clone support for Unity Catalog allows you to create tables with access control privileges independent from their parent tables without needing to copy underlying data files.

For more on Delta clone, see Clone a table on Databricks.

For more on Unity Catalog managed tables, see Managed tables.

Create a shallow clone on Unity Catalog

You can create a shallow clone in Unity Catalog using the same syntax available for shallow clones throughout the product, as shown in the following syntax example:

CREATE TABLE <catalog-name>.<schema-name>.<target-table-name> SHALLOW CLONE <catalog-name>.<schema-name>.<source-table-name>

To create a shallow clone on Unity Catalog, you must have sufficient privileges on both the source and target resources, as detailed in the following table:

Resource	Permissions required
Source table	`SELECT`
Source schema	`USE SCHEMA`
Source catalog	`USE CATALOG`
Target schema	`USE SCHEMA`, `CREATE TABLE`
Target catalog	`USE CATALOG`

Like other create table statements, the user who creates a shallow clone is the owner of the target table. The owner of a target cloned table can control the access rights for that table independently of the source table.

Note

The owner of a cloned table might be different than the owner of a source table.

Query or modify a shallow cloned table on Unity Catalog

Important

The instructions in this section describe privileges needed for compute configured with shared access mode. For Single User access mode, see Work with shallow cloned tables in Single User access mode.

To query a shallow clone on Unity Catalog, you must have sufficient privileges on the table and containing resources, as detailed in the following table:

Resource	Permissions required
Catalog	`USE CATALOG`
Schema	`USE SCHEMA`
Table	`SELECT`

You must also have MODIFY permissions on the target of the clone operation to complete the following actions:

Vacuum and Unity Catalog shallow clones

Important

In Databricks Runtime 13.2 and above, this behavior is in Public Preview.

When you use Unity Catalog managed tables for the source and target of a shallow clone operation, Unity Catalog manages the underlying data files to improve reliability for the source and target of the clone operation. Running VACUUM on the source of a shallow clone does not break the cloned table.

Normally, when VACUUM identifies valid files for a given retention threshold, only the metadata for the current table is considered. Shallow clone support for Unity Catalog tracks the relationships between all cloned tables and the source data files, so valid files are expanded to include data files necessary for returning queries for any shallow cloned table as well as the source table.

Note

This means that for Unity Catalog shallow clone VACUUM semantics, a valid data file is any file within the specified retention threshold for the source table or any cloned table.

This enhanced tracking of metadata changes how VACUUM operations impact data files backing the Delta tables, with the following semantics:

VACUUM operations against either the source or target of a shallow clone operation might delete data files from the source table.
Only data files not considered valid for the source table or any shallow clone against the source are removed.
If multiple shallow clones are defined against a single source table, running VACUUM on any of the cloned tables does not remove valid data files for other cloned tables.

Note

Databricks recommends never running VACUUM with a retention setting of less than 7 days to avoid corrupting ongoing long-running transactions. If you need to run VACUUM with a lower retention threshold, make sure you understand how VACUUM on shallow clones in Unity Catalog differs from how VACUUM interacts with other cloned tables on Databricks. See Clone a table on Databricks.

Work with shallow cloned tables in Single User access mode

When working with Unity Catalog shallow clones in Single User access mode, you must have permissions on the resources for the cloned table source as well as the target table.

This means that for simple queries in addition to the required permissions on the target table, you must have USE permissions on the source catalog and schema and SELECT permissions on the source table. For any queries that would update or insert records to the target table, you must also have MODIFY permissions on the source table.

Databricks recommends working with Unity Catalog clones on compute with shared access mode as this allows independent evolution of permissions for Unity Catalog shallow clone targets and their source tables.

Limitations

For shallow clones in Unity Catalog, only Unity Catalog managed tables are supported for both the source and target.
You cannot share shallow clones using Delta Sharing.
You cannot nest shallow clones, meaning you cannot make a shallow clone from a shallow clone.
In Databricks Runtime 13.1, running VACUUM on the source table breaks the target table.
Dropping the source table breaks the target table for shallow clones.
Unity Catalog allows users to UNDROP tables for around 7 days after a DROP TABLE command. In Databricks Runtime 13.2 and above, shallow clones based on a dropped table continue to work during this time. If you do not UNDROP the source table in this window, the shallow clone stops functioning once the source table’s data files are garbage collected.