Shallow clone support for Unity Catalog managed tables is in Public Preview in Databricks Runtime 13.1 and above. Shallow clone support for Unity Catalog external table is in Public Preview in Databricks Runtime 14.2 and above.
You can use shallow clone to create new Unity Catalog tables from existing Unity Catalog tables. Shallow clone support for Unity Catalog allows you to create tables with access control privileges independent from their parent tables without needing to copy underlying data files.
You can only clone Unity Catalog managed tables to Unity Catalog managed tables and Unity Catalog external tables to Unity Catalog external tables.
VACUUM behavior differs between managed and external tables. See Vacuum and Unity Catalog shallow clones.
For more on Delta clone, see Clone a table on Databricks.
For more on Unity Catalog tables, see Create tables in Unity Catalog.
You can create a shallow clone in Unity Catalog using the same syntax available for shallow clones throughout the product, as shown in the following syntax example:
CREATE TABLE <catalog-name>.<schema-name>.<target-table-name> SHALLOW CLONE <catalog-name>.<schema-name>.<source-table-name>
To create a shallow clone on Unity Catalog, you must have sufficient privileges on both the source and target resources, as detailed in the following table:
Target external location (external tables only)
Like other create table statements, the user who creates a shallow clone is the owner of the target table. The owner of a target cloned table can control the access rights for that table independently of the source table.
The owner of a cloned table might be different than the owner of a source table.
The instructions in this section describe privileges needed for compute configured with shared access mode. For Single User access mode, see Work with shallow cloned tables in Single User access mode.
To query a shallow clone on Unity Catalog, you must have sufficient privileges on the table and containing resources, as detailed in the following table:
You must also have
MODIFY permissions on the target of the clone operation to complete the following actions:
CREATE OR REPLACE TABLE
This behavior is in Public Preview in Databricks Runtime 13.2 and above for managed tables and Databricks Runtime 14.2 and above for external tables.
When you use Unity Catalog tables for the source and target of a shallow clone operation, Unity Catalog manages the underlying data files to improve reliability for the source and target of the clone operation. Running
VACUUM on the source of a shallow clone does not break the cloned table.
VACUUM identifies valid files for a given retention threshold, only the metadata for the current table is considered. Shallow clone support for Unity Catalog tracks the relationships between all cloned tables and the source data files, so valid files are expanded to include data files necessary for returning queries for any shallow cloned table as well as the source table.
This means that for Unity Catalog shallow clone
VACUUM semantics, a valid data file is any file within the specified retention threshold for the source table or any cloned table. Managed tables and external tables have slightly different semantics.
This enhanced tracking of metadata changes how
VACUUM operations impact data files backing the Delta tables, with the following semantics:
For managed tables,
VACUUMoperations against either the source or target of a shallow clone operation might delete data files from the source table.
For external tables,
VACUUMoperations only remove data files from the source table when run against the source table.
Only data files not considered valid for the source table or any shallow clone against the source are removed.
If multiple shallow clones are defined against a single source table, running
VACUUMon any of the cloned tables does not remove valid data files for other cloned tables.
Databricks recommends never running
VACUUM with a retention setting of less than 7 days to avoid corrupting ongoing long-running transactions. If you need to run
VACUUM with a lower retention threshold, make sure you understand how
VACUUM on shallow clones in Unity Catalog differs from how
VACUUM interacts with other cloned tables on Databricks. See Clone a table on Databricks.
When working with Unity Catalog shallow clones in Single User access mode, you must have permissions on the resources for the cloned table source as well as the target table.
This means that for simple queries in addition to the required permissions on the target table, you must have
USE permissions on the source catalog and schema and
SELECT permissions on the source table. For any queries that would update or insert records to the target table, you must also have
MODIFY permissions on the source table.
Databricks recommends working with Unity Catalog clones on compute with shared access mode as this allows independent evolution of permissions for Unity Catalog shallow clone targets and their source tables.
Shallow clones on external tables must be external tables. Shallow clones on managed tables must be managed tables.
You cannot share shallow clones using Delta Sharing.
You cannot nest shallow clones, meaning you cannot make a shallow clone from a shallow clone.
In Databricks Runtime 13.1, running
VACUUMon the source table breaks the target table.
For managed tables, dropping the source table breaks the target table for shallow clones. Data files backing external tables are not removed by
DROP TABLEoperations, and so shallow clones of external tables are not impacted by dropping the source.
Unity Catalog allows users to
UNDROPmanaged tables for around 7 days after a
DROP TABLEcommand. In Databricks Runtime 13.2 and above, managed shallow clones based on a dropped managed table continue to work during this time. If you do not
UNDROPthe source table in this window, the shallow clone stops functioning once the source table’s data files are garbage collected.