Understand and manage Delta Live Tables upgrades

Preview

This feature is in Public Preview.

Delta Live Tables clusters use a runtime based on Databricks Runtime. Databricks automatically upgrades the Delta Live Tables runtime to support enhancements and upgrades to the platform. As with any software upgrade, a Delta Live Tables runtime upgrade may result in errors or issues running your pipelines. This article describes best practices to test your pipeline with upcoming releases of the Delta Live Tables runtime, and Delta Live Tables features that enhance the stability of your pipelines.

Delta Live Tables runtime channels

The channel field in the Delta Live Tables pipeline settings controls the Delta Live Tables runtime version that runs your pipeline. The supported values are:

  • preview to test your pipeline with upcoming changes to the runtime version.

  • current to use the current runtime version.

By default, your pipelines run using the current runtime version. Databricks recommends using the current runtime for production workloads. To learn how to use the preview setting to test your pipelines with the next runtime version, see Automate testing of your pipelines with the next runtime version.

Delta Live Tables upgrade process

Delta Live Tables automatically upgrades the runtime in your Databricks workspaces and monitors the health of your pipelines after the upgrade. If Delta Live Tables detects that a pipeline cannot start because of an upgrade, the runtime version for the pipeline reverts to the previous known-good version, and the following steps are triggered automatically:

Note

Delta Live Tables reverts only pipelines running in production mode and with the channel set to current.

  • The pipeline‚Äôs Delta Live Tables runtime is pinned to the previous known-good version.

  • The Delta Live Tables UI shows a visual indicator that the pipeline is pinned to a previous version because of an upgrade failure.

  • Databricks support is notified of the issue. If the issue is related to a regression in the runtime, Databricks will resolve the issue. If the issue is caused by a custom library or package used by the pipeline, Databricks will contact you to resolve the issue.

  • When the issue is resolved, Databricks will initiate the upgrade again.

Best practices

Automate testing of your pipelines with the next runtime version

To ensure changes in the next Delta Live Tables runtime version do not impact your pipelines, use the Delta Live Tables channels feature:

  1. Create a staging pipeline and set the channel to preview.

  2. In the Delta Live Tables UI, create a schedule to run the pipeline weekly and enable alerts to receive an email notification for pipeline failures.

  3. If you receive a notification of a failure and are unable to resolve it, open a support ticket with Databricks.

Pipeline dependencies

Delta Live Tables supports external dependencies in your pipelines; for example, you can install any Python package using the %pip install command. Delta Live Tables also supports using global and cluster-scoped init scripts. However, these external dependencies, particularly init scripts, increase the risk of issues with runtime upgrades. To mitigate these risks, minimize using init scripts in your pipelines. If your processing requires init scripts, automate testing of your pipeline to detect problems early; see Automate testing of your pipelines with the next runtime version. If you use init scripts, Databricks recommends increasing your testing frequency.