February 2022

These features and Databricks platform improvements were released in February 2022.

Note

Releases are staged. Your Databricks account may not be updated until a week or more after the initial release date.

Easier scheduling for your Delta Live Tables pipelines (Public Preview)

February 28 - March 2, 2022

You can now create a scheduled Databricks job for your triggered pipelines directly from the Delta Live Tables UI. Previously, you needed to create the job and the schedule to run the job from the jobs UI.

Easily browse the history of your Delta Live Tables pipeline updates (Public Preview)

February 28 - March 2, 2022

You can now see a history of pipeline updates, including the update status, details, and events, with the Update history drop-down in the Delta Live Tables UI.

Ensure job idempotency for the Jobs API Run now request

February 22-28, 2022: Version 3.66

You can now use the optional idempotency_token parameter with the Jobs API Run now operation to ensure exactly one run of a job is launched. Previously, submitting multiple Run now requests for the same job may have launched more than one run.

To learn more about job idempotency, see How to ensure idempotency for jobs.

Jobs service stability and scalability improvements

February 22-28, 2022: Version 3.66

The following changes increase the stability and scalability of the Jobs service:

  • Each new job is assigned a longer, unique, numeric, non-sequential identifier. Clients that use the Jobs API and depend on a fixed identifier length or sequential or monotonically increasing identifiers must be modified to accept identifiers that are longer, non-sequential, and unordered. The identifier type of int64 remains unchanged, and compatibility is preserved for clients that use IEEE 754 64-bit floating-point numbers, for example, JavaScript clients.

  • The value of the number_in_job field, included in the response to some Jobs API requests, is now set to the same value as run_id.

Compare MLflow runs from different experiments

February 22-28, 2022: Version 3.66

You can now display and compare runs from multiple experiments. See Compare runs from multiple experiments.

Improvements to MLflow compare runs display

February 22-28, 2022: Version 3.66

The compare runs display has been improved. Visualizations now appear at the top of the page. You can now scroll through or collapse the tables that show run information, parameters, and metrics. You can now also simplify the display by hiding parameters and metrics that do not vary across runs.

For information about the compare runs page, see Compare runs. For instructions on how to compare runs, see Compare runs.

Improved visibility into job run owners in the clusters UI

February 22-28, 2022: Version 3.66

The Job Clusters tab in the Compute UI has been updated to show the Job run as user instead of Job owner. This update is based on changes to the jobs permissions model, and provides improved visibility into the owner of a job run. See View compute.

Drop dataset columns in Mosaic AutoML

February 22-28, 2022: Version 3.66

In Databricks Runtime 10.3 ML and above, for classification and regression problems, you can now specify columns in the dataset that AutoML should not use for training. See Mosaic AutoML Python API reference.

Experiments page is GA

February 22-28, 2022: Version 3.66

The experiments page is now generally available.

Support for temporary tables in the Delta Live Tables Python interface

February 22-28, 2022: Version 3.66

You can now use the temporary property when declaring tables with the Delta Live Tables Python interface. Defining a temporary table instructs Delta Live Tables not to persist metadata for the table and prevents publishing the table when the target setting is configured. See the Python specification for information on defining Python tables.

User interface improvements for Delta Live Tables (Public Preview)

February 22-28, 2022: Version 3.66

This release includes the following enhancements to the Delta Live Tables UI:

  • You can now edit your pipeline settings in the Settings dialog in the Delta Live Tables UI, providing a more user-friendly interface. Previously, you were limited to changing settings by editing the JSON specification. To select the UI or JSON view to edit pipeline settings, click the UI or JSON button on the Settings dialog.

  • The event log on the Pipeline Details page now highlights error records, making it easier to identify and address errors in your pipelines.

  • The progress display of running pipelines has been improved. Now, when you view the graph for your pipeline, nodes in the RUNNING state have an animated incoming edge and an animated top border.

Databricks Runtime 9.0 series support ends

February 17, 2022

Support for Databricks Runtime 9.0 and Databricks Runtime 9.0 for Machine Learning ended on February 17. See Databricks support lifecycles.

Data Science & Engineering landing page updates

February 10, 2022

We have updated the look and feel of the Data Science & Engineering landing page. Some links have changed, but there are no other notable functionality changes.

Databricks Repos now supports AWS CodeCommit for Git integration

February 7-14, 2022: Version 3.65

You can now use AWS CodeCommit repositories with Databricks Repos.

Improved visualization for your Delta Live Tables pipelines (Public Preview)

February 7-14, 2022: Version 3.65

The pipeline DAG visualization is redesigned to improve usability and navigation. The improvements include a change to display graphs from left-to-right instead of top-to-bottom, providing a more intuitive view of your pipeline execution.

Updated Markdown parser

February 7-14, 2022: Version 3.65

The Markdown parser used in Databricks notebooks has been updated. Some of the issues noted with the previous update have been fixed. Specifically, it is no longer necessary to put a space between the hashtags and the text of a heading, and links with spaces now render correctly.

Delta Live Tables now supports change data capture processing (Public Preview)

February 7-14, 2022: Version 3.65

You can now implement change data capture (CDC) processing in your Delta Live Tables pipelines. CDC processing is supported with the SQL and Python interfaces. See The APPLY CHANGES APIs: Simplify change data capture with Delta Live Tables.

Select algorithm frameworks to use with AutoML

February 7-14, 2022: Version 3.65

In Databricks Runtime 10.3 ML and above, you can specify algorithm frameworks, such as scikit-learn, that AutoML should not consider as it develops models. By default, AutoML considers models from scikit-learn, XGBoost, and LightGBM for classification and regression problems, and from Prophet and Auto-ARIMA for forecasting problems. See Mosaic AutoML Python API reference.

Databricks Runtime 10.3 and 10.3 ML are GA; 10.3 Photon is Public Preview

February 2, 2022

Databricks Runtime 10.3 and Databricks Runtime 10.3 ML are now generally available. Databricks Runtime 10.3 Photon is in Public Preview.

Repos now supports arbitrary file types (Public Preview)

February 28, 2022

In addition to syncing notebooks with a remote Git repository, Files in Repos lets you sync any type of file, such as .py files, data files in .csv or .json format, or .yaml configuration files. You can import and read these files within a Databricks Repo. You can also view and edit text-encoded files in the UI. For more information, see Git integration for Databricks Git folders.