March 2023

These features and Databricks platform improvements were released in March 2023.

Note

Releases are staged. Your Databricks account might not be updated until a week or more after the initial release date.

Databricks Terraform provider updated to version 1.14.0

March 31, 2023

Version 1.14.0 adds an example for automatically rotating an access token, a run_if condition and a file arrival trigger to the databricks_job resource, and more. For more details, see the changelog for version 1.14.0.

Local storage autoscaling feature added to clusters

March 28, 2023

Databricks compute instances now support storage autoscaling at the worker level using zonal solid state persistent disks. For more information, see Enable autoscaling.

Note

This feature is not yet available in europe-west3.

Distributed training with TorchDistributor

March 27, 2023

With Databricks Runtime 13.0 ML and above, you can perform distributed training on PyTorch ML models using TorchDistributor. See Distributed training with TorchDistributor.

TorchDistributor is an open-source module in PySpark that helps users do distributed training with PyTorch on their Spark clusters, so it lets you launch PyTorch training jobs as Spark jobs.

Databricks Runtime 13.0 (Beta)

March 24, 2023

Databricks Runtime 13.0 and Databricks Runtime 13.0 ML are now available as Beta releases.

See Databricks Runtime 13.0 (EoS) and Databricks Runtime 13.0 for Machine Learning (EoS).

Improved file editor

March 24, 2023

The file editor has been updated to bring many of the notebook features to your file editing experience. Additionally, you now can directly execute Python, SQL, R, and Scala files from the file editor. For details, see Use the Databricks notebook and file editor.

.ipynb (Jupyter) notebook support in Repos (preview)

March 22, 2023

Support for Jupyter notebooks (.ipynb files) is available in Repos. You can clone repositories with .ipynb notebooks, work in Databricks UI, and then commit and push as .ipynb notebooks. Metadata such as a notebook dashboard is preserved. Admins can control whether outputs can be committed or not.

You can also:

  • Create new .ipynb notebooks.

  • Convert notebooks to .ipynb file format.

  • View diffs as Code diff (code changes in cells) or Raw diff (code changes in JSON, including metadata).

See Allow committing .ipynb notebook output.

Support for reload4j

March 21, 2023

Reload4j 1.2.19 is replacing log4j in Databricks Runtime versions 10.4 and earlier.

Unity Catalog, Delta Sharing, and identity federation are GA on Google Cloud

March 20, 2023

Unity Catalog is a fine-grained governance solution for data and AI on the Databricks lakehouse. It helps simplify security and governance of your data by providing a central place to administer and audit data access. See What is Unity Catalog?.

Delta Sharing is a secure data sharing platform that lets you share data in Databricks with users outside your organization, as well as between workspaces on separate metastores within your Databricks account. See What is Delta Sharing?.

Identity federation centralizes identity management at the Databricks account level. You manage users, groups, and service principals in your account and use account-level interfaces to assign them to workspaces. Identity federation is available for all workspaces that are enabled for Unity Catalog. See How do admins assign users to workspaces?.

Execute SQL cells in the notebook in parallel

March 15, 2023

You can now run SQL cells in Databricks notebooks in parallel while attached to an interactive cluster. See Execute SQL cells in parallel.

Create job tasks using Python code stored in a Git repo

March 14, 2023

You can now retrieve Python code from a Git provider when you add a Python task to a Databricks job, simplifying integration of your jobs with your CI/CD workflows. See Use Git with jobs.

Databricks Terraform provider updated to version 1.13.0

March 14, 2023

Version 1.13.0 adds a databricks_sql_alert resource, experimental support for additional resources, and more. For more details, see the changelog for version 1.13.0.

Databricks Terraform provider updated to version 1.12.0

March 9, 2023

Version 1.12.0 adds support for Unity Catalog on Google Cloud, deprecates the schedule block within the databricks_sql_query resource, and more. For more details, see the changelog for version 1.12.0.

SQL admin console and workspace admin console combined

March 9, 2023

The SQL admin console has been combined with the general admin settings to create a unified experience for admin users. All SQL admin settings are now accessed from the admin console.

New Catalog Explorer availability

March 6, 2023

A new version of Catalog Explorer is now available in all workspaces that use a supported version of Databricks Runtime 7.3 or later.

Databricks Runtime 12.2 LTS and Databricks Runtime 12.2 LTS ML are GA

March 2, 2023

Databricks Runtime 12.2 LTS and Databricks Runtime 12.2 LTS ML are now generally available.

See Databricks Runtime 12.2 LTS and Databricks Runtime 12.2 LTS for Machine Learning.

Workspace files are now in Public Preview

March 6, 2023

You can now work with non-notebook files in Databricks. Workspaces files are enabled by default in all workspaces. See What are workspace files?.