December 2021

These features and Databricks platform improvements were released in December 2021.

Note

Releases are staged. Your Databricks account may not be updated until a week or more after the initial release date.

Databricks JDBC driver 2.6.21

December 24, 2021

We have released version 2.6.21 of the Databricks JDBC driver (download). This release includes an upgrade of the Log4j library to version 2.17.0.

Databricks Connector for Tableau 2021.4

December 23, 2021

We have released a new version of the Databricks connector in Tableau 2021.4. This release adds support for navigating a three-level namespace in the Unity Catalog.

Databricks Runtime 10.2 and 10.2 ML are GA; 10.2 Photon is Public Preview

December 22, 2021

Databricks Runtime 10.2 and Databricks Runtime 10.2 ML are now generally available. Databricks Runtime 10.2 Photon is in Public Preview.

See Databricks Runtime 10.2 (EoS) and Databricks Runtime 10.2 for ML (EoS).

Updated Markdown parser

December 13, 2021 - January 6, 2022: Version 3.61

The Markdown parser used in Databricks notebooks has been updated. The new version is faster when rendering. Some behavior has changed. The new behavior is closer to the commonmark spec.

The primary changes are:

You must now put a space between the hashtags and the text of a heading.
When you mix HTML and Markdown code in a cell, you must put a newline after each HTML start tag.

For example, the Markdown code to produce this result:

is as shown:

%md

## Heading

<del>

*example text*
</del>

User interface improvements for Delta Live Tables

December 13, 2021 - January 6, 2022: Version 3.61

This release includes the following enhancements to the Delta Live Tables UI:

When you create a pipeline, you can directly edit the JSON specification in the Create Pipeline dialog by clicking the JSON button.
You can view tooltips for data quality metrics by hovering over data quality values in the processing graph view for your pipeline.

New workspaces do not rely upon Google Cloud Deployment Manager

December 14, 2021

New workspaces do not use Google Cloud Deployment Manager to deploy network and storage cloud resources. There are two consequences of this change:

When you create a workspace, Databricks no longer enables the Google Cloud Deployment Manager API if it is not enabled. For the list of required APIs, see Enabling Google APIs on a workspace’s project
When you create a workspace, Databricks no longer includes the ten permissions with the deploymentmanager. prefix in the custom role that Databricks automatically creates while launching a workspace. For the full list of permissions, see Required permissions for the workspace service account.

This change does not affect existing workspaces. The custom role created for existing workspaces continues to include the ten deploymentmanager. permissions. If the workspace is deleted, the workspace deployment and cloud resources are deleted.

Databricks Runtime 8.3 series support extended

December 8, 2021

Support for Databricks Runtime 8.3 and Databricks Runtime 8.3 for Machine Learning has been extended from December 8, 2021 to January 20, 2022. See All supported Databricks Runtime releases.

Support for GPU clusters (Public Preview)

December 7, 2021

GPU-based clusters are now available for Databricks clusters running Databricks Runtime 9.1 ML and above. Databricks supports the accelerator-optimized A2 machine type, which uses A100 GPUs. For more information about the properties of each machine type, see the Google Cloud GPU models documentation. Also see GPU-enabled compute for more information.

Databricks Runtime 10.2 (Beta)

December 6, 2021

Databricks Runtime 10.2, 10.2 Photon, and 10.2 ML are now available as Beta releases.

See Databricks Runtime 10.2 (EoS) and Databricks Runtime 10.2 for ML (EoS).

Revert of recent breaking change that removed escaping and quotes from $ in environment variable values for cluster creation

December 3, 2021

The breaking change to environment variable behavior that was introduced in some workspaces last month will be reverted on December 3, 2021 from 01:00-03:00 UTC.

Accelerate workloads with Photon (Public Preview)

December 2, 2021

Photon is the new native vectorized engine on Databricks, directly compatible with Apache Spark APIs. It is included as part of a new high-performance runtime designed to run your workloads faster and reduce your total cost per workload. This runtime is a variant of the standard Databricks Runtime for version 9.1 and above. Photon is selectable when you create clusters in the UI, the Clusters API, or the Jobs API (specified in the APIs as 9.1.x-photon-scala2.12).

Photon supports a limited set of instance types on the driver and worker nodes. Instance types with Photon consume DBUs at a different rate than the same instance type running a non-Photon runtime. For more information about Photon instance types and Photon DBU consumption, see the Databricks pricing page.