August 2023

These features and Databricks platform improvements were released in August 2023.

Note

Releases are staged. Your Databricks workspace might not be updated until a week or more after the initial release date.

Databricks CLI updated to version 0.203.3 (Public Preview)

August 30, 2023

The Databricks command-line interface (Databricks CLI) has been updated to version 0.203.3. For details, see the changelog for version 0.203.3.

Databricks JDBC driver 2.6.34

August 30, 2023

We have released version 2.6.34 of the Databricks JDBC driver (download). This release addresses a number of vulnerability issues by upgrading third-party libraries. In addition, this release fixes the following issues:

  • Thread leakage when a query fails to connect to the server.

  • Missing comments in the REMARKS column of the table metadata.

  • When a resultset closure operation returns an error, the connector does not clean up the operation handle entries from the heartbeat thread.

The release also improves error message handling by honoring SQLState and error codes from the server, and checks the server protocol version correctly.

New Google Cloud region: northamerica-northeast1 (Montréal, Canada)

August 31, 2023

Databricks on Google Cloud is now available in the northamerica-northeast1 (Montréal, Canada) region.

See Databricks clouds and regions.

Databricks SDK for Go updated to version 0.18.0 (Beta)

August 29, 2023

Databricks SDK for Go version 0.18.0 introduces 17 breaking API changes, adds 30 APIs, and adds about a dozen fields. For details, see the changelog for version 0.18.0.

Databricks SDK for Python updated to version 0.7.0 (Beta)

August 29, 2023

Databricks SDK for Python version 0.7.0 introduces a breaking change for the files API as well as 17 additional breaking changes. This release also adds 14 fields and adds 27 dataclasses. For details, see the changelog for version 0.7.0.

Databricks Terraform provider updated to version 1.24.1

August 29, 2023

Databricks Terraform provider version 1.24.1 includes a fix for verification of workspace reachability. For details, see the changelog for version 1.24.1.

Init scripts on DBFS end of life extended to Dec 1, 2023

August 29, 2023

Databricks previously announced that init scripts on DBFS would reach end of life on Sept 1, 2023. The end of life date has been extended to Dec 1, 2023.

Databricks Runtime 14.0 (Beta)

August 29, 2023

Databricks Runtime 14.0 and Databricks Runtime 14.0 ML are now available as Beta releases.

See Databricks Runtime 14.0 (EoS) and Databricks Runtime 14.0 for Machine Learning (EoS).

Unified navigation experience is GA

August 28, 2023

The unified Databricks navigation experience is now generally available. Users can no longer revert to the classic navigation experience. For more information, see Navigate the workspace.

Databricks Terraform provider updated to version 1.24.0

August 25, 2023

Databricks Terraform provider version 1.24.0 adds account-level API support for Unity Catalog objects, adds acl_principal_id to several data sources, and more. For details, see the changelog for version 1.24.0.

Shared access mode clusters can now connect to public networks outside of Databricks VPC

August 25, 2023

Shared access mode clusters using Databricks Runtime 12.0 and above can now connect to public networks outside of the Databricks VPC. Users are no longer limited to creating network connections in ports 80 and 443. See Compute access mode limitations for Unity Catalog.

Databricks CLI updated to version 0.203.2 (Public Preview)

August 24, 2023

The Databricks command-line interface (Databricks CLI) has been updated to version 0.203.2. For details, see the changelog for version 0.203.2.

Go to definition for functions and variables in Python notebooks

August 24, 2023

In Databricks Runtime 12.2 LTS and above, when a Python notebook is attached to a cluster, you can automatically navigate to the definition of a variable or function that is defined in that notebook. See Go to definition for details.

Databricks Runtime 13.3 LTS is GA

August 22, 2023

Databricks Runtime 13.3 LTS and Databricks Runtime 13.3 LTS ML are now generally available.

See Databricks Runtime 13.3 LTS and Databricks Runtime 13.3 LTS for Machine Learning.

Introducing tags with Unity Catalog

August 22, 2023

You can now apply tags to your securable objects in Unity Catalog. Tags help you organize securable objects and simplify search and discovery of your data assets. See Apply tags to Unity Catalog securable objects.

Allowlist for JARs and Maven coordinates on Unity Catalog shared clusters is in Public Preview

August 23, 2023

You can now use the Unity Catalog allowlist to control installation of JARs and Maven coordinates on compute with shared access mode running Databricks Runtime 13.3 and above. See Allowlist libraries and init scripts on shared compute.

Volumes support for JARs is in Public Preview

August 23, 2023

You can now use Unity Catalog volumes to store JARs on compute with assigned or shared access modes running Databricks Runtime 13.3 and above. See Install libraries from a volume.

Easier Databricks Repos .ipynb file output commits

August 21, 2023

Databricks Repos has added features to make .ipynb file output commits easier to control and see:

  • Repos can create an commit_output config file for you.

  • Notifications in the UI let you know the status of output commits.

  • From the File menu, you can control whether noteook outputs are committed.

See Control .ipynb notebook output commits.

IPYNB notebook support in Databricks Repos is GA

August 21, 2023

IPYNB notebook file support in Databricks Repos is GA. For more information about Repos, see:

Databricks SDK for Go updated to version 0.17.0 (Beta)

August 18, 2023

Databricks SDK for Go version 0.17.0 adds over 30 APIs and renames about 10 APIs. For details, see the changelog for version 0.17.0.

Databricks SDK for Python updated to version 0.6.0 (Beta)

August 18, 2023

Databricks SDK for Python version 0.6.0 adds about 20 dataclasses, adds about 10 APIs, and includes about 20 renames. For details, see the changelog for version 0.6.0.

Databricks CLI updated to version 0.203.1 (Public Preview)

August 16, 2023

The Databricks command-line interface (Databricks CLI) has been updated to version 0.203.1. For details, see the changelog for version 0.203.1.

Unified schema browser is now GA

August 16, 2023

A unified way to navigate your data is now available for notebooks, SQL Editor, and Catalog Explorer, including the following new capabilities:

  • Condensed version of the tree.

  • Display of Unity Catalog object count at the schema level when filtering.

  • Right-click menu to view actions.

Databricks SDK for Go updated to version 0.16.0 (Beta)

August 14, 2023

Databricks SDK for Go version 0.16.0 adds about a dozen fields, removes about a half-dozen fields, and changes the signatures of about a half-dozen APIs. For details, see the changelog for version 0.16.0.

Partner Connect supports Census

August 14, 2023

You can now use Partner Connect to connect your Databricks workspace to Census. For more information, see Connect to Census.

Databricks SDK for Python updated to version 0.5.0 (Beta)

August 11, 2023

Databricks SDK for Python version 0.5.0 changes the signatures of a half-dozen APIs, adds a dozen fields, removes a half-dozen fields, and adds a few dataclasses. For details, see the changelog for version 0.5.0.

Programmatic write support for workspace files

August 11, 2023

Clusters using Databricks Runtime 11.2 and above support writing to workspace files. You can programmatically create, update, and delete files and directories within your workspace.

Databricks CLI updated to version 0.203.0 (Public Preview)

August 10, 2023

The Databricks command-line interface (Databricks CLI) version 0.203.0 improves the auth login command, and more. For details, see the changelog for version 0.203.0.

Databricks Terraform provider updated to version 1.23.0

August 9, 2023

Databricks Terraform provider version 1.23.0 extends RunJobTask with additional supported fields in databricks_job, adds exporter support for databricks_model_serving, and more. For details, see the changelog for version 1.23.0.

Groups can now be renamed

August 8, 2023

You can now rename existing account groups and workspace-local groups. See Change the name of a group.

Databricks SDK for Go updated to version 0.15.0 (Beta)

August 7, 2023

Databricks SDK for Go version 0.15.0 refactors permissions-related APIs to make them more understandable and easier to use. For details, see the changelog for version 0.15.0.

Databricks SDK for Python updated to version 0.4.0 (Beta)

August 7, 2023

Databricks SDK for Python version 0.4.0 refactors permissions-related APIs to make them more understandable and easier to use. For details, see the changelog for version 0.4.0.

Third-party iFraming prevention configuration setting was removed

August 7, 2023

Databricks reduces the risk of third-party domains from iFraming Databricks by adding appropriate HTTP headers. Third-party iFraming prevention is still enabled by default but Databricks removed the workspace admin setting to disable it. If you previously disabled third-party iFraming for a workspace, the feature continues to be disabled for that workspace. To re-enable third-party iFraming, contact your Databricks account team.

Databricks extension for Visual Studio Code updated to version 1.1.1

August 4, 2023

The Databricks extension for Visual Studio Code version 1.1.1 provides several minor fixes, for instance correctly handling the HOME environment variable on Windows, and more. For details, see the changelog for version 1.1.1.

LangChain available in 13.1 and above

August 4, 2023

LangChain is available as an experimental MLflow flavor which allows LangChain customers to leverage the robust tools and experiment tracking capabilities of MLflow directly from the Databricks environment.

LangChain is a software framework designed to help create applications that utilize large language models (LLMs) and combine them with external data to bring more training context for LLMs.

Databricks Runtime for Machine Learning includes langchain in Databricks Runtime 13.1 ML and above.

Feature Engineering in Unity Catalog is Public Preview

August 3, 2023

With Feature Engineering in Unity Catalog, Unity Catalog becomes your feature store. You can use any Delta table with a primary key as a feature table for model training or inference. Unity Catalog provides feature discovery and governance.

Improved error handling for repeated continuous job failures

August 3, 2023

Continuous jobs exceeding a threshold for consecutive failures now use an exponential backoff scheme to efficiently retry job runs and allow jobs to recover from transient issues. Previously, continuous jobs exceeding the failure threshold were paused and not retried.

Share schemas using Delta Sharing (Public Preview)

August 2, 2023

You can now use Delta Sharing to share an entire schema, giving the recipient access to all of the tables and views in the schema at the moment you share it, along with any tables and views that are added to the schema in the future. Adding schemas to a share using SQL commands requires a SQL warehouse or a cluster running Databricks Runtime 13.2 or above. Doing the same using Catalog Explorer has no compute requirements. See Add schemas to a share.

Run tasks conditionally in your Databricks jobs

August 2, 2023

You can use the new Run if condition to run a job task based on the status of the task’s dependencies, providing more flexibility when you implement a workflow with a Databricks job. For example, you can use the Run if condition to run a task when some or all of the task’s dependencies have failed. See Configure task dependencies.

Delta Live Tables is now generally available

August 2, 2023

Delta Live Tables is now generally available on Google Cloud. See What is Delta Live Tables?.

Databricks Terraform provider updated to version 1.22.0

August 1, 2023

Databricks Terraform provider version 1.22.0 adds a databricks_access_control_rule_set resource, adds databricks_metastore and databricks_metastore data sources, and more. For details, see the changelog for version 1.22.0.

Databricks Runtime 13.3 LTS (Beta)

August 1, 2023

Databricks Runtime 13.3 LTS and Databricks Runtime 13.3 LTS ML are now available as Beta releases.

See Databricks Runtime 13.3 LTS and Databricks Runtime 13.3 LTS for Machine Learning.

New Git operations are generally available: Merge branches, rebase and pull with conflict resolution

August 1, 2023

The following Git operations are now generally available in Databricks Repos:

  • Merge and rebase: You can now Git merge or rebase two branches directly in Databricks. This simplifies the process of combining branches and provides the ability to resolve conflicts that arise during the operation.

  • Pull with conflict resolution: You can now use Databricks Repos to manage and resolve conflicts that happen when you pull from a remote Git repo using Databricks.

With these powerful new conflict resolution tools, collaborating on code using Databricks Repos becomes even more streamlined and efficient.