Google
Cloud Platform
Amazon Web Services
Microsoft AzureSeptember 2023
These features and Databricks platform improvements were released in September 2023.
Note
Releases are staged. Your Databricks workspace might not be updated until a week or more after the initial release date.
Lakehouse Federation available on single user clusters
September 29, 2023
Lakehouse Federation is now available on single user clusters using Databricks Runtime 13.1 and above. Only the connection owner can run queries on federated catalogs. See What is Lakehouse Federation?.
Delegate the ability to create a storage credential in Unity Catalog
September 29, 2023
You can now grant users permissions to create a storage credential in a Unity Catalog metastore using the new CREATE STORAGE CREDENTIAL privilege. See CREATE STORAGE CREDENTIAL.
Databricks Runtime 14.1 (Beta)
September 28, 2023
Databricks Runtime 14.1 and Databricks Runtime 14.1 ML are now available as Beta releases.
See Databricks Runtime 14.1 and Databricks Runtime 14.1 for Machine Learning.
Prevent runs of a job from being skipped when you reach concurrency limits
September 27, 2023 - October 2, 2023
To prevent runs of a Databricks job from being skipped because of concurrency limits, you can now enable queueing on the job. When queueing is enabled, if a concurrency limit is reached, the job is placed in a queue until capacity is available to run the job. See Enable queueing of job runs.
Databricks Terraform provider updated to version 1.27.0
September 27, 2023
Databricks Terraform provider version 1.27.0 adds owner and force_destroy to databricks_metastore_data_access, includes several additional improvements and bug fixes, and more. For details, see the changelog for version 1.27.0.
Databricks CLI updated to version 0.206.0 (Public Preview)
September 27, 2023
The Databricks command-line interface (Databricks CLI) has been updated to version 0.206.0. This release enables target overrides for pipeline clusters in Databricks Asset Bundles, improves the bundle template initialization experience, and more. For details, see the changelog for version 0.206.0.
Databricks SDK for Go updated to version 0.21.0 (Beta)
September 27, 2023
Databricks SDK for Go version 0.21.0 introduces over a dozen breaking API changes, adds over a dozen APIs, and adds a dozen fields. For details, see the changelog for version 0.21.0.
Databricks ODBC driver 2.7.5
September 23, 2023
We have released version 2.7.5 of the Databricks ODBC driver (download). This release adds the following new features and enhancements:
OAuth 2.0 client authentication enhancement and OIDC endpoint discovery support.
Server-side encryption with customer provided keys.
Timedate function.
Improved error messaging for Cloud Fetch.
Security updates.
This release also resolves the following regression in 2.7.3:
The
SQLForeignKeysfunction returns an empty result if the caller passes aNULLvalue for the catalog of the primary table or the foreign table.
The release 2.7.5 now returns all the primary keys or foreign keys associated with the related table if the catalog of the other table is set NULL.
Databricks extension for Visual Studio Code updated to version 1.1.3
September 21, 2023
The Databricks extension for Visual Studio Code version 1.1.3 adds a few minor fixes. For details, see the changelog for version 1.1.3.
Running jobs as a service principal is GA
September 21, 2023
Roles on service principals and the job’s setting to run jobs using the service principal’s identity is now GA.
See Roles for managing service principals and Configure identity for job runs.
Databricks CLI updated to version 0.205.2 (Public Preview)
September 21, 2023
The Databricks command-line interface (Databricks CLI) has been updated to version 0.205.2. This release contains a few minor updates. For details, see the changelog for version 0.205.2.
Databricks CLI updated to version 0.205.1 (Public Preview)
September 20, 2023
The Databricks command-line interface (Databricks CLI) has been updated to version 0.205.1. This release enables target overrides for job tasks in Databricks Asset Bundles, and more. For details, see the changelog for version 0.205.1.
Databricks SDK for Go updated to version 0.20.0 (Beta)
September 20, 2023
Databricks SDK for Go version 0.20.0 adds over a dozen APIs, adds a half-dozen fields, changes a few fields and methods, and removes a few APIs. For details, see the changelog for version 0.20.0.
Databricks SDK for Python updated to version 0.9.0 (Beta)
September 20, 2023
Databricks SDK for Python version 0.9.0 changes a few methods and fields, adds about 10 fields, adds about 20 dataclasses, and introduces additional changes and removals. For details, see the changelog for version 0.9.0.
Databricks SDK for Python updated to version 0.7.1 (Beta)
September 20, 2023
Databricks SDK for Python version 0.7.1 includes a minor fix to improve file download performance. For details, see the changelog for version 0.7.1.
Databricks Terraform provider updated to version 1.26.0
September 20, 2023
Databricks Terraform provider version 1.26.0 adds support for init scripts from Unity Catalog volumes, includes several additional improvements and bug fixes, and more. For details, see the changelog for version 1.26.0.
Databricks Connect for Databricks Runtime 14.0
September 20, 2023
Databricks Connect for Databricks Runtime 14.0 supports full Structured Streaming (including foreachBatch) and long running queries that reattach to the query after interruption.
Databricks Connect V2 is Public Preview for Scala
September 19, 2023
Databricks Connect for Databricks Runtime 13.3 LTS now supports Scala in Public Preview, including DataFrames, Delta, Structured Streaming, and more.
For Python, query cancellation is now supported, and the DatabricksSession class can now be used in Databricks notebooks and jobs.
Databricks Runtime 14.0 is GA
September 14, 2023
Databricks Runtime 14.0 and Databricks Runtime 14.0 ML are now generally available.
See Databricks Runtime 14.0 (EoS) and Databricks Runtime 14.0 for Machine Learning (EoS).
Account email notifications upgrade
September 14, 2023
Databricks no longer sends email notifications when users are added to a Databricks account. Users will continue to receive email notifications when they are granted access to a workspace.
Databricks Terraform provider updated to version 1.25.1
September 13, 2023
Databricks Terraform provider version 1.25.1 has been released with minor improvements and bug fixes. For details, see the changelog for version 1.25.1.
Databricks SDK for Go updated to version 0.19.2 (Beta)
September 13, 2023
Databricks SDK for Go version 0.19.2 contains one minor fix. For details, see the changelog for version 0.19.2.
Databricks CLI updated to version 0.205.0 (Public Preview)
September 13, 2023
The Databricks command-line interface (Databricks CLI) has been updated to version 0.205.0. This is the first release of the Databricks CLI to support Databricks Asset Bundles in Public Preview. For details, see the changelog for version 0.205.0.
Reserved location for /Workspace
September 13, 2023
The location /Workspace is now reserved. You can no longer define a directory named /Workspace relative to the root path in Databricks.
Databricks Asset Bundles (Public Preview)
September 12, 2023
Databricks Asset Bundles, now in Public Preview, enable end-to-end data, analytics, and ML projects to be expressed as a collection of source files. This makes it simpler to apply data engineering best practices such as source control, code review, testing, and CI/CD.
Databricks Terraform provider updated to version 1.25.0
September 12, 2023
Databricks Terraform provider version 1.25.0 includes the addition of IS_OWNER permission for Databricks SQL warehouses, minor fixes, documentation updates, and more. For details, see the changelog for version 1.25.0.
Databricks extension for Visual Studio Code updated to version 1.1.2
September 12, 2023
The Databricks extension for Visual Studio Code version 1.1.2 adds experimental support for %run and # MAGIC in Databricks notebooks, fixes path issues with multiple local Databricks CLI installations, and more. For details, see the changelog for version 1.1.2.
Partner Connect supports Snowplow
September 12, 2023
You can now use Partner Connect to connect your Databricks workspace to Snowplow. For more information, see Connect to Snowplow.
Databricks CLI updated to version 0.204.1 (Public Preview)
September 11, 2023
The Databricks command-line interface (Databricks CLI) has been updated to version 0.204.1. For details, see the changelog for version 0.204.1.
Filter sensitive data with row filters and column masks (Public Preview)
September 11, 2023
You can now apply row filters and column masks to tables to prevent access to sensitive data by certain users. These filters and masks are implemented as SQL user-defined functions (UDFs). See Filter sensitive table data using row filters and column masks.
Databricks SDK for Go updated to version 0.19.1 (Beta)
September 7, 2023
Databricks SDK for Go version 0.19.1 continues a few minor updates and fixes. For details, see the changelog for version 0.19.1.
Databricks ODBC driver 2.7.3
September 7, 2023
We have released version 2.7.3 of the Databricks ODBC driver (download). This release includes the following features and fixes:
Vulnerability fixes by upgrading third-party dependencies.
SQL_ATTR_MAX_ROWSsupport that allows a client to limit the number of rows to fetch in a query to expedite fast loading (such as preview).Updated MacOS support. The driver now supports Apple Silicon/M1.
OAuth2 browser-based authentication support to facilitate single sign-on (SSO).
Important
If you use primary key and foreign key APIs, the SQLForeignKeys function returns an empty result if the caller passes a NULL value for the catalog of the primary table or the foreign table. This issue is known to developers.
Databricks CLI updated to version 0.204.0 (Public Preview)
September 6, 2023
The Databricks command-line interface (Databricks CLI) has been updated to version 0.204.0. This release includes the grouping of permission related commands, adds and changes to several commands, and more. For details, see the changelog for version 0.204.0.
Data Explorer is now Catalog Explorer
September 6, 2023
Data Explorer is renamed to Catalog Explorer to recognize the fact that you can use it to work with all securable objects in Unity Catalog, not just data objects. See What is Catalog Explorer?.
Databricks SDK for Go updated to version 0.19.0 (Beta)
September 4, 2023
Databricks SDK for Go version 0.19.0 renames permissions APIs, adds 8 APIs, adds 5 fields, changes and removes a few fields, and changes a few methods.. For details, see the changelog for version 0.19.0.
Databricks SDK for Python updated to version 0.8.0 (Beta)
September 4, 2023
Databricks SDK for Python version 0.8.0 renames permissions APIs, adds a half-dozen fields, adds a half-dozen dataclasses, and introduces several changes and removals. For details, see the changelog for version 0.8.0.