Google
Cloud Platform
Amazon Web Services
Microsoft AzureJuly 2023
These features and Databricks platform improvements were released in July 2023.
Note
Releases are staged. Your Databricks workspace might not be updated until a week or more after the initial release date.
Workspace admins can now create account groups
July 31, 2023
Workspace admins can now create account groups from their identity federated workspaces. Previously workspace admins could create only workspace-local groups. Account groups can be granted access to data in a Unity Catalog metastore and access to identity-federated workspaces. Databricks recommends that you use account groups instead of workspace-local groups to take advantage of Unity Catalog and a central place to administer identity.
See Manage account groups using the workspace admin settings page.
Group manager role is in Public Preview
July 31, 2023
You can now grant Databricks users, service principals, and groups permissions to manage groups. Group managers can manage group membership. They can also assign other users the group manager role. Account admins have the group manager role on all groups in the account. Workspace admins have the group manager role on account groups that they create.
Databricks CLI updated to version 0.202.0 (Public Preview)
July 27, 2023
The Databricks command-line interface (Databricks CLI) has been updated to version 0.202.0. For details, see the changelog for version 0.202.0.
Databricks SDK for Python updated to version 0.3.0 (Beta)
July 27, 2023
Databricks SDK for Python version 0.3.0 introduces support for the Account Network Policy service; handles nested query parameters; adds, removes, and creates various methods, fields, dataclasses, and services; and more. For details, see the changelog for version 0.3.0.
Databricks SDK for Go updated to version 0.14.1 (Beta)
July 27, 2023
Databricks SDK for Go version 0.14.1 handles nested query parameters appropriately, fixing filtering in QueryHistory listings. For details, see the changelog for version 0.14.1.
Databricks SDK for Go updated to version 0.14.0 (Beta)
July 26, 2023
Databricks SDK for Go version 0.14.0 adds support for the Account Network Policy service, and adds, removes, and changes several methods, fields, and types. For details, see the changelog for version 0.14.0.
Run another job as a task in a Databricks job
July 25, 2023
You can use the new Run Job task to add another job as a task in your Databricks job, allowing you to decompose a large processing workflow into multiple component jobs, or create reusable components to use in multiple jobs. For information about using the task in the Jobs UI, see Schedule and orchestrate workflows. For information about using the task with the Jobs REST API, see Jobs in the REST API 2.1 reference or the Jobs API 2.0 reference.
Enable customer-managed keys for managed services (Public Preview)
July 24, 2023
You can add your own key to protect and control access to managed services data, which includes notebooks, secrets, and Databricks SQL queries. To compare the two customer-managed key features and what data they help protect, see Customer-managed keys for encryption.
Account API is generally available
July 24, 2023
The Account API is generally available. Use it to programmatically create and manage workspaces and related configurations for workspace networks and Private Service Connect objects such as private access settings and VPC endpoints. Support for encryption keys remains in Public Preview.
To use the API to create a workspace, also see Create a workspace using the Account API.
Private Service Connect launches to Limited Availability with GA-level functionality
July 24, 2023
Databricks support for private connectivity using Private Service Connect (PSC) is now in Limited Availability, with GA-level functionality. Contact your Databricks account team to request access.
All users can access data products in Databricks Marketplace by default
July 24, 2023
Starting on July 24 and rolling out gradually to all eligible workspaces, all users will be able to install free or previously purchased datasets using Databricks Marketplace. This is made possible by granting the USE MARKETPLACE ASSETS privilege to all users in all Unity Catalog metastores. This new privilege has no cost impact, since all Marketplace transactions take place outside of Databricks. If you’d like to disable access, see Disable Marketplace access.
Classic keyboard shortcuts mode
July 21, 2023
A new setting allows you to use features from the new editor (such as autocomplete as you type, syntax highlighting, code formatting, and more) while retaining the same familiar keyboard shortcuts from the previous editor. To access this setting, open the editor configuration and toggle Notebook editor shortcuts mode to Classic (Code Mirror).
Lakehouse Federation lets you run queries against external database providers (Public Preview)
July 21, 2023
Lakehouse Federation is the query federation platform for Databricks, taking advantage of Unity Catalog to enable users and systems to run queries against multiple external data sources without needing to migrate all data to a unified system.
Lakehouse Federation is intended for use cases like:
Ad hoc reporting.
Proof-of-concept work.
The exploratory phase of new ETL pipelines or reports.
Supporting workloads during incremental migration.
In each of these scenarios, query federation gets you to insights faster, because you can query the data in place and avoid complex and time-consuming ETL processing. Plus you get the advantages of Unity Catalog interfaces and data governance, including fine-grained access control, data lineage, and search.
Move to trash enabled for Repos
July 19, 2023
Deleting repos now works the same way as deleting other workspace assets. In the Repos list, right-click the repo name and select Move to Trash. After 30 days, the Trash folder is automatically deleted permanently.
See Delete an object.
Create alerts for slow-running or stuck jobs
July 18, 2023
You can now configure an expected duration for a Databricks job or job task and add notifications if a job or task exceeds the expected duration. This feature lets you get alerts for slow-running jobs without the run being canceled. To configure an expected duration in the Databricks Jobs UI, see Configure thresholds for job run duration or streaming backlog metrics. To configure the expected duration with the Jobs 2.1 API, see Jobs in the REST API reference.
Databricks SDK for Go updated to version 0.13.0 (Beta)
July 18, 2023
Databricks SDK for Go version 0.13.0 adds, changes, and removes several methods and fields, and more. For details, see the changelog for version 0.13.0.
Databricks SDK for Python updated to version 0.2.0 (Beta)
July 18, 2023
The Databricks SDK for Python version 0.2.0 adds a local implementation of dbutils.widgets, adds, changes, and removes several methods, fields, and dataclasses, and more. For details, see the changelog for version 0.2.0.
Databricks CLI updated to version 0.201.0 (Public Preview)
July 18, 2023
The Databricks command-line interface (Databricks CLI) version 0.201.0 improves the auth login experience, supports tab completion for referencing Databricks authentication configuration profiles, makes additions, changes, and removals of several command groups and commands, and more. For details, see the changelog for version 0.201.0.
Databricks SDK for Python updated to version 0.2.1 (Beta)
July 18, 2023
Databricks SDK for Python version 0.2.1 supports older versions of urllib. For details, see the changelog for version 0.2.1.
Databricks Assistant is in Public Preview
July 18, 2023
The Databricks Assistant works as an AI-based companion pair-programmer to make you more efficient as you create notebooks, queries, and files. It can help you rapidly answer questions by generating, optimizing, completing, explaining, and fixing code and queries. See the Databricks Assistant FAQ for more information and for instructions on how to enable the Assistant.
Deactivate users and service principals from your account
July 13, 2023
You can now deactivate users and service principals from your Databricks account. A deactivated user cannot log in to the Databricks account or identity federated workspaces. However, all of the user’s permissions and workspace objects remain unchanged. For more information, see Deactivate a user in your Databricks account and Deactivate a service principal in your Databricks account.
Account-level SCIM provisioning now deactivates users when they are deactivated in the identity provider
July 13, 2023
Account-level SCIM provisioning now deactivates users when they are deactivated in your identity provider. Previously, when a user was deactivated in an identity provider, account-level SCIM provisioning deleted them from the Databricks account. For more information, see Sync users and groups from your identity provider and Deactivate a user in your Databricks account.
Trash directory admin access
July 13, 2023
Workspace admins can now access other users’ Trash directories.
Prevention of MIME type sniffing and XSS attack page rendering are now always enabled
July 12, 2023
Databricks reduces the risk of MIME type sniffing and XSS attack page rendering by adding appropriate HTTP headers. These features were enabled by default and previously could be disabled. For improved security, Databricks now always enables both features and there are no longer workspace admin settings to disable them. For workspaces that previously disabled one or both of these features, both features are now enabled.
Unity Catalog volumes are in Public Preview
July 12, 2023
You can now use volumes in Unity Catalog to manage access to cloud object storage locations on Databricks Runtime 13.2 and above. Volumes provide capabilities for accessing, storing, governing, and organizing files. See What are Unity Catalog volumes?.
Simplified experience for submitting product feedback from the workspace
July 11, 2023
It’s now easier to send product feedback from your workspace. You can submit feedback and attach screenshots with fewer steps and without leaving the workspace UI. For more information, see Submit feedback from the workspace.
Databricks extension for Visual Studio Code updated to version 1.1.0
July 10, 2023
The Databricks extension for Visual Studio Code version 1.1.0 enables Databricks Connect integration by default, adds experimental features for Jupyter notebooks, and more. For details, see the changelog for version 1.1.0.
Functions now displayed in Catalog Explorer (Public Preview)
July 10, 2023
SQL and Python user-defined functions are now visible in Catalog Explorer. You can view function details and manage permissions using this new UI.
Databricks Terraform provider updated to version 1.21.0
July 7, 2023
Databricks Terraform provider version 1.21.0 adds support for subscriptions in dashboards and alert SQL tasks in databricks_job, defines generic Databricks data utilities for defining workspace and account-level data sources, and more. For details, see the changelog for version 1.21.0.
The maximum offset for the List all jobs and List job runs API requests is now limited
July 6, 2023
The maximum offset you can specify in the List all jobs and List job runs requests in the Jobs API is now limited. To avoid this limit when you use these requests, use token-based pagination which does not have this limit. See GET /api/2.1/jobs/list and GET /api/2.1/jobs/runs/list in the REST API reference.
Databricks Runtime 13.2 is GA
July 6, 2023
Databricks Runtime 13.2 and Databricks Runtime 13.2 ML are now generally available.
See Databricks Runtime 13.2 (EoS) and Databricks Runtime 13.2 for Machine Learning (EoS).
Delta Sharing and Databricks Marketplace support view sharing (Public Preview)
July 6, 2023
You can now share views in Delta Sharing and Databricks Marketplace. See Add views to a share.
Init scripts on DBFS reach end of life on Sept 1, 2023
July 5, 2023
On Sept 1, 2023, support for init scripts on DBFS will reach end of life and the feature will no longer function. Store init scripts in workspace files to ensure that they continue to function after Sept 1, 2023.