Google
Cloud Platform
Amazon Web Services
Microsoft AzureJune 2023
These features and Databricks platform improvements were released in June 2023.
Note
Releases are staged. Your Databricks workspace might not be updated until a week or more after the initial release date.
Databricks Terraform provider updated to version 1.20.0
June 29, 2023
Databricks Terraform provider version 1.20.0 adds support for the Unity Catalog databricks_catalog_workspace_binding resource, improves the export of users and groups at the account level, and more. For details, see the changelog for version 1.20.0.
Databricks CLI updated to version 0.200.1 (Public Preview)
June 28, 2023
Databricks CLI version 0.200.1 updates the fs ls and fs cp commands, supports local files as the default for fs commands, adds provider detection to the repos create command, and more. For details, see the changelog for version 0.200.1.
Databricks SDK for Go updated to version 0.12.0 (Beta)
June 28, 2023
Databricks SDK for Go version 0.12.0 adds support for job-level parameters and parameter definitions, job task conditional execution, and several bug fixes for other services. For details, see the changelog for version 0.12.0.
Databricks SDK for Go updated to version 0.11.0 (Beta)
June 28, 2023
Databricks SDK for Go version 0.11.0 introduces several breaking changes to the Go APIs, including dozens of additions, renames, and consolidations. For details, see the changelog for version 0.11.0.
Databricks SDK for Python (Beta)
June 28, 2023
The Databricks SDK for Python is now available as a Beta. The Databricks SDK for Python enables you to automate Databricks accounts, workspaces, and resources by running Python code.
See the Databricks SDK for Python.
Databricks SDK for Go (Beta)
June 28, 2023
The Databricks SDK for Go is now available as a Beta. The Databricks SDK for Go enables you to automate Databricks accounts, workspaces, and resources by running Go code.
See the Databricks SDK for Go.
Models in Unity Catalog (Public Preview)
June 28, 2023
You can now upgrade MLflow Model Registry workflows to govern models through Unity Catalog. Unity Catalog provides centralized access control, auditing, lineage, model sharing across workspaces, and better MLOps deployment workflows. Databricks recommends using Models in Unity Catalog instead of the existing workspace model registry, which will be deprecated in the future. See Manage model lifecycle in Unity Catalog for details.
Databricks Marketplace is now GA
June 27, 2023
You can now use Databricks Marketplace to discover, access, and deploy hundreds of data products and notebooks from top providers into your AI and analytics workflows. Highlights since the beginning of the public preview include new providers—including Dun and Bradstreet, LexisNexis, Refinitiv (London Stock Exchange Group), YipitData, and ZoomInfo—along with the addition of private exchanges and simplified consumer permission management using the USE_MARKETPLACE_ASSETS privilege.
Databricks Runtime 13.2 (Beta)
June 26, 2023
Databricks Runtime 13.2 and Databricks Runtime 13.2 ML are now available as Beta releases.
See Databricks Runtime 13.2 (EoS) and Databricks Runtime 13.2 for Machine Learning (EoS).
Databricks extension for Visual Studio Code (General Availability)
June 26, 2023
The Databricks extension for Visual Studio Code is now generally available. Connect to the lakehouse, perform interactive debugging, and run Jupyter notebooks, all without leaving your IDE. Use your IDE’s code navigation, refactoring, completion, and error identification.
See a visual overview of completed job runs in the Databricks Jobs UI
June 22, 2023
An enhanced Databricks Jobs UI provides a visual overview of completed job runs, filterable by run status and time, and a list of the most common job errors. The default time filter covers the previous 48 hours. See View recent job runs.
Databricks CLI (Public Preview)
June 21, 2023
The Databricks command-line interface (Databricks CLI) has undergone a major overhaul. The new CLI covers all Databricks REST API operations and supports all Databricks authentication types. macOS and Linux users can install the new CLI with Homebrew. Windows is also supported.
Improved pagination of results from List all jobs and List job runs API requests
June 14, 2023
You can now use the page_token parameter to page the results returned from the GET /api/2.1/jobs/list and GET /api/2.1/jobs/runs/list Jobs API requests. This token-based pagination replaces the deprecated offset and limit parameters and improves the service scalability for these requests.
Full-page workspace browser includes Repos
June 14, 2023
The full-page workspace browser experience now extends to Repos. You can browse content in Databricks Repos alongside your workspace content by clicking Workspace in the sidebar. You can also browse content from within a notebook using a contextual browser. For more information about the full-page workspace browser, see Navigate the workspace and Workspace browser.
Databricks Terraform provider updated to version 1.19.0
June 14, 2023
Databricks Terraform provider 1.19.0 adds run_as and run_as_user_name to databricks_job, adds more attributes to databricks_cluster_policy, adds force_destroy to databricks_external_location, and more. For more details, see the changelog for version 1.19.0.
Configure the number of local SSDs in your cluster or instance pool
June 14, 2023
You can now configure the number of local SSDs to attach to your compute resources when you create the resource with the Databricks API.
Run jobs as a service principal (Public Preview)
June 13, 2023
You can now grant Databricks users, service principals, and groups permissions to use a service principal. This allows users to run jobs as the service principal, instead of running as the job owner’s identity, to prevent jobs from failing if a user leaves your organization or a group is modified.
See Roles for managing service principals and Configure identity for job runs.
New service principal UI provides better management experience
June 13, 2023
You can now use the new service principal UI to manage service principals in your workspace. The service principal UI lets you add service principals to your workspace, configure entitlements, and manage service principal permissions. See Manage service principals.
Home folders restored when users are re-added to workspaces
June 9, 2023
When an admin removes a user from a workspace and then re-adds them, the user’s home folder is now restored. Previously, upon being re-added to the workspace, the user received a new empty home folder. Their old home folder was retained as a backup. For more information about workspace folders, see Organize workspace objects into folders.
Databricks Marketplace: Private exchanges are now available
June 7, 2023
Private exchanges enable data providers to share data products with a select group of invited consumers. Private listings appear on the invited consumers’ Marketplace home page, but do not appear in the public marketplace. See Create and manage private exchanges in Databricks Marketplace.
Databricks Marketplace: Consumers can uninstall data products using the UI
June 5, 2023
Data consumers can now uninstall any data product using the Marketplace My requests interface. See Delete installed data products.
Databricks Marketplace: providers can create their own profiles
June 5, 2023
Data providers can now create and update their own profiles, once their provider application is approved. See Create your Marketplace provider profile.
Databricks Connect V2 is GA for Python
June 2, 2023
Beginning with Databricks Runtime 13.1, Databricks Connect is now generally available for Python. This release adds support for Structured Streaming, all Delta features, and Pandas on Spark. It also includes support for OAuth (Experimental), by relying on the included Databricks SDK for Python.
Databricks Terraform provider updated to version 1.18.0
June 2, 2023
Databricks Terraform provider 1.18.0 adds a databricks_volume resource, databricks_model_serving support to databricks_permissions, and more. For more details, see the changelog for version 1.18.0.
New Databricks Marketplace providers
June 1, 2023
Eight new providers have published listings in Databricks Marketplace in the past month. These include:
IQVIA: Healthcare and life sciences data.
Morningstar: Investment offering data and research insights such as ESG risk rating data.
Vaisala: Weather and environment data.
Sucali: Real estate performance metrics for hedge funds and investors.
Veritas Data Research: Comprehensive index of US mortality data from 2010 to present.
Collectors Data Store: Sports and trading card data for advertisers and consumers.
Tego Cyber: Cybersecurity and threat intelligence data
Ribbon Health: Healthcare provider data.
If you have questions about these providers or Marketplace, reach out to dataconsumers@databricks.com. See also What is Databricks Marketplace?.
Databricks Runtime 13.1 is GA
June 1, 2023
Databricks Runtime 13.1 and Databricks Runtime 13.1 ML are now generally available.
See Databricks Runtime 13.1 (EoS) and Databricks Runtime 13.1 for Machine Learning (EoS).
View data from notebooks, SQL editor, and Catalog Explorer with the unified schema browser (Public Preview)
June 1, 2023
The new unified schema browser lets you view all of the data in the Unity Catalog metastore without leaving a notebook or the SQL editor. You can select For you to limit the view to only active data you’re using and your favorites.
For details about Catalog Explorer, see Catalog Explorer.
For information about accessing Catalog Explorer from a notebook, see Browse data.