July 2024

These features and Databricks platform improvements were released in July 2024.

Note

Releases are staged. Your Databricks account might not be updated until a week or more after the initial release date.

Increased limit for simultaneous tasks

July 31, 2024

The workspace limit for tasks running simultaneously has been raised to 2000. See Resource limits.

Embed and drag & drop images in notebooks

July 31, 2024

You can now display images in notebooks by embedding them directly in markdown cells. Drag and drop images from your desktop directly into markdown cells to automatically upload and display them. See Display images and Drag and drop images.

Command palette available in notebooks

July 31, 2024

You can now quickly perform actions in the notebook using the command palette. Press Cmd + Shift + P on MacOS or Ctrl + Shift + P on Windows while in a notebook to access frequently used actions. See Command palette.

Workflow system schema renamed to lakeflow

July 31, 2024

The workflow schema is being updated to lakeflow. We recommend that you switch to lakeflow as it will include all the current tables plus new ones in the future, like pipelines. Customers must opt-in to the lakeflow schema to make it visible in their metastore. See Jobs system table reference.

Support for Cloudflare R2 storage is GA

July 30, 2024

The ability to use Cloudflare R2 as cloud storage for data registered in Unity Catalog is now generally available. Cloudflare R2 is intended primarily for Delta Sharing use cases in which you want to avoid the data egress fees charged by cloud providers when data crosses regions. R2 storage supports all of the Databricks data and AI assets supported in AWS S3, Azure Data Lake Storage Gen2, and Google Cloud Storage. Support for R2 requires a SQL Warehouse or Databricks Runtime 14.3 or above. See Use Cloudflare R2 replicas or migrate storage to R2 and Create a storage credential for connecting to Cloudflare R2.

Monitor Databricks Assistant activities with system tables (Public Preview)

July 30, 2024

You can now monitor Databricks Assistant activities in a dashboard by using system tables. For more information, see Databricks Assistant system table reference and example.

Sharing schemas using Delta Sharing is now GA

July 30, 2024

The ability to share schemas using Delta Sharing is GA. Sharing entire schema gives the recipient access to all of the tables and views in the schema at the moment you share it, along with any tables and views that are added to the schema in the future. Adding schemas to a share using SQL commands requires a SQL warehouse or a cluster running Databricks Runtime 13.2 or above. Doing the same using Catalog Explorer has no compute requirements. See Add schemas to a share.

Databricks Assistant can diagnose issues with jobs (Public Preview)

July 29, 2024

Databricks Assistant can now diagnose issues with failed jobs. See Diagnose errors in jobs.

Updates to Databricks Git folders authentication and sharing behaviors

July 29, 2024

• Git folder dialog-based authentication handling: The user experience is now streamlined to assist users in recovering from authentication errors when opening the Git folder dialog. In the dialog, you can update Git credentials directly, which triggers an automatic retry. You can use this approach to assist in resolving authentication errors.

  • When an authentication error occurs, the Git folder dialog now shows the Git folder’s provider and URL in the error. Previously this was hidden, making it difficult to know which Git credential should be used to resolve the error.

• Git folder sharing: Users can now share a URL link to other workspace users. When the URL is opened in the recipient’s browser, Databricks opens and launches the existing Add Git folder dialog with pre-filled values (such as the Git provider and the Git repository URL). This simplifies Git folder cloning for commonly used Git repositories among your workspace users. See Best practice: Collaborating in Git folders for more details. - Users are now prompted to create their own Git folders in their own workspace rather than working collaboratively in another user’s Git folder. - The Git folder dialog state is now persisted in your URL. If you copy the URL from your browser when the Git folder dialog is open, it can be opened later or shared with another user and the same information will be displayed.

• Git folder diff view: In the Git folder diff view, darker red and green highlighting was added for replaced text and for multiple lines of changes, making it easier to determine what was changed across their uncommitted changes.

  • Opening the Git folder dialog from a notebook or file editor selects that notebook or file in the Git folder dialog and displays the changes (diffs) by default.

Cluster library installation timeout

July 29, 2024

Library installation on clusters now has a timeout of 2 hours. A library that has taken more than 2 hours to install will be marked as failed. For information on cluster libraries, see Cluster libraries.

Compute plane outbound IP addresses must be added to a workspace IP allow list

July 29, 2024

When you configure IP access lists on a new workspace, you must add to an allowlist all public IPs that the compute plane uses to access the control plane or configure back-end Private Service Connect. This change will impact all new workspaces on July 29, 2024, and existing workspaces on August 26, 2024. For more information, see the Databricks Community post.

For example, when you configure a customer-managed VPC, subnets must have outbound access to the public network using a Google Cloud NAT or use a similar approach. Those public IPs must be included in an allowlist. See Egress requirements. Alternatively, if you use a Databricks-managed VPC and you configure the managed NAT gateway to access public IPs, those IPs must be in an allowlist.

See Configure IP access lists for workspaces.

Databricks Runtime 9.1 series support extended

July 26, 2024

Support for Databricks Runtime 9.1 LTS and Databricks Runtime 9.1 LTS for Machine Learning has been extended from September 23, 2024 to December 19, 2024.

Single sign-on (SSO) is supported in Lakehouse Federation for SQL Server

July 25, 2024

Unity Catalog now allows you to create SQL Server connections using SSO authentication. See Run federated queries on Microsoft SQL Server.

Enable cross-Geo processing

July 26, 2024

Account admins can now enable cross-Geo processing to allow data processing in data centers outside of a workspace Geo for Designated Services. If a Designated Service is not available in your workspace Geo, an account admin may be able to use the feature by explicitly giving permission to process relevant data in another Geo. See Enable cross-geo processing.

Model sharing using Delta Sharing is now generally available

July 26, 2024

Delta Sharing support for AI model sharing is now GA. Both the provider and recipient workspaces must be enabled for Unity Catalog, and models must be registered in Unity Catalog.

See Add models to a share.

Share comments and primary key constraints using Delta Sharing

July 25, 2024

Delta Sharing now supports the sharing of object metadata, including comments and primary key constraints:

• Model comments and model version comments have been included in Databricks-to-Databricks shares for some time, but not announced.

• Table comments, column comments, primary key constraints, and volume comments are now included in Databricks-to-Databricks shares that were shared with the recipient on or after July 25, 2024.

  If you want to include comments or constraints in a share that was shared with a recipient before that date, you must revoke and re-grant recipient access to trigger comment and constraint sharing.

See Create and manage shares for Delta Sharing.

New Databricks JDBC Driver (OSS)

July 25, 2024

A new open-source Databricks JDBC driver has been released for Public Preview. This driver has implemented the JDBC APIs and provides other core functionality including OAuth, Cloud Fetch, and features such as Unity Catalog volume ingestion. For more information, see Databricks JDBC Driver (OSS).

Databricks Runtime 15.4 LTS (Beta)

July 23, 2024

Databricks Runtime 15.4 LTS and Databricks Runtime 15.4 LTS ML are now available as Beta releases.

See Databricks Runtime 15.4 LTS and Databricks Runtime 15.4 LTS for Machine Learning.

Scala is GA on Unity Catalog shared compute

July 23, 2024

In Databricks Runtime 15.4 LTS and above, Scala is generally available on shared access mode Unity Catalog-enabled compute, including support for scalar user-defined functions (UDFs). Structured Streaming, Hive UDFs, and Hive user-defined aggregate functions are not supported. For a complete list of limitations, see Compute access mode limitations for Unity Catalog.

Notebooks: toggle more visible cell titles

July 18, 2024

Users can enable Show promoted cell titles in their developer settings to make notebook cell titles more visible in the UI. See Promoted cell titles

/ in workspace asset names is deprecated

July 17, 2024

To avoid ambiguity in path strings, the use of ‘/’ in the names of new workspace assets (such as notebooks, folders, and queries) has been deprecated. Existing assets with ‘/’ in their names are not affected, but renaming of existing assets follows the same rules as new assets.

Delta Sharing lets you share tables that use liquid clustering

July 16, 2024

Delta Sharing now lets you share tables that are enabled for liquid clustering, and recipients can run batch queries against them. Liquid clustering simplifies data layout decisions and optimizes query performance. See Use liquid clustering for Delta tables and Delta Lake feature support matrix.

Query history system table is now available (Public Preview)

July 16, 2024

Databricks system tables now include a query history table. This table logs detailed records of each query run on a SQL warehouse in your account. To access the table, admins must enable the new query system schema. See Query history system table reference.

Partition metadata logging for Unity Catalog external tables

July 15, 2024

In Databricks Runtime 13.3 LTS and above, you can optionally enable partition metadata logging for external tables registered to Unity Catalog that use Parquet, ORC, CSV, or JSON. Partition metatdata logging is a partition discovery strategy consistent with Hive metastore . See Partition discovery for external tables.

Filter data outputs using natural language prompts

July 11, 2024

You can now use the Databricks Assistant to filter data outputs using natural language prompts. For instance, to filter the Titanic survivors data table, you could type “Show me only males over 70.” See Filter data with natural language prompts.

Lakehouse Federation supports Salesforce Data Cloud (Public Preview)

July 10, 2024

You can now run federated queries on data managed by Salesforce Data Cloud. See Run federated queries on Salesforce Data Cloud.

Databricks Assistant system table now available (Public Preview)

July 10, 2024

Databricks Assistant events are now logged in a system table located at system.access.assistant_events. See Databricks Assistant system table reference and example.

Account SCIM API v2.1 (Public Preview)

July 10, 2024

The Account SCIM APIs are updated from v2.0 to v2.1 for speed and reliability. You can download a PDF of the Account SCIM v2.1 API reference.

Resource quota increase for tables per Unity Catalog metastore

July 3, 2024

Your Unity Catalog metastore can now register up to one million tables. See Resource quotas.

Databricks Assistant can diagnose notebook errors automatically

July 2, 2024

Databricks Assistant can now run /fix in notebooks automatically when it detects an error message. Assistant uses generative AI to analyze your code and the error message to suggest a fix directly in your notebook. For more information, see Debug code: Python and SQL examples.