June 2024

These features and Databricks platform improvements were released in June 2024.

Note

Releases are staged. Your Databricks account might not be updated until a week or more after the initial release date.

Notebooks - out of order syntax highlighting

June 16, 2024

Python syntax error highlighting in notebooks now warns users when variables are used before they are declared and suggests a fix.

Notebooks Assistant automatically diagnoses errors

June 16, 2024

The Databricks Assistant now automatically diagnoses code errors in notebooks. This uses generative AI to analyze your code and the error message to suggest a fix directly in your notebook. For more, see Use Databricks Assistant

Catalog Explorer improvements

June 28, 2024

The following improvements have been made in Catalog Explorer:

  • Added a Quick access section that displays recent Unity Catalog objects, favorite Unity Catalog objects, and catalogs and schemas for browsing.

  • Removed the top bar and accordion structure, and moved the entry points for the following to a gear icon menu in the sidebar with the schema browser for simpler navigation:

    • Compute selection

    • Delta Sharing

    • Storage credentials

    • External locations

    • Connections

    • Browse DBFS

    You can also access Delta Sharing and External data at the top of Catalog Explorer above the Quick access section.

Databricks Assistant is now GA

June 27, 2024

Databricks Assistant is released as GA. Assistant capabilities include:

  • Generate, debug, optimize, and explain code.

  • Create visualizations from data. See Create visualizations with Databricks Assistant.

  • Debug jobs (Public Preview feature).

  • Code and edit SQL queries.

  • Find relevant help in Databricks documentation and knowledge base.

For more information, see What is Databricks Assistant?.

Databricks Assistant now gives titles to threads

June 27, 2024

To more easily navigate help threads, Databricks Assistant titles the threads based on the context of your prompt.

See Use Databricks Assistant.

Serverless notebooks environment manager

June 27, 2024

You can now install Python dependencies for serverless notebooks using the Environment side panel, which provides a single place to edit, view, and export the library requirements for a notebook. See Install notebook dependencies.

New intermediate DNS name for Private Service Connect

June 27, 2024

Databricks is updating the intermediate DNS name for workspaces that enable Private Service Connect to add <workspace-gcp-region>.psc.gcp.databricks.com. This allows you to separate out traffic for the workspaces that they need to access, from other Databricks services that don’t support Private Service Connect, such as documentation. See Enable Private Service Connect for your workspace.

Notebooks debugging console

June 27, 2024

With the notebook debug console, you can now inspect live variable values and execute code snippets during a notebook debugging session.

Quickly add a periodic schedule to a Databricks workflow

June 28, 2024

You can now add a simple schedule to a Databricks workflow by specifying only an interval and unit of time (for example, run the workflow every 4 hours). This new schedule type complements the existing advanced schedule type by allowing you to add a schedule with just a few clicks. See Run jobs on a schedule.

Databricks Runtime 15.3 is GA

June 24, 2024

Databricks Runtime 15.3 and Databricks Runtime 15.3 ML are now generally available.

See Databricks Runtime 15.3 and Databricks Runtime 15.3 for Machine Learning.

Databricks Geos is GA

June 24, 2024

Databricks Geos is now generally available. Databricks Geos manages data residency when processing customer content for some AI capabilities.

For more information, see Databricks Geos: Data residency.

New improved catalog creation UI

June 18, 2024

The new catalog creation UI takes you stepwise through the entire process of creating and configuring a catalog, providing helpful prompts and separating required from optional configurations. You can use the same user interface to create standard, shared, and foreign catalogs.

Catalog creation stepper UI

See Create catalogs.

The billing system tables are enabled by default

June 18, 2024

The system.billing schema is now enabled by default in all Unity Catalog workspaces. See Billable usage system table reference.

Volume sharing using Delta Sharing is now generally available

June 13, 2024

Delta Sharing support for sharing non-tabular data as a volume is now GA. You can share volumes between Databricks workspaces on different Unity Catalog metastores (including workspaces on different Databricks accounts and different clouds).

Volumes are Unity Catalog objects that represent a logical volume of storage in a cloud object storage location. They are intended primarily to provide governance over non-tabular data assets. Delta Sharing on Databricks provides a native integration with Unity Catalog that allows you to manage, govern, audit, and track usage of shared volumes data.

See Add volumes to a share.

Column mapping is now GA

June 10, 2024

Delta Lake column mapping is now generally available. You can now use DROP FEATURE to disable column mapping and downgrade table protocol. See Rename and drop columns with Delta Lake column mapping.

Workflow system tables are now available (Public Preview)

June 10, 2024

The system.lakeflow schema contains four new tables you can use to monitor jobs in your account. See Jobs system table reference.

If you don’t have access to these system tables, ensure you have enabled the system.lakeflow schema in your account. See Enable system table schemas.

Databricks Assistant: Threads & queries experience enabled by default

June 6, 2024

Databricks Assistant (Public Preview) now tracks query threads and history throughout editor contexts in your session experience by default. You can manage enablement for all workspaces in an account or individual workspaces.

For more information, see What is Databricks Assistant?

New entry points for in-product help

June 5, 2024

You can now access the Contact Support option by clicking your user icon in the top bar of the workspace. See Get help.

Improved notebook Markdown editor

June 5, 2024

You can now see a live preview of your Markdown cells in notebooks and access a toolbar for common Markdown elements like headers, lists, and links. Spend more time communicating your ideas and less time remembering syntax.

Quickly access Catalog Explorer tables from notebooks

June 5, 2024

You can now easily access Catalog Explorer tables directly from notebooks. Hover over a table name in a notebook cell to get a link to its catalog explorer page to see details like columns, permissions, and more.

Databricks ODBC driver 2.8.2

June 4, 2024

Databricks ODBC Driver (download) version 2.8.2 is now available. This release adds the following new features and enhancements:

  • Customers can now use extended SQL PUT, GET, and REMOVE commands to ingest and manage large datasets in Unity Catalog volumes.

  • Default OAuth configuration values are now updated and compliant. The updated configuration options include the authorization endpoint, OAuth client ID, OAuth user-to-machine scope, and OAuth machine-to-machine scope. For more details, see the release notes.

  • Third-party libraries have been updated, including expat 2.6.2 (previously 2.6.0) and libCURL 8.7.1 (previously 8.6.0).

  • The driver now accepts a private signed certification authority (CA) if it is added to the Windows Trusted Root CA certificate store.

This release also fixes the following issues:

  • The client previously returned an error if a table comment’s length exceeded MaxCommentLen. The default behavior now truncates the table comment to conform to the limit specified in MaxCommentLen.

  • The client previously returned an error when issuing SQLCancel under certain circumstances.

The new Databricks Notebook UI is now generally available

June 4, 2024

The new Databricks Notebooks UI provides developers with a streamlined, data-focused authoring experience. Key features include:

  • Modern UX: Enjoy a streamlined coding experience with an updated UI and other features to enhance Notebook organization.

  • New results table: Perform no-code data exploration with search and filtering capabilities directly on result outputs.