Databricks SQL Release notes 2024
The following outlines the improvements and updates in Databricks SQL from January through December 2024.
November 21, 2024
SQL warehouse system tables (Public Preview)
The
system.compute.warehouses
table records when SQL warehouses are created, edited, and deleted. You can use the table to monitor changes to warehouse settings, including the warehouse name, type, size, channel, tags, auto-stop, and autoscaling settings. See Warehouses system table reference.
Data discovery
The Lineage tab in Catalog Explorer has been redesigned with an improved UI for entity filtering.
November 13, 2024
Legacy dashboards:
Resolved an issue where templated tooltips were not displaying detailed content for dual-axis and multi-field axis charts.
October 31, 2024
User interface updates
New SQL editor (Public Preview)
You can now run the active SQL query using the keyboard shortcut
Command
(orCtrl
) +Shift
+Enter
.The parameters input area now shows a scrollbar when the text extends outside of the display window.
Fixed an issue that prevented the query profile details page from opening fully.
You can now rename queries by typing the new name into the tab title.
The Schedule button is now disabled for queries that have never been saved before.
October 24, 2024
Release notes for AI/BI tools
The release notes for AI/BI dashboards and AI/BI Genie have moved to AI/BI release notes. Future releases and updates will be documented there.
October 17, 2024
Notification destinations are now generally available
You can create and configure notification destinations that workspace users can add to certain workflows, like alerts, Databricks jobs, and AI/BI dashboard schedules, to send emails or webhooks when an event runs. See Manage notification destinations.
October 10, 2024
Query insights
The new columns
query_source
,executed_as
, andexecuted_as_user_id
have been added to the query history system table. See Query history system table reference.
October 3, 2024
User interface updates
The features listed in this section are independent of the SQL warehouse compute versions described above.
Catalog Explorer
AI-generated comments are now supported for catalogs, schemas, volumes, models, and functions and users can use the inline chat Assistant to help edit their comments.
September 11, 2024
User interface updates
The features listed in this section are independent of the SQL Warehouse compute versions described above.
SQL editor
You can now use named parameter marker syntax in the SQL editor. Named parameter marker syntax can be used across the SQL editor, notebooks, and AI/BI dashboards. See Work with query parameters.
Queries and legacy dashboards
For SQL queries and legacy dashboards, deleted items no longer appear in the listing pages. Find deleted items in the workspace trash folder. Workspace admins can view deleted items in all users’ trash folders.
SQL warehouse
The system.compute.warehouse_events
table records when SQL warehouses start, stop, scale up and down. You can use the table to monitor the SQL warehouses in your workspaces. See Warehouse events system table reference.
September 5, 2024
Changes in 2024.40
Databricks SQL version 2024.40 includes the following behavioral changes, new features, and improvements.
Behavioral changes
Change to the default schema binding mode for views
Views now adapt to schema changes in the underlying query by using schema compensation with regular casting rules. This is a change from the previous default of
BINDING
mode, which raised errors when a safe cast could not be performed when referencing the view.See CREATE VIEW and cast function.
Disallow using the undocumented
!
syntax instead ofNOT
outside boolean expressionsWith this release, the use of
!
as a synonym forNOT
outside of boolean expressions is no longer allowed. For example, statements such as the following:CREATE ... IF ! EXISTS
, IS ! NULL, a! NULL
column or field property,! IN
and ! BETWEEN, must be replaced with:CREATE ... IF NOT EXISTS
,IS NOT NULL
, aNOT NULL
column or field property,NOT IN
andNOT BETWEEN
.This change ensures consistency, aligns with the SQL standard, and makes your SQL more portable.
The boolean prefix operator
!
(for example,!is_mgr
or!(true AND false)
) is unaffected by this change.Disallow undocumented column definition syntax in views
Databricks supports CREATE VIEW with named columns and column comments. Previously, the specification of column types,
NOT NULL
constraints, orDEFAULT
has been allowed. With this release, you can no longer use this syntax.This change ensures consistency, aligns with the SQL standard, and supports future enhancements.
Adding a
CHECK
constraint on an invalid column now returns theUNRESOLVED_COLUMN.WITH_SUGGESTION
error classTo provide more useful error messaging, in Databricks Runtime 15.3 and above, an
ALTER TABLE ADD CONSTRAINT
statement that includes aCHECK
constraint referencing an invalid column name returns the UNRESOLVED_COLUMN.WITH_SUGGESTION error class. Previously, anINTERNAL_ERROR
was returned.
New features and improvements
Enable UniForm Iceberg using ALTER TABLE
You can now enable UniForm Iceberg on existing tables without rewriting data files. See Enable Iceberg reads on an existing table.
UTF-8 validation functions
This release introduces the following functions for validating UTF-8 strings:
is_valid_utf8 verified whether a string is a valid UTF-8 string.
make_valid_utf8 converts a potentially invalid UTF-8 string to a valid UTF-8 string using substitution characters
validate_utf8 raises an error if the input is not a valid UTF-8 string.
try_validate_utf8 returns
NULL
if the input is not a valid UTF-8 string.
to_avro and from_avro functions
The to_avro and from_avro functions allow conversion of SQL types to Avro binary data and back.
try_url_decode function
This release introduces the try_url_decode function, which decodes a URL-encoded string. If the string is not in the correct format, the function returns
NULL
instead of raising an error.Optionally allow the optimizer to rely on unenforced foreign key constraints
To improve query performance, you can now specify the
RELY
keyword onFOREIGN KEY
constraints when you CREATE or ALTER a table.Support for dropping the check constraints table feature
Selective overwrites using
replaceWhere
now run jobs that delete data and insert new data in parallel, improving query performance and cluster utilization.Parallelized job runs for selective overwrites
Selective overwrites using
replaceWhere
now run jobs that delete data and insert new data in parallel, improving query performance and cluster utilization.Improved performance for change data feed with selective overwrites
Selective overwrites using
replaceWhere
on tables with change data feed no longer write separate change data files for inserted data. These operations use a hidden_change_type
column present in the underlying Parquet data files to record changes without write amplification.Improved query latency for the COPY INTO command
This release includes a change that improves the query latency for the
COPY INTO
command. This improvement is implemented by making the loading of state by the RocksDB state store asynchronous. With this change, you should see an improvement in start times for queries with large states, such as queries with a large number of already ingested files.
August 22, 2024
Visualizations
For grouped and multi-field configurations, tooltips now show totals when you hover over chart elements.
August 15, 2024
Visualizations
Fixed an issue where row numbers in table visualizations didn’t update after changing the page size.
Data discovery
The ability to expand and collapse nested complex column types in Unity Catalog tables is now supported.
August 1, 2024
API tools
The SQL Statement Execution API is now generally available. See Statement Execution API: Run SQL on warehouses.
Visualizations:
Table sorting is now preserved when data changes due to filtering.
SQL Editor:
Increased readability by adding additional padding between the last line of a query and the result output.
July 25, 2024
Databricks REST API:
APIs for managing queries, alerts, data sources, and permissions have changed. The legacy version will continue to be supported for six months. This transition period is intended to give you sufficient time to migrate your applications and integrations to the new version before the older version is phased out. See Update to the latest Databricks SQL API version
July 18, 2024
User interface updates
Catalog explorer:
A new catalog configuration wizard is now available for setting up workspace bindings, catalog privileges, and metadata when creating a catalog.
SQL Warehouse monitoring:
CAN MONITOR permission is now generally available. It allows privileged users to monitor SQL warehouses, including the associated query history and query profiles. See SQL warehouse ACLs.
Changes in 2024.35
Disable column mapping with drop feature
You can now use DROP FEATURE
to disable column mapping on Delta tables and downgrade the table protocol. See Disable column mapping.
Variant type syntax and functions in Public Preview
Built-in Apache Spark support for working with semi-structured data as VARIANT
type is now available in Spark DataFrames and SQL. See Query variant data.
Variant type support for Delta Lake in Public Preview
You can now use VARIANT
to store semi-structured data in tables backed by Delta Lake. See Variant support in Delta Lake.
Support for different modes of schema evolution in views
CREATE VIEW and ALTER VIEW now allow you to set a schema binding mode, enhancing how views handle schema changes in underlying objects. This feature enables views to either tolerate or adapt to schema changes in the underlying objects. It addresses changes in the query schema resulting from modifications to object definitions.
Performance improvement for some window functions
This release includes a change that improves the performance of some Spark window functions, specifically functions that do not include an ORDER BY
clause or a window_frame
parameter. In these cases, the system can rewrite the query to run it using an aggregate function. This change allows the query to run faster by using partial aggregation and avoiding the overhead of running window functions. The Spark configuration parameter spark.databricks.optimizer.replaceWindowsWithAggregates.enabled
controls this optimization and is set to true
by default. To turn this optimization off, set spark.databricks.optimizer.replaceWindowsWithAggregates.enabled
to false
.
Support for the try_mod
function added
This release adds support for the PySpark try_mod()
function. This function supports the ANSI SQL-compatible calculation of the integer remainder by dividing two numeric values. If the divisor argument is 0, the try_mod()
function returns null instead of throwing an exception. You can use the try_mod()
function instead of mod
or %
, which throws an exception if the divisor argument is 0 and ANSI SQL is enabled.
User interface updates
SQL Editor:
The inline assistant is now available in the SQL editor. Click the assistant icon in the editor box to toggle the input. Type a question or comment in English, then press Enter (not Shift+Enter, which runs a query) to generate a response with a different view directly in the editor.
Platform:
An API for notification destinations is now available. You can now programmatically manage webhook and email destinations for your alerts and job run notifications. See Notification Destinations.
Changes in 2024.30
Lakehouse Federation is generally available (GA)
Lakehouse Federation connectors across the following database types are now generally available (GA):
MySQL
PostgreSQL
Amazon Redshift
Snowflake
Microsoft SQL Server
Azure Synapse (SQL Data Warehouse)
Databricks
This release also introduces the following improvements:
Support for single sign-on (SSO) authentication in the Snowflake and Microsoft SQL Server connectors.
Support for additional pushdowns (string, math, miscellaneous functions).
Improved pushdown success rate across different query shapes.
Additional pushdown debugging capabilities:
The
EXPLAIN FORMATTED
output displays the pushed-down query text.The query profile UI displays the pushed-down query text, federated node identifiers, and JDBC query execution times (in verbose mode). See View system-generated federated queries.
DESCRIBE HISTORY
now shows clustering columns for tables that use liquid clustering
When you run a DESCRIBE HISTORY
query, the operationParameters
column shows a clusterBy
field by default for CREATE OR REPLACE
and OPTIMIZE
operations. For a Delta table that uses liquid clustering, the clusterBy
field is populated with the table’s clustering columns. If the table does not use liquid clustering, the field is empty.
Support for primary and foreign keys is generally available
Support for primary and foreign keys in Databricks Runtime is generally available. The GA release includes the following changes to the privileges required to use primary and foreign keys:
To define a foreign key, you must have the
SELECT
privilege on the table with the primary key that the foreign key refers to. You do not need to own the table with the primary key, which was previously required.Dropping a primary key using the
CASCADE
clause does not require privileges on the tables that define foreign keys that reference the primary key. Previously, you needed to own the referencing tables.Dropping a table that includes constraints now requires the same privileges as dropping tables that do not include constraints.
To learn how to use primary and foreign keys with tables or views, see CONSTRAINT clause, ADD CONSTRAINT clause, and DROP CONSTRAINT clause.
Liquid clustering is GA
Support for liquid clustering is now generally available using Databricks Runtime 15.2 and above. See Use liquid clustering for Delta tables.
Type widening is in Public Preview
You can now enable type widening on tables backed by Delta Lake. Tables with type widening enabled allow changing the type of columns to a wider data type without rewriting underlying data files. See Type widening.
Schema evolution clause added to SQL merge syntax
You can now add the WITH SCHEMA EVOLUTION
clause to a SQL merge statement to enable schema evolution for the operation. See Schema evolution syntax for merge.
Vacuum inventory support
You can now specify an inventory of files to consider when running the VACUUM
command on a Delta table. See the OSS Delta docs.
Support for Zstandard compression functions
You can now use the zst_compress, zstd_decompress, and try_zstd_decompress functions to compress and decompress BINARY
data.
Query plans in the SQL UI now correctly display PhotonWriteStage
When displayed in the SQL UI, write
commands in query plans incorrectly showed PhotonWriteStage
as an operator. With this release, the UI is updated to show PhotonWriteStage
as a stage. This is a UI change only and does not affect how queries are run.
Serverless SQL warehouses are now generally available.
Serverless SQL warehouses are now generally available in supported regions. They provide instant compute, minimal management, and cost optimization for SQL queries. Create serverless SQL warehouses or convert pro or classic SQL warehouses to serverless.
User interface updates
API support:
You can now manage notification destinations using the REST API. See Notification destinations.
June 27, 2024
User interface updates
Visualizations:
Improved interactivity in displaying tooltips when hovering over pie, scatter, and heatmap charts with many data points.
Catalog Explorer:
A revamped Catalog Explorer UI makes it easier to discover and favorite recent Unity Catalog assets from the Quick Access view. The navigation experience has also been simplified, allowing you to explore compute, storage, credentials, connections, DBFS, and management details using the Settings in the upper-left corner of the screen. Delta Sharing, Clean Rooms, and External Data now have dedicated pages.
June 6, 2024
User interface updates
Dashboards:
Account users can now download visualization data from published dashboards
Unpublished dashboards can now be published using the Draft/Publish dropdown near the top of a dashboard.
Fixed an issue where parameters named limit were not detected.
Visualizations:
Improved box-plot rendering in dark mode.
Query insights:
For all notebooks attached to SQL warehouses, you can access the query profile by clicking See performance under the cell that contains the query. If the cell includes multiple queries, a link to the query profile for each is provided for each statement.
May 30, 2024
Serverless SQL warehouses are in Public Preview in the following regions:
us-west1
us-east4
europe-west2
asia-southeast1
asia-south1
New permission level for SQL warehouses
Can monitor permission allows users to monitor SQL warehouses, including the associated query history and query profiles. The Can monitor permission is now in Public Preview. See SQL warehouse ACLs.
May 23, 2024
New dashboard tool
Dashboards (previously known as Lakeview dashboards) are now in Public Preview. They are named Dashboards in the UI. Databricks SQL dashboards are now called Legacy dashboards. The names of the related API tools have not changed.
User interface updates
You can now select multiple items in the workspace to move or delete. When multiple objects are selected, an action bar appears and provides options to move or delete items. Additionally, you can select multiple items using your mouse and drag them to a new location. Existing permissions on objects still apply during bulk move and delete operations.
You can now mark Unity Catalog assets as favorites in the Catalog Explorer and Schema Browser. This includes catalogs, schemas, tables, models, volumes, and functions. Unity Catalog assets that you mark as favorites are easily accessible from the Databricks homepage.
Dashboard updates:
Dual-axis combo charts now correctly display bar legends on the right axis and line legends accordingly.
Dual-axis charts now correctly show labels on bars.
Visualizations updates:
The table editor’s conditional format labels for if and then now support dark mode.
The redundant open link icon has been removed from the table editor’s format tooltips.
The default font color’s label in the table editor now aligns automatically.
May 16, 2024
Rollout schedule
Preview rollout for 2024.25: Completed May 1st
Current rollout for 2024.25: Between May 14th and May 21st
Note
An upgrade to the panda Python library (version 2.0.3) caused a breaking change in Databricks SQL version 2024.20. Databricks did not roll out version 2024.20 to the current channel. Instead, the preview channel was upgraded to 2024.25 on May 1, 2025. The current channel rollout goes directly from 2024.15 to 2024.25.
Changes in 2024.25
SQL language features
The * (star) clause is now supported in the WHERE clause.
You can now use the star (*
) clause in the WHERE
clause to reference all columns from the SELECT
list.
For example, SELECT * FROM VALUES(1, 2) AS T(a1, a2) WHERE 1 IN(T.*)
.
Support for Cloudflare R2 storage to avoid cross-region egress fees since 2024.15
You can now use Cloudflare R2 as cloud storage for data registered in Unity Catalog. Cloudflare R2 is intended primarily for Delta Sharing use cases in which you want to avoid the data egress fees charged by cloud providers when data crosses regions.
Cloudflare R2 storage supports all of the Databricks data and AI assets supported in Google Cloud Storage.
See Use Cloudflare R2 replicas or migrate storage to R2 and Create a storage credential for connecting to Cloudflare R2.
User interface updates
The features listed in this section are independent of the SQL Warehouse compute versions described above.
Data discovery updates:
The Hive metastore to Unity Catalog update wizard supports upgrading Hive metastore managed tables using all-purpose compute or SQL warehouses. Updating more than 20 tables creates a new notebook that contains the SYNC
AND ALTER TABLE
commands that perform the conversion.
May 9, 2024
SQL Editor fixes:
The admin setting Results table clipboard features now applies to the SQL editor’s New result table.
May 2, 2024
Visualization updates:
User-selected color for tables now persists across light and dark modes in legacy charts.
Data truncation logic has been improved to enhance performance in combo, pie, heatmap, and histogram charts.
A tick mark is now always displayed at the top of a quantitative axis for basic charts.
April 23, 2024
UI updates:
For all Share dialogs in the UI, the
All Users
group has been renamed toAll Workspace Users
. The new name more accurately reflects the scope of the group, which has always included users assigned to the workspace. No change is made to group membership as part of this rename.
Visualization updates:
Custom tooltop formats now function correctly for multi-axis charts.
The New charts preview tag is removed when users have not changed the toggle in the past 14 days.
April 18, 2024
User interface updates
The features listed in this section are independent of the SQL warehouse compute versions described above.
Improvements:
The tooltips on stacked charts now display the stack value and percentage by default.
The tooltips for multi-axis charts now highlight the hovered item.
Table visualizations for Databricks SQL now adapt a new query result’s data type when edited in the SQL editor.
The Catalog Explorer’s Query History table shows a tree-like view for Query Source attribution. You can use this to see which entities have triggered the query statement to run.
April 11, 2024
Serverless SQL warehouses are now in Public Preview.
Using the Databricks serverless architecture, a serverless SQL warehouse supports all of the performance features of Databricks SQL. With a serverless SQL warehouse and its performance features, you get:
Rapid startup time (typically between 2 and 6 seconds).
Rapid upscaling to acquire more compute when needed for maintaining low latency.
Query admittance closer to the hardware’s limitation rather than the virtual machine.
Quick downscaling to minimize costs when demand is low, providing consistent performance with optimized costs and resources.
User interface updates
The features listed in this section are independent of the SQL warehouse compute versions described above.
Improvements:
You can now group by percentage when creating visualizations in Databricks SQL and notebooks.
For new charts (in Public Preview), you can zoom in along a single axis by clicking and dragging in a straight line parallel to the axis.
The Unity Catalog shared cluster Allowlist UI is now generally available. You can access it on the Metastore details page in Catalog Explorer. See How to add items to the allowlist.
Forms to create and edit external locations now open as a full page. They include the option to include a storage credential.
Fixes:
Corrected an issue for Histogram charts where negative values were erroneously marked as positive.
April 4, 2024
User interface updates
The features listed in this section are independent of the SQL warehouse compute versions described above.
Improvements:
Switching visualizations between heat maps and other chart types now preserves the relevant fields better.
Fixes:
Bar charts with color encodings now correctly restrict adding multiple Y-axis fields.
Resolved an issue where the Download as PNG button was missing from some visualizations.
Corrected formatting for negative big integers previously missing thousands of separators.
Fixed incorrect hover line placement when hovering over labels on line charts.
Changes in 2024.15
Delta updates
Delta UniForm is now generally available: UniForm is now generally available and uses the IcebergCompatV2 table feature. You can now enable or upgrade UniForm on existing tables. See Read Delta tables with Iceberg clients.
Recompute data skipping statistics for Delta tables: You can now recompute statistics stored in the Delta log after changing columns used for data skipping. See Specify Delta statistics columns.
SQL language updates
Declare temporary variables in a SQL session: This release introduces the ability to declare temporary variables in a session that can be set and then referred to from in queries. See Variables.
Native XML file format support (Public Preview): Native XML file format support is now in Public Preview. XML file format support enables ingestion, querying, and parsing of XML data for batch processing or streaming. It can automatically infer and evolve schema and data types, supports SQL expressions like
from_xml
, and can generate XML documents. It doesn’t require external jars and works seamlessly with Auto Loader,read_files
,COPY INTO
, and Delta Live Tables. See Read and write XML files.
Apache Spark SQL updates
Databricks SQL 2024.15 include Apache Spark 3.5.0. Additional bug fixes and improvements for SQL are listed on the Databricks Runtime 14.3 release note. See Apache Spark and look for the [SQL]
tag for a complete list.
User interface updates
The features listed in this section are independent of the SQL warehouse compute versions described above.
A new overview tab in the entity page of Catalog Explorer shows important metadata like filesize, data source, owner, table schema, and comments.
Switching visualizations between heatmaps and other chart types now better preserves the relevant fields.
Bar charts with color encodings now correctly restrict adding multiple Y-axis fields.
March 21, 2024
Unity Catalog model lineage is now in Public Preview.
The table view in Catalog Explorer now has an Overview tab to describe its primary metadata.
March 14, 2024
New charts now apply aliases and custom colors for null values in numeric columns.
New charts now render tick marks to show the top of the y-axis.
March 7, 2024
Fixed an issue where tooltips in charts with over 100 series incorrectly showed all series. Now, only the focused series is shown.
Reduced typing latency in the SQL editor by 30% through performance optimizations.
When managing queries in the SQL editor, moving a query to trash automatically closes the tab.
Fixed an issue in the SQL editor where text was accidentally selected when adjusting the side panel width.
February 29, 2024
The schema browser in Catalog Explorer now displays column primary and foreign key constraints.
The retention time shown in the Lineage tab in Catalog Explorer has been increased to one year.
Tooltips on new charts in notebooks are now always rendered inside the visualization boundary.
Feburary 22, 2024
Improvements to the Sample Data tab in the Catalog Explorer table view enable you to sort columns, Copy selected data to your clipboard, and view line numbers. It can now better display special values, like JSON objects, dates, numeric and null values.
February 15, 2024
The documentation for code-based query filters, such as
SELECT action AS 'action::filter'
, has been removed. Databricks recommends updating queries to remove this pattern.
February 8, 2024
You can now request access when opening a link to a Lakeview dashboard you do not have permissions on.
Lakeview dashboard filters now have explicit All and None options. Authors can choose to hide the All option in single select filters.
You can now set minimum and maximum values for axes on Lakeview dashboard charts.
February 1, 2024
Databricks SQL Version 2024.10 Available
Rollout Schedule
Preview rollout for 2024.10: Between Jan 30, 2024 and Feb 5, 2024
Current rollout for 2024.10: Between Feb 13, 2023 and Feb 20, 2024
Changes in 2024.10
Fixed corrupt file handling in DML commands: The DML commands
DELETE
,UPDATE
, andMERGE INTO
no longer respect the read optionsignoreCorruptFiles
andignoreMissingFiles
. When encountering an unreadable file in a table, these commands now fail even if these options are specified.Row-level concurrency is Generally Available and on by default: Row-level concurrency reduces conflicts between concurrent write operations by detecting changes at the row-level. Row-level concurrency is only supported on tables without partitioning, which includes tables with liquid clustering. Row-level concurrency is enabled by default on Delta tables with deletion vectors enabled. See Write conflicts with row-level concurrency.
Shallow clone for Unity Catalog external tables (Public Preview): You can now use shallow clone with Unity Catalog external tables. See Shallow clone for Unity Catalog tables.
Faster multi-threaded statistics collection: Statistics collection is up to 10 times faster on small clusters when running
CONVERT TO DELTA
or cloning from Iceberg and Parquet tables. See Convert to Delta Lake and Incrementally clone Parquet and Iceberg tables to Delta Lake.Pushdown filters in the DeltaSource on Delta files: For better utilization, partition filters on Delta tables streaming queries are now pushed down to Delta before rate limiting.
User interface updates
The features listed in this section are independent of the SQL Warehouse compute versions described above.
The Admin view tab on listing pages for Databricks SQL objects (queries, dashboards, and alerts) has been removed. Workspace admin users can view all objects from their respective listing pages. See Access and manage saved queries, Legacy dashboards, and What are Databricks SQL alerts?.
The query history page displays queries from the past 24 hours by default. See Query history.
January 18, 2024
Fixed a rendering issue for visualizations where bar charts showing a single date on the x-axis resulted in a very thin bar. New chart visualizations render as expected.
January 11, 2024
Databricks SQL Queries and Dashboard APIs support changing the Run as role setting programatically.
January 4, 2024
Introduced primary key and foreign key entity relationship diagrams in Catalog Explorer. See View the Entity Relationship Diagram.