Databricks SQL release notes
This article lists new Databricks SQL features and improvements, along with known issues and FAQs.
Release process
Databricks releases updates to the Databricks SQL web application user interface on an ongoing basis, with all users getting the same updates rolled out over a short period of time.
In addition, Databricks typically releases new SQL warehouse compute versions regularly. Two channels are always available: Preview and Current.
Note
Releases are staged. Your Databricks account might not be updated with a new SQL warehouse version or Databricks SQL feature until a week or more after the initial release date.
Channels
Channels let you choose between the Current SQL warehouse compute version or the Preview version. Preview versions let you try out functionality before it becomes the Databricks SQL standard. Take advantage of preview versions to test your production queries and dashboards against upcoming changes.
Typically, a preview version is promoted to the current channel approximately two weeks after being released to the preview channel. Some features, such as security features, maintenance updates, and bug fixes, may be released directly to the current channel. From time to time, Databricks may promote a preview version to the current channel on a different schedule. Each new version will be announced in the following sections.
To learn how to switch an existing SQL warehouse to the preview channel, see Preview Channels. The features listed in the user interface updates sections are independent of the SQL Warehouse compute versions described in the Channels section of the release notes.
Available Databricks SQL versions
Current channel: Databricks SQL version 2024.40
See features in 2024.40.
October 31, 2024
User interface updates
New SQL editor (Public Preview)
You can now run the active SQL query using the keyboard shortcut
Command
(orCtrl
) +Shift
+Enter
.The parameters input area now shows a scrollbar when the text extends outside of the display window.
Fixed an issue that prevented the query profile details page from opening fully.
You can now rename queries by typing the new name into the tab title.
The Schedule button is now disabled for queries that have never been saved before.
October 24, 2024
Release notes for AI/BI tools
The release notes for AI/BI dashboards and AI/BI Genie have moved to AI/BI release notes. Future releases and updates will be documented there.
October 17, 2024
Notification destinations are now generally available
You can create and configure notification destinations that workspace users can add to certain workflows, like alerts, Databricks jobs, and AI/BI dashboard schedules, to send emails or webhooks when an event runs. See Manage notification destinations.
October 10, 2024
Query insights
The new columns
query_source
,executed_as
, andexecuted_as_user_id
have been added to the query history system table. See Query history system table reference.
October 3, 2024
User interface updates
The features listed in this section are independent of the SQL warehouse compute versions described above.
Catalog Explorer
AI-generated comments are now supported for catalogs, schemas, volumes, models, and functions and users can use the inline chat Assistant to help edit their comments.
September 11, 2024
User interface updates
The features listed in this section are independent of the SQL Warehouse compute versions described above.
SQL editor
You can now use named parameter marker syntax in the SQL editor. Named parameter marker syntax can be used across the SQL editor, notebooks, and AI/BI dashboards. See Work with query parameters.
Queries and legacy dashboards
For SQL queries and legacy dashboards, deleted items no longer appear in the listing pages. Find deleted items in the workspace trash folder. Workspace admins can view deleted items in all users’ trash folders.
SQL warehouse
The system.compute.warehouse_events
table records when SQL warehouses start, stop, scale up and down. You can use the table to monitor the SQL warehouses in your workspaces. See Warehouse events system table reference.
September 5, 2024
Changes in 2024.40
Databricks SQL version 2024.40 includes the following behavioral changes, new features, and improvements.
Behavioral changes
Change to the default schema binding mode for views
Views now adapt to schema changes in the underlying query by using schema compensation with regular casting rules. This is a change from the previous default of
BINDING
mode, which raised errors when a safe cast could not be performed when referencing the view.See CREATE VIEW and cast function.
Disallow using the undocumented
!
syntax instead ofNOT
outside boolean expressionsWith this release, the use of
!
as a synonym forNOT
outside of boolean expressions is no longer allowed. For example, statements such as the following:CREATE ... IF ! EXISTS
, IS ! NULL, a! NULL
column or field property,! IN
and ! BETWEEN, must be replaced with:CREATE ... IF NOT EXISTS
,IS NOT NULL
, aNOT NULL
column or field property,NOT IN
andNOT BETWEEN
.This change ensures consistency, aligns with the SQL standard, and makes your SQL more portable.
The boolean prefix operator
!
(for example,!is_mgr
or!(true AND false)
) is unaffected by this change.Disallow undocumented column definition syntax in views
Databricks supports CREATE VIEW with named columns and column comments. Previously, the specification of column types,
NOT NULL
constraints, orDEFAULT
has been allowed. With this release, you can no longer use this syntax.This change ensures consistency, aligns with the SQL standard, and supports future enhancements.
Adding a
CHECK
constraint on an invalid column now returns theUNRESOLVED_COLUMN.WITH_SUGGESTION
error classTo provide more useful error messaging, in Databricks Runtime 15.3 and above, an
ALTER TABLE ADD CONSTRAINT
statement that includes aCHECK
constraint referencing an invalid column name returns the UNRESOLVED_COLUMN.WITH_SUGGESTION error class. Previously, anINTERNAL_ERROR
was returned.
New features and improvements
Enable UniForm Iceberg using ALTER TABLE
You can now enable UniForm Iceberg on existing tables without rewriting data files. See Enable by altering an existing table.
UTF-8 validation functions
This release introduces the following functions for validating UTF-8 strings:
is_valid_utf8 verified whether a string is a valid UTF-8 string.
make_valid_utf8 converts a potentially invalid UTF-8 string to a valid UTF-8 string using substitution characters
validate_utf8 raises an error if the input is not a valid UTF-8 string.
try_validate_utf8 returns
NULL
if the input is not a valid UTF-8 string.
to_avro and from_avro functions
The to_avro and from_avro functions allow conversion of SQL types to Avro binary data and back.
try_url_decode function
This release introduces the try_url_decode function, which decodes a URL-encoded string. If the string is not in the correct format, the function returns
NULL
instead of raising an error.Optionally allow the optimizer to rely on unenforced foreign key constraints
To improve query performance, you can now specify the
RELY
keyword onFOREIGN KEY
constraints when you CREATE or ALTER a table.Support for dropping the check constraints table feature
Selective overwrites using
replaceWhere
now run jobs that delete data and insert new data in parallel, improving query performance and cluster utilization.Parallelized job runs for selective overwrites
Selective overwrites using
replaceWhere
now run jobs that delete data and insert new data in parallel, improving query performance and cluster utilization.Improved performance for change data feed with selective overwrites
Selective overwrites using
replaceWhere
on tables with change data feed no longer write separate change data files for inserted data. These operations use a hidden_change_type
column present in the underlying Parquet data files to record changes without write amplification.Improved query latency for the COPY INTO command
This release includes a change that improves the query latency for the
COPY INTO
command. This improvement is implemented by making the loading of state by the RocksDB state store asynchronous. With this change, you should see an improvement in start times for queries with large states, such as queries with a large number of already ingested files.
August 22, 2024
Visualizations
For grouped and multi-field configurations, tooltips now show totals when you hover over chart elements.
August 15, 2024
Visualizations
Fixed an issue where row numbers in table visualizations didn’t update after changing the page size.
Data discovery
The ability to expand and collapse nested complex column types in Unity Catalog tables is now supported.
August 1, 2024
API tools
The SQL Statement Execution API is now generally available. See Statement Execution API: Run SQL on warehouses.
Visualizations:
Table sorting is now preserved when data changes due to filtering.
SQL Editor:
Increased readability by adding additional padding between the last line of a query and the result output.
July 25, 2024
Databricks REST API:
APIs for managing queries, alerts, data sources, and permissions have changed. The legacy version will continue to be supported for six months. This transition period is intended to give you sufficient time to migrate your applications and integrations to the new version before the older version is phased out. See Update to the latest Databricks SQL API version
July 18, 2024
User interface updates
Catalog explorer:
A new catalog configuration wizard is now available for setting up workspace bindings, catalog privileges, and metadata when creating a catalog.
SQL Warehouse monitoring:
CAN MONITOR permission is now generally available. It allows privileged users to monitor SQL warehouses, including the associated query history and query profiles. See SQL warehouse ACLs.
Changes in 2024.35
Disable column mapping with drop feature
You can now use DROP FEATURE
to disable column mapping on Delta tables and downgrade the table protocol. See Disable column mapping.
Variant type syntax and functions in Public Preview
Built-in Apache Spark support for working with semi-structured data as VARIANT
type is now available in Spark DataFrames and SQL. See Query variant data.
Variant type support for Delta Lake in Public Preview
You can now use VARIANT
to store semi-structured data in tables backed by Delta Lake. See Variant support in Delta Lake.
Support for different modes of schema evolution in views
CREATE VIEW and ALTER VIEW now allow you to set a schema binding mode, enhancing how views handle schema changes in underlying objects. This feature enables views to either tolerate or adapt to schema changes in the underlying objects. It addresses changes in the query schema resulting from modifications to object definitions.
Performance improvement for some window functions
This release includes a change that improves the performance of some Spark window functions, specifically functions that do not include an ORDER BY
clause or a window_frame
parameter. In these cases, the system can rewrite the query to run it using an aggregate function. This change allows the query to run faster by using partial aggregation and avoiding the overhead of running window functions. The Spark configuration parameter spark.databricks.optimizer.replaceWindowsWithAggregates.enabled
controls this optimization and is set to true
by default. To turn this optimization off, set spark.databricks.optimizer.replaceWindowsWithAggregates.enabled
to false
.
Support for the try_mod
function added
This release adds support for the PySpark try_mod()
function. This function supports the ANSI SQL-compatible calculation of the integer remainder by dividing two numeric values. If the divisor argument is 0, the try_mod()
function returns null instead of throwing an exception. You can use the try_mod()
function instead of mod
or %
, which throws an exception if the divisor argument is 0 and ANSI SQL is enabled.
User interface updates
SQL Editor:
The inline assistant is now available in the SQL editor. Click the assistant icon in the editor box to toggle the input. Type a question or comment in English, then press Enter (not Shift+Enter, which runs a query) to generate a response with a different view directly in the editor.
Platform:
An API for notification destinations is now available. You can now programmatically manage webhook and email destinations for your alerts and job run notifications. See Notification Destinations.
Changes in 2024.30
Lakehouse Federation is generally available (GA)
Lakehouse Federation connectors across the following database types are now generally available (GA):
MySQL
PostgreSQL
Amazon Redshift
Snowflake
Microsoft SQL Server
Azure Synapse (SQL Data Warehouse)
Databricks
This release also introduces the following improvements:
Support for single sign-on (SSO) authentication in the Snowflake and Microsoft SQL Server connectors.
Support for additional pushdowns (string, math, miscellaneous functions).
Improved pushdown success rate across different query shapes.
Additional pushdown debugging capabilities:
The
EXPLAIN FORMATTED
output displays the pushed-down query text.The query profile UI displays the pushed-down query text, federated node identifiers, and JDBC query execution times (in verbose mode). See View system-generated federated queries.
DESCRIBE HISTORY
now shows clustering columns for tables that use liquid clustering
When you run a DESCRIBE HISTORY
query, the operationParameters
column shows a clusterBy
field by default for CREATE OR REPLACE
and OPTIMIZE
operations. For a Delta table that uses liquid clustering, the clusterBy
field is populated with the table’s clustering columns. If the table does not use liquid clustering, the field is empty.
Support for primary and foreign keys is generally available
Support for primary and foreign keys in Databricks Runtime is generally available. The GA release includes the following changes to the privileges required to use primary and foreign keys:
To define a foreign key, you must have the
SELECT
privilege on the table with the primary key that the foreign key refers to. You do not need to own the table with the primary key, which was previously required.Dropping a primary key using the
CASCADE
clause does not require privileges on the tables that define foreign keys that reference the primary key. Previously, you needed to own the referencing tables.Dropping a table that includes constraints now requires the same privileges as dropping tables that do not include constraints.
To learn how to use primary and foreign keys with tables or views, see CONSTRAINT clause, ADD CONSTRAINT clause, and DROP CONSTRAINT clause.
Liquid clustering is GA
Support for liquid clustering is now generally available using Databricks Runtime 15.2 and above. See Use liquid clustering for Delta tables.
Type widening is in Public Preview
You can now enable type widening on tables backed by Delta Lake. Tables with type widening enabled allow changing the type of columns to a wider data type without rewriting underlying data files. See Type widening.
Schema evolution clause added to SQL merge syntax
You can now add the WITH SCHEMA EVOLUTION
clause to a SQL merge statement to enable schema evolution for the operation. See Schema evolution syntax for merge.
Vacuum inventory support
You can now specify an inventory of files to consider when running the VACUUM
command on a Delta table. See the OSS Delta docs.
Support for Zstandard compression functions
You can now use the zst_compress, zstd_decompress, and try_zstd_decompress functions to compress and decompress BINARY
data.
Query plans in the SQL UI now correctly display PhotonWriteStage
When displayed in the SQL UI, write
commands in query plans incorrectly showed PhotonWriteStage
as an operator. With this release, the UI is updated to show PhotonWriteStage
as a stage. This is a UI change only and does not affect how queries are run.
Serverless SQL warehouses are now generally available.
Serverless SQL warehouses are now generally available in supported regions. They provide instant compute, minimal management, and cost optimization for SQL queries. Create serverless SQL warehouses or convert pro or classic SQL warehouses to serverless.
User interface updates
API support:
You can now manage notification destinations using the REST API. See Notification destinations.
June 27, 2024
User interface updates
Visualizations:
Improved interactivity in displaying tooltips when hovering over pie, scatter, and heatmap charts with many data points.
Catalog Explorer:
A revamped Catalog Explorer UI makes it easier to discover and favorite recent Unity Catalog assets from the Quick Access view. The navigation experience has also been simplified, allowing you to explore compute, storage, credentials, connections, DBFS, and management details using the Settings in the upper-left corner of the screen. Delta Sharing, Clean Rooms, and External Data now have dedicated pages.
June 6, 2024
User interface updates
Dashboards:
Account users can now download visualization data from published dashboards
Unpublished dashboards can now be published using the Draft/Publish dropdown near the top of a dashboard.
Fixed an issue where parameters named limit were not detected.
Visualizations:
Improved box-plot rendering in dark mode.
Query insights:
For all notebooks attached to SQL warehouses, you can access the query profile by clicking See performance under the cell that contains the query. If the cell includes multiple queries, a link to the query profile for each is provided for each statement.
May 30, 2024
Serverless SQL warehouses are in Public Preview in the following regions:
us-west1
us-east4
europe-west2
asia-southeast1
asia-south1
New permission level for SQL warehouses
Can monitor permission allows users to monitor SQL warehouses, including the associated query history and query profiles. The Can monitor permission is now in Public Preview. See SQL warehouse ACLs.
May 23, 2024
New dashboard tool
Dashboards (previously known as Lakeview dashboards) are now in Public Preview. They are named Dashboards in the UI. Databricks SQL dashboards are now called Legacy dashboards. The names of the related API tools have not changed.
User interface updates
You can now select multiple items in the workspace to move or delete. When multiple objects are selected, an action bar appears and provides options to move or delete items. Additionally, you can select multiple items using your mouse and drag them to a new location. Existing permissions on objects still apply during bulk move and delete operations.
You can now mark Unity Catalog assets as favorites in the Catalog Explorer and Schema Browser. This includes catalogs, schemas, tables, models, volumes, and functions. Unity Catalog assets that you mark as favorites are easily accessible from the Databricks homepage.
Dashboard updates:
Dual-axis combo charts now correctly display bar legends on the right axis and line legends accordingly.
Dual-axis charts now correctly show labels on bars.
Visualizations updates:
The table editor’s conditional format labels for if and then now support dark mode.
The redundant open link icon has been removed from the table editor’s format tooltips.
The default font color’s label in the table editor now aligns automatically.
May 16, 2024
Rollout schedule
Preview rollout for 2024.25: Completed May 1st
Current rollout for 2024.25: Between May 14th and May 21st
Note
An upgrade to the panda Python library (version 2.0.3) caused a breaking change in Databricks SQL version 2024.20. Databricks did not roll out version 2024.20 to the current channel. Instead, the preview channel was upgraded to 2024.25 on May 1, 2025. The current channel rollout goes directly from 2024.15 to 2024.25.
Changes in 2024.25
SQL language features
The * (star) clause is now supported in the WHERE clause.
You can now use the star (*
) clause in the WHERE
clause to reference all columns from the SELECT
list.
For example, SELECT * FROM VALUES(1, 2) AS T(a1, a2) WHERE 1 IN(T.*)
.
Support for Cloudflare R2 storage to avoid cross-region egress fees since 2024.15
You can now use Cloudflare R2 as cloud storage for data registered in Unity Catalog. Cloudflare R2 is intended primarily for Delta Sharing use cases in which you want to avoid the data egress fees charged by cloud providers when data crosses regions.
Cloudflare R2 storage supports all of the Databricks data and AI assets supported in Google Cloud Storage.
See Use Cloudflare R2 replicas or migrate storage to R2 and Create a storage credential for connecting to Cloudflare R2.
User interface updates
The features listed in this section are independent of the SQL Warehouse compute versions described above.
Data discovery updates:
The Hive metastore to Unity Catalog update wizard supports upgrading Hive metastore managed tables using all-purpose compute or SQL warehouses. Updating more than 20 tables creates a new notebook that contains the SYNC
AND ALTER TABLE
commands that perform the conversion.
May 9, 2024
SQL Editor fixes:
The admin setting Results table clipboard features now applies to the SQL editor’s New result table.
May 2, 2024
Visualization updates:
User-selected color for tables now persists across light and dark modes in legacy charts.
Data truncation logic has been improved to enhance performance in combo, pie, heatmap, and histogram charts.
A tick mark is now always displayed at the top of a quantitative axis for basic charts.
April 23, 2024
UI updates:
For all Share dialogs in the UI, the
All Users
group has been renamed toAll Workspace Users
. The new name more accurately reflects the scope of the group, which has always included users assigned to the workspace. No change is made to group membership as part of this rename.
Visualization updates:
Custom tooltop formats now function correctly for multi-axis charts.
The New charts preview tag is removed when users have not changed the toggle in the past 14 days.
April 18, 2024
User interface updates
The features listed in this section are independent of the SQL warehouse compute versions described above.
Improvements:
The tooltips on stacked charts now display the stack value and percentage by default.
The tooltips for multi-axis charts now highlight the hovered item.
Table visualizations for Databricks SQL now adapt a new query result’s data type when edited in the SQL editor.
The Catalog Explorer’s Query History table shows a tree-like view for Query Source attribution. You can use this to see which entities have triggered the query statement to run.
April 11, 2024
Serverless SQL warehouses are now in Public Preview.
Using the Databricks serverless architecture, a serverless SQL warehouse supports all of the performance features of Databricks SQL. With a serverless SQL warehouse and its performance features, you get:
Rapid startup time (typically between 2 and 6 seconds).
Rapid upscaling to acquire more compute when needed for maintaining low latency.
Query admittance closer to the hardware’s limitation rather than the virtual machine.
Quick downscaling to minimize costs when demand is low, providing consistent performance with optimized costs and resources.
User interface updates
The features listed in this section are independent of the SQL warehouse compute versions described above.
Improvements:
You can now group by percentage when creating visualizations in Databricks SQL and notebooks.
For new charts (in Public Preview), you can zoom in along a single axis by clicking and dragging in a straight line parallel to the axis.
The Unity Catalog shared cluster Allowlist UI is now generally available. You can access it on the Metastore details page in Catalog Explorer. See How to add items to the allowlist.
Forms to create and edit external locations now open as a full page. They include the option to include a storage credential.
Fixes:
Corrected an issue for Histogram charts where negative values were erroneously marked as positive.
April 4, 2024
User interface updates
The features listed in this section are independent of the SQL warehouse compute versions described above.
Improvements:
Switching visualizations between heat maps and other chart types now preserves the relevant fields better.
Fixes:
Bar charts with color encodings now correctly restrict adding multiple Y-axis fields.
Resolved an issue where the Download as PNG button was missing from some visualizations.
Corrected formatting for negative big integers previously missing thousands of separators.
Fixed incorrect hover line placement when hovering over labels on line charts.
Changes in 2024.15
Delta updates
Delta UniForm is now generally available: UniForm is now generally available and uses the IcebergCompatV2 table feature. You can now enable or upgrade UniForm on existing tables. See Use UniForm to read Delta tables with Iceberg clients.
Recompute data skipping statistics for Delta tables: You can now recompute statistics stored in the Delta log after changing columns used for data skipping. See Specify Delta statistics columns.
SQL language updates
Declare temporary variables in a SQL session: This release introduces the ability to declare temporary variables in a session that can be set and then referred to from in queries. See Variables.
Native XML file format support (Public Preview): Native XML file format support is now in Public Preview. XML file format support enables ingestion, querying, and parsing of XML data for batch processing or streaming. It can automatically infer and evolve schema and data types, supports SQL expressions like
from_xml
, and can generate XML documents. It doesn’t require external jars and works seamlessly with Auto Loader,read_files
,COPY INTO
, and Delta Live Tables. See Read and write XML files.
Apache Spark SQL updates
Databricks SQL 2024.15 include Apache Spark 3.5.0. Additional bug fixes and improvements for SQL are listed on the Databricks Runtime 14.3 release note. See Apache Spark and look for the [SQL]
tag for a complete list.
User interface updates
The features listed in this section are independent of the SQL warehouse compute versions described above.
A new overview tab in the entity page of Catalog Explorer shows important metadata like filesize, data source, owner, table schema, and comments.
Switching visualizations between heatmaps and other chart types now better preserves the relevant fields.
Bar charts with color encodings now correctly restrict adding multiple Y-axis fields.
March 21, 2024
Unity Catalog model lineage is now in Public Preview.
The table view in Catalog Explorer now has an Overview tab to describe its primary metadata.
March 14, 2024
New charts now apply aliases and custom colors for null values in numeric columns.
New charts now render tick marks to show the top of the y-axis.
March 7, 2024
Fixed an issue where tooltips in charts with over 100 series incorrectly showed all series. Now, only the focused series is shown.
Reduced typing latency in the SQL editor by 30% through performance optimizations.
When managing queries in the SQL editor, moving a query to trash automatically closes the tab.
Fixed an issue in the SQL editor where text was accidentally selected when adjusting the side panel width.
February 29, 2024
The schema browser in Catalog Explorer now displays column primary and foreign key constraints.
The retention time shown in the Lineage tab in Catalog Explorer has been increased to one year.
Tooltips on new charts in notebooks are now always rendered inside the visualization boundary.
Feburary 22, 2024
Improvements to the Sample Data tab in the Catalog Explorer table view enable you to sort columns, Copy selected data to your clipboard, and view line numbers. It can now better display special values, like JSON objects, dates, numeric and null values.
February 15, 2024
The documentation for code-based query filters, such as
SELECT action AS 'action::filter'
, has been removed. Databricks recommends updating queries to remove this pattern.
February 8, 2024
You can now request access when opening a link to a Lakeview dashboard you do not have permissions on.
Lakeview dashboard filters now have explicit All and None options. Authors can choose to hide the All option in single select filters.
You can now set minimum and maximum values for axes on Lakeview dashboard charts.
February 1, 2024
Databricks SQL Version 2024.10 Available
Rollout Schedule
Preview rollout for 2024.10: Between Jan 30, 2024 and Feb 5, 2024
Current rollout for 2024.10: Between Feb 13, 2023 and Feb 20, 2024
Changes in 2024.10
Fixed corrupt file handling in DML commands: The DML commands
DELETE
,UPDATE
, andMERGE INTO
no longer respect the read optionsignoreCorruptFiles
andignoreMissingFiles
. When encountering an unreadable file in a table, these commands now fail even if these options are specified.Row-level concurrency is Generally Available and on by default: Row-level concurrency reduces conflicts between concurrent write operations by detecting changes at the row-level. Row-level concurrency is only supported on tables without partitioning, which includes tables with liquid clustering. Row-level concurrency is enabled by default on Delta tables with deletion vectors enabled. See Write conflicts with row-level concurrency.
Shallow clone for Unity Catalog external tables (Public Preview): You can now use shallow clone with Unity Catalog external tables. See Shallow clone for Unity Catalog tables.
Faster multi-threaded statistics collection: Statistics collection is up to 10 times faster on small clusters when running
CONVERT TO DELTA
or cloning from Iceberg and Parquet tables. See Convert to Delta Lake and Incrementally clone Parquet and Iceberg tables to Delta Lake.Pushdown filters in the DeltaSource on Delta files: For better utilization, partition filters on Delta tables streaming queries are now pushed down to Delta before rate limiting.
User interface updates
The features listed in this section are independent of the SQL Warehouse compute versions described above.
The Admin view tab on listing pages for Databricks SQL objects (queries, dashboards, and alerts) has been removed. Workspace admin users can view all objects from their respective listing pages. See Access and manage saved queries, Legacy dashboards, and What are Databricks SQL alerts?.
The query history page displays queries from the past 24 hours by default. See Query history.
January 18, 2024
Fixed a rendering issue for visualizations where bar charts showing a single date on the x-axis resulted in a very thin bar. New chart visualizations render as expected.
January 11, 2024
Databricks SQL Queries and Dashboard APIs support changing the Run as role setting programatically.
January 4, 2024
Introduced primary key and foreign key entity relationship diagrams in Catalog Explorer. See View the Entity Relationship Diagram.
December 21, 2023
For new charts, heatmap-type charts respect the reverseY setting.
Fixed a rendering performance issue for notebooks with a large number of visualizations.
December 7, 2023
User interface updates
Corrected visualization formatting issue where large integer values were mistakenly displayed as floats.
Fixed an issue with Databricks SQL dashboards where expanded chart views were sometimes showing blank charts.
Bar charts with quantitative fields on both X and Y axes render more legible data labels.
Fixed issue in the SQL Editor so that tables with the word
stream
in the title no longer conflict with the reserved keyword. These tables now appear as expected in the schema browser and are not error-highlighted.The query history page now supports column resize and column selections.
The query history page supports two new columns: Query source and Query source type.
BI options, like Tableau and Power BI, are easier to find in Catalog Explorer on eligible pages.
November 30, 2023
User interface updates
The features listed in this section are independent of the SQL Warehouse compute versions described above.
Improved dark mode compatibility for labels in new charts.
New charts now consistently prioritize label display inside bars.
Fixed a bug where some right-click menu actions weren’t working in the SQL Editor.
November 16, 2023
User interface updates
The features listed in this section are independent of the SQL Warehouse compute versions described above.
Databricks SQL queries, alerts, and dashboards have a new scheduler and scheduling interface.
November 9, 2023
Changes in 2023.50:
Highlights:
You can now use named parameter invocation on SQL and Python UDF.
SQL Language updates:
The following builtin functions have been added:
- `from_xml`: Parses an XML STRING
into a STRUCT
.
- `schema_of_xml`: Derives a schema from an XML STRING
.
- `session_user`: Returns the logged-in user.
- `try_reflect`: Returns NULL
instead of the exception if a Java method fails.
The following builtin functions have been enhanced:
function invocation. Table arguments to functions support partitioning and ordering: You can now use
PARTITION BY
andORDER BY
clauses to control how table arguments are passed to a function.`mode`: Support for an optional parameter forcing a deterministic result.
`to_char`: New support for
DATE
,TIMESTAMP
, andBINARY
.`to_varchar`: New support for
DATE
,TIMESTAMP
, andBINARY
.`array_insert()` is 1-based for negative indexes: The
array_insert
function is 1-based for both positive and negative indexes. It now inserts a new element at the end of input arrays for the index -1.
User interface updates
The features listed in this section are independent of the SQL warehouse compute versions described above.
Visualizations:
Tooltips for new charts show only hovered items for high-cardinality charts.
New charts automatically align dual-axis charts at zero.
Charts changed rotated label and axis text angle direction from -90 to 90 degrees.
Charts now use the label’s width to decide rotation.
SQL Editor: Autocomplete is now less aggressive and dismisses automatically at the end of statements
October 26, 2023
Changes in Databricks SQL version 2023.45
Highlights:
Predictive I/O for updates is now generally available. See What is predictive I/O?
Deletion vectors are now generally available. See What are deletion vectors?
Query optimizations:
Removed the outer join if they are all distinct aggregate functions. SPARK-42583
Optimized the order of filtering predicates. SPARK-40045
SQL function updates:
Added support for implicit lateral column alias resolution on
Aggregate
. SPARK-41631Support for implicit lateral column alias in queries with Window. SPARK-42217
Support for for Datasketches
HLLSketch
. See hll_sketch_agg aggregate function.Added the
try_aes_decrypt()
function. See try_aes_decrypt function.Support for CBC mode for
aes_encrypt()
andaes_decrypt()
. See aes_decrypt function.Added support for aes_encrypt IVs and AAD. SPARK-43290
Implement bitmap functions. SPARK-44154
Added the
to_varchar
alias forto_char
. See to_varchar function.Added
array_compact
support. See array_compact function.Support for udf
luhn_check
. See luhn_check function.Added analyzer support of named arguments for built-in functions. SPARK-44059
Support for
TABLE
argument parser rule forTableValuedFunction
. SPARK-44200array_insert
now fails with 0 index. SPARK-43011Added
NULL
values forINSERT
with user-specified lists of fewer columns than the target table. SPARK-42521DECODE
function returns wrong results when passed NULL. SPARK-41668
User interface updates
Improvements:
Periods in data field names no longer result in blank charts.
The legacy schema browser now works with Unity Catalog.
Implemented performance improvements to load only the active tab in the SQL Editor, decreasing initial load time by up to 20%.
AI-generated table comments in Catalog Explorer are now generally available.
Fixes:
Bars on temporal bar charts are now centered over the date tick mark.
Data label templates with aggregate expressions now use the proper numeric formatters.
Zooming on new charts now works with a scale set to categorical using temporal data.
October 19, 2023
Improvements:
Drag and drop in schema browser is now available.
Fixes:
Legend selection tooltip instructions specify Mac users use
cmd
and Windows users usectrl
.
Charts with truncated data display consistent colors as seen in the editor.
October 12, 2023
Improvements:
New charts support
@@name
data labels on scatter plots.Customized percentage formats apply to grouped chart tooltips in new charts.
Workspace binding extension UI is GA.
Lineage is GA.
Fixes:
Improved label rendering on new charts to prevent labels from spilling outside chart boundaries.
Increased the contrast of tick and grid lines on new charts for improved visibility.
Increased the axis label spacing to improve readability on new charts.
October 5, 2023
Improvements:
Hovering over a series in a chart now dims the surrounding series in the tooltip to improve readability in new charts.
New charts using percentage values now display tooltips with absolute values.
Added autocomplete support for creating volumes.
Closing a non-active tab no longer switches tabs.
Selected run clearly indicates when highlighting text.
Fixes:
Improved error message wording in Pivot Tables when the data is truncated.
Fixed a rendering error in Pivot Tables where colors were not showing when using BigInt data types.
When downloading PNGs in new charts with numerous legend items, removed the color symbol for overflow legend entries.
Lines in new charts will maintain a consistent thickness even at the topmost view boundary.
Delta Live Tables are properly detected by the SQL Parser and won’t show-up as invalid tables in Schema Browser.
Tooltips was added to sidebar.
September 28, 2023
Improvements:
Delta tables history improvement in Catalog Explorer includes adding filters for date range, user, and operation type, as well as sortable columns, and inline links to associated Jobs and Notebooks.
Dark mode support added across legends, tooltips and table visualization.
September 21, 2023
Improvements:
Pivot Table rendering performance has been improved.
New DuBois pattern for lineage tabular views in the UI.
September 14, 2023
Improvement:
File names are now preserved when downloading PNGs in new chart visualizations. See New chart visualizations in Databricks.
September 7, 2023
Databricks SQL version 2023.40 available
Rollout schedule
Preview rollout for 2023.40: Between Sep 5, 2023 and Sep 11, 2023
Current rollout for 2023.40: Between Sep 18, 2023 and Sep 25, 2023
Changes in 2023.40:
Tags are now available with Unity Catalog.
Databricks Runtime returns an error if a file is modified between query planning and invocation.
Databricks ODBC/JDBC driver support.
Enable time series column labeling.
New bitmap SQL functions.
Improved encryption functions.
Unity Catalog support for
REFRESH FOREIGN.
INSERT BY NAME
is now supported.Share materialized views with Delta Sharing.
User interface updates
Improvements:
New charts are now available, featuring faster render performance, beautiful colors, and improved interactivity. See New chart visualizations in Databricks.
In the graph view of Query Profile, you can now view the Join type on any node containing a join in the query plan.
Data Explorer is renamed to Catalog Explorer to recognize the fact that you can use it to work with all securable objects in Unity Catalog, not just data objects. See What is Catalog Explorer?.
August 31, 2023
New feature:
Tagging for Unity Catalog is in Public Preview. You can use tags to simplify search and discovery of your data assets. See Apply tags to Unity Catalog securable objects.
August 24, 2023
Improvement:
Autocomplete stops suggesting recommendations after you press the spacebar.
Schema Browser no longer sees
live
Delta Live Tables as broken tables.
August 16, 2023
Improvement:
The Catalog dropdown in the SQL editor now closes when you switch tabs. Previously, when you switched tabs, the dropdown would remain open.
August 10, 2023
Improvement:
Autocomplete now supports the new syntax for setting Unity Catalog tags. For information on commands, see SQL language reference.
August 3, 2023
Improvements:
The underlying Monaco Editor now uses version 37.1.
Autocomplete support for
SHOW ARCHIVED FILES FOR
Delta commands.
July 27, 2023
Improvements:
The SQL editor is now compatible with Windows newline characters, ensuring that query formatting works as expected in all cases.
You can open the query profile navigation from notebook results. For queries, run your query with SQL warehouse.
July 20, 2023
Databricks SQL version 2023.35 available
Rollout schedule
Preview rollout for 2023.35: Between Jul 18, 2023 and Jul 24, 2023
Current rollout for 2023.35: Between Jul 31, 2023 and Aug 8, 2023
Changes in 2023.35:
Enhanced reliability for
VACUUM
with shallow clone in Unity Catalog.Support for Python UDFs in SQL.
Delta Lake UniForm for Iceberg is in Public Preview.
Delta Lake liquid clustering is in Public Preview.
Archival support for Delta Lake.
IDENTIFIER clause support.
Unity Catalog support for Python and Pandas User-Defined Functions (UDFs).
Improvement:
Table popularity in Catalog Explorer is available to all Unity Catalog users.
July 13, 2023
Improvement:
Unity Catalog users can now view additional table insights in Catalog Explorer such as frequently joined tables, frequent users of a given table, and frequently used notebooks and dashboards.
June 15, 2023
New feature:
SQL tasks in Jobs are now generally available. You can orchestrate Queries, Dashboards, and Alerts from the Jobs page. See SQL task for jobs.
A new schema browser is now in Public Preview, featuring an updated UX, a For You tab, and improved filters. The schema browser is available in Databricks SQL, Catalog Explorer, and notebooks. See Browse data.
June 8, 2023
DBSQL version 2023.30 available
Changes in 2023.30
New SQL built-in functions, such as
array_prepend(array, elem)
,try_aes_decrypt(expr, key [, mode [, padding]])
, andsql_keywords()
.You can now use shallow clone to create new Unity Catalog managed tables from existing Unity Catalog managed tables. See Shallow clone for Unity Catalog tables.
You can now use
CLONE
andCONVERT TO DELTA
with Iceberg tables that have partitions defined on truncated columns of typesint
,long
, andstring
. Truncated columns of typedecimal
are not supported.START VERSION
is now deprecated forALTER SHARE
.
June 1, 2023
Improvements:
Binary data will now render as a hex string when using the Arrow format.
In the SQL Statement API, the CSV format is now supported for the
EXTERNAL_LINKS
disposition. This allows clients to extract up to 100 GiB of data in CSV format with pre-signed URLs, whereas theINLINE
limit for JSON is 16 MiB.
Databricks SQL Pro is available in all regions. SQL Pro is now the default warehouse in the UI. The API default is still Classic.
May 29, 2023
New feature:
You can now use the add data UI to load data from a cloud object storage path that’s defined as a Unity Catalog external location. For more information, see Load data using a Unity Catalog external location.
May 25, 2023
Improvements:
You can now toggle the autocompletion result panel.
Disable the enter key so it no longer accepts autocomplete suggestions. Under DBSQL User Settings, click Editor Settings, then New Editor settings. Turn off *Enter key accepts autocomplete suggestions*.
Fixes:
Sorted table headers now have colors.
Chart lines now render correctly.
May 18, 2023
Improvement:
In the SQL Statement API, the
EXTERNAL_LINKS
disposition now supports theJSON_ARRAY
format. You can extract up to 100 GiB of data in JSON format with pre-signed URLs. TheINLINE
limit for JSON is 16 MiB.
May 11, 2023
New feature:
Schema Browser is now generally available in Catalog Explorer.
Improvements:
On-hover table detail panel showing is less sensitive.
The escape key now closes the autocomplete panel.
View definitions now have syntax highlighting in the Catalog Explorer details tab.
Fixes:
Pivot tables now correctly render on Windows devices.
Completion suggestions now properly follows the case of the first keyword.
May 4, 2023
Databricks SQL Version 2023.26 Available
Rollout Schedule
Preview rollout for 2023.26: Between April 19, 2023 and April 25, 2023
Current rollout for 2023.26: Between May 3, 2023 and May 10, 2023
Changes in 2023.26
Photon returns an error if a file is modified between query planning and execution.
New features and extended support for Predictive I/O features. See Databricks Runtime 13.0 (EoS).
Use the Databricks connector to connect to another Databricks workspace.
CREATE TABLE LIKE
feature for Delta tables.New metadata column fields denoting file block start and length.
New H3 geospatial functions. See H3 geospatial functions.
New SQL built-in functions. See Databricks Runtime 13.0 (EoS).
User interface updates
Improvements:
Administrators can change warehouse owners using the user interface or the API. See Manage a SQL warehouse.
Catalog Explorer now displays account service principals in user lists for assets in Unity Catalog. For example, account service principals are visible when editing privileges or changing owners in Catalog Explorer.
Custom chart labels support the ability to reference any column within the dataset.
Dashboard filters now load column names, even when using queries that don’t have catalog or schema info.
April 27, 2023
Improvements:
The SQL editor now relies on the Monaco editor for a more reliable editing experience.
SQL History list page (Queries) now includes the Dubois Design System.
April 20, 2023
Improvements:
Introduces new pivot tables that allow you to aggregate more than 64k results.
Databricks SQL tables and visualizations now support BigInt, 38bit Decimals, and non UTF-8 characters. For numbers, the default setting is now user-defined digit precision.
Autocomplete now suggests frequent past joins for Unity Catalog tables, powered by Unity Catalog lineage data in Databricks Runtime 12.0 and above.
April 13, 2023
New feature:
The
TIMESTAMP_NTZ
type represents values comprising of fields year, month, day, hour, minute, and second. All operations are performed regardless of time zone. See TIMESTAMP_NTZ type.
Improvements:
Users can now send formatted results within alerts by using the
QUERY_RESULT_TABLE
tag in a custom alerts message.Users can now view the file data size for Unity Catalog tables in Catalog Explorer.
April 6, 2023
Databricks SQL Version 2023.20 Available
Rollout Schedule
Preview rollout for 2023.20: Between Mar 15, 2023 and Mar 23, 2023
Current rollout for 2023.20: Between Mar 27, 2023 and Apr 3, 2023
Changes in 2023.20
Delta Lake schema evolution supports specifying source columns in merge statements.
Remove all NULL elements from an array using array_compact.
To append elements to an array, use array_append.
To anonymize sensitive string values, use the mask function.
Common error conditions now return SQLSTATE.
Invoke table-valued generator functions in the regular
FROM
clause of a query.Use the
from_protobuf
andto_protobuf
functions to exchange data between binary and struct types. See Read and write protocol buffers.Improved consistency for Delta commit behavior for empty transactions relating to
update
,delete
, andmerge
commands.Behavior change
The lateral column alias feature introduces behavior changes during name resolution. See Behavior changes.
April 3, 2023
New feature:
The Create or modify table from file upload page now supports JSON file uploads. For more information, see Create or modify a table using file upload.
March 30, 2023
Improvements:
On the warehouse monitoring page, you can now view the query history for the selected time range along with your scaling charts. You can also view currently running and queued queries, active SQL sessions, the warehouse status, and the current cluster count. See Monitor a SQL warehouse.
Map clustering is now off by default in Marker maps.
Tooltips for visualization truncation and render limits have been added.
Fixes:
Charts now respect the original order when sorting is disabled for the axis values and the chart has the group by column.
March 23, 2023
Improvements:
Visualizations now support time binning by week.
Total
now calculates all data beyond the 100 X 100 cells up to 64k results in notebooks pivot tables.Users can now format cell values in the new notebooks pivot table.
File, query, and feature store lineage are available.
March 9, 2023
New feature:
The Databricks SQL Statement Execution API is now available in Public Preview. Statement Execution provides endpoints that are running SQL statements on a Databricks SQL warehouse. It can also check the status, fetch results, and cancel a running SQL statement.
Improvement:
The SQL admin console has been combined with the general admin settings to create a unified experience for admin users. All SQL admin settings are now accessed from the admin console.
Alert destinations are now called notification destinations.
Fixes:
Tables no longer display two scrollbars.
Blank widget titles no longer get overwritten on dashboards.
February 23, 2023
Improvement:
TINYINT
is now supported in results tables in notebooks.
Fix:
Fixed a bug where scrolling on the create dashboard filter modal resulted in an error.
February 16, 2023
Improvements:
Data labels on charts now show more characters to avoid truncating descriptions.
Autocomplete now recognizes
range()
and Python UDFcreate
functions.Autocomplete now avoids initiating snippets on decimals and within code comments.
Fixes:
Users can now zoom in on maps.
In notebooks, colors are now correctly assigned to charts.
February 2, 2023
Improvements:
Support for
DESCRIBE DETAILS
in the editor.Improved schema browser loading speed.
You can now view a list of possible columns on the side panel of a
SELECT *
.
January 26, 2023
Improvement:
Your query’s error messages now include links to the related documentation topic that describes the error.
January 19, 2023
Improvements:
You can now find a What’s New panel that highlights key enhancements. You can open and close this panel by clicking the gift icon.
Admins can now change ownership of SQL warehouses.
You can now filter across multiple visualizations in a dashboard by clicking Add > Filter and selecting the query and columns for filtering.
January 12, 2023
Improvements:
Visualization widget titles on dashboards are now formatted as Visualization name - Query name by default.
Added H3 Geospatial functions to the inline panel reference.
Added inline references for SQL syntax like
CREATE TABLE
andOVER
.
Fixes:
Sorting and reverse toggles are now available when the X-axis scale is set to automatic.
Heat maps and pivot tables are now responsive for certain edge cases.
The Categorical Y-axis tick marks are now sorted by default.
Query drafts are no longer lost after doing a browser refresh.
December 8, 2022
Databricks SQL alerts now support alerts for aggregations of query result columns such as
SUM
,COUNT
, andAVG
.The default visualization title is now ‘VisualizationName - QueryName’ when creating new widgets on dashboards.
November 17, 2022
Alerts:
Chatworks, Mattermost, and Google Hangouts Chat are no longer notification destinations.
Improvement:
The y-axis now displays numbers as percentages when checking percent values.
The workspace administrator setting to disable the upload data UI now applies to the new upload data UI. This setting applies to the Data Science & Engineering, Databricks Mosaic AI, and Databricks SQL personas.
Fixes:
Fixed an issue in Databricks SQL alerts where comparing against null values evaluated incorrectly.
Fixed an issue where scrollbars on pivot tables disappeared.
Fixed an issue where the schema browser couldn’t resize with overflowed tabs.
November 10, 2022
Improvement:
You can now create a dashboard filter that works across multiple queries at the same time. In Edit dashboard mode, choose Add, then Filter, then New Dashboard Filter.
Autocomplete now supports
CREATE MATERIALIZED VIEW
.
Fix:
Fixed an issue where scrolling to the end of a set of dashboard paged results would send an error.
Fixed an issue where switching from a stacked bar chart to a line chart kept the stacking property.
Fixed duplicated fetch calls.
November 3, 2022
Improvement:
When requesting access in Databricks SQL, the default permission is now “can run”.
Fixes:
Fixed an issue where sorting by
created_at
using the Queries and dashboards API did not return the correct sort order.Fixed an issue where columns containing URLs with HTML formatting had overly wide column widths.
Fixed an issue where
WHERE
keyword wasn’t highlighted.
October 27, 2022
Improvements:
The row limit for downloading query results to Excel has been increased from 64,000 rows to 100,000 rows. CSV & TSV download limits remain unchanged (about 1 GB of data).
Autocomplete now supports
LIST
syntax, URLs, and credentials.Consolidated and modernized Fix-me suggestion panels.
Fixes:
Fixed an issue where the warning banner in the editor overlapped full-height visualizations.
Fixed an issue where table column width was not preserved when columns of the table were moved.
Fixed an issue where the link to the dashboard in pop-up notifications was broken if a visualization was added from the SQL editor.
October 20, 2022
Improvements:
You can now find the query progress bar in the footer and the editing a visualization action in the kebab menu.
Autocomplete now supports Delta time travel, and provides column autocomplete when defining a foreign key.
Fix:
Fixed an issue where adding multiple visualizations to a dashboard in quick succession would result in visualizations not appearing on the dashboard.
October 13, 2022
Improvements:
You can now remove reported error messages.
COMMENT ON
instruction is now supported in the editor.You can now use Cmd+P or Ctr+P (for PC) as a shortcut for Top search. Use Cmd+I or Ctr+I (for PC) for Add parameters.
October 11, 2022
Improvements:
The add data UI provides access to common data sources configurations and file upload UIs. See Upload files to Databricks.
You can uploading small files to Delta Lake using a UI. See Create or modify a table using file upload.
October 6, 2022
Improvements:
EXTERNAL
is now a reserved table property. CommandsCREATE TABLE ... TBLPROPERTIES
andALTER TABLE ... SET TBLPROPERTIES
fail ifEXTERNAL
is specified in the properties.The
strfmt
informat_string(strfmt, obj, ...)
andprintf(strfmt, obj, ...)
no longer supports the use of0$
as the first argument. The first argument should be referenced by1$
when using an argument index to indicate the position of the argument in the argument list.Pie chart segments now have a thin border to delineate different segments.
You can now use Cmd+I (for Mac) or Ctr+I (for PC) as a shortcut for Add parameter. Use Cmd+P or Ctr+P (for PC) as a shortcut for Global search.
A feedback button is available for good or bad query error messages.
Fix me suggestions are now available as Quick fix.
Fixes:
lpad
andrpad
functions now work correctly withBINARY
string inputs. The output oflpad
andrpad
forBINARY
string inputs is now aBINARY
string.Fixed an issue where manual alert refreshes were not working.
Rolled back changes to automatic counter sizing to fix formatting issues.
September 29, 2022
Improvements:
You can now request access to Databricks SQL Queries, alerts, and dashboards from owners of those assets.
You can now filter by query duration and statement type on the Query history page.
To use the query duration filter, enter an amount of time and choose a time unit. The history result returns queries that run longer than the time specified.
To use the statement type filter, choose a statement type from the dropdown. The history result returns queries containing that statement.
Fix:
Fixed an issue where not all supported HTML tags were working in custom alert templates. All tags are available as documented.
Fixed an issue where Visualization notification toasts. For example, double clicking to zoom out on a visualization was previously not showing.
Fixed an issue where swapping the axes on a chart was not reflected in the chart name.
September 22, 2022
Improvements:
Counter widgets of the same size will have the same font sizing when multiple counter widgets are displayed on a dashboard.
Updated combination charts so that when using dual axis, only the same chart type (e.g., line, bar) can be used on the same axis. Series aliases are also applied to the axis.
Added autocomplete support for surrogate keys and
LIST
operations.
Fix:
Fixed issue where text parameters did not accept
Null
as a valid value.
September 15, 2022
Fix:
Fixed an issue where viewing query history from the SQL warehouses listing page did not work.
September 8, 2022
Improvement:
Introducing the new ‘Open Source Integrations’ card in DSE/SQL homepages that displays open source integration options such as Delta Live Tables and dbt core.
Fix:
Fixed an issue where parameter dropdown menus were blocked by the visualization tab.
September 1, 2022
Improvements:
Introducing a new simplified UI to add parameters and filters. Choose <button>+</button> and choose to add a filter or parameter.
The parentheses of SQL tokens, such as ‘OVER()’ now get autocompleted.
Fixes:
Fixed an issue where viewing the dashboard in full-screen ignored the color palette.
Fixed an issue where typing quickly and then using the Run shortcut ran the previous query text, instead of the newly typed query text.
Fixed issue where using the keyboard command, ctrl+enter to run queries would submit duplicate queries.
August 25, 2022
Fix:
Fixed an issue where dashboard filters were not updating when query parameters changed.
August 18, 2022
Documentation: Alerts API documentation has been released.
Visualizations: Users can now set default values for date filters. Any time the filter is refreshed on a query or dashboard, the default value is applied.
Fixes:
Fixed an issue where apply changes did not work if a dashboard was still reloading.
Fixed an issue where columns were too narrow when a query returns no results.
August 11, 2022
Improvements:
Users can receive emails when their refreshes fail. To enable such notifications, navigate to the SQL settings tab of the admin console. Under Failure Emails, choose the type of object (Query, Dashboard, or Alert) for which you wish to receive failure notifications. Failure reports are sent hourly.
Visualizations
Introducing a new, modern color palette for visualizations and dashboards. To change a dashboard to the new color palette, go to your dashboard, click on <button>Edit</button> -> <button>Colors</button> -> <button>Import</button> and select the Databricks Color Palette. SQL Admins can also set the new color palette as the default option for a workspace by going to <button>Settings</button> -> <button>QL Admin Console</button> -> <button>Workspace Colors</button> -> <button>Import</button> and selecting the new palette.
Fixes:
Fixed an issue where previously selecting <button>Apply Changes</button> to apply a filter did not work if a query was already being executed.
August 4, 2022
Improvements:
On cloning a dashboard, there is now an option for whether or not queries should be cloned as well.
Tab content is synced across browser tabs. The state of your query will now be in sync across all browser tabs. This means that if you are working on query1 in browser tab 1 and then switch to browser tab 2, you’ll see query1 in the state you left it in while in the original browser tab.
Fix:
Labels for empty strings in pie chart now reflect that the string is empty rather than the index of the value.
July 28, 2022
Alerts
Custom alert email templates have been updated to disallow certain HTML tags that may pose a security risk. Disallowed HTML tags and attributes are automatically sanitized. For example, <button> is a disallowed HTML tag, so instead of rendering a button, the text “button” displays. See Alerts for the list of allowed HTML tags and attributes.
Users can now subscribe other users to alerts without needing to create a notification destination, which requires admin permissions.
Downloads: Users can now download up to approximately 1GB of results data from Databricks SQL in CSV and TSV format, up from 64,000 rows previously.
Visualizations
You can now edit visualizations directly on the dashboard. In edit mode, click on the kebab menu and select Edit visualization to begin editing the visualization.
When downloading results associated with a visualization leveraging aggregations, the downloaded results are also aggregated. The option to download is moving from bottom kebab to the kebab associated with the tab. The downloaded results are from the most recent execution of the query that created the visualization.
SQL editor: Results tables now display a message when data displayed by the in-browser table has been limited to 64,000 rows. TSV and CSV download will still be up to approximately 1GB of data.
Query filters:
Query filters have been updated to work dynamically on either client- or server-side to optimize performance. Previous query filters (now legacy) operated client-side only. Users can still use legacy filters with the
::
syntax, if desired.The updated filters are simpler: Users click a +Add Filter button and select a column from a dropdown. Previously, users had to modify the query text directly.
Relevant values are highlighted to make it easier to see which selections within a filter will return results given other filter selections.
Query history: Query details in Query History now show the Query Source, which is the origin of the executed query.
July 21, 2022
Notifications on share: Users will now be notified by email whenever a dashboard, query, or alert is shared with them.
Enhanced SQL editor experience via the new embedded editor toolkit
Live syntax error highlighting (for example, wrong keyword, table does not exist, and suggestions for fixing the error)
In context help: on hover (for example, full table name, detailed Function panel) and inline execution error messages (for example, highlight row with error post execution)
Intelligent ranking of suggestions (for example, parameter autocompletion, ranking formula, less noisy matching)
July 14, 2022
You can now upload TSV files using the
Create Table
UI in addition to CSV files.Databricks SQL now provides the option to notify users by email whenever a dashboard, query, or alert is shared with them.
Visualization tables now optionally include row numbers displayed next to results.
When you select a geographic region for the Chloropleth visualization, you now get inline hints for accepted values.
June 23, 2022
SQL endpoint name change: Databricks changed the name from SQL endpoint to SQL warehouse because it is more than just an API entry point for running SQL commands. A SQL warehouse is a computation resource for all your data warehousing needs, an integral part of the Databricks platform. Compute resources are infrastructure resources that provide processing capabilities in the cloud.
For Choropleth visualizations, the Key column and Target field selections in the visualization editor have been renamed to Geographic Column and Geographic Type. This renaming for understandability does not introduce any behavior changes to new or existing Choropleths.
The limit 1000 query option has moved from a checkbox in the SQL query editor to a checkbox in the run button.
Cached queries in Query History table are now marked with a Cache tag.
Manually refreshing a dashboard uses the dashboard’s warehouse (if available) instead of each individual query’s warehouse.
Refreshing an alert always uses the alert’s warehouse, regardless of the Run as Viewer/Owner setting.
June 9, 2022
When you hover in the endpoint selector, the full endpoint name is displayed as a tooltip.
When you filter in the SQL editor schema browser, the search term is now highlighted in the search results.
The Close All dialog box in the SQL editor now displays a list of unsaved queries.
To reopen the last closed tab in the SQL editor, use this new keyboard shortcut:
<Cmd> + <Shift> + <Option> + T
You can now add data labels to combination charts.
The list of visualization aggregations operations now includes variance and standard deviation.
May 26, 2022
Authoring improvements:
You can now bypass aggregations when you author visualizations. This is particularly useful when your query already includes an aggregation. For example, if your query is
SELECT AVG(price_per_sqft), isStudio, location GROUP BY location, isStudio
, the chart editor previously required explicitly specifying another layer of aggregation.When you author dashboards, you now have the ability to:
Duplicate textbox widgets
Expand the size of the edit textbox panel
The default aggregation for the error column when you author visualizations is standard deviation.
Fixes:
Edit actions for visualizations are only available when the dashboard is in edit mode. Edit actions are no longer available as a view mode action.
When you create a new query, it opens in a tab to the immediate right of the tab in focus rather than at the end of the list.
The open query modal shows which query is already open and provides the option to switch focus to that query tab.
The Sankey & Sunburst charts no longer treat 0 as null.
May 19, 2022
Fixed issue: When you have the focus of the SQL editor open on a specific visualization tab and share the link to another user, the user will have the same focus in the SQL editor when they click the shared link.
Improvements:
Microsoft Teams is now a supported notification destination.
The Date Range, Date and Time Range, and Date and Time Range (with seconds) parameters now support the option to designate the starting day of the week, with Sunday as the default.
May 12, 2022
Visualizations now support time binning directly in the UI. You can now easily switch between yearly, monthly, daily, or hourly bins of your data by changing a dropdown value rather than adding and modifying a
date_trunc()
function in the query text itself.Dashboards now have color consistency by default. If you have the same series across multiple charts, the series is always colored the same across all charts – without requiring any manual configuration.
May 3, 2022
When sharing a dashboard with a user or group, we now also provide the ability to share all upstream queries used by visualizations and parameters.
When you do not have permission to share one or more of the upstream queries, you will receive a warning message that not all queries could be shared.
The permissions granted when sharing a dashboard do not override, negate, or expand upon existing permissions on the upstream queries. For example, if a user or group has CAN RUN as Owner permissions on the shared dashboard but only has Run as Viewer permissions on an upstream query, the effective permissions on that upstream query will be Run as Viewer.
Known issues
Reads from data sources other than Delta Lake in multi-cluster load balanced SQL endpoints can be inconsistent.
Delta tables accessed in Databricks SQL upload their schema and table properties to the configured metastore. If you are using an external metastore, you will be able to see Delta Lake information in the metastore. Delta Lake tries to keep this information as up-to-date as possible on a best-effort basis. You can also use the
DESCRIBE <table>
command to ensure that the information is updated in your metastore.Databricks SQL does not support zone offsets like ‘GMT+8’ as session time zones. The workaround is to use a region based time zone https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) like ‘Etc/GMT+8’ instead. See SET TIME ZONE for more information about setting time zones.
Frequently asked questions (FAQ)
How are Databricks SQL workloads charged?
Databricks SQL workloads are charged according to the Premium Jobs Compute SKU.
Where do SQL endpoints run?
Like Databricks clusters, SQL endpoints are created and managed in your Google Cloud account. SQL endpoints manage SQL-optimized clusters automatically in your account and scale to match end-user demand.
Can I use SQL endpoints from Data Science & Engineering workspace SQL notebooks?
No. You can use SQL endpoints from Databricks SQL queries, BI tools, and other JDBC and ODBC clients.
I have been granted access to data using a cloud provider credential. Why can’t I access this data in Databricks SQL?
In Databricks SQL, all access to data is subject to data access control, and an administrator or data owner must first grant you the appropriate privileges.