Databricks SQL release notes

This article lists new Databricks SQL features and improvements, along with known issues and FAQs.

Release process

Databricks releases updates to the Databricks SQL web application user interface on an ongoing basis, with all users getting the same updates, rolled out over a short period of time.

In addition, Databricks typically releases new SQL warehouse compute versions regularly. Two channels are always available: Preview and Current.

Note

Releases are staged. Your Databricks account may not be updated with a new SQL warehouse version or Databricks SQL feature until a week or more after the initial release date.

Channels

Channels let you choose whether to use the Current SQL warehouse compute version or the Preview version. Preview versions let you try out functionality before it becomes the Databricks SQL standard. Take advantage of preview versions to test your production queries and dashboards against upcoming changes.

Typically, a preview version is promoted to the current channel approximately two weeks after being released to the preview channel. Some features, such as security features, maintenance updates, and bug fixes, may be released directly to the current channel. From time to time, Databricks may promote a preview version to the current channel on a different schedule. Each new version will be announced in the following sections.

To learn how to switch an existing SQL warehouse to the preview channel, see Preview Channels. The features listed in the user interface updates sections are independent of the SQL Warehouse compute versions described in the Channels section of the release notes.

Available Databricks SQL versions

Current channel: Databricks SQL version 2023.20 Preview channel: Databricks SQL version 2023.26

June 1, 2023

Improvements:

  • Binary data will now render as a hex string when using the Arrow format.

  • In the SQL Statement API, the CSV format is now supported for the EXTERNAL_LINKS disposition. This allows clients to extract up to 100 GiB of data in CSV format with pre-signed URLs, whereas the INLINE limit for JSON is 16 MiB.

  • Databricks SQL Pro is available in all regions. SQL Pro is now the default warehouse in the UI. The API default is still Classic.

May 29, 2023

New feature:

May 25, 2023

Improvements:

  • You can now toggle the autocompletion result panel.

  • Disable the enter key so it no longer accepts autocomplete suggestions. Under DBSQL User Settings, click Editor Settings, then New Editor settings. Turn off *Enter key accepts autocomplete suggestions*.

Fixes:

  • Sorted table headers now have colors.

  • Chart lines now render correctly.

May 18, 2023

Improvement:

  • In the SQL Statement API, the EXTERNAL_LINKS disposition now supports the JSON_ARRAY format. You can extract up to 100 GiB of data in JSON format with pre-signed URLs. The INLINE limit for JSON is 16 MiB.

May 11, 2023

New feature:

  • Schema Browser is now generally available in Data Explorer.

Improvements:

  • On-hover table detail panel showing is less sensitive.

  • The escape key now closes the autocomplete panel.

  • View definitions now have syntax highlighting in the Data Explorer details tab.

Fixes:

  • Pivot tables now correctly render on Windows devices.

  • Completion suggestions now properly follows the case of the first keyword.

May 4, 2023

Databricks SQL Version 2023.26 Available

Rollout Schedule

  • Preview rollout for 2023.26: Between April 19, 2023 and April 25, 2023

  • Current rollout for 2023.26: Between May 3, 2023 and May 10, 2023

Changes in 2023.26

  • Photon returns an error if a file is modified between query planning and execution.

  • New features and extended support for Predictive I/O features. See Databricks Runtime 13.0.

  • Use the Databricks connector to connect to another Databricks workspace.

  • CREATE TABLE LIKE feature for Delta tables.

  • New metadata column fields denoting file block start and length.

  • New H3 geospatial functions. See H3 geospatial functions.

  • New SQL built-in functions. See Databricks Runtime 13.0.

User interface updates

Improvements:

  • Administrators can change warehouse owners using the user interface or the API. See Transfer ownership of Databricks SQL objects.

  • Data Explorer now displays account service principals in user lists for assets in Unity Catalog. For example, account service principals are visible when editing privileges or changing owners in Data Explorer.

  • Custom chart labels support the ability to reference any column within the dataset.

  • Dashboard filters now load column names, even when using queries that don’t have catalog or schema info.

April 27, 2023

Improvements:

  • The SQL editor now relies on the Monaco editor for a more reliable editing experience.

  • SQL History list page (Queries) now includes the Dubois Design System.

April 20, 2023

Improvements:

  • Introduces new pivot tables that allow you to aggregate more than 64k results.

  • Databricks SQL tables and visualizations now support BigInt, 38bit Decimals, and non UTF-8 characters. For numbers, the default setting is now user-defined digit precision.

  • Autocomplete now suggests frequent past joins for Unity Catalog tables, powered by Unity Catalog lineage data in Databricks Runtime 12.0 and above.

New feature:

  • Return text generated by a selected large language model (LLM) given the prompt with ai_generate_text. This function is only available as public preview on Databricks SQL Pro and Serverless. To participate in the public preview, populate and submit the AI Functions Public Preview enrollment form.

April 13, 2023

New feature:

  • The TIMESTAMP_NTZ type represents values comprising of fields year, month, day, hour, minute, and second. All operations are performed regardless of time zone. See TIMESTAMP_NTZ type.

Improvements:

  • Users can now send formatted results within alerts by using the QUERY_RESULT_TABLE tag in a custom alerts message.

  • Users can now view the file data size for Unity Catalog tables in Data Explorer.

April 6, 2023

Databricks SQL Version 2023.20 Available

Rollout Schedule

  • Preview rollout for 2023.20: Between Mar 15, 2023 and Mar 23, 2023

  • Current rollout for 2023.20: Between Mar 27, 2023 and Apr 3, 2023

Changes in 2023.20

  • Delta Lake schema evolution supports specifying source columns in merge statements.

  • Remove all NULL elements from an array using array_compact.

  • To append elements to an array, use array_append.

  • To anonymize sensitive string values, use the mask function.

  • Common error conditions now return SQLSTATE.

  • Invoke table-valued generator functions in the regular FROM clause of a query.

  • Use the from_protobuf and to_protobuf functions to exchange data between binary and struct types. See Read and write protocol buffers.

  • Improved consistency for Delta commit behavior for empty transactions relating to update, delete, and merge commands.

  • Behavior change

    • The lateral column alias feature introduces behavior changes during name resolution. See Behavior changes.

User interface updates

Improvements:

  • Data explorer now supports Hive metastore delete.

  • The message for the visualization editor’s initial state changed from Missing required columns to No Data.

April 3, 2023

New feature:

March 30, 2023

Improvements:

  • On the warehouse monitoring page, you can now view the query history for the selected time range along with your scaling charts. You can also view currently running and queued queries, active SQL sessions, the warehouse status, and the current cluster count. See Monitor a SQL warehouse.

  • Map clustering is now off by default in Marker maps.

  • Tooltips for visualization truncation and render limits have been added.

Fixes:

  • Charts now respect the original order when sorting is disabled for the axis values and the chart has the group by column.

March 23, 2023

Improvements:

  • Visualizations now support time binning by week.

  • Total now calculates all data beyond the 100 X 100 cells up to 64k results in notebooks pivot tables.

  • Users can now format cell values in the new notebooks pivot table.

  • File, query, and feature store lineage are available.

March 16, 2023

Improvement:

  • Support for GROUP BY ALL syntax.

March 9, 2023

New feature:

  • The Databricks SQL Statement Execution API is now available in Public Preview. Statement Execution provides endpoints that are running SQL statements on a Databricks SQL warehouse. It can also check the status, fetch results, and cancel a running SQL statement.

Improvement:

  • The SQL admin console has been combined with the general admin settings to create a unified experience for admin users. All SQL admin settings are now accessed from the admin console.

  • Alert destinations are now called notification destinations.

Fixes:

  • Tables no longer display two scrollbars.

  • Blank widget titles no longer get overwritten on dashboards.

March 2, 2023

Fix:

  • You can now use relative URLs in tables to access dashboards.

February 23, 2023

Improvement:

  • TINYINT is now supported in results tables in notebooks.

Fix:

  • Fixed a bug where scrolling on the create dashboard filter modal resulted in an error.

February 16, 2023

Improvements:

  • Data labels on charts now show more characters to avoid truncating descriptions.

  • Autocomplete now recognizes range() and Python UDF create functions.

  • Autocomplete now avoids initiating snippets on decimals and within code comments.

Fixes:

  • Users can now zoom in on maps.

  • In notebooks, colors are now correctly assigned to charts.

February 2, 2023

Improvements:

  • Support for DESCRIBE DETAILS in the editor.

  • Improved schema browser loading speed.

  • You can now view a list of possible columns on the side panel of a SELECT *.

January 26, 2023

Improvement:

  • Your query’s error messages now include links to the related documentation topic that describes the error.

January 19, 2023

Improvements:

  • You can now find a What’s New panel that highlights key enhancements. You can open and close this panel by clicking the gift icon.

  • Admins can now change ownership of SQL warehouses.

  • You can now filter across multiple visualizations in a dashboard by clicking Add > Filter and selecting the query and columns for filtering.

January 12, 2023

Improvements:

  • Visualization widget titles on dashboards are now formatted as Visualization name - Query name by default.

  • Added H3 Geospatial functions to the inline panel reference.

  • Added inline references for SQL syntax like CREATE TABLE and OVER.

Fixes:

  • Sorting and reverse toggles are now available when the X-axis scale is set to automatic.

  • Heat maps and pivot tables are now responsive for certain edge cases.

  • The Categorical Y-axis tick marks are now sorted by default.

  • Query drafts are no longer lost after doing a browser refresh.

December 8, 2022

  • Databricks SQL alerts now support alerts for aggregations of query result columns such as SUM, COUNT, and AVG.

  • The default visualization title is now ‘VisualizationName - QueryName’ when creating new widgets on dashboards.

November 17, 2022

Alerts:

  • Chatworks, Mattermost, and Google Hangouts Chat are no longer notification destinations.

Improvement:

  • The y-axis now displays numbers as percentages when checking percent values.

  • The workspace administrator setting to disable the upload data UI now applies to the new upload data UI. This setting applies to the Data Science & Engineering, Databricks Machine Learning, and Databricks SQL personas.

Fixes:

  • Fixed an issue in Databricks SQL alerts where comparing against null values evaluated incorrectly.

  • Fixed an issue where scrollbars on pivot tables disappeared.

  • Fixed an issue where the schema browser couldn’t resize with overflowed tabs.

November 10, 2022

Improvement:

  • You can now create a dashboard filter that works across multiple queries at the same time. In Edit dashboard mode, choose Add, then Filter, then New Dashboard Filter.

  • Autocomplete now supports CREATE MATERIALIZED VIEW.

Fix:

  • Fixed an issue where scrolling to the end of a set of dashboard paged results would send an error.

  • Fixed an issue where switching from a stacked bar chart to a line chart kept the stacking property.

  • Fixed duplicated fetch calls.

November 3, 2022

Improvement:

  • When requesting access in Databricks SQL, the default permission is now “can run”.

Fixes:

  • Fixed an issue where sorting by created_at using the Queries and dashboards API did not return the correct sort order.

  • Fixed an issue where columns containing URLs with HTML formatting had overly wide column widths.

  • Fixed an issue where WHERE keyword wasn’t highlighted.

October 27, 2022

Improvements:

  • The row limit for downloading query results to Excel has been increased from 64,000 rows to 100,000 rows. CSV & TSV download limits remain unchanged (about 1 GB of data).

  • Autocomplete now supports LIST syntax, URLs, and credentials.

  • Consolidated and modernized Fix-me suggestion panels.

Fixes:

  • Fixed an issue where the warning banner in the editor overlapped full-height visualizations.

  • Fixed an issue where table column width was not preserved when columns of the table were moved.

  • Fixed an issue where the link to the dashboard in pop-up notifications was broken if a visualization was added from the SQL editor.

October 20, 2022

Improvements:

  • You can now find the query progress bar in the footer and the editing a visualization action in the kebab menu.

  • Autocomplete now supports Delta time travel, and provides column autocomplete when defining a foreign key.

Fix:

  • Fixed an issue where adding multiple visualizations to a dashboard in quick succession would result in visualizations not appearing on the dashboard.

October 13, 2022

Improvements:

  • You can now remove reported error messages.

  • COMMENT ON instruction is now supported in the editor.

  • You can now use Cmd+P or Ctr+P (for PC) as a shortcut for Top search. Use Cmd+I or Ctr+I (for PC) for Add parameters.

October 11, 2022

Improvements:

October 6, 2022

Improvements:

  • EXTERNAL is now a reserved table property. Commands CREATE TABLE ... TBLPROPERTIES and ALTER TABLE ... SET TBLPROPERTIES fail if EXTERNAL is specified in the properties.

  • The strfmt in format_string(strfmt, obj, ...) and printf(strfmt, obj, ...) no longer supports the use of 0$ as the first argument. The first argument should be referenced by 1$ when using an argument index to indicate the position of the argument in the argument list.

  • Pie chart segments now have a thin border to delineate different segments.

  • You can now use Cmd+I (for Mac) or Ctr+I (for PC) as a shortcut for Add parameter. Use Cmd+P or Ctr+P (for PC) as a shortcut for Global search.

  • A feedback button is available for good or bad query error messages.

  • Fix me suggestions are now available as Quick fix.

Fixes:

  • lpad and rpad functions now work correctly with BINARY string inputs. The output of lpad and rpad for BINARY string inputs is now a BINARY string.

  • Fixed an issue where manual alert refreshes were not working.

  • Rolled back changes to automatic counter sizing to fix formatting issues.

September 29, 2022

Improvements:

  • You can now request access to Databricks SQL Queries, alerts, and dashboards from owners of those assets.

  • You can now filter by query duration and statement type on the Query history page.

    • To use the query duration filter, enter an amount of time and choose a time unit. The history result returns queries that run longer than the time specified.

    • To use the statement type filter, choose a statement type from the dropdown. The history result returns queries containing that statement.

Fix:

  • Fixed an issue where not all supported HTML tags were working in custom alert templates. All tags are available as documented.

  • Fixed an issue where Visualization notification toasts. For example, double clicking to zoom out on a visualization was previously not showing.

  • Fixed an issue where swapping the axes on a chart was not reflected in the chart name.

September 22, 2022

Improvements:

  • Counter widgets of the same size will have the same font sizing when multiple counter widgets are displayed on a dashboard.

  • Updated combination charts so that when using dual axis, only the same chart type (e.g., line, bar) can be used on the same axis. Series aliases are also applied to the axis.

  • Added autocomplete support for surrogate keys and LIST operations.

Fix:

  • Fixed issue where text parameters did not accept Null as a valid value.

September 15, 2022

Fix:

  • Fixed an issue where viewing query history from the SQL warehouses listing page did not work.

September 8, 2022

Improvement:

  • Introducing the new ‘Open Source Integrations’ card in DSE/SQL homepages that displays open source integration options such as Delta Live Tables and dbt core.

Fix:

  • Fixed an issue where parameter dropdown menus were blocked by the visualization tab.

September 1, 2022

Improvements:

  • Introducing a new simplified UI to add parameters and filters. Choose <button>+</button> and choose to add a filter or parameter.

  • The parentheses of SQL tokens, such as ‘OVER()’ now get autocompleted.

Fixes:

  • Fixed an issue where viewing the dashboard in full-screen ignored the color palette.

  • Fixed an issue where typing quickly and then using the Run shortcut ran the previous query text, instead of the newly typed query text.

  • Fixed issue where using the keyboard command, ctrl+enter to run queries would submit duplicate queries.

August 25, 2022

Fix:

  • Fixed an issue where dashboard filters were not updating when query parameters changed.

August 18, 2022

  • Documentation: Alerts API documentation has been released.

  • Visualizations: Users can now set default values for date filters. Any time the filter is refreshed on a query or dashboard, the default value is applied.

  • Fixes:

    • Fixed an issue where apply changes did not work if a dashboard was still reloading.

    • Fixed an issue where columns were too narrow when a query returns no results.

August 11, 2022

Improvements:

  • Users can receive emails when their refreshes fail. To enable such notifications, navigate to the SQL settings tab of the admin console. Under Failure Emails, choose the type of object (Query, Dashboard, or Alert) for which you wish to receive failure notifications. Failure reports are sent hourly.

  • Visualizations

    • Introducing a new, modern color palette for visualizations and dashboards. To change a dashboard to the new color palette, go to your dashboard, click on <button>Edit</button> -> <button>Colors</button> -> <button>Import</button> and select the Databricks Color Palette. SQL Admins can also set the new color palette as the default option for a workspace by going to <button>Settings</button> -> <button>QL Admin Console</button> -> <button>Workspace Colors</button> -> <button>Import</button> and selecting the new palette.

Fixes:

  • Fixed an issue where previously selecting <button>Apply Changes</button> to apply a filter did not work if a query was already being executed.

August 4, 2022

Improvements:

  • On cloning a dashboard, there is now an option for whether or not queries should be cloned as well.

  • Tab content is synced across browser tabs. The state of your query will now be in sync across all browser tabs. This means that if you are working on query1 in browser tab 1 and then switch to browser tab 2, you’ll see query1 in the state you left it in while in the original browser tab.

Fix:

  • Labels for empty strings in pie chart now reflect that the string is empty rather than the index of the value.

July 28, 2022

  • Alerts

    • Custom alert email templates have been updated to disallow certain HTML tags that may pose a security risk. Disallowed HTML tags and attributes are automatically sanitized. For example, <button> is a disallowed HTML tag, so instead of rendering a button, the text “button” displays. See Alerts for the list of allowed HTML tags and attributes.

    • Users can now subscribe other users to alerts without needing to create a notification destination, which requires admin permissions.

  • Downloads: Users can now download up to approximately 1GB of results data from Databricks SQL in CSV and TSV format, up from 64,000 rows previously.

  • Visualizations

    • You can now edit visualizations directly on the dashboard. In edit mode, click on the kebab menu and select Edit visualization to begin editing the visualization.

    • When downloading results associated with a visualization leveraging aggregations, the downloaded results are also aggregated. The option to download is moving from bottom kebab to the kebab associated with the tab. The downloaded results are from the most recent execution of the query that created the visualization.

  • SQL editor: Results tables now display a message when data displayed by the in-browser table has been limited to 64,000 rows. TSV and CSV download will still be up to approximately 1GB of data.

  • Query filters:

    • Query filters have been updated to work dynamically on either client- or server-side to optimize performance. Previous query filters (now legacy) operated client-side only. Users can still use legacy filters with the :: syntax, if desired.

    • The updated filters are simpler: Users click a +Add Filter button and select a column from a dropdown. Previously, users had to modify the query text directly.

    • Relevant values are highlighted to make it easier to see which selections within a filter will return results given other filter selections.

  • Query history: Query details in Query History now show the Query Source, which is the origin of the executed query.

July 21, 2022

  • Notifications on share: Users will now be notified by email whenever a dashboard, query, or alert is shared with them.

  • Enhanced SQL editor experience via the new embedded editor toolkit

    • Live syntax error highlighting (for example, wrong keyword, table does not exist, and suggestions for fixing the error)

    • In context help: on hover (for example, full table name, detailed Function panel) and inline execution error messages (for example, highlight row with error post execution)

    • Intelligent ranking of suggestions (for example, parameter autocompletion, ranking formula, less noisy matching)

July 14, 2022

  • You can now upload TSV files using the Create Table UI in addition to CSV files.

  • Databricks SQL now provides the option to notify users by email whenever a dashboard, query, or alert is shared with them.

  • Visualization tables now optionally include row numbers displayed next to results.

  • When you select a geographic region for the Chloropleth visualization, you now get inline hints for accepted values.

June 23, 2022

  • SQL endpoint name change: Databricks changed the name from SQL endpoint to SQL warehouse because it is more than just an API entry point for running SQL commands. A SQL warehouse is a computation resource for all your data warehousing needs, an integral part of the Lakehouse Platform. Compute resources are infrastructure resources that provide processing capabilities in the cloud.

  • For Choropleth visualizations, the Key column and Target field selections in the visualization editor have been renamed to Geographic Column and Geographic Type. This renaming for understandability does not introduce any behavior changes to new or existing Choropleths.

  • The limit 1000 query option has moved from a checkbox in the SQL query editor to a checkbox in the run button.

  • Cached queries in Query History table are now marked with a Cache tag.

  • Manually refreshing a dashboard uses the dashboard’s warehouse (if available) instead of each individual query’s warehouse.

  • Refreshing an alert always uses the alert’s warehouse, regardless of the Run as Viewer/Owner setting.

June 9, 2022

  • When you hover in the endpoint selector, the full endpoint name is displayed as a tooltip.

  • When you filter in the SQL editor schema browser, the search term is now highlighted in the search results.

  • The Close All dialog box in the SQL editor now displays a list of unsaved queries.

  • To reopen the last closed tab in the SQL editor, use this new keyboard shortcut: <Cmd> + <Shift> + <Option> + T

  • You can now add data labels to combination charts.

  • The list of visualization aggregations operations now includes variance and standard deviation.

May 26, 2022

Authoring improvements:

  • You can now bypass aggregations when you author visualizations. This is particularly useful when your query already includes an aggregation. For example, if your query is SELECT AVG(price_per_sqft), isStudio, location GROUP BY location, isStudio, the chart editor previously required explicitly specifying another layer of aggregation.

  • When you author dashboards, you now have the ability to:

    • Duplicate textbox widgets

    • Expand the size of the edit textbox panel

  • The default aggregation for the error column when you author visualizations is standard deviation.

Fixes:

  • Edit actions for visualizations are only available when the dashboard is in edit mode. Edit actions are no longer available as a view mode action.

  • When you create a new query, it opens in a tab to the immediate right of the tab in focus rather than at the end of the list.

  • The open query modal shows which query is already open and provides the option to switch focus to that query tab.

  • The Sankey & Sunburst charts no longer treat 0 as null.

May 19, 2022

  • Fixed issue: When you have the focus of the SQL editor open on a specific visualization tab and share the link to another user, the user will have the same focus in the SQL editor when they click the shared link.

  • Improvements:

    • Microsoft Teams is now a supported notification destination.

    • The Date Range, Date and Time Range, and Date and Time Range (with seconds) parameters now support the option to designate the starting day of the week, with Sunday as the default.

May 12, 2022

  • Visualizations now support time binning directly in the UI. You can now easily switch between yearly, monthly, daily, or hourly bins of your data by changing a dropdown value rather than adding and modifying a date_trunc() function in the query text itself.

  • Dashboards now have color consistency by default. If you have the same series across multiple charts, the series is always colored the same across all charts – without requiring any manual configuration.

May 3, 2022

  • When sharing a dashboard with a user or group, we now also provide the ability to share all upstream queries used by visualizations and parameters.

    • When you do not have permission to share one or more of the upstream queries, you will receive a warning message that not all queries could be shared.

    • The permissions granted when sharing a dashboard do not override, negate, or expand upon existing permissions on the upstream queries. For example, if a user or group has Can Run as Owner permissions on the shared dashboard but only has Run as Viewer permissions on an upstream query, the effective permissions on that upstream query will be Run as Viewer.

April 27, 2022

  • Your dashboard layout is now retained when exporting to PDF on demand and generating scheduled subscription emails.

April 25, 2022

  • Databricks SQL is now available in Public Preview on Google Cloud Platform.

Known issues

  • Reads from data sources other than Delta Lake in multi-cluster load balanced SQL endpoints can be inconsistent.

  • Delta tables accessed in Databricks SQL upload their schema and table properties to the configured metastore. If you are using an external metastore, you will be able to see Delta Lake information in the metastore. Delta Lake tries to keep this information as up-to-date as possible on a best-effort basis. You can also use the DESCRIBE <table> command to ensure that the information is updated in your metastore.

  • Databricks SQL does not support zone offsets like ‘GMT+8’ as session time zones. The workaround is to use a region based time zone https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) like ‘Etc/GMT+8’ instead. See SET TIME ZONE for more information about setting time zones.

Frequently asked questions (FAQ)

How are Databricks SQL workloads charged?

Databricks SQL workloads are charged according to the Premium Jobs Compute SKU.

Where do SQL endpoints run?

Like Databricks clusters, SQL endpoints are created and managed in your Google Cloud account. SQL endpoints manage SQL-optimized clusters automatically in your account and scale to match end-user demand.

Can I use SQL endpoints from Data Science & Engineering workspace SQL notebooks?

No. You can use SQL endpoints from Databricks SQL queries, BI tools, and other JDBC and ODBC clients.

I have been granted access to data using a cloud provider credential. Why can’t I access this data in Databricks SQL?

In Databricks SQL, all access to data is subject to data access control, and an administrator or data owner must first grant you the appropriate privileges.