These features and Databricks platform improvements were released in July 2022.
Releases are staged. Your Databricks account may not be updated until a week or more after the initial release date.
July 28, 2022
You can now configure audit logs to record when a notebook command or Databricks SQL query runs. To do so, use the workspace configuration setting verbose audit logs. See Enable verbose audit logs.
July 27, 2022
Databricks Runtime 11.1 and Databricks Runtime 11.1 ML are now generally available.
July 27, 2022
Photon is now generally available, beginning with Databricks Runtime 11.1.
Photon is the native vectorized query engine on Databricks, written to be directly compatible with Apache Spark APIs so it works with your existing code. Photon is developed in C++ to take advantage of modern hardware, and uses the latest techniques in vectorized query processing to capitalize on data- and instruction-level parallelism in CPUs, enhancing performance on real-world data and applications—all natively on your data lake.
Photon is part of a high-performance runtime that runs your existing SQL and DataFrame API calls faster and reduces your total cost per workload. Photon is used by default in Databricks SQL warehouses.
See Photon is GA in the Databricks Runtime 11.1 release notes.
July 27 - August 15, 2022
If you navigate away from a notebook while it’s running, a notification now appears when it finishes. You can enable or disable these notifications in your browser.
July 19-22, 2022
You can now create up to 10,000 jobs in a Databricks workspace when you enable the Increased number of jobs toggle in the Databricks admin console. This feature also enables a paginated Jobs list when you view jobs in the Jobs user interface, reducing the time required to load the Jobs list view.
Databricks recommends Jobs API 2.1 if you enable the increased jobs limit. Using the Jobs 2.0 API results in degraded performance when returning large numbers of jobs.
To learn how to enable this feature, see Increase the saved jobs limit in a workspace.
July 11, 2022
We have released version 2.6.27 of the Databricks JDBC driver (download). This release upgrades the third-party libraries such as Apache Arrow 7.0.0. This release also includes updates to the license file and configuration installation guide.
In addition, this release resolves the following issues:
In some cases, when an HTTP connection has been active for a long period of time, the connector returns a
In some cases, unshaded
Log4jclasses in the connector JAR file conflict with files deployed in the end user’s environment. The unshaded classes have been removed from the connector JAR file.
When using a DataSource class, the subprotocol names are incorrect.
July 11, 2022
We have released version 2.6.25 of the Databricks ODBC driver (download). This release adds support to configure the connector to mitigate straggling result file download operations.
July 8, 2022
Databricks Runtime 11.1, 11.1 Photon, and 11.1 ML are now available as Beta releases.
See the full release notes at Databricks Runtime 11.1 (unsupported) and Databricks Runtime 11.1 for Machine Learning (unsupported).
July 5-11, 2022: Version 3.75
The built-in support for charts and visualizations in notebooks now supports multiple charts and data profiles. When you run a SQL cell in a notebook or use the
display function on a DataFrame, the notebook shows tabbed results, similar to the results display in Databricks SQL. You can now create the same rich visualizations in a notebook that you can create in Databricks SQL. For details, see Visualizations in Databricks notebooks.