These features and Databricks platform improvements were released in November 2023.
Releases are staged. Your Databricks workspace might not be updated until a week or more after the initial release date.
November 29, 2023
The following settings are enabled on all Premium plan workspaces and can no longer be disabled from the workspace admin settings page or the Workspace Conf API:
Workspace access control: enables users to configure permissions to workspace objects.
Cluster, pool, jobs access control: enables users to configure permissions to clusters, pools, and jobs.
Cluster visibility control: prevents clusters from being visible to users that have no permissions on them.
Jobs visibility control: prevents jobs being from visible to users that have no permissions on them.
Workspace visibility control setting: prevents notebooks and folders from being from visible to users that have no permissions on them.
Access control settings and visibility control settings were previously enabled by default on all new workspaces.
For more information, see Access control overview.
November 28, 2023
AI assistive features are now enabled by default. These include:
Admins can opt out of AI assistive features for all workspaces in their account or a subset of workspaces. See Enable or disable Databricks Assistant.
November 22, 2023
You can now create a new volume in the target schema if none exists from the Upload to volume UI. See Upload files to a Unity Catalog volume.
Actions such as Delete file and Download file have been added to Catalog Explorer for files in volumes.
November 22, 2023
Databricks Runtime 14.2 and Databricks Runtime 14.2 ML are now generally available.
November 17, 2023
You can now run
pip install databricks-sql-connector==3.0.0.
Removes support for Python 3.7
Adds support for native parameterized SQL queries. Requires DBR 14.2 and above. For more information, see Using Native Parameters in the databricks-sql-python GitHub repo.
Includes our completely rewritten SQLAlchemy dialect:
Adds support for SQLAlchemy >= 2.0 and drops support for SQLAlchemy 1.x
Full end-to-end test coverage of all supported features
Detailed usage notes available here
Adds support for:
Numerictype scale and precision, like
Reading and writing
Reading and writing composite keys
Reading and writing from views
Identityto tables (autoincrementing primary keys)
OFFSETfor paging through results
Caching metadata calls
Enables Cloud Fetch by default. To disable, set
Add integration tests for Databricks Unity Catalog volumes ingestion queries
_enable_v3_retries=Trueand warn if users override it.
Security: Bumps minimum pyarrow version to 14.0.1 (CVE-2023-47248)
November 16, 2023
You can now enable deletion vectors on all new Delta tables in Databricks Runtime 14.0 and above with the workspace admin setting Auto-Enable Deletion Vectors. This behavior is currently opt-in but will change to opt-out in the future. Databricks recommends manually configuring an option for this setting. See Auto-enable deletion vectors.
November 15, 2023
Workspace libraries are now deprecated and creating new workspace libraries has been disabled. Existing workspace libraries continue to function. Databricks recommends adding libraries directly to clusters or jobs. See Libraries.
November 13, 2023
You can now find data sets, AI models, and Databricks Solution Accelerators in Databricks Marketplace using the global search bar in the Databricks workspace UI. See What is Databricks Marketplace? and Search for workspace objects.
November 10, 2023
Consumers without a Unity Catalog-enabled Databricks workspace can now access data products in Databricks Marketplace. You can use Delta Sharing open sharing connectors to access Marketplace data using a number of common platforms, including Microsoft Power BI, Microsoft Excel, pandas, Apache Spark, and non-Unity Catalog Databricks workspaces. Only tabular data sets are available on external platforms (not Databricks notebooks or models, for example). See Request and access data products in Databricks Marketplace using external platforms.
November 8, 2023
Databricks Runtime 14.2 and Databricks Runtime 14.2 ML are now available as beta releases.
November 7, 2023
Databricks Marketplace now gives you access to Databricks Solution Accelerators, purpose-built guides that provide fully functional notebooks, best practices guidance, and sample data to get you set up quickly with many popular use cases on Databricks. See Get access to Databricks Solution Accelerators.
November 3, 2023
You can now configure Lakehouse Federation to run federated queries with the
BigQuery connector type. The new BigQuery connector is supported on Pro and Serverless SQL warehouses and Databricks Runtime 13.3 LTS and above. See Run federated queries on Google BigQuery.