Google
Cloud Platform
Amazon Web Services
Microsoft AzureDecember 2024
These features and Databricks platform improvements were released in December 2024.
Note
Releases are staged. Your Databricks account might not be updated until a week or more after the initial release date.
Databricks Runtime 16.1 is GA
December 20, 2024
Databricks Runtime 16.1 and Databricks Runtime 16.1 ML are now generally available.
See Databricks Runtime 16.1 and Databricks Runtime 16.1 for Machine Learning.
The default format for new notebooks is now IPYNB (Jupyter) format
December 20, 2024
The default format for new notebooks you create in your Databricks workspace is now IPYNB (.ipynb). Previously, the default format for notebooks was Source (.py, .sql, .scala, .r). To change the default format, use the Default file format for notebooks setting in the Developer pane of your workspace user settings. See Notebook formats.
Attribute serverless usage with budget policies (Public Preview)
December 19, 2024
To assist with serverless billing attribution, workspace admins can now create and assign budget policies to users, groups, or service principals. Budget policies enforce custom tags on all serverless usage incurred by the policy assignee, allowing granular billing attribution for serverless usage in notebooks, jobs, and pipelines.
For more information, see Attribute serverless usage with budget policies.
databricks-agents SDK 0.13.0 release
December 18, 2024
Version 0.13.0 of the databricks-agents SDK has been released to PyPI, containing the following changes:
Honor the current active Databricks CLI profile and MLflow model registry URI when calling
agents.deploy()and otherdatabricks.agentsAPIs. In particular, you can now specify a combination ofDATABRICKS_CONFIG_PROFILE=my-profileandMLFLOW_REGISTRY_URI=databricks-uc://my-profilebefore callingagents.deploy()to specify the Databricks CLI profile to use to deploy and access agents.In
mlflow.evaluate(), only run retrieval and guidelines metrics if retrieval and guideline context are present, respectively.Add secret-based authentication to clients for
mlflow.evaluate().
External groups are now labeled and immutable
December 18, 2024
External groups are groups that are created in Databricks from your identity provider. These groups are created using a SCIM provisioning connector and stay in sync with your identity provider. External groups are now explicitly labeled as External and can no longer be updated from the Databricks account console or workspace admin settings page by default. To update external group membership from the Databricks UI, an account admin can disable Immutable external groups in the account console preview page.
Model Serving now supports provisioned throughput and GPU serving
December 18, 2024
Model Serving now supports provisioned throughput and GPU serving.
Serverless compute for Delta Live Tables pipelines is in public preview
December 18, 2024
With serverless DLT pipelines, you can run your Delta Live Tables pipelines without configuring and deploying infrastructure. Databricks efficiently manages the compute resources that run your pipeline, including optimizing and scaling compute for your data processing workloads. See Configure a serverless Delta Live Tables pipeline.
Assign compute resources to groups (Public Preview)
December 17, 2024
The new Dedicated access mode (previously Single user) allows you to assign a dedicated all-purpose compute to a group or single user. See Assign compute resources to a group.
This Public Preview also gives your workspace access to the new simplified compute UI. See Use the simple form to manage compute.
A workspace admin must enable this preview. See Manage Databricks Previews.
Version 2.2 of the Jobs API is released
December 16, 2024
The Jobs API version is updated from 2.1 to 2.2. Updates in version 2.2 of the Jobs API include default queueing of new or updated jobs, and enhancing paging of job and job run responses that include fields with large numbers of values. To learn more about the updates in this version, see Updating from Jobs API 2.1 to 2.2. To see the full Jobs API 2.2 documentation, see Jobs (latest). Although Databricks recommends using version 2.2 of the Jobs API, you can still access versions 2.1 and 2.0. See Jobs (2.1) and Jobs API 2.0.
Unity Catalog MANAGE privilege (Public Preview)
December 14, 2024
You can now grant users the MANAGE privilege on Unity Catalog securable objects. The MANAGE privilege allows users to perform key actions on a Unity Catalog object, including:
Managing privileges
Dropping the object
Renaming the object
Transferring ownership
See MANAGE.
View streaming workload metrics for your job runs (Public Preview)
December 12, 2024
When you view job runs in the Databricks Jobs UI, you can now view metrics such as backlog seconds, backlog bytes, backlog records, and backlog files for sources supported by Spark Structured Streaming, including Apache Kafka, Amazon Kinesis, and Auto Loader. See View metrics for streaming tasks.
View streaming workload metrics for your Delta Live Tables pipeline updates (Public Preview)
December 12, 2024
When you view pipeline updates in the Delta Live Tables UI, you can now view metrics such as backlog seconds, backlog bytes, backlog records, and backlog files for each streaming flow in the pipeline. Streaming metrics are supported for Spark Structured Streaming sources, including Apache Kafka, Amazon Kinesis, and Auto Loader. See View streaming metrics.
Lakehouse Federation supports Oracle (Public Preview)
December 12, 2024
You can now run federated queries on data managed by Oracle. See Run federated queries on Oracle.
Databricks Runtime 16.1 (Beta)
December 11, 2024
Databricks Runtime 16.1 and Databricks Runtime 16.1 ML are now available as Beta releases.
See Databricks Runtime 16.1 and Databricks Runtime 16.1 for Machine Learning
Monitor and revoke personal access tokens in your account (Public Preview)
December 11, 2024
Account admins can now view a token report to monitor and revoke personal access tokens (PATs) in the account console. Databricks recommends you use OAuth access tokens instead of PATs for greater security and convenience. To join this preview, contact your Databricks account team. See Monitor and revoke personal access tokens in the account.
Unity Catalog can federate to Hive metastores
December 11, 2024
You can now use Unity Catalog to access and govern data that is registered in a Hive metastore. This includes both externally-managed Hive metastores and legacy internal Databricks Hive Hive metastores.
See Hive metastore federation: enable Unity Catalog to govern tables registered in a Hive metastore.
Remove metastore-level storage to enforce catalog-level storage isolation
December 11, 2024
If you have metastore-level storage for managed tables and volumes (also known as the metastore storage root), but you want to enforce data storage isolation at the catalog or schema level, you can now remove that metastore-level storage without interrupting existing workloads. See Remove metastore-level storage.
bamboolib is now deprecated
December 10, 2024
bamboolib is now deprecated. Users can still access bamboolib to perform low-code data analysis within notebooks, but Databricks is no longer actively developing nor supporting this tool. For assistance with code generation, use the Databricks Assistant.
Improve Databricks-to-Databricks Delta Sharing table read performance with history sharing (Public Preview)
December 5, 2024
Improve performance for Databricks-to-Databricks table shares by enabling history sharing. See Improve table read performance with history sharing.
Maximum personal access token lifetime now 730 days (two years)
December 5, 2024
The default maximum lifetime for newly created Databricks-issued personal access tokens is now set to 730 days (two years). Previously, personal access tokens could be created with no expiration by default. With this update, users cannot generate new tokens with a lifetime exceeding 730 days, and tokens created without a specified lifetime are set to a 730-day duration. If you configured the maximum token lifetime for your workspace to less than 730 days, the configuration remains unchanged. See Monitor and revoke personal access tokens and Databricks personal access token authentication.
Mosaic AI Model Training - serverless forecasting (Public Preview)
December 5, 2024
Mosaic AI Model Training - forecasting improves upon the existing AutoML forecasting experience with managed serverless compute, Unity Catalog support, access to deep learning algorithms, and an upgraded interface. See Forecasting (serverless) with AutoML.