February 2024

These features and Databricks platform improvements were released in February 2024.

Note

Releases are staged. Your Databricks account might not be updated until a week or more after the initial release date.

New region: asia-south1 (Mumbai)

February 29, 2024

Databricks is now available in GCP region asia-south1, Mumbai, India. See Databricks clouds and regions.

Use Delta Live Tables in Feature Engineering (Public Preview)

February 29, 2024

Tables that are published to Unity Catalog from Delta Live Tables pipelines and that include a primary key can now be used as feature tables for model training or inference. Unity Catalog provides feature discovery, governance, lineage, and cross-workspace access. For more information, see Create a feature table in Unity Catalog.

Restrict creating a personal access token for a service principal

February 29, 2024

Workspace admins can create personal access tokens on behalf of any service principal in their workspace. Account admins can now enforce that workspace admins have the Service Principal User role on a service principal in order to create a personal access token on behalf of the service principal. You can automate enabling this setting after workspace creation. See Restrict workspace admins.

Restrict changing a job owner and the run as setting

February 29, 2024

Workspace admins can change a job owner to any user or service principal in their workspace by default and the job run as setting to any user in their workspace or service principals that they have the Service Principal User role on.

Account admins can now restrict workspace admins to only change a job owner to themselves and the job run as setting to service principals that they have the Service Principal User role on or to themselves. You can automate enabling this setting after workspace creation. See Restrict workspace admins.

Support for Cloudflare R2 storage to avoid cross-region egress fees (Public Preview)

February 26, 2024

You can now use Cloudflare R2 as cloud storage for data registered in Unity Catalog. Cloudflare R2 is intended primarily for Delta Sharing use cases in which you want to avoid the data egress fees charged by cloud providers when data crosses regions. R2 storage supports all of the Databricks data and AI assets supported in AWS S3, Azure Data Lake Storage Gen2, and Google Cloud Storage. Support for R2 requires Databricks Runtime 14.3 or SQL Warehouse 2024.15. See Use Cloudflare R2 replicas or migrate storage to R2 and Create a storage credential for connecting to Cloudflare R2.

Add data UI supports XML file format

February 23, 2024

The add data UI now supports XML file upload and ingestion from cloud object storage. See Upload files to Databricks.

Unity Catalog volumes are GA

February 22, 2024

Unity Catalog volumes are now generally available in Databricks Runtime 14.3 LTS and above. Databricks recommends using volumes for managing read and write access to non-tabular data in cloud object storage. See What are Unity Catalog volumes?.

Full-page AI-powered search

February 16-22, 2024

Databricks search now leverages DatabricksIQ to provide a more intelligent AI-powered search experience. The new full-page search experience gives you more space to see results, more metadata for your objects, and more filters to narrow down your results.

See Search for workspace objects.

UI experience for OAuth app registration

February 14, 2024

You can now enable custom OAuth applications in the Databricks UI. You can also disable and modify both published and custom OAuth applications in the UI. For more information, see Enable custom OAuth applications using the Databricks UI.

File arrival triggers in Databricks Jobs is GA

February 7, 2024

File arrival triggers is now generally available in all cloud providers. With this release, you can use file arrival triggers to run a Databricks job when new files arrive in a Unity Catalog volume in addition to the existing support for Unity Catalog external locations. See Trigger jobs when new files arrive.

Databricks Git server proxy is GA

February 6, 2024

You can use Databricks Git server proxy to configure secure access from Databricks Repos to your on-premises Git server. See Set up private Git connectivity for Databricks Git folders (Repos).

Databricks Git server proxy no longer requires CAN_ATTACH_TO permissions

February 6, 2024

The GA release of Databricks Git server proxy has removed the need to provide all users with CAN_ATTACH_TO permissions.

If you previously configured Databricks Git server proxy with CAN_ATTACH_TO privileges, see Set up private Git connectivity for Databricks Git folders (Repos).

Workspace file support for the dbt and SQL file tasks is GA

February 1, 2024

Support for using dbt projects and SQL files stored in Databricks workspace folders is now generally available, adding to the existing support for using files stored in a remote Git repository. See dbt task for jobs and SQL task for jobs.

Databricks Connect is GA for Scala

February 1, 2024

Beginning with Databricks Runtime 14.3 LTS, Databricks Connect is now generally available for Scala.

Create tables from files in volumes

February 1, 2024

You can now use the UI to create Unity Catalog managed tables from data files stored in Unity Catalog volumes. See Create a table from data in a volume.

Databricks Runtime 14.3 LTS is GA

February 1, 2024

Databricks Runtime 14.3 LTS and Databricks Runtime 14.3 LTS ML are now generally available.

See Databricks Runtime 14.3 LTS and Databricks Runtime 14.3 LTS for Machine Learning.

Delta Sharing supports tables that use deletion vectors (Public Preview)

February 1, 2024

Delta Sharing now supports the sharing of tables that use deletion vectors, which are a storage optimization feature that you can enable on Delta tables. Recipients can read tables that use deletion vectors using a SQL warehouse, a cluster running Databricks Runtime 14.1 or above, or compute that is running open source delta-sharing-spark 3.1 or above. Support is rolling out gradually over the coming month.

See Add tables with deletion vectors or column mapping to a share, Read tables with deletion vectors or column mapping enabled, and Read tables with deletion vectors or column mapping enabled.