July 2021

These features and Databricks platform improvements were released in July 2021.

Note

Releases are staged. Your Databricks account may not be updated until a week or more after the initial release date.

Manage MLflow experiment permissions with the Databricks REST API

July 29 - Aug 2, 2021: Version 3.51

You can now use the Databricks REST APIs’ Permissions API to manage which users can read, edit, or manage MLflow experiments. See Permissions API.

Restrict access to Databricks using IP access lists

July 29 - Aug 2, 2021: Version 3.51

You can now configure Databricks workspaces so that users and API clients connect to the service only through authorized IP addresses, such as existing corporate networks with a secure perimeter. This feature requires the Premium plan. See Configure IP access lists for workspaces.

New workspaces automatically enable the required Google APIs on your project

July 27, 2021

When you create a new workspace, Databricks automatically enables the required Google APIs on the project associated with the workspace. See Enabling Google APIs on a workspace’s project.

Feature freshness information available in Databricks Feature Store UI (Public Preview)

July 26 - August 2, 2021: Version 3.51

The Databricks Feature Store UI now shows the last time that data was written to a feature table. It also shows whether a feature table is updated by a scheduled job, and if so, the schedule and status of the job.

Display up to 10,000 result rows

July 26 - August 2, 2021: Version 3.51

Previously a maximum of 1000 rows was displayed in a notebook. Now, if a table returns more than 1000 rows, you can choose to re-execute the query and display up to 10,000 rows. Requires Databricks Runtime 8.4 or above. For more information, see Download results.

Bulk import and export notebooks in a folder as source files

July 26 - August 2, 2021: Version 3.51

You can now export the source for all notebooks in a workspace folder as a ZIP archive. You can import the ZIP archive into another workspace or another folder in the same workspace to recreate all the exported notebooks there. A notebook’s source file is in the notebook’s default language. For example, a Python notebook’s source is a Python file.

Notebook source files do not include notebook command results and are appropriate for using in a CI/CD pipeline. Previously, you could export the source for only one notebook at a time, from within the notebook editor, or you could export all notebooks in a folder as HTML files or as a Databricks archive (with the extension .dbc), which would include notebook command results unless the notebook was cleared before exporting.

For more information about importing and exporting notebooks, see Manage notebooks.

Autocomplete in SQL notebooks now uses all-caps for SQL keywords

July 26 - August 2, 2021: Version 3.51

When you use TAB to autocomplete a SQL keyword in a notebook, the autocompleted result is now capitalized. For example, SELECT is shown, rather than select. This will not effect the query since SQL keywords are case-insensitive.

Reorderable and resizable widgets in notebooks

July 26 - August 2, 2021: Version 3.51

You can now change the order and sizes of Databricks widgets in a notebook by clicking edit icon in the notebook’s widgets bar. See Configure widget settings.

Databricks UI usability fixes

July 26 - August 2, 2021: Version 3.51

This release resolves an issue that prevented using the ESC key to close some dialogs in the Databricks UI, including:

  • The View JSON dialog in the jobs UI.

  • The Edit Schedule dialog in the jobs UI.

  • The Single Logout Configurations dialog in the admin UI.

  • The Create Notebook dialog.

Store secrets in environment variables

July 22, 2021

You can specify a secret path in an environment variable and use it in a cluster-scoped init script. See Use a secret in a Spark configuration property or environment variable.

Databricks Runtime 8.4 and 8.4 ML are GA; 8.4 Photon is Public Preview

July 20, 2021

Databricks Runtime 8.4 and 8.4 ML are now generally available. 8.4 Photon is in Public Preview.

For information, see the full release notes at Databricks Runtime 8.4 (EoS) and Databricks Runtime 8.4 for ML (EoS).

High CPU instance types no longer have local SSDs

July 12-19, 2021: Version 3.50

High CPU instance types no longer have local SSDs, which also means that you cannot enable disk cache. For the latest list of instance types and their configuration, see the Google Cloud pricing estimator.

Enhanced data processing and analysis with Databricks jobs (Public Preview)

July 12-19, 2021: Version 3.50

Databricks is pleased to announce an update to Databricks jobs that allows you to orchestrate and run a job with multiple tasks. You can use this to define a directed acyclic graph (DAG) of task dependencies, making it easier to build and monitor reliable data and machine learning workflows. This feature is in public preview and is disabled by default.

Important

Once you enable this feature, you cannot disable it.

See Schedule and orchestrate workflows for details.

Changes to Compute page

July 12-19, 2021: Version 3.50

The Libraries, Job Run, Spark UI, and Logs links from the Compute page have been removed. To see information about libraries, or to view the Spark UI and logs, use the corresponding tabs on the cluster detail page. To see information about a job run, click Go To Job Run on that page.

Repos API (Public Preview)

July 6, 2021

You can now programmatically update a Databricks repo to the latest version of a specific Git branch or to a tag using the new Repos API. For details, see Repos API.

Databricks Runtime 8.4 (Beta)

July 1, 2021

Databricks Runtime 8.4 and Databricks Runtime 8.4 ML are now available as Beta releases.

For information, see the full release notes at Databricks Runtime 8.4 (EoS) and Databricks Runtime 8.4 for ML (EoS).

Improved security for cluster connectivity (Public Preview)

July 1, 2021: Version 3.49

With secure cluster connectivity (Public Preview), customer VPCs in the compute plane have no open ports and Databricks Runtime cluster nodes have no public IP addresses. Databricks secure cluster connectivity on Google Cloud is implemented by two features: no public IP addresses on cluster nodes, enabled by default, and the new secure cluster connectivity relay. See What is secure cluster connectivity?.

For workspaces in all regions except us-west4, clusters created or restarted after July 1, 2021 automatically use the new secure cluster connectivity relay. Clusters that are already running on that day do not use the secure cluster connectivity relay until the cluster terminates or restarts. For workspaces in the region us-west4, secure cluster connectivity is not enabled. Contact your Databricks account team for any questions.

New Google Cloud regions

July 1, 2021

Databricks is now available in the regions us-west4, europe-west1, and australia-southeast1. See Databricks clouds and regions.