Databricks platform release notes
- November 2024
- Expanded support for Python debugger in notebooks
- Query history and query profile support for Delta Live Tables
- Reuse SQL outputs as
_sqldf
in subsequent SQL cells - Warehouses system table is now available (Public Preview)
- Cross-platform view sharing (Public Preview)
- Databricks Runtime 16.0 is GA
- Breaking change: Hosted RStudio is end-of-life
- Updated and unified read-only code blocks
- Increased Quick Fix recommendations
- Databricks Runtime 14.2 series support ends
- Serverless compute for workflows (Public Preview)
- Serverless compute for notebooks (Public Preview)
- October 2024
- Databricks Assistant helps optimize SQL queries
- Mosaic AI Model Serving is now available on GCP (Public Preview)
- Databricks Runtime 15.1 series support ends
- Enable a metastore to be assigned automatically to new workspaces
- Compute system tables now available in all GCP regions
- Databricks Assistant Autocomplete is now generally available
- Filters on notebook results tables now persist
- Improvements to comments in notebooks
- Databricks Runtime 16.0 (Beta)
- More reliable reloading of modified Python modules with improvements to
autoreload
- September 2024
- Publish to Power BI is now generally available
- Use
@
to reference tables in Databricks Assistant prompts - Control external access to data in Unity Catalog using the new
EXTERNAL USE SCHEMA
privilege - Databricks Assistant Quick Fix debugs code inline
- Warehouse events system table is now available (Public Preview)
- Extended AI-generated comments support
- Databricks extension for Visual Studio Code is GA
- System tables are now generally available
- August 2024
- Monitor Unity Catalog object usage against quotas using the new Resource Quotas APIs
- Compute system tables are now available (Public Preview)
- AskSupport replaced by Databricks assistant
- Lakeflow system tables are extend with additional columns
- Python code formatting error highlights
- Delta Sharing: More Delta Lake features now supported by the Python and Power BI connectors
- Delta Sharing adds support for TimestampNTZ
- The Databricks Jobs
For each
task is GA - Databricks Runtime 15.4 LTS is GA
- Personalized notebook autocomplete
- Configure your workspace’s default access mode for jobs compute
- New slash commands for Databricks Assistant
- Workspace search now supports volumes
- Databricks JDBC driver 2.6.40
- Databricks personal access tokens revoked if unused after 90 days
- Wrap lines in notebook cells
- Create budgets to monitor account spending (Public Preview)
- Files can no longer have identical names in workspace folders
- Compute policy enforcement now available
- Format columns in notebook and query results tables
- Row filters and column masks are now GA, with improvements
- Lakehouse Federation is generally available (GA)
- July 2024
- Increased limit for simultaneous tasks
- Embed and drag & drop images in notebooks
- Command palette available in notebooks
- Workflow system schema renamed to
lakeflow
- Support for Cloudflare R2 storage is GA
- Monitor Databricks Assistant activities with system tables (Public Preview)
- Sharing schemas using Delta Sharing is now GA
- Databricks Assistant can diagnose issues with jobs (Public Preview)
- Updates to Databricks Git folders authentication and sharing behaviors
- Cluster library installation timeout
- Compute plane outbound IP addresses must be added to a workspace IP allow list
- Databricks Runtime 9.1 series support extended
- Single sign-on (SSO) is supported in Lakehouse Federation for SQL Server
- Enable cross-Geo processing
- Model sharing using Delta Sharing is now generally available
- Share comments and primary key constraints using Delta Sharing
- New Databricks JDBC Driver (OSS)
- Databricks Runtime 15.4 LTS (Beta)
- Scala is GA on Unity Catalog shared compute
- Notebooks: toggle more visible cell titles
/
in workspace asset names is deprecated- Delta Sharing lets you share tables that use liquid clustering
- Query history system table is now available (Public Preview)
- Partition metadata logging for Unity Catalog external tables
- Filter data outputs using natural language prompts
- Lakehouse Federation supports Salesforce Data Cloud (Public Preview)
- Databricks Assistant system table now available (Public Preview)
- Account SCIM API v2.1 (Public Preview)
- Resource quota increase for tables per Unity Catalog metastore
- Databricks Assistant can diagnose notebook errors automatically
- June 2024
- Notebooks - out of order syntax highlighting
- Notebooks Assistant automatically diagnoses errors
- Catalog Explorer improvements
- Databricks Assistant is now GA
- Databricks Assistant now gives titles to threads
- Serverless notebooks environment manager
- New intermediate DNS name for Private Service Connect
- Notebooks debugging console
- Quickly add a periodic schedule to a Databricks workflow
- Databricks Runtime 15.3 is GA
- Databricks Geos is GA
- New improved catalog creation UI
- The billing system tables are enabled by default
- Volume sharing using Delta Sharing is now generally available
- Column mapping is now GA
- Workflow system tables are now available (Public Preview)
- Databricks Assistant: Threads & queries experience enabled by default
- New entry points for in-product help
- Improved notebook Markdown editor
- Quickly access Catalog Explorer tables from notebooks
- Databricks ODBC driver 2.8.2
- The new Databricks Notebook UI is now generally available
- May 2024
- Databricks Runtime 15.0 series support ends
- Databricks Runtime 15.3 (Beta)
- The compute metrics UI is now available on all Databricks Runtime versions
- Improved search and filtering in notebook and SQL editor results tables
- New dashboard helps Databricks Marketplace providers monitor listing usage
- Notebooks on SQL warehouses are now GA
- View system-generated federated queries in
Query Profile
- Compute plane outbound IP addresses must be added to a workspace IP allow list
- OAuth is supported in Lakehouse Federation for Snowflake
- Bulk move and delete workspace objects from the workspace browser
- Unity Catalog objects are available in recents and favorites
- New dbt-databricks connector 1.8.0 adopts decoupled dbt architecture
- New compliance and security settings APIs (Public Preview)
- Databricks Runtime 15.2 is GA
- New Tableau connector for Delta Sharing
- New deep learning recommendation model examples
- Bind storage credentials and external locations to specific workspaces (Public Preview)
- Git folders are GA
- Databricks Assistant autocomplete (Public Preview)
- New changes to Git folder UI
- Allow users to copy data to the clipboard from results table
- Attribute tag values for Unity Catalog objects can now be 1000 characters long (Public Preview)
- New Previews page
- Compute cloning now clones any libraries installed on the original compute resource
- Hive metastore table access controls are deprecated
- Databricks JDBC driver 2.6.38
- Databricks Runtime 15.2 (Beta)
- Notebooks now detect and auto-complete column names for Spark Connect DataFrames
- April 2024
- Databricks Runtime 15.1 is GA
- Databricks Assistant: Threads & history
- Data lineage now captures reads on tables with column masks and row-level security
- Notebooks now automatically detect SQL
- New columns added to the billable usage system table (Public Preview)
- Delta Sharing supports tables that use column mapping (Public Preview)
- Creation and installation of workspace libraries is no longer available
- Jobs created through the UI are now queued by default
- Lakehouse Federation supports foreign tables with case-sensitive identifiers
- Serverless SQL warehouses (Public Preview)
- Delta Live Tables notebook developer experience improvements (Public Preview)
- March 2024
- MLflow now enforces quota limits for experiments and runs
- The Jobs UI is updated to better manage jobs deployed by Databricks Asset Bundles
- Interactive notebook debugging
- Self-service sign-up for private exchange providers in Marketplace
- Databricks Runtime 15.0 is GA
- Databricks Repos changed to Git folders
- Databricks Runtime 14.1 and 14.2 series support extended
- Databricks ODBC driver 2.8.0
- Delegate the ability to view an object’s metadata in Unity Catalog (Public Preview)
- System tables now include Marketplace tables (Public Preview)
- System tables now available (Public Preview)
- Databricks Runtime 15.0 (Beta)
- Databricks Runtime 14.0 series support ends
- New computation for sys.path and CWD in Repos
- Automatic enablement of Unity Catalog for new workspaces
- February 2024
- New region: asia-south1 (Mumbai)
- Use Delta Live Tables in Feature Engineering (Public Preview)
- Restrict creating a personal access token for a service principal
- Restrict changing a job owner and the run as setting
- Support for Cloudflare R2 storage to avoid cross-region egress fees (Public Preview)
- Add data UI supports XML file format
- Unity Catalog volumes are GA
- Full-page AI-powered search
- UI experience for OAuth app registration
- File arrival triggers in Databricks Jobs is GA
- Search for machine learning models in Unity Catalog using global workspace search
- Databricks Git server proxy is GA
- Databricks Git server proxy no longer requires CAN_ATTACH_TO permissions
- Workspace file support for the dbt and SQL file tasks is GA
- Databricks Connect is GA for Scala
- Create tables from files in volumes
- Databricks Runtime 14.3 LTS is GA
- Delta Sharing supports tables that use deletion vectors (Public Preview)
- January 2024
- Native XML file format support (Public Preview)
- Share AI models using Databricks Marketplace (Public Preview)
- Workspace path update
- Streamlined creation of Databricks jobs
- URI path-based access to Unity Catalog external volumes
- Access controls lists can be enabled on upgraded workspaces
- Updated UI for notebook cells (Public Preview)
- Quick Fix help with syntax errors in the notebook
- Databricks Runtime 14.3 LTS (Beta)
- Share AI models using Delta Sharing (Public Preview)
- Databricks Marketplace supports volume sharing
- Create widgets from the Databricks UI
- Libraries now supported in compute policies (Public Preview)
- Workspace file size limit is now 500MB
- Feature removal notice for legacy Git integration in Databricks
- Databricks ODBC driver 2.7.7
- Databricks Runtime 13.2 series support ends
- December 2023
- Share dynamic views using Delta Sharing (Public Preview)
- Share volumes using Delta Sharing (Public Preview)
- Entity Relationship Diagram for primary keys and foreign keys
- Compute created in the UI now uses the “Auto” availability zone by default
- Unity Catalog volume file upload size limit increase
- New notebook cell results rendering available (Public Preview)
- Notebook editor themes available
- New unified admin settings UI
- Init scripts on DBFS are end-of-life
- November 2023
- Access controls lists can no longer be disabled
- AI assistive features are enabled by default
- New behaviors and actions in Catalog Explorer for volumes
- Databricks Runtime 14.2 is GA
- Databricks SQL Connector for Python version 3.0.0
- Auto-enable deletion vectors
- Libraries in workspace files supported on no-isolation shared clusters
- Deprecation of workspace libraries
- Search for Databricks Marketplace listings using global workspace search
- Consume data products in Databricks Marketplace using external platforms
- Databricks Runtime 14.2 (beta)
- Databricks Marketplace includes Databricks Solution Accelerators
- Lakehouse Federation adds support for Google BigQuery
- October 2023
- Configure the availability zone of instance pools
- View the YAML source for a Databricks job
- Add conditional logic to your Databricks workflows
- Configure parameters on a Databricks job that can be referenced by all job tasks
- Init scripts are now supported on Unity Catalog volumes
- Unity Catalog support for
UNDROP TABLE
is GA - Partner Connect supports Dataiku
- AutoML generated notebooks are now saved as MLflow artifacts
- Unity Catalog support for Python and PySpark UDFs
- Feature Engineering in Unity Catalog is GA
- AI-generated table comments (Public Preview)
- Models in Unity Catalog is GA
- Partner Connect supports Monte Carlo
- Semantic search (Public Preview)
- Enable Databricks Assistant at the workspace level
- IP access lists for the account console is GA
- New Photon defaults
- Databricks Runtime 14.1 is GA
- Developer tools release notes have moved
- Databricks extension for Visual Studio Code updated to version 1.1.5
- Predictive I/O for updates is GA
- Deletion vectors are GA
- Customer-managed keys are generally available
- Personal Compute cluster policy is available by default
- Partner Connect supports RudderStack
- Databricks CLI updated to version 0.207.0 (Public Preview)
- Capture data lineage using Unity Catalog
- Run selected cells in a notebook
- Use workspace-catalog binding to give read-only access to a catalog
- New in-product Help experience (Public Preview)
- Databricks extension for Visual Studio Code updated to version 1.1.4
- Databricks SDK for Python updated to version 0.10.0 (Beta)
- Databricks SDK for Go updated to version 0.22.0 (Beta)
- September 2023
- Lakehouse Federation available on single user clusters
- Delegate the ability to create a storage credential in Unity Catalog
- Databricks Runtime 14.1 (Beta)
- Prevent runs of a job from being skipped when you reach concurrency limits
- Databricks Terraform provider updated to version 1.27.0
- Databricks CLI updated to version 0.206.0 (Public Preview)
- Databricks SDK for Go updated to version 0.21.0 (Beta)
- Databricks ODBC driver 2.7.5
- Databricks extension for Visual Studio Code updated to version 1.1.3
- Running jobs as a service principal is GA
- Databricks CLI updated to version 0.205.2 (Public Preview)
- Databricks CLI updated to version 0.205.1 (Public Preview)
- Databricks SDK for Go updated to version 0.20.0 (Beta)
- Databricks SDK for Python updated to version 0.9.0 (Beta)
- Databricks SDK for Python updated to version 0.7.1 (Beta)
- Databricks Terraform provider updated to version 1.26.0
- Databricks Connect for Databricks Runtime 14.0
- Databricks Connect V2 is Public Preview for Scala
- Databricks Runtime 14.0 is GA
- Account email notifications upgrade
- Databricks Terraform provider updated to version 1.25.1
- Databricks SDK for Go updated to version 0.19.2 (Beta)
- Databricks CLI updated to version 0.205.0 (Public Preview)
- Reserved location for
/Workspace
- Databricks Asset Bundles (Public Preview)
- Databricks Terraform provider updated to version 1.25.0
- Databricks extension for Visual Studio Code updated to version 1.1.2
- Partner Connect supports Snowplow
- Databricks CLI updated to version 0.204.1 (Public Preview)
- Filter sensitive data with row filters and column masks (Public Preview)
- Databricks SDK for Go updated to version 0.19.1 (Beta)
- Databricks ODBC driver 2.7.3
- Databricks CLI updated to version 0.204.0 (Public Preview)
- Data Explorer is now Catalog Explorer
- Databricks SDK for Go updated to version 0.19.0 (Beta)
- Databricks SDK for Python updated to version 0.8.0 (Beta)
- GitHub apps integration in Repos is GA
- August 2023
- Tables now appear in navigational search
- Databricks CLI updated to version 0.203.3 (Public Preview)
- Databricks JDBC driver 2.6.34
- New Google Cloud region: northamerica-northeast1 (Montréal, Canada)
- Databricks SDK for Go updated to version 0.18.0 (Beta)
- Databricks SDK for Python updated to version 0.7.0 (Beta)
- Databricks Terraform provider updated to version 1.24.1
- Init scripts on DBFS end of life extended to Dec 1, 2023
- Databricks Runtime 14.0 (Beta)
- Unified navigation experience is GA
- Databricks Terraform provider updated to version 1.24.0
- Shared access mode clusters can now connect to public networks outside of Databricks VPC
- Databricks CLI updated to version 0.203.2 (Public Preview)
- Go to definition for functions and variables in Python notebooks
- Databricks Runtime 13.3 LTS is GA
- Introducing tags with Unity Catalog
- Allowlist for JARs and Maven coordinates on Unity Catalog shared clusters is in Public Preview
- Volumes support for JARs is in Public Preview
- Easier Databricks Repos .ipynb file output commits
- IPYNB notebook support in Databricks Repos is GA
- Databricks SDK for Go updated to version 0.17.0 (Beta)
- Databricks SDK for Python updated to version 0.6.0 (Beta)
- Databricks CLI updated to version 0.203.1 (Public Preview)
- Unified schema browser is now GA
- Databricks SDK for Go updated to version 0.16.0 (Beta)
- Partner Connect supports Census
- Databricks SDK for Python updated to version 0.5.0 (Beta)
- Programmatic write support for workspace files
- Databricks CLI updated to version 0.203.0 (Public Preview)
- Databricks Terraform provider updated to version 1.23.0
- Groups can now be renamed
- Databricks SDK for Go updated to version 0.15.0 (Beta)
- Databricks SDK for Python updated to version 0.4.0 (Beta)
- Third-party iFraming prevention configuration setting was removed
- Databricks extension for Visual Studio Code updated to version 1.1.1
- LangChain available in 13.1 and above
- Feature Engineering in Unity Catalog is Public Preview
- Improved error handling for repeated continuous job failures
- Share schemas using Delta Sharing (Public Preview)
- Run tasks conditionally in your Databricks jobs
- Delta Live Tables is now generally available
- Databricks Terraform provider updated to version 1.22.0
- Databricks Runtime 13.3 LTS (Beta)
- New Git operations are generally available: Merge branches, rebase and pull with conflict resolution
- July 2023
- Workspace admins can now create account groups
- Group manager role is in Public Preview
- Databricks CLI updated to version 0.202.0 (Public Preview)
- Databricks SDK for Python updated to version 0.3.0 (Beta)
- Databricks SDK for Go updated to version 0.14.1 (Beta)
- Databricks SDK for Go updated to version 0.14.0 (Beta)
- Run another job as a task in a Databricks job
- Enable customer-managed keys for managed services (Public Preview)
- Account API is generally available
- Private Service Connect launches to Limited Availability with GA-level functionality
- All users can access data products in Databricks Marketplace by default
- Classic keyboard shortcuts mode
- Lakehouse Federation lets you run queries against external database providers (Public Preview)
- Move to trash enabled for Repos
- Create alerts for slow-running or stuck jobs
- Databricks SDK for Go updated to version 0.13.0 (Beta)
- Databricks SDK for Python updated to version 0.2.0 (Beta)
- Databricks CLI updated to version 0.201.0 (Public Preview)
- Databricks SDK for Python updated to version 0.2.1 (Beta)
- Databricks Assistant is in Public Preview
- Deactivate users and service principals from your account
- Account-level SCIM provisioning now deactivates users when they are deactivated in the identity provider
- Trash directory admin access
- Prevention of MIME type sniffing and XSS attack page rendering are now always enabled
- Unity Catalog volumes are in Public Preview
- Simplified experience for submitting product feedback from the workspace
- Databricks extension for Visual Studio Code updated to version 1.1.0
- Functions now displayed in Catalog Explorer (Public Preview)
- Databricks Terraform provider updated to version 1.21.0
- The maximum offset for the
List all jobs
andList job runs
API requests is now limited - Databricks Runtime 13.2 is GA
- Delta Sharing and Databricks Marketplace support view sharing (Public Preview)
- Init scripts on DBFS reach end of life on Sept 1, 2023
- June 2023
- Databricks Terraform provider updated to version 1.20.0
- Databricks CLI updated to version 0.200.1 (Public Preview)
- Databricks SDK for Go updated to version 0.12.0 (Beta)
- Databricks SDK for Go updated to version 0.11.0 (Beta)
- Databricks SDK for Python (Beta)
- Databricks SDK for Go (Beta)
- Models in Unity Catalog (Public Preview)
- Databricks Marketplace is now GA
- Databricks Runtime 13.2 (Beta)
- Databricks extension for Visual Studio Code (General Availability)
- See a visual overview of completed job runs in the Databricks Jobs UI
- Databricks CLI (Public Preview)
- Improved pagination of results from
List all jobs
andList job runs
API requests - Full-page workspace browser includes Repos
- Databricks Terraform provider updated to version 1.19.0
- Configure the number of local SSDs in your cluster or instance pool
- Run jobs as a service principal (Public Preview)
- New service principal UI provides better management experience
- Home folders restored when users are re-added to workspaces
- Databricks Marketplace: Private exchanges are now available
- Databricks Marketplace: Consumers can uninstall data products using the UI
- Databricks Marketplace: providers can create their own profiles
- Databricks Connect V2 is GA for Python
- Databricks Terraform provider updated to version 1.18.0
- New Databricks Marketplace providers
- Databricks Runtime 13.1 is GA
- View data from notebooks, SQL editor, and Catalog Explorer with the unified schema browser (Public Preview)
- May 2023
- Create or modify table UI supports Avro, Parquet, and text file uploads
- Use the add data UI to load data using Unity Catalog external locations (Public Preview)
- Prevent Enter key from accepting autocomplete suggestion
- Databricks Terraform provider updated to version 1.17.0
- Upload data UI supports new column data types
- Bind Unity Catalog catalogs to specific workspaces
- All users can connect to Fivetran using Partner Connect (Public Preview)
- Databricks Terraform provider updated to version 1.16.1
- Databricks JDBC driver 2.6.33
- New default theme for editor
- Databricks Terraform provider updated to version 1.16.0
- Databricks Runtime 13.1 (Beta)
- Databricks Terraform provider updated to version 1.15.0
- Account nicknames now available in the account console
- Share notebooks using Delta Sharing
- Deprecation of cluster-scoped init scripts on DBFS
- Unified navigation (Public Preview)
- April 2023
- Databricks Marketplace (Public Preview): an open marketplace for data, analytics, and AI
- Customer-managed keys for workspace storage (Public Preview)
- Configure the Python formatter
- Workspace files are GA
- Cluster-scoped init scripts can now be stored in workspace files
- New Delta Sharing privileges enable delegation of share, recipient, and provider management tasks
- Databricks Runtime 13.0 is GA
- Compute resource usage for Google Cloud Marketplace aggregates under a single SKU
- Control access to the account console by IP address ranges (Public Preview)
- Databricks Terraform provider updated to version 1.14.3
- Audit log entries for changed admin settings for workspaces and accounts
- Databricks Terraform provider updated to versions 1.14.1 and 1.14.2
- Combined SQL user settings and general user settings
- Legacy notebook visualizations deprecated
- Create or modify table from file upload page supports JSON file uploads
- New API support for configuring Private Service Connect (Public Preview)
- March 2023
- Databricks Terraform provider updated to version 1.14.0
- Local storage autoscaling feature added to clusters
- Distributed training with TorchDistributor
- Databricks Runtime 13.0 (Beta)
- Improved file editor
- .ipynb (Jupyter) notebook support in Repos (preview)
- Support for reload4j
- Unity Catalog, Delta Sharing, and identity federation are GA on Google Cloud
- Execute SQL cells in the notebook in parallel
- Create job tasks using Python code stored in a Git repo
- Databricks Terraform provider updated to version 1.13.0
- Databricks Terraform provider updated to version 1.12.0
- SQL admin console and workspace admin console combined
- New Catalog Explorer availability
- Exact match search is available in global search
- Databricks Runtime 12.2 LTS and Databricks Runtime 12.2 LTS ML are GA
- Workspace files are now in Public Preview
- February 2023
- Updated required permissions for new workspaces
- Secure a workspace with private connectivity using Google Private Service Connect (Public Preview)
- Ray on Databricks (Public Preview)
- Notebook cell output results limit increased
- Databricks Jobs now supports running continuous jobs
- Databricks Terraform provider updated to version 1.10.0
- Improvements to the MLflow experiment UI
- Databricks Runtime 12.2 (Beta)
- Databricks extension for Visual Studio Code (Public Preview)
- Databricks Terraform provider updated to version 1.9.2
- Partner Connect is GA
- Variable explorer in Databricks notebooks
- January 2023
- Create a workspace using Terraform (Public Preview)
- Create a workspace using the Account API (Public Preview)
- Authenticate to Databricks REST APIs using a Google ID
- Account SCIM is now GA
- Easier creation and editing of Databricks jobs in the UI
- Improvements to the Databricks Jobs UI when viewing job runs
- REST API Reference is now available for browsing API documentation
- Databricks Terraform provider updated to version 1.9.1
- Account users can update email preferences in the account console
- New account console home screen provides better account management experience
- Region support consolidated onto one page
- Databricks Runtime 12.1 and Databricks Runtime 12.1 ML are GA
- Cluster policies now support limiting the max number of clusters per user
- Databricks Terraform provider updated to version 1.9.0
- Databricks Terraform provider updated to version 1.8.0
- Databricks Runtime 12.1 (Beta)
- Global workspace search
- Partner Connect supports Sigma
- New left and right sidebars in Databricks notebooks
- December 2022
- Databricks SQL Driver for Go is Generally Available
- Prevent concurrent workspace updates
- Databricks Terraform provider updated to version 1.7.0
- Databricks Runtime 12.0 and 12.0 ML are GA
- Billable usage graphs can now aggregate by individual tags
- Databricks ODBC driver 2.6.29
- Databricks JDBC driver 2.6.32
- November 2022
- Enhanced notifications for your Databricks jobs (Public Preview)
- Databricks Runtime 12.0 (Beta)
- Enforce user isolation cluster types on a workspace
- New Google Cloud region: asia-northeast1 (Tokyo)
- Upload data UI can now be disabled via admin settings
- Work with large repositories with Sparse Checkout
- Databricks Terraform provider updated to version 1.6.5
- Databricks Terraform provider updated to versions 1.6.3 and 1.6.4
- Create or modify table from file upload page now supports multiple files
- Create or modify table from file upload page now supports overwrite
- Search for jobs by name with the Jobs API 2.1
- Databricks Terraform provider updated to version 1.6.2
- Use a Google ID token to authenticate to workspace REST APIs (Public Preview)
- October 2022
- GA: Repos support for non-notebook files
- Deploy models for streaming inference with Delta Live Tables notebooks
- Connect to Fivetran from the add data UI
- Databricks SQL Driver for Node.js is Generally Available
- Partner Connect supports connecting to erwin Data Modeler by Quest
- Optimize performance and control costs by using different pools for the driver node and worker nodes
- Databricks Runtime 11.3 LTS and 11.3 LTS ML are GA
- Format Python code in notebooks (Public Preview)
- DBFS Rest API is available for all customers
- Add data UI provides a central UI for loading data to Databricks
- Create or modify table from file upload page unifies experience for small file upload to Delta Lake
- Partner Connect Public Preview
- Cluster UI preview and access mode replaces security mode
- Partner Connect supports connecting to Hevo Data
- Account users can access the account console
- Enable admin protection for Standard clusters (or High Concurrency clusters without table ACLs)
- SQL persona integrated with new search experience
- Service principals provide API-only access to Databricks resources (Public Preview)
- Top navigation bar in the UI
- Databricks Runtime 11.3 (Beta)
- The account console is available in multiple languages
- September 2022
- New reference solution for natural language processing
- Audit logs now include events for web terminal
- Audit logs now include events for managing credentials for Git repos
- Audit logs are GA
- Cluster and pool tags now propagate to GCE labels
- Select cluster policies directly in the Delta Live Tables UI
- New data trasformation card on workspace landing pages
- Delta cache is now disk cache
- View and organize assets in the workspace browser across personas
- Databricks Runtime 11.2 and 11.2 ML are GA
- Enhanced Spark egress network policy
- August 2022
- Databricks ODBC driver 2.6.26
- Databricks JDBC driver 2.6.29
- Databricks Feature Store client now available on PyPI
- Databricks Runtime 11.2 (Beta)
- Decreased storage costs by reducing default boot disk size
- Reduced message volume in the Delta Live Tables UI for continuous pipelines
- Easier cluster configuration for your Delta Live Tables pipelines
- Orchestrate dbt tasks in your Databricks workflows (Public Preview)
- Control workspace access with Customer Approved Workspace Login
- Deploy a workspace in a customer-managed VPC (GA)
- Databricks is now available in region
europe-west3
- Use generated columns when you create Delta Live Tables datasets
- Improved editing for notebooks with Monaco-based editor (Experimental)
- Compliance controls for HIPAA (GA)
- Databricks Runtime 10.3 series support ends
- Delta Live Tables now supports refreshing only selected tables in pipeline updates
- Job execution now waits for cluster libraries to finish installing
- July 2022
- Verbose audit logs now record when a notebook command or Databricks SQL query runs
- Databricks Runtime 11.1 and 11.1 ML are GA
- Photon is GA
- Notification upon notebook completion
- Increased limit for the number of jobs in your Databricks workspaces
- Databricks JDBC driver 2.6.27
- Databricks ODBC driver 2.6.25
- Databricks Runtime 11.1 (Beta)
- Improved notebook visualizations
- June 2022
- Databricks Runtime 6.4 Extended Support reaches end of support
- Databricks Runtime 10.2 series support ends
- Databricks ODBC driver 2.6.24
- Databricks Terraform provider is now GA
- HIPAA compliance controls (Public Preview)
- Databricks Runtime 11.0 and 11.0 ML are GA; 11.0 Photon is Public Preview
- Change to Repos default working directory in Databricks Runtime 11.0
- Databricks Runtime 10.1 series support ends
- Delta Live Tables now supports SCD type 2
- Create Delta Live Tables pipelines directly in the Databricks UI
- Select the Delta Live Tables channel when you create or edit a pipeline
- Communicate between tasks in your Databricks jobs with task values
- Enable account switching in the Databricks UI
- Default instance type is now
n2-highmem-4
- May 2022
- Copy and paste notebook cells between tabs and windows
- Databricks Runtime 11.0 (Beta)
- Explore SQL cell results in Python notebooks natively using Python
- Databricks Repos: Support for more files in a repo
- Databricks Repos: Fix to issue with MLflow experiment data loss
- Power BI Desktop system-wide HTTP proxy support
- Databricks Runtime 10.5 and 10.5 ML are GA; 10.5 Photon is Public Preview
- Databricks JDBC driver 2.6.25
- See the user a pipeline runs as in the Delta Live Tables UI
- April 2022
- Use tags to better manage your Databricks jobs
- Databricks Runtime 10.0 series support ends
- Get a visual overview of your job runs with the new jobs matrix view
- Save time and resources when your Databricks job runs are unsuccessful
- View the run history for job tasks
- Assign a new cluster in the jobs UI when the Single User access no longer exists
- Databricks Runtime 10.5 (Beta)
- New email provider for jobs alerts
- The Delta Live Tables UI is enhanced to disable unauthorized actions
- AutoML is generally available
- Delta Live Tables is GA on AWS and Azure, and in Public Preview on GCP
- Delta Live Tables SQL interface: non-breaking change to table names
- March 2022
- Better performance and cost for your Delta Live Tables pipelines with Databricks enhanced autoscaling
- Customer-managed VPC is Public Preview
- Files in Repos enabled by default in new workspaces
- Databricks Feature Store is generally available
- Share an experiment from the experiment page
- RStudio Workbench bug fix
- New workspace language options
- Databricks Runtime 10.4 LTS and 10.4 LTS ML are GA; 10.4 Photon is Public Preview
- Enhanced access control for Delta Live Tables pipelines
- Test Delta Live Tables preview functionality with the new
channel
setting (Public Preview) - Improved error handling for Delta Live Tables Python functions (Public Preview)
- Improvements to Databricks Repos
- Audit logging for cluster policy changes
- Databricks Runtime 10.4 (Beta)
- February 2022
- Easier scheduling for your Delta Live Tables pipelines (Public Preview)
- Easily browse the history of your Delta Live Tables pipeline updates (Public Preview)
- Ensure job idempotency for the Jobs API Run now request
- Jobs service stability and scalability improvements
- Compare MLflow runs from different experiments
- Improvements to MLflow compare runs display
- Improved visibility into job run owners in the clusters UI
- Drop dataset columns in AutoML
- Experiments page is GA
- Support for temporary tables in the Delta Live Tables Python interface
- User interface improvements for Delta Live Tables (Public Preview)
- Databricks Runtime 9.0 series support ends
- Data Science & Engineering landing page updates
- Databricks Repos now supports AWS CodeCommit for Git integration
- Improved visualization for your Delta Live Tables pipelines (Public Preview)
- Updated Markdown parser
- Delta Live Tables now supports change data capture processing (Public Preview)
- Select algorithm frameworks to use with AutoML
- Databricks Runtime 10.3 and 10.3 ML are GA; 10.3 Photon is Public Preview
- Repos now supports arbitrary file types (Public Preview)
- January 2022
- MLflow Model Registry Webhooks on Databricks (Public Preview)
- Databricks Runtime 10.3 (Beta)
- View information on recent job runs
- Use Markdown in Databricks Repos file editor
- Improved cluster management for jobs that orchestrate multiple tasks
- Databricks Runtime 8.3 and Databricks Runtime 8.4 series support ends
- Databricks JDBC driver 2.6.22
- MLflow Model Serving on Databricks
- New Share button replaces Permissions icon in notebooks
- New workspace language options
- December 2021
- Databricks JDBC driver 2.6.21
- Databricks Connector for Tableau 2021.4
- Databricks Runtime 10.2 and 10.2 ML are GA; 10.2 Photon is Public Preview
- Updated Markdown parser
- User interface improvements for Delta Live Tables
- New workspaces do not rely upon Google Cloud Deployment Manager
- Databricks Runtime 8.3 series support extended
- Support for GPU clusters (Public Preview)
- Databricks Runtime 10.2 (Beta)
- Revert of recent breaking change that removed escaping and quotes from $ in environment variable values for cluster creation
- Accelerate workloads with Photon (Public Preview)
- November 2021
- Create tags for feature tables (Public Preview)
- Syntax highlighting and autocomplete for SQL commands in Python cells
- Rename, delete, and change permissions for MLflow experiments from experiment page (Public Preview)
- New data profiles in notebooks: tabular and graphic summaries of your data (Public Preview)
- Improved logging when schemas evolve while running a Delta Live Tables pipeline
- Breaking change: remove escaping and quotes from $ in environment variable values for cluster creation
- Ease of use improvements for Files in Repos
- Support for legacy SQL widgets ends on January 15, 2022
- User interface improvements for Databricks jobs
- Delta Sharing Connector for Power BI
- Databricks ODBC driver 2.6.19
- Databricks Runtime 10.1 and 10.1 ML are GA; 10.1 Photon is Public Preview
- Databricks Runtime 10.1 (Beta)
- Rename and delete MLflow experiments (Public Preview)
- Support for creating instance pools with preemptible VM instances using the UI
- You can now create a cluster policy by cloning an existing policy
- Change the default language of notebooks and notebook cells more easily
- Use Files in Repos from the web terminal
- October 2021
- Databricks Runtime 8.2 series support ends
- Access Databricks File System (DBFS) mounts using the local file system
- Databricks Runtime 10.0 and 10.0 ML are GA; 10.0 Photon is Public Preview
- Databricks is available in region
asia-southeast1
- User interface improvements for Delta Live Tables (Public Preview)
- Specify a fixed-size cluster when you create a new pipeline in Delta Live Tables (Public Preview)
- View data quality metrics for tables in Delta Live Tables triggered pipelines (Public Preview)
- Improved security for cluster connectivity in all regions
- Jobs orchestration is now GA
- Databricks Connector for Power BI
- More detailed job run output with the Jobs API
- Improved readability of notebook paths in the Jobs UI
- Open your Delta Live Tables pipeline in a new tab or window
- New escape sequence for
$
in legacy input widgets in SQL - Faster model deployment with automatically generated batch inference notebook
- September 2021
- Databricks Runtime 10.0 (Beta)
- Databricks ODBC driver 2.6.18
- Databricks Runtime 9.1 LTS and 9.1 LTS ML are GA; 9.1 LTS Photon is Public Preview
- Databricks Runtime 8.1 series support ends
- Databricks JDBC driver 2.6.19
- Repos now supports
.gitignore
- Enhanced jobs UI is now standard for all workspaces
- Databricks Runtime 9.1 (Beta)
- Streamlined management of settings for Databricks jobs
- Delete feature tables from Feature Store
- Grant view pipeline permissions in the Delta Live Tables UI (Public Preview)
- Reduce cluster resource usage with Delta Live Tables (Public Preview)
- Use MLflow models in your Delta Live Tables pipelines (Public Preview)
- Find Delta Live Tables pipelines by name (Public Preview)
- PyTorch TorchScript and other third-party libraries are now supported in Databricks jobs
- Databricks Runtime 8.0 series support ends
- August 2021
- Databricks Repos GA
- New Google Cloud region
us-east1
- Increased timeout for long-running notebook jobs
- Jobs service stability and scalability improvements
- User entitlements granted by group membership are displayed in the admin console
- Manage MLflow experiment permissions (Public Preview)
- Improved job creation from notebooks
- Improved support for collapsing notebook headings
- Databricks Runtime 9.0 and 9.0 ML are GA; 9.0 Photon is Public Preview
- Databricks Runtime 9.0 (Beta)
- Simplify and control cluster creation using cluster policies (Public Preview)
- Manage repos programmatically with the Databricks REST API (Public Preview)
- Databricks Runtime 7.6 series support ends
- Improved job creation and management workflow
- Simplified instructions for setting Git credentials (Public Preview)
- Import multiple notebooks in
.html
format
- July 2021
- Manage MLflow experiment permissions with the Databricks REST API
- Restrict access to Databricks using IP access lists
- New workspaces automatically enable the required Google APIs on your project
- Feature freshness information available in Databricks Feature Store UI (Public Preview)
- Display up to 10,000 result rows
- Bulk import and export notebooks in a folder as source files
- Autocomplete in SQL notebooks now uses all-caps for SQL keywords
- Reorderable and resizable widgets in notebooks
- Databricks UI usability fixes
- Store secrets in environment variables
- Databricks Runtime 8.4 and 8.4 ML are GA; 8.4 Photon is Public Preview
- High CPU instance types no longer have local SSDs
- Enhanced data processing and analysis with Databricks jobs (Public Preview)
- Changes to Compute page
- Repos API (Public Preview)
- Databricks Runtime 8.4 (Beta)
- Improved security for cluster connectivity (Public Preview)
- New Google Cloud regions
- June 2021
- Change to Feature Store permissions
- Improved access to results in the MLflow runs table
- Databricks ODBC driver 2.6.17
- Databricks Runtime 7.5 series support ends
- Registry-wide permissions for Model Registry
- A user’s home directory is no longer protected when you delete a user using the SCIM API
- Store the path to a secret in a Spark configuration property
- Databricks Runtime 8.3 and 8.3 ML are GA; 8.3 Photon is Public Preview
- Python and SQL table access control (GA)
- Jobs UI and API now show the owner of a job run
- New Google Cloud region
us-west4
- Single Node clusters are now available
- Deliver cluster logs to a GCS bucket using a DBFS path
- Cluster-scoped init scripts can now be accessed using a DBFS path
- Redesigned Workspace Settings UI
- Updates to
ListTokens
andListAllTokens
database queries expired tokens - Confirmation now required when granting or revoking Admin permissions
- Changes to keyboard shortcuts in the web UI
- May 2021
- Databricks Mosaic AI: a data-native and collaborative solution for the full ML lifecycle
- Additional instance types and more local SSD support
- Faster local SSD performance with disk caching
- Audit log delivery (Public Preview)
- Increased limit for the number of terminated all-purpose clusters
- Increased limit for the number of pinned clusters
- Manage where notebook results are stored (Public Preview)
- Spark UI displays information about terminated clusters
- Create and manage pools using the UI
- Databricks on Google Cloud is GA
- Subnet address range size calculator for new workspaces
- Databricks Runtime 7.4 series support ends
- Jobs service stability and scalability improvements (Public Preview)
- April 2021
- Databricks Runtime 8.2 (GA)
- Easier job management with the enhanced jobs user interface
- MLflow sidebar reflects the most recent experiment
- Change to default channel for
conda.yaml
files in MLflow - Secret management
- Notebook-scoped libraries
- Databricks Runtime 8.2 (Beta)
- User and group limits
- Easier monitoring of job run status
- Receive email notification about activity in Model Registry
- March 2021
- Databricks now supports dark mode for viewing notebooks
- Databricks on Google Cloud (Public Preview)
- Easier job creation and management with the enhanced jobs user interface (Public Preview)
- Track job retry attempts with a new sequential value returned for each job run attempt
- Easier way to connect to Databricks from your favorite BI tools and SQL clients
- Databricks Runtime 8.1 (Beta)
- Increased limit for the number of terminated all-purpose clusters
- Increased limit for the number of pinned clusters in a workspace
- Databricks Runtime 8.0 (GA)
- February 2021