Databricks platform release notes

• November 2024
  • Reuse SQL outputs as _sqldf in subsequent SQL cells
  • Warehouses system table is now available (Public Preview)
  • Cross-platform view sharing (Public Preview)
  • Databricks Runtime 16.0 is GA
  • Breaking change: Hosted RStudio is end-of-life
  • Updated and unified read-only code blocks
  • Increased Quick Fix recommendations
  • Databricks Runtime 14.2 series support ends
  • Serverless compute for workflows (Public Preview)
  • Serverless compute for notebooks (Public Preview)
• October 2024
  • Databricks Assistant helps optimize SQL queries
  • Mosaic AI Model Serving is now available on GCP (Public Preview)
  • Databricks Runtime 15.1 series support ends
  • Enable a metastore to be assigned automatically to new workspaces
  • Compute system tables now available in all GCP regions
  • Databricks Assistant Autocomplete is now generally available
  • Filters on notebook results tables now persist
  • Improvements to comments in notebooks
  • Databricks Runtime 16.0 (Beta)
  • More reliable reloading of modified Python modules with improvements to autoreload
• September 2024
  • Publish to Power BI is now generally available
  • Use @ to reference tables in Databricks Assistant prompts
  • Control external access to data in Unity Catalog using the new EXTERNAL USE SCHEMA privilege
  • Databricks Assistant Quick Fix debugs code inline
  • Warehouse events system table is now available (Public Preview)
  • Extended AI-generated comments support
  • Databricks extension for Visual Studio Code is GA
  • System tables are now generally available
• August 2024
  • Monitor Unity Catalog object usage against quotas using the new Resource Quotas APIs
  • Compute system tables are now available (Public Preview)
  • AskSupport replaced by Databricks assistant
  • Lakeflow system tables are extend with additional columns
  • Python code formatting error highlights
  • Delta Sharing: More Delta Lake features now supported by the Python and Power BI connectors
  • Delta Sharing adds support for TimestampNTZ
  • The Databricks Jobs For each task is GA
  • Databricks Runtime 15.4 LTS is GA
  • Personalized notebook autocomplete
  • Configure your workspace’s default access mode for jobs compute
  • New slash commands for Databricks Assistant
  • Workspace search now supports volumes
  • Databricks JDBC driver 2.6.40
  • Databricks personal access tokens revoked if unused after 90 days
  • Wrap lines in notebook cells
  • Create budgets to monitor account spending (Public Preview)
  • Files can no longer have identical names in workspace folders
  • Compute policy enforcement now available
  • Format columns in notebook and query results tables
  • Row filters and column masks are now GA, with improvements
  • Lakehouse Federation is generally available (GA)
• July 2024
  • Increased limit for simultaneous tasks
  • Embed and drag & drop images in notebooks
  • Command palette available in notebooks
  • Workflow system schema renamed to lakeflow
  • Support for Cloudflare R2 storage is GA
  • Monitor Databricks Assistant activities with system tables (Public Preview)
  • Sharing schemas using Delta Sharing is now GA
  • Databricks Assistant can diagnose issues with jobs (Public Preview)
  • Updates to Databricks Git folders authentication and sharing behaviors
  • Cluster library installation timeout
  • Compute plane outbound IP addresses must be added to a workspace IP allow list
  • Databricks Runtime 9.1 series support extended
  • Single sign-on (SSO) is supported in Lakehouse Federation for SQL Server
  • Enable cross-Geo processing
  • Model sharing using Delta Sharing is now generally available
  • Share comments and primary key constraints using Delta Sharing
  • New Databricks JDBC Driver (OSS)
  • Databricks Runtime 15.4 LTS (Beta)
  • Scala is GA on Unity Catalog shared compute
  • Notebooks: toggle more visible cell titles
  • / in workspace asset names is deprecated
  • Delta Sharing lets you share tables that use liquid clustering
  • Query history system table is now available (Public Preview)
  • Partition metadata logging for Unity Catalog external tables
  • Filter data outputs using natural language prompts
  • Lakehouse Federation supports Salesforce Data Cloud (Public Preview)
  • Databricks Assistant system table now available (Public Preview)
  • Account SCIM API v2.1 (Public Preview)
  • Resource quota increase for tables per Unity Catalog metastore
  • Databricks Assistant can diagnose notebook errors automatically
• June 2024
  • Notebooks - out of order syntax highlighting
  • Notebooks Assistant automatically diagnoses errors
  • Catalog Explorer improvements
  • Databricks Assistant is now GA
  • Databricks Assistant now gives titles to threads
  • Serverless notebooks environment manager
  • New intermediate DNS name for Private Service Connect
  • Notebooks debugging console
  • Quickly add a periodic schedule to a Databricks workflow
  • Databricks Runtime 15.3 is GA
  • Databricks Geos is GA
  • New improved catalog creation UI
  • The billing system tables are enabled by default
  • Volume sharing using Delta Sharing is now generally available
  • Column mapping is now GA
  • Workflow system tables are now available (Public Preview)
  • Databricks Assistant: Threads & queries experience enabled by default
  • New entry points for in-product help
  • Improved notebook Markdown editor
  • Quickly access Catalog Explorer tables from notebooks
  • Databricks ODBC driver 2.8.2
  • The new Databricks Notebook UI is now generally available
• May 2024
  • Databricks Runtime 15.0 series support ends
  • Databricks Runtime 15.3 (Beta)
  • The compute metrics UI is now available on all Databricks Runtime versions
  • Improved search and filtering in notebook and SQL editor results tables
  • New dashboard helps Databricks Marketplace providers monitor listing usage
  • Notebooks on SQL warehouses are now GA
  • View system-generated federated queries in Query Profile
  • Compute plane outbound IP addresses must be added to a workspace IP allow list
  • OAuth is supported in Lakehouse Federation for Snowflake
  • Bulk move and delete workspace objects from the workspace browser
  • Unity Catalog objects are available in recents and favorites
  • New dbt-databricks connector 1.8.0 adopts decoupled dbt architecture
  • New compliance and security settings APIs (Public Preview)
  • Databricks Runtime 15.2 is GA
  • New Tableau connector for Delta Sharing
  • New deep learning recommendation model examples
  • Bind storage credentials and external locations to specific workspaces (Public Preview)
  • Git folders are GA
  • Databricks Assistant autocomplete (Public Preview)
  • New changes to Git folder UI
  • Allow users to copy data to the clipboard from results table
  • Attribute tag values for Unity Catalog objects can now be 1000 characters long (Public Preview)
  • New Previews page
  • Compute cloning now clones any libraries installed on the original compute resource
  • Hive metastore table access controls are deprecated
  • Databricks JDBC driver 2.6.38
  • Databricks Runtime 15.2 (Beta)
  • Notebooks now detect and auto-complete column names for Spark Connect DataFrames
• April 2024
  • Databricks Runtime 15.1 is GA
  • Databricks Assistant: Threads & history
  • Data lineage now captures reads on tables with column masks and row-level security
  • Notebooks now automatically detect SQL
  • New columns added to the billable usage system table (Public Preview)
  • Delta Sharing supports tables that use column mapping (Public Preview)
  • Creation and installation of workspace libraries is no longer available
  • Jobs created through the UI are now queued by default
  • Lakehouse Federation supports foreign tables with case-sensitive identifiers
  • Serverless SQL warehouses (Public Preview)
  • Delta Live Tables notebook developer experience improvements (Public Preview)
• March 2024
  • MLflow now enforces quota limits for experiments and runs
  • The Jobs UI is updated to better manage jobs deployed by Databricks Asset Bundles
  • Interactive notebook debugging
  • Self-service sign-up for private exchange providers in Marketplace
  • Databricks Runtime 15.0 is GA
  • Databricks Repos changed to Git folders
  • Databricks Runtime 14.1 and 14.2 series support extended
  • Databricks ODBC driver 2.8.0
  • Delegate the ability to view an object’s metadata in Unity Catalog (Public Preview)
  • System tables now include Marketplace tables (Public Preview)
  • System tables now available (Public Preview)
  • Databricks Runtime 15.0 (Beta)
  • Databricks Runtime 14.0 series support ends
  • New computation for sys.path and CWD in Repos
  • Automatic enablement of Unity Catalog for new workspaces
• February 2024
  • New region: asia-south1 (Mumbai)
  • Use Delta Live Tables in Feature Engineering (Public Preview)
  • Restrict creating a personal access token for a service principal
  • Restrict changing a job owner and the run as setting
  • Support for Cloudflare R2 storage to avoid cross-region egress fees (Public Preview)
  • Add data UI supports XML file format
  • Unity Catalog volumes are GA
  • Full-page AI-powered search
  • UI experience for OAuth app registration
  • File arrival triggers in Databricks Jobs is GA
  • Search for machine learning models in Unity Catalog using global workspace search
  • Databricks Git server proxy is GA
  • Databricks Git server proxy no longer requires CAN_ATTACH_TO permissions
  • Workspace file support for the dbt and SQL file tasks is GA
  • Databricks Connect is GA for Scala
  • Create tables from files in volumes
  • Databricks Runtime 14.3 LTS is GA
  • Delta Sharing supports tables that use deletion vectors (Public Preview)
• January 2024
  • Native XML file format support (Public Preview)
  • Share AI models using Databricks Marketplace (Public Preview)
  • Workspace path update
  • Streamlined creation of Databricks jobs
  • URI path-based access to Unity Catalog external volumes
  • Access controls lists can be enabled on upgraded workspaces
  • Updated UI for notebook cells (Public Preview)
  • Quick Fix help with syntax errors in the notebook
  • Databricks Runtime 14.3 LTS (Beta)
  • Share AI models using Delta Sharing (Public Preview)
  • Databricks Marketplace supports volume sharing
  • Create widgets from the Databricks UI
  • Libraries now supported in compute policies (Public Preview)
  • Workspace file size limit is now 500MB
  • Feature removal notice for legacy Git integration in Databricks
  • Databricks ODBC driver 2.7.7
  • Databricks Runtime 13.2 series support ends
• December 2023
  • Share dynamic views using Delta Sharing (Public Preview)
  • Share volumes using Delta Sharing (Public Preview)
  • Entity Relationship Diagram for primary keys and foreign keys
  • Compute created in the UI now uses the “Auto” availability zone by default
  • Unity Catalog volume file upload size limit increase
  • New notebook cell results rendering available (Public Preview)
  • Notebook editor themes available
  • New unified admin settings UI
  • Init scripts on DBFS are end-of-life
• November 2023
  • Access controls lists can no longer be disabled
  • AI assistive features are enabled by default
  • New behaviors and actions in Catalog Explorer for volumes
  • Databricks Runtime 14.2 is GA
  • Databricks SQL Connector for Python version 3.0.0
  • Auto-enable deletion vectors
  • Libraries in workspace files supported on no-isolation shared clusters
  • Deprecation of workspace libraries
  • Search for Databricks Marketplace listings using global workspace search
  • Consume data products in Databricks Marketplace using external platforms
  • Databricks Runtime 14.2 (beta)
  • Databricks Marketplace includes Databricks Solution Accelerators
  • Lakehouse Federation adds support for Google BigQuery
• October 2023
  • Configure the availability zone of instance pools
  • View the YAML source for a Databricks job
  • Add conditional logic to your Databricks workflows
  • Configure parameters on a Databricks job that can be referenced by all job tasks
  • Init scripts are now supported on Unity Catalog volumes
  • Unity Catalog support for UNDROP TABLE is GA
  • Partner Connect supports Dataiku
  • Mosaic AutoML generated notebooks are now saved as MLflow artifacts
  • Unity Catalog support for Python and PySpark UDFs
  • Feature Engineering in Unity Catalog is GA
  • AI-generated table comments (Public Preview)
  • Models in Unity Catalog is GA
  • Partner Connect supports Monte Carlo
  • Semantic search (Public Preview)
  • Enable Databricks Assistant at the workspace level
  • IP access lists for the account console is GA
  • New Photon defaults
  • Databricks Runtime 14.1 is GA
  • Developer tools release notes have moved
  • Databricks extension for Visual Studio Code updated to version 1.1.5
  • Predictive I/O for updates is GA
  • Deletion vectors are GA
  • Customer-managed keys are generally available
  • Personal Compute cluster policy is available by default
  • Partner Connect supports RudderStack
  • Databricks CLI updated to version 0.207.0 (Public Preview)
  • Capture data lineage using Unity Catalog
  • Run selected cells in a notebook
  • Use workspace-catalog binding to give read-only access to a catalog
  • New in-product Help experience (Public Preview)
  • Databricks extension for Visual Studio Code updated to version 1.1.4
  • Databricks SDK for Python updated to version 0.10.0 (Beta)
  • Databricks SDK for Go updated to version 0.22.0 (Beta)
• September 2023
  • Lakehouse Federation available on single user clusters
  • Delegate the ability to create a storage credential in Unity Catalog
  • Databricks Runtime 14.1 (Beta)
  • Prevent runs of a job from being skipped when you reach concurrency limits
  • Databricks Terraform provider updated to version 1.27.0
  • Databricks CLI updated to version 0.206.0 (Public Preview)
  • Databricks SDK for Go updated to version 0.21.0 (Beta)
  • Databricks ODBC driver 2.7.5
  • Databricks extension for Visual Studio Code updated to version 1.1.3
  • Running jobs as a service principal is GA
  • Databricks CLI updated to version 0.205.2 (Public Preview)
  • Databricks CLI updated to version 0.205.1 (Public Preview)
  • Databricks SDK for Go updated to version 0.20.0 (Beta)
  • Databricks SDK for Python updated to version 0.9.0 (Beta)
  • Databricks SDK for Python updated to version 0.7.1 (Beta)
  • Databricks Terraform provider updated to version 1.26.0
  • Databricks Connect for Databricks Runtime 14.0
  • Databricks Connect V2 is Public Preview for Scala
  • Databricks Runtime 14.0 is GA
  • Account email notifications upgrade
  • Databricks Terraform provider updated to version 1.25.1
  • Databricks SDK for Go updated to version 0.19.2 (Beta)
  • Databricks CLI updated to version 0.205.0 (Public Preview)
  • Reserved location for /Workspace
  • Databricks Asset Bundles (Public Preview)
  • Databricks Terraform provider updated to version 1.25.0
  • Databricks extension for Visual Studio Code updated to version 1.1.2
  • Partner Connect supports Snowplow
  • Databricks CLI updated to version 0.204.1 (Public Preview)
  • Filter sensitive data with row filters and column masks (Public Preview)
  • Databricks SDK for Go updated to version 0.19.1 (Beta)
  • Databricks ODBC driver 2.7.3
  • Databricks CLI updated to version 0.204.0 (Public Preview)
  • Data Explorer is now Catalog Explorer
  • Databricks SDK for Go updated to version 0.19.0 (Beta)
  • Databricks SDK for Python updated to version 0.8.0 (Beta)
  • GitHub apps integration in Repos is GA
• August 2023
  • Tables now appear in navigational search
  • Databricks CLI updated to version 0.203.3 (Public Preview)
  • Databricks JDBC driver 2.6.34
  • New Google Cloud region: northamerica-northeast1 (Montréal, Canada)
  • Databricks SDK for Go updated to version 0.18.0 (Beta)
  • Databricks SDK for Python updated to version 0.7.0 (Beta)
  • Databricks Terraform provider updated to version 1.24.1
  • Init scripts on DBFS end of life extended to Dec 1, 2023
  • Databricks Runtime 14.0 (Beta)
  • Unified navigation experience is GA
  • Databricks Terraform provider updated to version 1.24.0
  • Shared access mode clusters can now connect to public networks outside of Databricks VPC
  • Databricks CLI updated to version 0.203.2 (Public Preview)
  • Go to definition for functions and variables in Python notebooks
  • Databricks Runtime 13.3 LTS is GA
  • Introducing tags with Unity Catalog
  • Allowlist for JARs and Maven coordinates on Unity Catalog shared clusters is in Public Preview
  • Volumes support for JARs is in Public Preview
  • Easier Databricks Repos .ipynb file output commits
  • IPYNB notebook support in Databricks Repos is GA
  • Databricks SDK for Go updated to version 0.17.0 (Beta)
  • Databricks SDK for Python updated to version 0.6.0 (Beta)
  • Databricks CLI updated to version 0.203.1 (Public Preview)
  • Unified schema browser is now GA
  • Databricks SDK for Go updated to version 0.16.0 (Beta)
  • Partner Connect supports Census
  • Databricks SDK for Python updated to version 0.5.0 (Beta)
  • Programmatic write support for workspace files
  • Databricks CLI updated to version 0.203.0 (Public Preview)
  • Databricks Terraform provider updated to version 1.23.0
  • Groups can now be renamed
  • Databricks SDK for Go updated to version 0.15.0 (Beta)
  • Databricks SDK for Python updated to version 0.4.0 (Beta)
  • Third-party iFraming prevention configuration setting was removed
  • Databricks extension for Visual Studio Code updated to version 1.1.1
  • LangChain available in 13.1 and above
  • Feature Engineering in Unity Catalog is Public Preview
  • Improved error handling for repeated continuous job failures
  • Share schemas using Delta Sharing (Public Preview)
  • Run tasks conditionally in your Databricks jobs
  • Delta Live Tables is now generally available
  • Databricks Terraform provider updated to version 1.22.0
  • Databricks Runtime 13.3 LTS (Beta)
  • New Git operations are generally available: Merge branches, rebase and pull with conflict resolution
• July 2023
  • Workspace admins can now create account groups
  • Group manager role is in Public Preview
  • Databricks CLI updated to version 0.202.0 (Public Preview)
  • Databricks SDK for Python updated to version 0.3.0 (Beta)
  • Databricks SDK for Go updated to version 0.14.1 (Beta)
  • Databricks SDK for Go updated to version 0.14.0 (Beta)
  • Run another job as a task in a Databricks job
  • Enable customer-managed keys for managed services (Public Preview)
  • Account API is generally available
  • Private Service Connect launches to Limited Availability with GA-level functionality
  • All users can access data products in Databricks Marketplace by default
  • Classic keyboard shortcuts mode
  • Lakehouse Federation lets you run queries against external database providers (Public Preview)
  • Move to trash enabled for Repos
  • Create alerts for slow-running or stuck jobs
  • Databricks SDK for Go updated to version 0.13.0 (Beta)
  • Databricks SDK for Python updated to version 0.2.0 (Beta)
  • Databricks CLI updated to version 0.201.0 (Public Preview)
  • Databricks SDK for Python updated to version 0.2.1 (Beta)
  • Databricks Assistant is in Public Preview
  • Deactivate users and service principals from your account
  • Account-level SCIM provisioning now deactivates users when they are deactivated in the identity provider
  • Trash directory admin access
  • Prevention of MIME type sniffing and XSS attack page rendering are now always enabled
  • Unity Catalog volumes are in Public Preview
  • Simplified experience for submitting product feedback from the workspace
  • Databricks extension for Visual Studio Code updated to version 1.1.0
  • Functions now displayed in Catalog Explorer (Public Preview)
  • Databricks Terraform provider updated to version 1.21.0
  • The maximum offset for the List all jobs and List job runs API requests is now limited
  • Databricks Runtime 13.2 is GA
  • Delta Sharing and Databricks Marketplace support view sharing (Public Preview)
  • Init scripts on DBFS reach end of life on Sept 1, 2023
• June 2023
  • Databricks Terraform provider updated to version 1.20.0
  • Databricks CLI updated to version 0.200.1 (Public Preview)
  • Databricks SDK for Go updated to version 0.12.0 (Beta)
  • Databricks SDK for Go updated to version 0.11.0 (Beta)
  • Databricks SDK for Python (Beta)
  • Databricks SDK for Go (Beta)
  • Models in Unity Catalog (Public Preview)
  • Databricks Marketplace is now GA
  • Databricks Runtime 13.2 (Beta)
  • Databricks extension for Visual Studio Code (General Availability)
  • See a visual overview of completed job runs in the Databricks Jobs UI
  • Databricks CLI (Public Preview)
  • Improved pagination of results from List all jobs and List job runs API requests
  • Full-page workspace browser includes Repos
  • Databricks Terraform provider updated to version 1.19.0
  • Configure the number of local SSDs in your cluster or instance pool
  • Run jobs as a service principal (Public Preview)
  • New service principal UI provides better management experience
  • Home folders restored when users are re-added to workspaces
  • Databricks Marketplace: Private exchanges are now available
  • Databricks Marketplace: Consumers can uninstall data products using the UI
  • Databricks Marketplace: providers can create their own profiles
  • Databricks Connect V2 is GA for Python
  • Databricks Terraform provider updated to version 1.18.0
  • New Databricks Marketplace providers
  • Databricks Runtime 13.1 is GA
  • View data from notebooks, SQL editor, and Catalog Explorer with the unified schema browser (Public Preview)
• May 2023
  • Create or modify table UI supports Avro, Parquet, and text file uploads
  • Use the add data UI to load data using Unity Catalog external locations (Public Preview)
  • Prevent Enter key from accepting autocomplete suggestion
  • Databricks Terraform provider updated to version 1.17.0
  • Upload data UI supports new column data types
  • Bind Unity Catalog catalogs to specific workspaces
  • All users can connect to Fivetran using Partner Connect (Public Preview)
  • Databricks Terraform provider updated to version 1.16.1
  • Databricks JDBC driver 2.6.33
  • New default theme for editor
  • Databricks Terraform provider updated to version 1.16.0
  • Databricks Runtime 13.1 (Beta)
  • Databricks Terraform provider updated to version 1.15.0
  • Account nicknames now available in the account console
  • Share notebooks using Delta Sharing
  • Deprecation of cluster-scoped init scripts on DBFS
  • Unified navigation (Public Preview)
• April 2023
  • Databricks Marketplace (Public Preview): an open marketplace for data, analytics, and AI
  • Customer-managed keys for workspace storage (Public Preview)
  • Configure the Python formatter
  • Workspace files are GA
  • Cluster-scoped init scripts can now be stored in workspace files
  • New Delta Sharing privileges enable delegation of share, recipient, and provider management tasks
  • Databricks Runtime 13.0 is GA
  • Compute resource usage for Google Cloud Marketplace aggregates under a single SKU
  • Control access to the account console by IP address ranges (Public Preview)
  • Databricks Terraform provider updated to version 1.14.3
  • Audit log entries for changed admin settings for workspaces and accounts
  • Databricks Terraform provider updated to versions 1.14.1 and 1.14.2
  • Combined SQL user settings and general user settings
  • Legacy notebook visualizations deprecated
  • Create or modify table from file upload page supports JSON file uploads
  • New API support for configuring Private Service Connect (Public Preview)
• March 2023
  • Databricks Terraform provider updated to version 1.14.0
  • Local storage autoscaling feature added to clusters
  • Distributed training with TorchDistributor
  • Databricks Runtime 13.0 (Beta)
  • Improved file editor
  • .ipynb (Jupyter) notebook support in Repos (preview)
  • Support for reload4j
  • Unity Catalog, Delta Sharing, and identity federation are GA on Google Cloud
  • Execute SQL cells in the notebook in parallel
  • Create job tasks using Python code stored in a Git repo
  • Databricks Terraform provider updated to version 1.13.0
  • Databricks Terraform provider updated to version 1.12.0
  • SQL admin console and workspace admin console combined
  • New Catalog Explorer availability
  • Exact match search is available in global search
  • Databricks Runtime 12.2 LTS and Databricks Runtime 12.2 LTS ML are GA
  • Workspace files are now in Public Preview
• February 2023
  • Updated required permissions for new workspaces
  • Secure a workspace with private connectivity using Google Private Service Connect (Public Preview)
  • Ray on Databricks (Public Preview)
  • Notebook cell output results limit increased
  • Databricks Jobs now supports running continuous jobs
  • Databricks Terraform provider updated to version 1.10.0
  • Improvements to the MLflow experiment UI
  • Databricks Runtime 12.2 (Beta)
  • Databricks extension for Visual Studio Code (Public Preview)
  • Databricks Terraform provider updated to version 1.9.2
  • Partner Connect is GA
  • Variable explorer in Databricks notebooks
• January 2023
  • Create a workspace using Terraform (Public Preview)
  • Create a workspace using the Account API (Public Preview)
  • Authenticate to Databricks REST APIs using a Google ID
  • Account SCIM is now GA
  • Easier creation and editing of Databricks jobs in the UI
  • Improvements to the Databricks Jobs UI when viewing job runs
  • REST API Reference is now available for browsing API documentation
  • Databricks Terraform provider updated to version 1.9.1
  • Account users can update email preferences in the account console
  • New account console home screen provides better account management experience
  • Region support consolidated onto one page
  • Databricks Runtime 12.1 and Databricks Runtime 12.1 ML are GA
  • Cluster policies now support limiting the max number of clusters per user
  • Databricks Terraform provider updated to version 1.9.0
  • Databricks Terraform provider updated to version 1.8.0
  • Databricks Runtime 12.1 (Beta)
  • Global workspace search
  • Partner Connect supports Sigma
  • New left and right sidebars in Databricks notebooks
• December 2022
  • Databricks SQL Driver for Go is Generally Available
  • Prevent concurrent workspace updates
  • Databricks Terraform provider updated to version 1.7.0
  • Databricks Runtime 12.0 and 12.0 ML are GA
  • Billable usage graphs can now aggregate by individual tags
  • Databricks ODBC driver 2.6.29
  • Databricks JDBC driver 2.6.32
• November 2022
  • Enhanced notifications for your Databricks jobs (Public Preview)
  • Databricks Runtime 12.0 (Beta)
  • Enforce user isolation cluster types on a workspace
  • New Google Cloud region: asia-northeast1 (Tokyo)
  • Upload data UI can now be disabled via admin settings
  • Work with large repositories with Sparse Checkout
  • Databricks Terraform provider updated to version 1.6.5
  • Databricks Terraform provider updated to versions 1.6.3 and 1.6.4
  • Create or modify table from file upload page now supports multiple files
  • Create or modify table from file upload page now supports overwrite
  • Search for jobs by name with the Jobs API 2.1
  • Databricks Terraform provider updated to version 1.6.2
  • Use a Google ID token to authenticate to workspace REST APIs (Public Preview)
• October 2022
  • GA: Repos support for non-notebook files
  • Deploy models for streaming inference with Delta Live Tables notebooks
  • Connect to Fivetran from the add data UI
  • Databricks SQL Driver for Node.js is Generally Available
  • Partner Connect supports connecting to erwin Data Modeler by Quest
  • Optimize performance and control costs by using different pools for the driver node and worker nodes
  • Databricks Runtime 11.3 LTS and 11.3 LTS ML are GA
  • Format Python code in notebooks (Public Preview)
  • DBFS Rest API is available for all customers
  • Add data UI provides a central UI for loading data to Databricks
  • Create or modify table from file upload page unifies experience for small file upload to Delta Lake
  • Partner Connect Public Preview
  • Cluster UI preview and access mode replaces security mode
  • Partner Connect supports connecting to Hevo Data
  • Account users can access the account console
  • Enable admin protection for Standard clusters (or High Concurrency clusters without table ACLs)
  • SQL persona integrated with new search experience
  • Service principals provide API-only access to Databricks resources (Public Preview)
  • Top navigation bar in the UI
  • Databricks Runtime 11.3 (Beta)
  • The account console is available in multiple languages
• September 2022
  • New reference solution for natural language processing
  • Audit logs now include events for web terminal
  • Audit logs now include events for managing credentials for Git repos
  • Audit logs are GA
  • Cluster and pool tags now propagate to GCE labels
  • Select cluster policies directly in the Delta Live Tables UI
  • New data trasformation card on workspace landing pages
  • Delta cache is now disk cache
  • View and organize assets in the workspace browser across personas
  • Databricks Runtime 11.2 and 11.2 ML are GA
  • Enhanced Spark egress network policy
• August 2022
  • Databricks ODBC driver 2.6.26
  • Databricks JDBC driver 2.6.29
  • Databricks Feature Store client now available on PyPI
  • Databricks Runtime 11.2 (Beta)
  • Decreased storage costs by reducing default boot disk size
  • Reduced message volume in the Delta Live Tables UI for continuous pipelines
  • Easier cluster configuration for your Delta Live Tables pipelines
  • Orchestrate dbt tasks in your Databricks workflows (Public Preview)
  • Control workspace access with Customer Approved Workspace Login
  • Deploy a workspace in a customer-managed VPC (GA)
  • Databricks is now available in region europe-west3
  • Use generated columns when you create Delta Live Tables datasets
  • Improved editing for notebooks with Monaco-based editor (Experimental)
  • Compliance controls for HIPAA (GA)
  • Databricks Runtime 10.3 series support ends
  • Delta Live Tables now supports refreshing only selected tables in pipeline updates
  • Job execution now waits for cluster libraries to finish installing
• July 2022
  • Verbose audit logs now record when a notebook command or Databricks SQL query runs
  • Databricks Runtime 11.1 and 11.1 ML are GA
  • Photon is GA
  • Notification upon notebook completion
  • Increased limit for the number of jobs in your Databricks workspaces
  • Databricks JDBC driver 2.6.27
  • Databricks ODBC driver 2.6.25
  • Databricks Runtime 11.1 (Beta)
  • Improved notebook visualizations
• June 2022
  • Databricks Runtime 6.4 Extended Support reaches end of support
  • Databricks Runtime 10.2 series support ends
  • Databricks ODBC driver 2.6.24
  • Databricks Terraform provider is now GA
  • HIPAA compliance controls (Public Preview)
  • Databricks Runtime 11.0 and 11.0 ML are GA; 11.0 Photon is Public Preview
  • Change to Repos default working directory in Databricks Runtime 11.0
  • Databricks Runtime 10.1 series support ends
  • Delta Live Tables now supports SCD type 2
  • Create Delta Live Tables pipelines directly in the Databricks UI
  • Select the Delta Live Tables channel when you create or edit a pipeline
  • Communicate between tasks in your Databricks jobs with task values
  • Enable account switching in the Databricks UI
  • Default instance type is now n2-highmem-4
• May 2022
  • Copy and paste notebook cells between tabs and windows
  • Databricks Runtime 11.0 (Beta)
  • Explore SQL cell results in Python notebooks natively using Python
  • Databricks Repos: Support for more files in a repo
  • Databricks Repos: Fix to issue with MLflow experiment data loss
  • Power BI Desktop system-wide HTTP proxy support
  • Databricks Runtime 10.5 and 10.5 ML are GA; 10.5 Photon is Public Preview
  • Databricks JDBC driver 2.6.25
  • See the user a pipeline runs as in the Delta Live Tables UI
• April 2022
  • Use tags to better manage your Databricks jobs
  • Databricks Runtime 10.0 series support ends
  • Get a visual overview of your job runs with the new jobs matrix view
  • Save time and resources when your Databricks job runs are unsuccessful
  • View the run history for job tasks
  • Assign a new cluster in the jobs UI when the Single User access no longer exists
  • Databricks Runtime 10.5 (Beta)
  • New email provider for jobs alerts
  • The Delta Live Tables UI is enhanced to disable unauthorized actions
  • Mosaic AutoML is generally available
  • Delta Live Tables is GA on AWS and Azure, and in Public Preview on GCP
  • Delta Live Tables SQL interface: non-breaking change to table names
• March 2022
  • Better performance and cost for your Delta Live Tables pipelines with Databricks enhanced autoscaling
  • Customer-managed VPC is Public Preview
  • Files in Repos enabled by default in new workspaces
  • Databricks Feature Store is generally available
  • Share an experiment from the experiment page
  • RStudio Workbench bug fix
  • New workspace language options
  • Databricks Runtime 10.4 LTS and 10.4 LTS ML are GA; 10.4 Photon is Public Preview
  • Enhanced access control for Delta Live Tables pipelines
  • Test Delta Live Tables preview functionality with the new channel setting (Public Preview)
  • Improved error handling for Delta Live Tables Python functions (Public Preview)
  • Improvements to Databricks Repos
  • Audit logging for cluster policy changes
  • Databricks Runtime 10.4 (Beta)
• February 2022
  • Easier scheduling for your Delta Live Tables pipelines (Public Preview)
  • Easily browse the history of your Delta Live Tables pipeline updates (Public Preview)
  • Ensure job idempotency for the Jobs API Run now request
  • Jobs service stability and scalability improvements
  • Compare MLflow runs from different experiments
  • Improvements to MLflow compare runs display
  • Improved visibility into job run owners in the clusters UI
  • Drop dataset columns in Mosaic AutoML
  • Experiments page is GA
  • Support for temporary tables in the Delta Live Tables Python interface
  • User interface improvements for Delta Live Tables (Public Preview)
  • Databricks Runtime 9.0 series support ends
  • Data Science & Engineering landing page updates
  • Databricks Repos now supports AWS CodeCommit for Git integration
  • Improved visualization for your Delta Live Tables pipelines (Public Preview)
  • Updated Markdown parser
  • Delta Live Tables now supports change data capture processing (Public Preview)
  • Select algorithm frameworks to use with AutoML
  • Databricks Runtime 10.3 and 10.3 ML are GA; 10.3 Photon is Public Preview
  • Repos now supports arbitrary file types (Public Preview)
• January 2022
  • MLflow Model Registry Webhooks on Databricks (Public Preview)
  • Databricks Runtime 10.3 (Beta)
  • View information on recent job runs
  • Use Markdown in Databricks Repos file editor
  • Improved cluster management for jobs that orchestrate multiple tasks
  • Databricks Runtime 8.3 and Databricks Runtime 8.4 series support ends
  • Databricks JDBC driver 2.6.22
  • MLflow Model Serving on Databricks
  • New Share button replaces Permissions icon in notebooks
  • New workspace language options
• December 2021
  • Databricks JDBC driver 2.6.21
  • Databricks Connector for Tableau 2021.4
  • Databricks Runtime 10.2 and 10.2 ML are GA; 10.2 Photon is Public Preview
  • Updated Markdown parser
  • User interface improvements for Delta Live Tables
  • New workspaces do not rely upon Google Cloud Deployment Manager
  • Databricks Runtime 8.3 series support extended
  • Support for GPU clusters (Public Preview)
  • Databricks Runtime 10.2 (Beta)
  • Revert of recent breaking change that removed escaping and quotes from $ in environment variable values for cluster creation
  • Accelerate workloads with Photon (Public Preview)
• November 2021
  • Create tags for feature tables (Public Preview)
  • Syntax highlighting and autocomplete for SQL commands in Python cells
  • Rename, delete, and change permissions for MLflow experiments from experiment page (Public Preview)
  • New data profiles in notebooks: tabular and graphic summaries of your data (Public Preview)
  • Improved logging when schemas evolve while running a Delta Live Tables pipeline
  • Breaking change: remove escaping and quotes from $ in environment variable values for cluster creation
  • Ease of use improvements for Files in Repos
  • Support for legacy SQL widgets ends on January 15, 2022
  • User interface improvements for Databricks jobs
  • Delta Sharing Connector for Power BI
  • Databricks ODBC driver 2.6.19
  • Databricks Runtime 10.1 and 10.1 ML are GA; 10.1 Photon is Public Preview
  • Databricks Runtime 10.1 (Beta)
  • Rename and delete MLflow experiments (Public Preview)
  • Support for creating instance pools with preemptible VM instances using the UI
  • You can now create a cluster policy by cloning an existing policy
  • Change the default language of notebooks and notebook cells more easily
  • Use Files in Repos from the web terminal
• October 2021
  • Databricks Runtime 8.2 series support ends
  • Access Databricks File System (DBFS) mounts using the local file system
  • Databricks Runtime 10.0 and 10.0 ML are GA; 10.0 Photon is Public Preview
  • Databricks is available in region asia-southeast1
  • User interface improvements for Delta Live Tables (Public Preview)
  • Specify a fixed-size cluster when you create a new pipeline in Delta Live Tables (Public Preview)
  • View data quality metrics for tables in Delta Live Tables triggered pipelines (Public Preview)
  • Improved security for cluster connectivity in all regions
  • Jobs orchestration is now GA
  • Databricks Connector for Power BI
  • More detailed job run output with the Jobs API
  • Improved readability of notebook paths in the Jobs UI
  • Open your Delta Live Tables pipeline in a new tab or window
  • New escape sequence for $ in legacy input widgets in SQL
  • Faster model deployment with automatically generated batch inference notebook
• September 2021
  • Databricks Runtime 10.0 (Beta)
  • Databricks ODBC driver 2.6.18
  • Databricks Runtime 9.1 LTS and 9.1 LTS ML are GA; 9.1 LTS Photon is Public Preview
  • Databricks Runtime 8.1 series support ends
  • Databricks JDBC driver 2.6.19
  • Repos now supports .gitignore
  • Enhanced jobs UI is now standard for all workspaces
  • Databricks Runtime 9.1 (Beta)
  • Streamlined management of settings for Databricks jobs
  • Delete feature tables from Feature Store
  • Grant view pipeline permissions in the Delta Live Tables UI (Public Preview)
  • Reduce cluster resource usage with Delta Live Tables (Public Preview)
  • Use MLflow models in your Delta Live Tables pipelines (Public Preview)
  • Find Delta Live Tables pipelines by name (Public Preview)
  • PyTorch TorchScript and other third-party libraries are now supported in Databricks jobs
  • Databricks Runtime 8.0 series support ends
• August 2021
  • Databricks Repos GA
  • New Google Cloud region us-east1
  • Increased timeout for long-running notebook jobs
  • Jobs service stability and scalability improvements
  • User entitlements granted by group membership are displayed in the admin console
  • Manage MLflow experiment permissions (Public Preview)
  • Improved job creation from notebooks
  • Improved support for collapsing notebook headings
  • Databricks Runtime 9.0 and 9.0 ML are GA; 9.0 Photon is Public Preview
  • Databricks Runtime 9.0 (Beta)
  • Simplify and control cluster creation using cluster policies (Public Preview)
  • Manage repos programmatically with the Databricks REST API (Public Preview)
  • Databricks Runtime 7.6 series support ends
  • Improved job creation and management workflow
  • Simplified instructions for setting Git credentials (Public Preview)
  • Import multiple notebooks in .html format
• July 2021
  • Manage MLflow experiment permissions with the Databricks REST API
  • Restrict access to Databricks using IP access lists
  • New workspaces automatically enable the required Google APIs on your project
  • Feature freshness information available in Databricks Feature Store UI (Public Preview)
  • Display up to 10,000 result rows
  • Bulk import and export notebooks in a folder as source files
  • Autocomplete in SQL notebooks now uses all-caps for SQL keywords
  • Reorderable and resizable widgets in notebooks
  • Databricks UI usability fixes
  • Store secrets in environment variables
  • Databricks Runtime 8.4 and 8.4 ML are GA; 8.4 Photon is Public Preview
  • High CPU instance types no longer have local SSDs
  • Enhanced data processing and analysis with Databricks jobs (Public Preview)
  • Changes to Compute page
  • Repos API (Public Preview)
  • Databricks Runtime 8.4 (Beta)
  • Improved security for cluster connectivity (Public Preview)
  • New Google Cloud regions
• June 2021
  • Change to Feature Store permissions
  • Improved access to results in the MLflow runs table
  • Databricks ODBC driver 2.6.17
  • Databricks Runtime 7.5 series support ends
  • Registry-wide permissions for Model Registry
  • A user’s home directory is no longer protected when you delete a user using the SCIM API
  • Store the path to a secret in a Spark configuration property
  • Databricks Runtime 8.3 and 8.3 ML are GA; 8.3 Photon is Public Preview
  • Python and SQL table access control (GA)
  • Jobs UI and API now show the owner of a job run
  • New Google Cloud region us-west4
  • Single Node clusters are now available
  • Deliver cluster logs to a GCS bucket using a DBFS path
  • Cluster-scoped init scripts can now be accessed using a DBFS path
  • Redesigned Workspace Settings UI
  • Updates to ListTokens and ListAllTokens database queries expired tokens
  • Confirmation now required when granting or revoking Admin permissions
  • Changes to keyboard shortcuts in the web UI
• May 2021
  • Databricks Mosaic AI: a data-native and collaborative solution for the full ML lifecycle
  • Additional instance types and more local SSD support
  • Faster local SSD performance with disk caching
  • Audit log delivery (Public Preview)
  • Increased limit for the number of terminated all-purpose clusters
  • Increased limit for the number of pinned clusters
  • Manage where notebook results are stored (Public Preview)
  • Spark UI displays information about terminated clusters
  • Create and manage pools using the UI
  • Databricks on Google Cloud is GA
  • Subnet address range size calculator for new workspaces
  • Databricks Runtime 7.4 series support ends
  • Jobs service stability and scalability improvements (Public Preview)
• April 2021
  • Databricks Runtime 8.2 (GA)
  • Easier job management with the enhanced jobs user interface
  • MLflow sidebar reflects the most recent experiment
  • Change to default channel for conda.yaml files in MLflow
  • Secret management
  • Notebook-scoped libraries
  • Databricks Runtime 8.2 (Beta)
  • User and group limits
  • Easier monitoring of job run status
  • Receive email notification about activity in Model Registry
• March 2021
  • Databricks now supports dark mode for viewing notebooks
  • Databricks on Google Cloud (Public Preview)
  • Easier job creation and management with the enhanced jobs user interface (Public Preview)
  • Track job retry attempts with a new sequential value returned for each job run attempt
  • Easier way to connect to Databricks from your favorite BI tools and SQL clients
  • Databricks Runtime 8.1 (Beta)
  • Increased limit for the number of terminated all-purpose clusters
  • Increased limit for the number of pinned clusters in a workspace
  • Databricks Runtime 8.0 (GA)
• February 2021
  • New Databricks Power BI connector (GA)
  • Databricks on Google Cloud public preview
    • Google Cloud technology integration
    • Initial account setup
    • What’s in this release?