Databricks Runtime 16.0 for Machine Learning (Beta)

Beta

Databricks Runtime 16.0 for Machine Learning is in Beta. The contents of the supported environments may change during the Beta. Changes can include the list of packages or versions of installed packages.

Databricks Runtime 16.0 for Machine Learning provides a ready-to-go environment for machine learning and data science based on Databricks Runtime 16.0 (Beta). Databricks Runtime ML contains many popular machine learning libraries, including TensorFlow, PyTorch, and XGBoost. Databricks Runtime ML includes AutoML, a tool to automatically train machine learning pipelines. Databricks Runtime ML also supports distributed deep learning training using TorchDistributor, DeepSpeed, and Ray.

Note

These release notes may include references to features that are not available on Google Cloud as of this release.

Tip

To see release notes for Databricks Runtime versions that have reached end-of-support (EoS), see End-of-support Databricks Runtime release notes. The EoS Databricks Runtime versions have been retired and might not be updated.

New features and improvements

Databricks Runtime 16.0 ML is built on top of Databricks Runtime 16.0. For information on what’s new in Databricks Runtime 16.0, including Apache Spark MLlib and SparkR, see the Databricks Runtime 16.0 (Beta) release notes.

New Python packages

The following Python packages have been added to Databricks Runtime ML:

  • composer 0.24.1

  • optuna 3.6.1

Mosaic AutoML sample weights for forecasting

AutoML now supports sample weights for forecasting, letting you adjust the importance of each time-series for training multi-time-series forecasting models. For more information, see the forecasting parameters for the AutoML Python API.

Use a view in Unity Catalog as a feature table

You can now use a view in Unity Catalog as a feature table. See Use an existing view in Unity Catalog as a feature table.

Other changes

Horovod, HorovodRunner, Petastorm, spark-tensorflow-distributor removed

The following packages that were included in previous versions of Databricks Runtime ML are not included in Databricks Runtime 16.0 ML:

  • Horovod

  • HorovodRunner

  • Petastorm

  • spark-tensorflow-distributor

Databricks recommends the following replacements:

  • For distributed deep learning, Databricks recommends using TorchDistributor for distributed training with PyTorch or the tf.distribute.Strategy API for distributed training with TensorFlow.

  • For loading large datasets from cloud storage, Databricks recommends using Mosaic Streaming.

  • For distributed training for a TensorFlow or Keras model, Databricks recommends using Ray. See Ray on Databricks and the Ray documentation.

System environment

The system environment in Databricks Runtime 16.0 ML differs from Databricks Runtime 16.0 as follows:

  • For GPU clusters, Databricks Runtime ML includes the following NVIDIA GPU libraries:

    • CUDA 12.6

    • cublas 12.6.0.22-1

    • cusolver 11.6.4.38-1

    • cupti 12.6.37-1

    • cusparse 12.5.2.23-1

    • cuDNN 9.3.0.75-1

    • NCCL 2.22.3

    • TensorRT 10.2.0.19-1

Libraries

The following sections list the libraries included in Databricks Runtime 16.0 ML that differ from those included in Databricks Runtime 16.0.

Python libraries

Databricks Runtime 16.0 ML uses virtualenv for Python package management and includes many popular ML packages.

In addition to the packages specified in the following sections, Databricks Runtime 16.0 ML also includes the following packages:

  • hyperopt 0.2.7+db5

  • automl 1.29.0

To reproduce the Databricks Runtime ML Python environment in your local Python virtual environment, download the requirements-16.0.txt file and run pip install -r requirements-16.0.txt. This command installs all of the open source libraries that Databricks Runtime ML uses, but does not install libraries developed by Databricks, such as databricks-automl, databricks-feature-engineering, or the Databricks fork of hyperopt.

Python libraries on CPU clusters

Library

Version

Library

Version

Library

Version

absl-py

1.0.0

accelerate

0.33.0

aiohttp

3.9.5

aiohttp-cors

0.7.0

aiosignal

1.2.0

alembic

1.13.3

annotated-types

0.7.0

anyio

4.2.0

argcomplete

3.5.0

argon2-cffi

21.3.0

argon2-cffi-bindings

21.2.0

arrow

1.2.3

astor

0.8.1

asttokens

2.0.5

astunparse

1.6.3

async-lru

2.0.4

attrs

23.1.0

audioread

3.0.1

autocommand

2.2.2

azure-core

1.31.0

azure-cosmos

4.3.1

azure-identity

1.18.0

azure-storage-blob

12.23.1

azure-storage-file-datalake

12.17.0

Babel

2.11.0

backoff

2.2.1

backports.tarfile

1.2.0

bcrypt

3.2.0

beautifulsoup4

4.12.3

black

24.4.2

bleach

4.1.0

blinker

1.7.0

blis

0.7.11

boto3

1.34.69

botocore

1.34.69

Brotli

1.0.9

cachetools

5.3.3

catalogue

2.0.10

category-encoders

2.6.3

certifi

2024.6.2

cffi

1.16.0

chardet

4.0.0

charset-normalizer

2.0.4

circuitbreaker

2.0.0

click

8.1.7

cloudpathlib

0.19.0

cloudpickle

2.2.1

cmdstanpy

1.2.4

colorful

0.5.6

colorlog

6.8.2

comm

0.2.1

composer

0.24.1

confection

0.1.5

configparser

5.2.0

contourpy

1.2.0

coolname

2.2.0

cryptography

42.0.5

cycler

0.11.0

cymem

2.0.8

Cython

3.0.11

dacite

1.8.1

databricks-automl-runtime

0.2.21

databricks-feature-engineering

0.7.0

databricks-sdk

0.30.0

datasets

2.20.0

dbl-tempo

0.1.26

dbus-python

1.3.2

debugpy

1.6.7

decorator

5.1.1

deepspeed

0.14.4

defusedxml

0.7.1

Deprecated

1.2.14

dill

0.3.8

distlib

0.3.8

dm-tree

0.1.8

docstring-to-markdown

0.11

entrypoints

0.4

evaluate

0.4.2

executing

0.8.3

facets-overview

1.1.1

Farama-Notifications

0.0.4

fastjsonschema

2.20.0

fasttext-wheel

0.9.2

filelock

3.13.1

Flask

2.2.5

flatbuffers

24.3.25

fonttools

4.51.0

fqdn

1.5.1

frozenlist

1.4.0

fsspec

2023.5.0

future

0.18.3

gast

0.4.0

gitdb

4.0.11

GitPython

3.1.37

google-api-core

2.20.0

google-auth

2.21.0

google-auth-oauthlib

1.0.0

google-cloud-core

2.4.1

google-cloud-storage

2.10.0

google-crc32c

1.6.0

google-pasta

0.2.0

google-resumable-media

2.7.2

googleapis-common-protos

1.65.0

gql

3.5.0

graphql-core

3.2.4

greenlet

3.0.1

grpcio

1.60.0

grpcio-status

1.60.0

gunicorn

20.1.0

gviz-api

1.10.0

gymnasium

0.28.1

h11

0.14.0

h5py

3.11.0

hjson

3.1.0

holidays

0.54

htmlmin

0.1.12

httpcore

1.0.5

httplib2

0.20.4

httpx

0.27.2

huggingface-hub

0.24.5

idna

3.7

ImageHash

4.3.1

imageio

2.33.1

imbalanced-learn

0.12.3

importlib-metadata

6.0.0

importlib_resources

6.4.5

inflect

7.3.1

ipyflow-core

0.0.201

ipykernel

6.28.0

ipython

8.25.0

ipython-genutils

0.2.0

ipywidgets

7.7.2

isodate

0.6.1

isoduration

20.11.0

itsdangerous

2.2.0

jaraco.context

5.3.0

jaraco.functools

4.0.1

jaraco.text

3.12.1

jax-jumpy

1.0.0

jedi

0.19.1

Jinja2

3.1.4

jiter

0.5.0

jmespath

1.0.1

joblib

1.4.2

joblibspark

0.5.1

json5

0.9.6

jsonpatch

1.33

jsonpointer

3.0.0

jsonschema

4.19.2

jsonschema-specifications

2023.7.1

jupyter-events

0.10.0

jupyter-lsp

2.2.0

jupyter_client

8.6.0

jupyter_core

5.7.2

jupyter_server

2.14.1

jupyter_server_terminals

0.4.4

jupyterlab

4.0.11

jupyterlab-pygments

0.1.2

jupyterlab_server

2.25.1

keras

3.5.0

kiwisolver

1.4.4

langchain

0.2.12

langchain-core

0.2.41

langchain-text-splitters

0.2.4

langcodes

3.4.1

langsmith

0.1.129

language_data

1.2.0

launchpadlib

1.11.0

lazr.restfulclient

0.14.6

lazr.uri

1.0.6

lazy_loader

0.4

libclang

15.0.6.1

librosa

0.10.2

lightgbm

4.5.0

lightning-utilities

0.11.7

linkify-it-py

2.0.0

llvmlite

0.42.0

lz4

4.3.2

Mako

1.2.0

marisa-trie

1.2.0

Markdown

3.4.1

markdown-it-py

2.2.0

MarkupSafe

2.1.3

matplotlib

3.8.4

matplotlib-inline

0.1.6

mccabe

0.7.0

mdit-py-plugins

0.3.0

mdurl

0.1.0

memray

1.14.0

mistune

2.0.4

ml-dtypes

0.4.1

mlflow-skinny

2.15.1

more-itertools

10.3.0

mosaicml-cli

0.6.41

mosaicml-streaming

0.8.0

mpmath

1.3.0

msal

1.31.0

msal-extensions

1.2.0

msgpack

1.1.0

multidict

6.0.4

multimethod

1.12

multiprocess

0.70.16

murmurhash

1.0.10

mypy

1.10.0

mypy-extensions

1.0.0

namex

0.0.8

nbclient

0.8.0

nbconvert

7.10.0

nbformat

5.9.2

nest-asyncio

1.6.0

networkx

3.2.1

ninja

1.11.1.1

nltk

3.8.1

nodeenv

1.9.1

notebook

7.0.8

notebook_shim

0.2.3

numba

0.59.1

numpy

1.26.4

nvidia-ml-py

12.560.30

oauthlib

3.2.0

oci

2.135.0

openai

1.40.2

opencensus

0.11.4

opencensus-context

0.1.3

opentelemetry-api

1.27.0

opentelemetry-sdk

1.27.0

opentelemetry-semantic-conventions

0.48b0

opt_einsum

3.4.0

optree

0.12.1

optuna

3.6.1

optuna-integration

3.6.0

orjson

3.10.7

overrides

7.4.0

packaging

24.1

pandas

1.5.3

pandocfilters

1.5.0

paramiko

3.4.0

parso

0.8.3

pathspec

0.10.3

patsy

0.5.6

pexpect

4.8.0

phik

0.12.4

pillow

10.3.0

pip

24.2

platformdirs

3.10.0

plotly

5.22.0

pluggy

1.0.0

pmdarima

2.0.4

pooch

1.8.2

portalocker

2.10.1

preshed

3.0.9

prometheus-client

0.14.1

prompt-toolkit

3.0.43

prophet

1.1.5

proto-plus

1.24.0

protobuf

4.24.1

psutil

5.9.0

psycopg2

2.9.3

ptyprocess

0.7.0

pure-eval

0.2.2

py-cpuinfo

9.0.0

py-spy

0.3.14

pyarrow

15.0.2

pyarrow-hotfix

0.6

pyasn1

0.4.8

pyasn1-modules

0.2.8

pybind11

2.13.6

pyccolo

0.0.65

pycparser

2.21

pydantic

2.8.2

pydantic_core

2.20.1

pyflakes

3.2.0

Pygments

2.15.1

PyGObject

3.48.2

PyJWT

2.7.0

PyNaCl

1.5.0

pyodbc

5.0.1

pyOpenSSL

24.0.0

pyparsing

3.0.9

pyright

1.1.294

pytesseract

0.3.10

python-dateutil

2.9.0.post0

python-editor

1.0.4

python-json-logger

2.0.7

python-lsp-jsonrpc

1.1.2

python-lsp-server

1.10.0

python-snappy

0.6.1

pytoolconfig

1.2.6

pytorch-ranger

0.1.1

pytz

2024.1

PyWavelets

1.5.0

PyYAML

6.0.1

pyzmq

25.1.2

questionary

1.10.0

ray

2.35.0

referencing

0.30.2

regex

2023.10.3

requests

2.32.2

requests-oauthlib

1.3.1

rfc3339-validator

0.1.4

rfc3986-validator

0.1.1

rich

13.3.5

rope

1.12.0

rpds-py

0.10.6

rsa

4.9

ruamel.yaml

0.18.6

ruamel.yaml.clib

0.2.8

s3transfer

0.10.2

safetensors

0.4.4

scikit-image

0.23.2

scikit-learn

1.4.2

scipy

1.13.1

seaborn

0.13.2

Send2Trash

1.8.2

sentence-transformers

3.0.1

sentencepiece

0.2.0

setuptools

74.0.0

shap

0.46.0

shellingham

1.5.4

simplejson

3.17.6

six

1.16.0

slicer

0.0.8

smart-open

5.2.1

smmap

5.0.0

sniffio

1.3.0

soundfile

0.12.1

soupsieve

2.5

soxr

0.5.0.post1

spacy

3.7.5

spacy-legacy

3.0.12

spacy-loggers

1.0.5

SQLAlchemy

2.0.30

sqlparse

0.4.2

srsly

2.4.8

ssh-import-id

5.11

stack-data

0.2.0

stanio

0.5.1

statsmodels

0.14.2

sympy

1.12

tabulate

0.9.0

tangled-up-in-unicode

0.2.0

tenacity

8.2.2

tensorboard

2.17.0

tensorboard-data-server

0.7.2

tensorboard-plugin-profile

2.17.0

tensorboardX

2.6.2.2

tensorflow

2.17.0

tensorflow-estimator

2.15.0

termcolor

2.4.0

terminado

0.17.1

textual

0.81.0

tf_keras

2.17.0

thinc

8.2.5

threadpoolctl

2.2.0

tifffile

2023.4.12

tiktoken

0.7.0

tinycss2

1.2.1

tokenize-rt

4.2.1

tokenizers

0.19.1

tomli

2.0.1

torch

2.4.0+cpu

torch-optimizer

0.3.0

torcheval

0.0.7

torchmetrics

1.4.0.post0

torchvision

0.19.0+cpu

tornado

6.4.1

tqdm

4.66.4

traitlets

5.14.3

transformers

4.44.0

typeguard

4.3.0

typer

0.12.5

types-protobuf

3.20.3

types-psutil

5.9.0

types-pytz

2023.3.1.1

types-PyYAML

6.0.0

types-requests

2.31.0.0

types-setuptools

68.0.0.0

types-six

1.16.0

types-urllib3

1.26.25.14

typing_extensions

4.11.0

uc-micro-py

1.0.1

ujson

5.10.0

unattended-upgrades

0.1

uri-template

1.3.0

urllib3

1.26.16

validators

0.34.0

virtualenv

20.26.2

visions

0.7.5

wadllib

1.3.6

wasabi

1.1.3

wcwidth

0.2.5

weasel

0.4.1

webcolors

24.8.0

webencodings

0.5.1

websocket-client

1.8.0

websockets

11.0.3

Werkzeug

3.0.3

whatthepatch

1.0.2

wheel

0.43.0

wordcloud

1.9.3

wrapt

1.14.1

xgboost

2.0.3

xgboost-ray

0.1.19

xxhash

3.4.1

yapf

0.33.0

yarl

1.9.3

ydata-profiling

4.9.0

zipp

3.17.0

zstd

1.5.5.1

Python libraries on GPU clusters

Library

Version

Library

Version

Library

Version

absl-py

1.0.0

accelerate

0.33.0

aiohttp

3.9.5

aiohttp-cors

0.7.0

aiosignal

1.2.0

annotated-types

0.7.0

anyio

4.2.0

argcomplete

3.5.0

argon2-cffi

21.3.0

argon2-cffi-bindings

21.2.0

arrow

1.2.3

astor

0.8.1

asttokens

2.0.5

astunparse

1.6.3

async-lru

2.0.4

attrs

23.1.0

audioread

3.0.1

autocommand

2.2.2

azure-core

1.31.0

azure-cosmos

4.3.1

azure-identity

1.18.0

azure-storage-blob

12.23.1

azure-storage-file-datalake

12.17.0

Babel

2.11.0

backoff

2.2.1

backports.tarfile

1.2.0

bcrypt

3.2.0

beautifulsoup4

4.12.3

black

24.4.2

bleach

4.1.0

blinker

1.7.0

blis

0.7.11

boto3

1.34.69

botocore

1.34.69

Brotli

1.0.9

cachetools

5.3.3

catalogue

2.0.10

category-encoders

2.6.3

certifi

2024.6.2

cffi

1.16.0

chardet

4.0.0

charset-normalizer

2.0.4

circuitbreaker

2.0.0

click

8.1.7

cloudpathlib

0.19.0

cloudpickle

2.2.1

cmdstanpy

1.2.4

colorful

0.5.6

colorlog

6.8.2

comm

0.2.1

composer

0.24.1

confection

0.1.5

configparser

5.2.0

contourpy

1.2.0

coolname

2.2.0

cryptography

42.0.5

cycler

0.11.0

cymem

2.0.8

Cython

3.0.11

dacite

1.8.1

databricks-automl-runtime

0.2.21

databricks-feature-engineering

0.7.0

databricks-sdk

0.30.0

datasets

2.20.0

dbl-tempo

0.1.26

dbus-python

1.3.2

debugpy

1.6.7

decorator

5.1.1

deepspeed

0.14.4

defusedxml

0.7.1

Deprecated

1.2.14

dill

0.3.8

distlib

0.3.8

dm-tree

0.1.8

docstring-to-markdown

0.11

einops

0.8.0

entrypoints

0.4

evaluate

0.4.2

executing

0.8.3

facets-overview

1.1.1

Farama-Notifications

0.0.4

fastjsonschema

2.20.0

fasttext-wheel

0.9.2

filelock

3.13.1

flash_attn

2.5.6

Flask

2.2.5

flatbuffers

24.3.25

fonttools

4.51.0

fqdn

1.5.1

frozenlist

1.4.0

fsspec

2023.5.0

future

0.18.3

gast

0.4.0

gitdb

4.0.11

GitPython

3.1.37

google-api-core

2.20.0

google-auth

2.21.0

google-auth-oauthlib

1.0.0

google-cloud-core

2.4.1

google-cloud-storage

2.10.0

google-crc32c

1.6.0

google-pasta

0.2.0

google-resumable-media

2.7.2

googleapis-common-protos

1.65.0

gql

3.5.0

graphql-core

3.2.4

greenlet

3.0.1

grpcio

1.60.0

grpcio-status

1.60.0

gunicorn

20.1.0

gviz-api

1.10.0

gymnasium

0.28.1

h11

0.14.0

h5py

3.11.0

hjson

3.1.0

holidays

0.54

htmlmin

0.1.12

httpcore

1.0.5

httplib2

0.20.4

httpx

0.27.2

huggingface-hub

0.24.5

idna

3.7

ImageHash

4.3.1

imageio

2.33.1

imbalanced-learn

0.12.3

importlib-metadata

6.0.0

importlib_resources

6.4.5

inflect

7.3.1

ipyflow-core

0.0.201

ipykernel

6.28.0

ipython

8.25.0

ipython-genutils

0.2.0

ipywidgets

7.7.2

isodate

0.6.1

isoduration

20.11.0

itsdangerous

2.2.0

jaraco.context

5.3.0

jaraco.functools

4.0.1

jaraco.text

3.12.1

jax-jumpy

1.0.0

jedi

0.19.1

Jinja2

3.1.4

jiter

0.5.0

jmespath

1.0.1

joblib

1.4.2

joblibspark

0.5.1

json5

0.9.6

jsonpatch

1.33

jsonpointer

3.0.0

jsonschema

4.19.2

jsonschema-specifications

2023.7.1

jupyter-events

0.10.0

jupyter-lsp

2.2.0

jupyter_client

8.6.0

jupyter_core

5.7.2

jupyter_server

2.14.1

jupyter_server_terminals

0.4.4

jupyterlab

4.0.11

jupyterlab-pygments

0.1.2

jupyterlab_server

2.25.1

keras

3.5.0

kiwisolver

1.4.4

langchain

0.2.12

langchain-core

0.2.41

langchain-text-splitters

0.2.4

langcodes

3.4.1

langsmith

0.1.129

language_data

1.2.0

launchpadlib

1.11.0

lazr.restfulclient

0.14.6

lazr.uri

1.0.6

lazy_loader

0.4

libclang

15.0.6.1

librosa

0.10.2

lightgbm

4.5.0

lightning-utilities

0.11.7

linkify-it-py

2.0.0

llvmlite

0.42.0

lz4

4.3.2

Mako

1.2.0

marisa-trie

1.2.0

Markdown

3.4.1

markdown-it-py

2.2.0

MarkupSafe

2.1.3

matplotlib

3.8.4

matplotlib-inline

0.1.6

mccabe

0.7.0

mdit-py-plugins

0.3.0

mdurl

0.1.0

memray

1.14.0

mistune

2.0.4

ml-dtypes

0.4.1

mlflow-skinny

2.15.1

more-itertools

10.3.0

mosaicml-cli

0.6.41

mosaicml-streaming

0.8.0

mpmath

1.3.0

msal

1.31.0

msal-extensions

1.2.0

msgpack

1.1.0

multidict

6.0.4

multimethod

1.12

multiprocess

0.70.16

murmurhash

1.0.10

mypy

1.10.0

mypy-extensions

1.0.0

namex

0.0.8

nbclient

0.8.0

nbconvert

7.10.0

nbformat

5.9.2

nest-asyncio

1.6.0

networkx

3.2.1

ninja

1.11.1.1

nltk

3.8.1

nodeenv

1.9.1

notebook

7.0.8

notebook_shim

0.2.3

numba

0.59.1

numpy

1.26.4

nvidia-cublas-cu12

12.4.2.65

nvidia-cuda-cupti-cu12

12.4.99

nvidia-cuda-nvrtc-cu12

12.4.99

nvidia-cuda-runtime-cu12

12.4.99

nvidia-cudnn-cu12

9.1.0.70

nvidia-cufft-cu12

11.2.0.44

nvidia-curand-cu12

10.3.5.119

nvidia-cusolver-cu12

11.6.0.99

nvidia-cusparse-cu12

12.3.0.142

nvidia-ml-py

12.560.30

nvidia-nccl-cu12

2.20.5

nvidia-nvjitlink-cu12

12.4.99

nvidia-nvtx-cu12

12.4.99

oauthlib

3.2.0

oci

2.135.0

openai

1.40.2

opencensus

0.11.4

opencensus-context

0.1.3

opentelemetry-api

1.27.0

opentelemetry-sdk

1.27.0

opentelemetry-semantic-conventions

0.48b0

opt_einsum

3.4.0

optree

0.12.1

optuna

3.6.1

optuna-integration

3.6.0

orjson

3.10.7

overrides

7.4.0

packaging

24.1

pandas

1.5.3

pandocfilters

1.5.0

paramiko

3.4.0

parso

0.8.3

pathspec

0.10.3

patsy

0.5.6

pexpect

4.8.0

phik

0.12.4

pillow

10.3.0

pip

24.2

platformdirs

3.10.0

plotly

5.22.0

pluggy

1.0.0

pmdarima

2.0.4

pooch

1.8.2

portalocker

2.10.1

preshed

3.0.9

prometheus-client

0.14.1

prompt-toolkit

3.0.43

prophet

1.1.5

proto-plus

1.24.0

protobuf

4.24.1

psutil

5.9.0

psycopg2

2.9.3

ptyprocess

0.7.0

pure-eval

0.2.2

py-cpuinfo

9.0.0

py-spy

0.3.14

pyarrow

15.0.2

pyarrow-hotfix

0.6

pyasn1

0.4.8

pyasn1-modules

0.2.8

pybind11

2.13.6

pyccolo

0.0.65

pycparser

2.21

pydantic

2.8.2

pydantic_core

2.20.1

pyflakes

3.2.0

Pygments

2.15.1

PyGObject

3.48.2

PyJWT

2.7.0

PyNaCl

1.5.0

pyodbc

5.0.1

pyOpenSSL

24.0.0

pyparsing

3.0.9

pyright

1.1.294

pytesseract

0.3.10

python-dateutil

2.9.0.post0

python-editor

1.0.4

python-json-logger

2.0.7

python-lsp-jsonrpc

1.1.2

python-lsp-server

1.10.0

python-snappy

0.6.1

pytoolconfig

1.2.6

pytorch-ranger

0.1.1

pytz

2024.1

PyWavelets

1.5.0

PyYAML

6.0.1

pyzmq

25.1.2

questionary

1.10.0

ray

2.35.0

referencing

0.30.2

regex

2023.10.3

requests

2.32.2

requests-oauthlib

1.3.1

rfc3339-validator

0.1.4

rfc3986-validator

0.1.1

rich

13.3.5

rope

1.12.0

rpds-py

0.10.6

rsa

4.9

ruamel.yaml

0.18.6

ruamel.yaml.clib

0.2.8

s3transfer

0.10.2

safetensors

0.4.4

scikit-image

0.23.2

scikit-learn

1.4.2

scipy

1.13.1

seaborn

0.13.2

Send2Trash

1.8.2

sentence-transformers

3.0.1

sentencepiece

0.2.0

setuptools

74.0.0

shap

0.46.0

shellingham

1.5.4

simplejson

3.17.6

six

1.16.0

slicer

0.0.8

smart-open

5.2.1

smmap

5.0.0

sniffio

1.3.0

soundfile

0.12.1

soupsieve

2.5

soxr

0.5.0.post1

spacy

3.7.5

spacy-legacy

3.0.12

spacy-loggers

1.0.5

SQLAlchemy

2.0.30

sqlparse

0.4.2

srsly

2.4.8

ssh-import-id

5.11

stack-data

0.2.0

stanio

0.5.1

statsmodels

0.14.2

sympy

1.12

tabulate

0.9.0

tangled-up-in-unicode

0.2.0

tenacity

8.2.2

tensorboard

2.17.0

tensorboard-data-server

0.7.2

tensorboard-plugin-profile

2.17.0

tensorboardX

2.6.2.2

tensorflow

2.17.0

tensorflow-estimator

2.15.0

termcolor

2.4.0

terminado

0.17.1

textual

0.81.0

tf_keras

2.17.0

thinc

8.2.5

threadpoolctl

2.2.0

tifffile

2023.4.12

tiktoken

0.7.0

tinycss2

1.2.1

tokenize-rt

4.2.1

tokenizers

0.19.1

tomli

2.0.1

torch

2.4.0+cu124

torch-optimizer

0.3.0

torcheval

0.0.7

torchmetrics

1.4.0.post0

torchvision

0.19.0+cu124

tornado

6.4.1

tqdm

4.66.4

traitlets

5.14.3

transformers

4.44.0

triton

3.0.0

typeguard

4.3.0

typer

0.12.5

types-protobuf

3.20.3

types-psutil

5.9.0

types-pytz

2023.3.1.1

types-PyYAML

6.0.0

types-requests

2.31.0.0

types-setuptools

68.0.0.0

types-six

1.16.0

types-urllib3

1.26.25.14

typing_extensions

4.11.0

uc-micro-py

1.0.1

ujson

5.10.0

unattended-upgrades

0.1

uri-template

1.3.0

urllib3

1.26.16

validators

0.34.0

virtualenv

20.26.2

visions

0.7.5

wadllib

1.3.6

wasabi

1.1.3

wcwidth

0.2.5

weasel

0.4.1

webcolors

24.8.0

webencodings

0.5.1

websocket-client

1.8.0

websockets

11.0.3

Werkzeug

3.0.3

whatthepatch

1.0.2

wheel

0.43.0

wordcloud

1.9.3

wrapt

1.14.1

xgboost

2.0.3

xgboost-ray

0.1.19

xxhash

3.4.1

yapf

0.33.0

yarl

1.9.3

ydata-profiling

4.9.0

zipp

3.17.0

zstd

1.5.5.1

R libraries

The R libraries are identical to the R Libraries in Databricks Runtime 16.0.

Java and Scala libraries (Scala 2.12 cluster)

In addition to Java and Scala libraries in Databricks Runtime 16.0, Databricks Runtime 16.0 ML contains the following JARs:

CPU clusters

Group ID

Artifact ID

Version

com.typesafe.akka

akka-actor_2.12

2.5.23

ml.dmlc

xgboost4j-spark_2.12

1.7.3

ml.dmlc

xgboost4j_2.12

1.7.3

org.graphframes

graphframes_2.12

0.8.4-db1-spark3.5

org.mlflow

mlflow-client

2.15.1

org.tensorflow

spark-tensorflow-connector_2.12

1.15.0

GPU clusters

Group ID

Artifact ID

Version

com.typesafe.akka

akka-actor_2.12

2.5.23

ml.dmlc

xgboost4j-gpu_2.12

1.7.3

ml.dmlc

xgboost4j-spark-gpu_2.12

1.7.3

org.graphframes

graphframes_2.12

0.8.4-db1-spark3.5

org.mlflow

mlflow-client

2.15.1

org.tensorflow

spark-tensorflow-connector_2.12

1.15.0