Databricks Runtime 8.0 (EoS)

Note

Support for this Databricks Runtime version has ended. For the end-of-support date, see End-of-support history. For all supported Databricks Runtime versions, see Databricks Runtime release notes versions and compatibility.

Databricks released this version in March 2021.

The following release notes provide information about Databricks Runtime 8.0, powered by Apache Spark 3.1.1.

Note

These release notes may include references to features that are not available on Google Cloud as of this release.

New features

Databricks Runtime 8.0 includes Apache Spark 3.1.1. For details, see Apache Spark.

Improvements

Delta is now the default format when a format is not specified

Databricks Runtime 8.0 changes the default format to delta to make it simpler to create a Delta table. When you create a table using SQL commands, or {Dataset|DataFrame}.{read|readStream|write|writeTo|writeStream} APIs, and you do not specify a format, the default format is delta.

With Delta Lake, you get better performance over Parquet, better data reliability with rich schema validation, quality constraints, and transactional guarantees. With Delta Lake, you can simplify your data pipelines with unified structured streaming and batch processing on a single data source.

While Databricks recommends using Delta Lake to store your data, you may have legacy workflows that require migration to Delta Lake. For information about migrating existing workflows, see What is Delta Lake?.

New Structured Streaming default trigger interval reduces costs

If you do not set a trigger interval using Trigger.ProcessingTime in your streaming query, the interval is set to 500 ms. Previously, the default interval was 0 ms. This change should reduce the number of empty triggers and reduce costs for cloud storage such as listing.

Use the LDA transform function with credential passthrough (Public Preview)

You can now use the LDA transform function on a cluster configured to use credential passthrough for authentication.

Single user clusters configured with credential passthrough no longer require trusted filesystems (Public Preview)

You no longer need to configure local filesystems as trusted filesystems when using a standard or job cluster configured for credential passthrough with a single user. This change removes unnecessary filesystem restrictions when running jobs on a single user cluster.

Library upgrades

Apache Spark

Databricks Runtime 8.0 includes Apache Spark 3.1.1.

Core and Spark SQL

Highlight

ANSI SQL Compatibility Enhancements

Performance enhancements

  • Host-local shuffle data reading without shuffle service (SPARK-32077)

  • Remove redundant sorts before repartition nodes (SPARK-32276)

  • Partially push down predicates (SPARK-32302, SPARK-32352)

  • Push down filters through expand (SPARK-33302)

  • Push more possible predicates through Join via CNF conversion (SPARK-31705)

  • Remove shuffle by preserving output partitioning of broadcast hash join (SPARK-31869)

  • Remove shuffle by improving reordering join keys (SPARK-32282)

  • Remove shuffle by normalizing output partitioning and sortorder (SPARK-33399)

  • Shuffled hash join improvement (SPARK-32461)

    • Preserve shuffled hash join build side partitioning (SPARK-32330)

    • Preserve hash join (BHJ and SHJ) stream side ordering (SPARK-32383)

    • Coalesce bucketed tables for sort merge join (SPARK-32286)

    • Add code-gen for shuffled hash join (SPARK-32421)

    • Support full outer join in shuffled hash join (SPARK-32399)

  • Support subexpression elimination in project with whole-stage-codegen (SPARK-33092)

  • Support subexpression elimination in conditional expressions (SPARK-33337)

  • Support subexpression elimination for interpreted expression evaluation (SPARK-33427)

  • Support subexpression elimination for interpreted predicate (SPARK-33540)

  • Other optimizer rules

    • Rule ExtractSingleColumnNullAwareAntiJoin (SPARK-32290)

    • Rule EliminateNullAwareAntiJoin (SPARK-32573)

    • Rule EliminateAggregateFilter (SPARK-32540)

    • Rule UnwrapCastInBinaryComparison (SPARK-32858)

    • Rule DisableUnnecessaryBucketedScan (SPARK-32859)

    • Rule CoalesceBucketsInJoin (SPARK-31350)

    • Prune unnecessary nested fields from generate without project (SPARK-29721)

    • Prune unnecessary nested fields from aggregate and expand (SPARK-27217)

    • Prune unnecessary nested fields from repartition-by-expression and join (SPARK-31736)

    • Prune unnecessary nested fields over cosmetic variations (SPARK-32163)

    • Prune unnecessary nested fields from window and sort (SPARK-32059)

    • Optimize size of CreateArray/CreateMap to be the size of its children (SPARK-33544)

Extensibility enhancements

  • Add SupportsPartitions APIs on DataSourceV2 (SPARK-31694)

  • Add SupportsMetadataColumns API on DataSourceV2 (SPARK-31255)

  • Make SQL cache serialization pluggable (SPARK-32274)

  • Introduce the purge option in TableCatalog.dropTable for v2 catalog (SPARK-33364)

Connector enhancements

Feature enhancements

Other notable changes

  • Provide Search Function in Spark docs site (SPARK-33166)

  • Upgrade Apache Arrow to 2.0.0 (SPARK-33213)

  • Enable Java 8 time API in thrift server (SPARK-31910)

  • Enable Java 8 time API in UDFs (SPARK-32154)

  • Overflow check for aggregate sum with decimals (SPARK-28067)

  • Fix commit collision in dynamic partition overwrite mode (SPARK-27194, SPARK-29302)

  • Removed references to slave, blacklist and whitelist (SPARK-32004, SPARK-32036, SPARK-32037)

  • Remove task result size check for shuffle map stage (SPARK-32470)

  • Generalize ExecutorSource to expose user-given file system schemes (SPARK-33476)

  • Add StorageLevel.DISK_ONLY_3 (SPARK-32517)

  • Expose executor memory metrics in the web UI for executors (SPARK-23432)

  • Expose executor memory metrics at the stage level, in the Stages tab (SPARK-26341)

  • Fix explicitly set of spark.ui.port in YARN cluster mode (SPARK-29465)

  • Add spark.submit.waitForCompletion configuration to control spark-submit exit in Standalone cluster mode (SPARK-31486)

  • Set up yarn.Client to print direct links to driver stdout/stderr (SPARK-33185)

  • Fix memory leak when fail to store pieces of broadcast (SPARK-32715)

  • Make BlockManagerMaster driver heartbeat timeout configurable (SPARK-34278)

  • Unify and complete cache behaviors (SPARK-33507)

PySpark

Project Zen

Other notable changes

  • Deduplicate deterministic PythonUDF calls (SPARK-33303)

  • Support higher order functions in PySpark functions(SPARK-30681)

  • Support data source v2x write APIs (SPARK-29157)

  • Support percentile_approx in PySpark functions(SPARK-30569)

  • Support inputFiles in PySpark DataFrame (SPARK-31763)

  • Support withField in PySpark Column (SPARK-32835)

  • Support dropFields in PySpark Column (SPARK-32511)

  • Support nth_value in PySpark functions (SPARK-33020)

  • Support acosh, asinh and atanh (SPARK-33563)

  • Support getCheckpointDir method in PySpark SparkContext (SPARK-33017)

  • Support to fill nulls for missing columns in unionByName (SPARK-32798)

  • Update cloudpickle to v1.5.0 (SPARK-32094)

  • Add MapType support for PySpark with Arrow (SPARK-24554)

  • DataStreamReader.table and DataStreamWriter.toTable (SPARK-33836)

Structured Streaming

Performance enhancements

  • Cache fetched list of files beyond maxFilesPerTrigger as unread file (SPARK-30866)

  • Streamline the logic on file stream source and sink metadata log (SPARK-30462)

  • Avoid reading compact metadata log twice if the query restarts from compact batch (SPARK-30900)

Feature enhancements

  • Add DataStreamReader.table API (SPARK-32885)

  • Add DataStreamWriter.toTable API (SPARK-32896)

  • Left semi stream-stream join (SPARK-32862)

  • Full outer stream-stream join (SPARK-32863)

  • Provide a new option to have retention on output files (SPARK-27188)

  • Add Spark Structured Streaming History Server Support (SPARK-31953)

  • Introduce State schema validation among query restart (SPARK-27237)

Other notable changes

  • Introduce schema validation for streaming state store (SPARK-31894)

  • Support to use a different compression codec in state store (SPARK-33263)

  • Kafka connector infinite wait because metadata never updated (SPARK-28367)

  • Upgrade Kafka to 2.6.0 (SPARK-32568)

  • Pagination support for Structured Streaming UI pages (SPARK-31642, SPARK-30119)

  • State information in Structured Streaming UI (SPARK-33223)

  • Watermark gap information in Structured Streaming UI (SPARK-33224)

  • Expose state custom metrics information on SS UI (SPARK-33287)

  • Add a new metric regarding number of rows later than watermark (SPARK-24634)

MLlib

Highlights

  • LinearSVC blockify input vectors (SPARK-30642)

  • LogisticRegression blockify input vectors (SPARK-30659)

  • LinearRegression blockify input vectors (SPARK-30660)

  • AFT blockify input vectors (SPARK-31656)

  • Add support for association rules in ML (SPARK-19939)

  • Add training summary for LinearSVCModel (SPARK-20249)

  • Add summary to RandomForestClassificationModel (SPARK-23631)

  • Add training summary to FMClassificationModel (SPARK-32140)

  • Add summary to MultilayerPerceptronClassificationModel (SPARK-32449)

  • Add FMClassifier to SparkR (SPARK-30820)

  • Add SparkR LinearRegression wrapper (SPARK-30818)

  • Add FMRegressor wrapper to SparkR (SPARK-30819)

  • Add SparkR wrapper for vector_to_array (SPARK-33040)

  • adaptively blockify instances - LinearSVC (SPARK-32907)

  • make CrossValidator/TrainValidateSplit/OneVsRest Reader/Writer support Python backend estimator/evaluator (SPARK-33520)

  • Improve performance of ML ALS recommendForAll by GEMV (SPARK-33518)

  • Add UnivariateFeatureSelector (SPARK-34080)

Other Notable Changes

  • GMM compute summary and update distributions in one job (SPARK-31032)

  • Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel (SPARK-31077)

  • Flatten the result dataframe of tests in testChiSquare (SPARK-31301)

  • MinHash keyDistance optimization (SPARK-31436)

  • KMeans optimization based on triangle-inequality (SPARK-31007)

  • Add weight support in ClusteringEvaluator (SPARK-31734)

  • Add getMetrics in Evaluators (SPARK-31768)

  • Add instance weight support in LinearRegressionSummary (SPARK-31944)

  • Add user-specified fold column to CrossValidator (SPARK-31777)

  • ML params default value parity in feature and tuning (SPARK-32310)

  • Fix double caching in KMeans/BiKMeans (SPARK-32676)

  • aft transform optimization (SPARK-33111)

  • FeatureHasher transform optimization (SPARK-32974)

  • Add array_to_vector function for dataframe column (SPARK-33556)

  • ML params default value parity in classification, regression, clustering and fpm (SPARK-32310)

  • Summary.totalIterations greater than maxIters (SPARK-31925)

  • tree models prediction optimization (SPARK-32298)

SparkR

  • Add SparkR interface for higher order functions (SPARK-30682)

  • Support to fill nulls for missing columns in unionByName (SPARK-32798)

  • Support withColumn in SparkR functions (SPARK-32946)

  • Support timestamp_seconds in SparkR functions (SPARK-32949)

  • Support nth_value in SparkR functions (SPARK-33030)

  • Minimum Arrow version bumped up to 1.0.0 (SPARK-32452)

  • Support array_to_vector in SparkR functions (SPARK-33622)

  • Support acosh, asinh and atanh (SPARK-33563)

  • Support from_avro and to_avro (SPARK-33304)

System environment

  • Operating System: Ubuntu 18.04.5 LTS

  • Java: Zulu 8.50.0.51-CA-linux64 (build 1.8.0_275-b01)

  • Scala: 2.12.10

  • Python: 3.8.8 (updated from 3.8.6 in May 26, 2021 maintenance update)

  • R: R version 4.0.3 (2020-10-10)

  • Delta Lake 0.8.0

Installed Python libraries

Library

Version

Library

Version

Library

Version

appdirs

1.4.4

asn1crypto

1.4.0

backcall

0.2.0

boto3

1.16.7

botocore

1.19.7

brotlipy

0.7.0

certifi

2020.12.5

cffi

1.14.3

chardet

3.0.4

cryptography

3.1.1

cycler

0.10.0

Cython

0.29.21

decorator

4.4.2

distlib

0.3.1

docutils

0.15.2

entrypoints

0.3

filelock

3.0.12

idna

2.10

ipykernel

5.3.4

ipython

7.19.0

ipython-genutils

0.2.0

jedi

0.17.2

jmespath

0.10.0

joblib

0.17.0

jupyter-client

6.1.7

jupyter-core

4.6.3

kiwisolver

1.3.0

koalas

1.5.0

matplotlib

3.2.2

numpy

1.19.2

pandas

1.1.3

parso

0.7.0

patsy

0.5.1

pexpect

4.8.0

pickleshare

0.7.5

pip

20.2.4

prompt-toolkit

3.0.8

psycopg2

2.8.5

ptyprocess

0.6.0

pyarrow

1.0.1

pycparser

2.20

Pygments

2.7.2

pyOpenSSL

19.1.0

pyparsing

2.4.7

PySocks

1.7.1

python-dateutil

2.8.1

pytz

2020.1

pyzmq

19.0.2

requests

2.24.0

s3transfer

0.3.3

scikit-learn

0.23.2

scipy

1.5.2

seaborn

0.10.0

setuptools

50.3.1

six

1.15.0

statsmodels

0.12.0

threadpoolctl

2.1.0

tornado

6.0.4

traitlets

5.0.5

urllib3

1.25.11

virtualenv

20.2.1

wcwidth

0.2.5

wheel

0.35.1

Installed R libraries

R libraries are installed from the Microsoft CRAN snapshot on 2020-11-02.

Library

Version

Library

Version

Library

Version

askpass

1.1

assertthat

0.2.1

backports

1.2.1

base

4.0.3

base64enc

0.1-3

BH

1.72.0-3

bit

4.0.4

bit64

4.0.5

blob

1.2.1

boot

1.3-25

brew

1.0-6

brio

1.1.0

broom

0.7.2

callr

3.5.1

caret

6.0-86

cellranger

1.1.0

chron

2.3-56

class

7.3-17

cli

2.2.0

clipr

0.7.1

cluster

2.1.0

codetools

0.2-18

colorspace

2.0-0

commonmark

1.7

compiler

4.0.3

config

0.3

covr

3.5.1

cpp11

0.2.4

crayon

1.3.4

credentials

1.3.0

crosstalk

1.1.0.1

curl

4.3

data.table

1.13.4

datasets

4.0.3

DBI

1.1.0

dbplyr

2.0.0

desc

1.2.0

devtools

2.3.2

diffobj

0.3.2

digest

0.6.27

dplyr

1.0.2

DT

0.16

ellipsis

0.3.1

evaluate

0.14

fansi

0.4.1

farver

2.0.3

fastmap

1.0.1

forcats

0.5.0

foreach

1.5.1

foreign

0.8-79

forge

0.2.0

fs

1.5.0

future

1.21.0

generics

0.1.0

gert

1.0.2

ggplot2

3.3.2

gh

1.2.0

gitcreds

0.1.1

glmnet

4.0-2

globals

0.14.0

glue

1.4.2

gower

0.2.2

graphics

4.0.3

grDevices

4.0.3

grid

4.0.3

gridExtra

2.3

gsubfn

0.7

gtable

0.3.0

haven

2.3.1

highr

0.8

hms

0.5.3

htmltools

0.5.0

htmlwidgets

1.5.3

httpuv

1.5.4

httr

1.4.2

hwriter

1.3.2

hwriterPlus

1.0-3

ini

0.3.1

ipred

0.9-9

isoband

0.2.3

iterators

1.0.13

jsonlite

1.7.2

KernSmooth

2.23-18

knitr

1.30

labeling

0.4.2

later

1.1.0.1

lattice

0.20-41

lava

1.6.8.1

lazyeval

0.2.2

lifecycle

0.2.0

listenv

0.8.0

lubridate

1.7.9.2

magrittr

2.0.1

markdown

1.1

MASS

7.3-53

Matrix

1.2-18

memoise

1.1.0

methods

4.0.3

mgcv

1.8-33

mime

0.9

ModelMetrics

1.2.2.2

modelr

0.1.8

munsell

0.5.0

nlme

3.1-151

nnet

7.3-14

numDeriv

2016.8-1.1

openssl

1.4.3

parallel

4.0.3

parallelly

1.22.0

pillar

1.4.7

pkgbuild

1.1.0

pkgconfig

2.0.3

pkgload

1.1.0

plogr

0.2.0

plyr

1.8.6

praise

1.0.0

prettyunits

1.1.1

pROC

1.16.2

processx

3.4.5

prodlim

2019.11.13

progress

1.2.2

promises

1.1.1

proto

1.0.0

ps

1.5.0

purrr

0.3.4

r2d3

0.2.3

R6

2.5.0

randomForest

4.6-14

rappdirs

0.3.1

rcmdcheck

1.3.3

RColorBrewer

1.1-2

Rcpp

1.0.5

readr

1.4.0

readxl

1.3.1

recipes

0.1.15

rematch

1.0.1

rematch2

2.1.2

remotes

2.2.0

reprex

0.3.0

reshape2

1.4.4

rex

1.2.0

rlang

0.4.9

rmarkdown

2.6

RODBC

1.3-17

roxygen2

7.1.1

rpart

4.1-15

rprojroot

2.0.2

Rserve

1.8-7

RSQLite

2.2.1

rstudioapi

0.13

rversions

2.0.2

rvest

0.3.6

scales

1.1.1

selectr

0.4-2

sessioninfo

1.1.1

shape

1.4.5

shiny

1.5.0

sourcetools

0.1.7

sparklyr

1.5.2

SparkR

3.1.0

spatial

7.3-11

splines

4.0.3

sqldf

0.4-11

SQUAREM

2020.5

stats

4.0.3

stats4

4.0.3

stringi

1.5.3

stringr

1.4.0

survival

3.2-7

sys

3.4

tcltk

4.0.3

TeachingDemos

2.10

testthat

3.0.0

tibble

3.0.4

tidyr

1.1.2

tidyselect

1.1.0

tidyverse

1.3.0

timeDate

3043.102

tinytex

0.28

tools

4.0.3

usethis

2.0.0

utf8

1.1.4

utils

4.0.3

uuid

0.1-4

vctrs

0.3.5

viridisLite

0.3.0

waldo

0.2.3

whisker

0.4

withr

2.3.0

xfun

0.19

xml2

1.3.2

xopen

1.0.0

xtable

1.8-4

yaml

2.2.1

zip

2.1.1

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID

Artifact ID

Version

antlr

antlr

2.7.7

com.amazonaws

amazon-kinesis-client

1.12.0

com.amazonaws

aws-java-sdk-autoscaling

1.11.655

com.amazonaws

aws-java-sdk-cloudformation

1.11.655

com.amazonaws

aws-java-sdk-cloudfront

1.11.655

com.amazonaws

aws-java-sdk-cloudhsm

1.11.655

com.amazonaws

aws-java-sdk-cloudsearch

1.11.655

com.amazonaws

aws-java-sdk-cloudtrail

1.11.655

com.amazonaws

aws-java-sdk-cloudwatch

1.11.655

com.amazonaws

aws-java-sdk-cloudwatchmetrics

1.11.655

com.amazonaws

aws-java-sdk-codedeploy

1.11.655

com.amazonaws

aws-java-sdk-cognitoidentity

1.11.655

com.amazonaws

aws-java-sdk-cognitosync

1.11.655

com.amazonaws

aws-java-sdk-config

1.11.655

com.amazonaws

aws-java-sdk-core

1.11.655

com.amazonaws

aws-java-sdk-datapipeline

1.11.655

com.amazonaws

aws-java-sdk-directconnect

1.11.655

com.amazonaws

aws-java-sdk-directory

1.11.655

com.amazonaws

aws-java-sdk-dynamodb

1.11.655

com.amazonaws

aws-java-sdk-ec2

1.11.655

com.amazonaws

aws-java-sdk-ecs

1.11.655

com.amazonaws

aws-java-sdk-efs

1.11.655

com.amazonaws

aws-java-sdk-elasticache

1.11.655

com.amazonaws

aws-java-sdk-elasticbeanstalk

1.11.655

com.amazonaws

aws-java-sdk-elasticloadbalancing

1.11.655

com.amazonaws

aws-java-sdk-elastictranscoder

1.11.655

com.amazonaws

aws-java-sdk-emr

1.11.655

com.amazonaws

aws-java-sdk-glacier

1.11.655

com.amazonaws

aws-java-sdk-iam

1.11.655

com.amazonaws

aws-java-sdk-importexport

1.11.655

com.amazonaws

aws-java-sdk-kinesis

1.11.655

com.amazonaws

aws-java-sdk-kms

1.11.655

com.amazonaws

aws-java-sdk-lambda

1.11.655

com.amazonaws

aws-java-sdk-logs

1.11.655

com.amazonaws

aws-java-sdk-machinelearning

1.11.655

com.amazonaws

aws-java-sdk-opsworks

1.11.655

com.amazonaws

aws-java-sdk-rds

1.11.655

com.amazonaws

aws-java-sdk-redshift

1.11.655

com.amazonaws

aws-java-sdk-route53

1.11.655

com.amazonaws

aws-java-sdk-s3

1.11.655

com.amazonaws

aws-java-sdk-ses

1.11.655

com.amazonaws

aws-java-sdk-simpledb

1.11.655

com.amazonaws

aws-java-sdk-simpleworkflow

1.11.655

com.amazonaws

aws-java-sdk-sns

1.11.655

com.amazonaws

aws-java-sdk-sqs

1.11.655

com.amazonaws

aws-java-sdk-ssm

1.11.655

com.amazonaws

aws-java-sdk-storagegateway

1.11.655

com.amazonaws

aws-java-sdk-sts

1.11.655

com.amazonaws

aws-java-sdk-support

1.11.655

com.amazonaws

aws-java-sdk-swf-libraries

1.11.22

com.amazonaws

aws-java-sdk-workspaces

1.11.655

com.amazonaws

jmespath-java

1.11.655

com.chuusai

shapeless_2.12

2.3.3

com.clearspring.analytics

stream

2.9.6

com.databricks

Rserve

1.8-3

com.databricks

jets3t

0.7.1-0

com.databricks.scalapb

compilerplugin_2.12

0.4.15-10

com.databricks.scalapb

scalapb-runtime_2.12

0.4.15-10

com.esotericsoftware

kryo-shaded

4.0.2

com.esotericsoftware

minlog

1.3.0

com.fasterxml

classmate

1.3.4

com.fasterxml.jackson.core

jackson-annotations

2.10.0

com.fasterxml.jackson.core

jackson-core

2.10.0

com.fasterxml.jackson.core

jackson-databind

2.10.0

com.fasterxml.jackson.dataformat

jackson-dataformat-cbor

2.10.0

com.fasterxml.jackson.datatype

jackson-datatype-joda

2.10.0

com.fasterxml.jackson.module

jackson-module-paranamer

2.10.0

com.fasterxml.jackson.module

jackson-module-scala_2.12

2.10.0

com.github.ben-manes.caffeine

caffeine

2.3.4

com.github.fommil

jniloader

1.1

com.github.fommil.netlib

core

1.1.2

com.github.fommil.netlib

native_ref-java

1.1

com.github.fommil.netlib

native_ref-java-natives

1.1

com.github.fommil.netlib

native_system-java

1.1

com.github.fommil.netlib

native_system-java-natives

1.1

com.github.fommil.netlib

netlib-native_ref-linux-x86_64-natives

1.1

com.github.fommil.netlib

netlib-native_system-linux-x86_64-natives

1.1

com.github.joshelser

dropwizard-metrics-hadoop-metrics2-reporter

0.1.2

com.github.luben

zstd-jni

1.4.8-1

com.github.wendykierp

JTransforms

3.1

com.google.code.findbugs

jsr305

3.0.0

com.google.code.gson

gson

2.2.4

com.google.flatbuffers

flatbuffers-java

1.9.0

com.google.guava

guava

15.0

com.google.protobuf

protobuf-java

2.6.1

com.h2database

h2

1.4.195

com.helger

profiler

1.1.1

com.jcraft

jsch

0.1.50

com.jolbox

bonecp

0.8.0.RELEASE

com.lihaoyi

sourcecode_2.12

0.1.9

com.microsoft.sqlserver

mssql-jdbc

8.2.1.jre8

com.microsoft.azure

azure-data-lake-store-sdk

2.3.9

com.ning

compress-lzf

1.0.3

com.sun.mail

javax.mail

1.5.2

com.tdunning

json

1.8

com.thoughtworks.paranamer

paranamer

2.8

com.trueaccord.lenses

lenses_2.12

0.4.12

com.twitter

chill-java

0.9.5

com.twitter

chill_2.12

0.9.5

com.twitter

util-app_2.12

7.1.0

com.twitter

util-core_2.12

7.1.0

com.twitter

util-function_2.12

7.1.0

com.twitter

util-jvm_2.12

7.1.0

com.twitter

util-lint_2.12

7.1.0

com.twitter

util-registry_2.12

7.1.0

com.twitter

util-stats_2.12

7.1.0

com.typesafe

config

1.2.1

com.typesafe.scala-logging

scala-logging_2.12

3.7.2

com.univocity

univocity-parsers

2.9.0

com.zaxxer

HikariCP

3.1.0

commons-beanutils

commons-beanutils

1.9.4

commons-cli

commons-cli

1.2

commons-codec

commons-codec

1.10

commons-collections

commons-collections

3.2.2

commons-configuration

commons-configuration

1.6

commons-dbcp

commons-dbcp

1.4

commons-digester

commons-digester

1.8

commons-fileupload

commons-fileupload

1.3.3

commons-httpclient

commons-httpclient

3.1

commons-io

commons-io

2.4

commons-lang

commons-lang

2.6

commons-logging

commons-logging

1.1.3

commons-net

commons-net

3.1

commons-pool

commons-pool

1.5.4

info.ganglia.gmetric4j

gmetric4j

1.0.10

io.airlift

aircompressor

0.10

io.dropwizard.metrics

metrics-core

4.1.1

io.dropwizard.metrics

metrics-graphite

4.1.1

io.dropwizard.metrics

metrics-healthchecks

4.1.1

io.dropwizard.metrics

metrics-jetty9

4.1.1

io.dropwizard.metrics

metrics-jmx

4.1.1

io.dropwizard.metrics

metrics-json

4.1.1

io.dropwizard.metrics

metrics-jvm

4.1.1

io.dropwizard.metrics

metrics-servlets

4.1.1

io.netty

netty-all

4.1.51.Final

io.prometheus

simpleclient

0.7.0

io.prometheus

simpleclient_common

0.7.0

io.prometheus

simpleclient_dropwizard

0.7.0

io.prometheus

simpleclient_pushgateway

0.7.0

io.prometheus

simpleclient_servlet

0.7.0

io.prometheus.jmx

collector

0.12.0

jakarta.annotation

jakarta.annotation-api

1.3.5

jakarta.validation

jakarta.validation-api

2.0.2

jakarta.ws.rs

jakarta.ws.rs-api

2.1.6

javax.activation

activation

1.1.1

javax.el

javax.el-api

2.2.4

javax.jdo

jdo-api

3.0.1

javax.servlet

javax.servlet-api

3.1.0

javax.servlet.jsp

jsp-api

2.1

javax.transaction

jta

1.1

javax.transaction

transaction-api

1.1

javax.xml.bind

jaxb-api

2.2.2

javax.xml.stream

stax-api

1.0-2

javolution

javolution

5.5.1

jline

jline

2.14.6

joda-time

joda-time

2.10.5

log4j

apache-log4j-extras

1.2.17

log4j

log4j

1.2.17

net.razorvine

pyrolite

4.30

net.sf.jpam

jpam

1.1

net.sf.opencsv

opencsv

2.3

net.sf.supercsv

super-csv

2.2.0

net.snowflake

snowflake-ingest-sdk

0.9.6

net.snowflake

snowflake-jdbc

3.12.8

net.snowflake

spark-snowflake_2.12

2.8.1-spark_3.0

net.sourceforge.f2j

arpack_combined_all

0.1

org.acplt.remotetea

remotetea-oncrpc

1.1.2

org.antlr

ST4

4.0.4

org.antlr

antlr-runtime

3.5.2

org.antlr

antlr4-runtime

4.8-1

org.antlr

stringtemplate

3.2.1

org.apache.ant

ant

1.9.2

org.apache.ant

ant-jsch

1.9.2

org.apache.ant

ant-launcher

1.9.2

org.apache.arrow

arrow-format

2.0.0

org.apache.arrow

arrow-memory-core

2.0.0

org.apache.arrow

arrow-memory-netty

2.0.0

org.apache.arrow

arrow-vector

2.0.0

org.apache.avro

avro

1.8.2

org.apache.avro

avro-ipc

1.8.2

org.apache.avro

avro-mapred-hadoop2

1.8.2

org.apache.commons

commons-compress

1.20

org.apache.commons

commons-crypto

1.1.0

org.apache.commons

commons-lang3

3.10

org.apache.commons

commons-math3

3.4.1

org.apache.commons

commons-text

1.6

org.apache.curator

curator-client

2.7.1

org.apache.curator

curator-framework

2.7.1

org.apache.curator

curator-recipes

2.7.1

org.apache.derby

derby

10.12.1.1

org.apache.directory.api

api-asn1-api

1.0.0-M20

org.apache.directory.api

api-util

1.0.0-M20

org.apache.directory.server

apacheds-i18n

2.0.0-M15

org.apache.directory.server

apacheds-kerberos-codec

2.0.0-M15

org.apache.hadoop

hadoop-annotations

2.7.4

org.apache.hadoop

hadoop-auth

2.7.4

org.apache.hadoop

hadoop-client

2.7.4

org.apache.hadoop

hadoop-common

2.7.4

org.apache.hadoop

hadoop-hdfs

2.7.4

org.apache.hadoop

hadoop-mapreduce-client-app

2.7.4

org.apache.hadoop

hadoop-mapreduce-client-common

2.7.4

org.apache.hadoop

hadoop-mapreduce-client-core

2.7.4

org.apache.hadoop

hadoop-mapreduce-client-jobclient

2.7.4

org.apache.hadoop

hadoop-mapreduce-client-shuffle

2.7.4

org.apache.hadoop

hadoop-yarn-api

2.7.4

org.apache.hadoop

hadoop-yarn-client

2.7.4

org.apache.hadoop

hadoop-yarn-common

2.7.4

org.apache.hadoop

hadoop-yarn-server-common

2.7.4

org.apache.hive

hive-beeline

2.3.7

org.apache.hive

hive-cli

2.3.7

org.apache.hive

hive-common

2.3.7

org.apache.hive

hive-exec-core

2.3.7

org.apache.hive

hive-jdbc

2.3.7

org.apache.hive

hive-llap-client

2.3.7

org.apache.hive

hive-llap-common

2.3.7

org.apache.hive

hive-metastore

2.3.7

org.apache.hive

hive-serde

2.3.7

org.apache.hive

hive-shims

2.3.7

org.apache.hive

hive-storage-api

2.7.2

org.apache.hive

hive-vector-code-gen

2.3.7

org.apache.hive.shims

hive-shims-0.23

2.3.7

org.apache.hive.shims

hive-shims-common

2.3.7

org.apache.hive.shims

hive-shims-scheduler

2.3.7

org.apache.htrace

htrace-core

3.1.0-incubating

org.apache.httpcomponents

httpclient

4.5.6

org.apache.httpcomponents

httpcore

4.4.12

org.apache.ivy

ivy

2.4.0

org.apache.mesos

mesos-shaded-protobuf

1.4.0

org.apache.orc

orc-core

1.5.12

org.apache.orc

orc-mapreduce

1.5.12

org.apache.orc

orc-shims

1.5.12

org.apache.parquet

parquet-column

1.10.1-databricks6

org.apache.parquet

parquet-common

1.10.1-databricks6

org.apache.parquet

parquet-encoding

1.10.1-databricks6

org.apache.parquet

parquet-format

2.4.0

org.apache.parquet

parquet-hadoop

1.10.1-databricks6

org.apache.parquet

parquet-jackson

1.10.1-databricks6

org.apache.thrift

libfb303

0.9.3

org.apache.thrift

libthrift

0.12.0

org.apache.velocity

velocity

1.5

org.apache.xbean

xbean-asm7-shaded

4.15

org.apache.yetus

audience-annotations

0.5.0

org.apache.zookeeper

zookeeper

3.4.14

org.codehaus.jackson

jackson-core-asl

1.9.13

org.codehaus.jackson

jackson-jaxrs

1.9.13

org.codehaus.jackson

jackson-mapper-asl

1.9.13

org.codehaus.jackson

jackson-xc

1.9.13

org.codehaus.janino

commons-compiler

3.0.16

org.codehaus.janino

janino

3.0.16

org.datanucleus

datanucleus-api-jdo

4.2.4

org.datanucleus

datanucleus-core

4.1.17

org.datanucleus

datanucleus-rdbms

4.1.19

org.datanucleus

javax.jdo

3.2.0-m3

org.eclipse.jetty

jetty-client

9.4.34.v20201102

org.eclipse.jetty

jetty-continuation

9.4.34.v20201102

org.eclipse.jetty

jetty-http

9.4.34.v20201102

org.eclipse.jetty

jetty-io

9.4.34.v20201102

org.eclipse.jetty

jetty-jndi

9.4.34.v20201102

org.eclipse.jetty

jetty-plus

9.4.34.v20201102

org.eclipse.jetty

jetty-proxy

9.4.34.v20201102

org.eclipse.jetty

jetty-security

9.4.34.v20201102

org.eclipse.jetty

jetty-server

9.4.34.v20201102

org.eclipse.jetty

jetty-servlet

9.4.34.v20201102

org.eclipse.jetty

jetty-servlets

9.4.34.v20201102

org.eclipse.jetty

jetty-util

9.4.34.v20201102

org.eclipse.jetty

jetty-webapp

9.4.34.v20201102

org.eclipse.jetty

jetty-xml

9.4.34.v20201102

org.fusesource.leveldbjni

leveldbjni-all

1.8

org.glassfish.hk2

hk2-api

2.6.1

org.glassfish.hk2

hk2-locator

2.6.1

org.glassfish.hk2

hk2-utils

2.6.1

org.glassfish.hk2

osgi-resource-locator

1.0.3

org.glassfish.hk2.external

aopalliance-repackaged

2.6.1

org.glassfish.hk2.external

jakarta.inject

2.6.1

org.glassfish.jersey.containers

jersey-container-servlet

2.30

org.glassfish.jersey.containers

jersey-container-servlet-core

2.30

org.glassfish.jersey.core

jersey-client

2.30

org.glassfish.jersey.core

jersey-common

2.30

org.glassfish.jersey.core

jersey-server

2.30

org.glassfish.jersey.inject

jersey-hk2

2.30

org.glassfish.jersey.media

jersey-media-jaxb

2.30

org.hibernate.validator

hibernate-validator

6.1.0.Final

org.javassist

javassist

3.25.0-GA

org.jboss.logging

jboss-logging

3.3.2.Final

org.jdbi

jdbi

2.63.1

org.joda

joda-convert

1.7

org.jodd

jodd-core

3.5.2

org.json4s

json4s-ast_2.12

3.7.0-M5

org.json4s

json4s-core_2.12

3.7.0-M5

org.json4s

json4s-jackson_2.12

3.7.0-M5

org.json4s

json4s-scalap_2.12

3.7.0-M5

org.lz4

lz4-java

1.7.1

org.mariadb.jdbc

mariadb-java-client

2.1.2

org.objenesis

objenesis

2.5.1

org.postgresql

postgresql

42.1.4

org.roaringbitmap

RoaringBitmap

0.9.0

org.roaringbitmap

shims

0.9.0

org.rocksdb

rocksdbjni

6.2.2

org.rosuda.REngine

REngine

2.1.0

org.scala-lang

scala-compiler_2.12

2.12.10

org.scala-lang

scala-library_2.12

2.12.10

org.scala-lang

scala-reflect_2.12

2.12.10

org.scala-lang.modules

scala-collection-compat_2.12

2.1.1

org.scala-lang.modules

scala-parser-combinators_2.12

1.1.2

org.scala-lang.modules

scala-xml_2.12

1.2.0

org.scala-sbt

test-interface

1.0

org.scalacheck

scalacheck_2.12

1.14.2

org.scalactic

scalactic_2.12

3.0.8

org.scalanlp

breeze-macros_2.12

1.0

org.scalanlp

breeze_2.12

1.0

org.scalatest

scalatest_2.12

3.0.8

org.slf4j

jcl-over-slf4j

1.7.30

org.slf4j

jul-to-slf4j

1.7.30

org.slf4j

slf4j-api

1.7.30

org.slf4j

slf4j-log4j12

1.7.30

org.spark-project.spark

unused

1.0.0

org.springframework

spring-core

4.1.4.RELEASE

org.springframework

spring-test

4.1.4.RELEASE

org.threeten

threeten-extra

1.5.0

org.tukaani

xz

1.5

org.typelevel

algebra_2.12

2.0.0-M2

org.typelevel

cats-kernel_2.12

2.0.0-M4

org.typelevel

machinist_2.12

0.6.8

org.typelevel

macro-compat_2.12

1.1.1

org.typelevel

spire-macros_2.12

0.17.0-M1

org.typelevel

spire-platform_2.12

0.17.0-M1

org.typelevel

spire-util_2.12

0.17.0-M1

org.typelevel

spire_2.12

0.17.0-M1

org.wildfly.openssl

wildfly-openssl

1.0.7.Final

org.xerial

sqlite-jdbc

3.8.11.2

org.xerial.snappy

snappy-java

1.1.8.2

org.yaml

snakeyaml

1.24

oro

oro

2.0.8

pl.edu.icm

JLargeArrays

1.5

software.amazon.ion

ion-java

1.0.2

stax

stax-api

1.0.1

xmlenc

xmlenc

0.52