Databricks Runtime 8.4 (EoS)

Note

Support for this Databricks Runtime version has ended. For the end-of-support date, see End-of-support history. For all supported Databricks Runtime versions, see Databricks Runtime release notes versions and compatibility.

The following release notes provide information about Databricks Runtime 8.4 and Databricks Runtime 8.4 Photon, powered by Apache Spark 3.1.2. Databricks released this version in July 2021. Photon is in Public Preview.

Note

These release notes may include references to features that are not available on Google Cloud as of this release.

New features and improvements

Delta Lake features and improvements

Delta table change data feed (GA)

The Delta table change data feed is now generally available. It represents the row level changes between different versions of the table. When enabled, additional information is recorded regarding row level changes for every write operation on the table. See Use Delta Lake change data feed on Databricks.

Load shared Delta tables easily with Databricks Runtime

The Apache Spark Connector for Delta Sharing 0.1.0 is now in Databricks Runtime. You can load a shared table using spark.read.format("deltaSharing").load(uri) directly without attaching the Delta Sharing Spark connector to your cluster.

More tables benefit from dynamic file pruning

The dynamic file pruning feature has been tuned to trigger on tables with fewer files. See Dynamic file pruning.

Better performance with automatic target file size tuning

The target file size for Delta tables is now automatically tuned based on the table size. Previously, the target file size for OPTIMIZE and OPTIMIZE ZORDER BY was 1GB. With autotuning based on table size, Delta tables up to 2.56TB will use 256MB as the target size. Tables larger than 10TB will use 1GB as before. Tables between these sizes will use target sizes that grow proportionally with the table size. See Configure Delta Lake to control data file size.

More ways to specify tables in DeltaTable.forName

DeltaTable.forName now supports using delta.`<path>` to identify tables.

Robust streaming multi-table writes using foreachBatch

Idempotent Delta streaming writes within the foreachBatch() command are now supported. For details, see Idempotent multi-table writes.

Improved read query performance in certain workloads due to tuned checkpoints

Delta Lake now tunes how frequently it does enhanced checkpoints. Instead of checkpointing at a fixed interval, Delta now dynamically adjusts the checkpoint frequency based on certain event triggers. This improves read query performance in workloads where some data skipping optimization couldn’t be applied before. To use these optimizations, upgrade your jobs that write to Delta Lake to Databricks Runtime 8.4. See Enable enhanced checkpoints for Structured Streaming queries.

Create GroupState to test user defined Structured Streaming functions

Until now, only the Structured Streaming engine could create instances of GroupState. Hence, any unit tests of the user-defined function required running a streaming query in Apache Spark.

Now you can create instances of GroupState using TestGroupState.create(…). This allow you to test a user-defined function in simple unit tests that do not require running Spark. See Test the mapGroupsWithState update function. Specifically, it produces instances of type TestGroupState that extends the interface GroupState with additional methods for introspecting the internal state after the user-defined function has been applied.

Auto Loader features and improvements

Configure backfilling to capture missed files

Auto Loader now supports performing backfills asynchronously to capture any files that could have been missed with file notifications. File storage systems and notification systems cannot guarantee 100% delivery of all file events. Therefore Databricks recommends enabling periodic backfills to capture all of your data with Auto Loader. Use the cloudFiles.backfillInterval option to schedule regular backfills over your data. See Common Auto Loader options.

Bound storage footprint for large volume streams

You can now configure Auto Loader to expire and remove entries in RocksDB to bound its storage footprint in the checkpoint location. Databricks does not recommend that you use unless you are ingesting data in the order of millions of files an hour. Setting this option incorrectly or attempting to tune it can lead to many data quality issues, such as unprocessed files being ignored or duplicate of some files instead of exactly once processing. For details, see Event retention.

Simplified configuration with pathless support

S3 buckets

You can now provide the SQS queue that receives events from multiple paths or S3 buckets. If you provide the SQS queue URL, the path option is not required for this use case. Auto Loader constructs S3 paths using the bucket and key from the S3 events. If you want to read the files through DBFS mount points, you can use cloudFiles.pathRewrites to change path prefixes to DBFS. This is not required unless you’re accessing data in different accounts with AssumeRole.

See File notification options.

Azure Data Lake Storage Gen2 containers

You can now provide the Azure queue that receives events from multiple containers. If you provide the Azure queue name, the path option is not required. By default, Auto Loader constructs Azure Data Lake Storage Gen2 paths using the container and key in the file events. If you want to use WASB paths or DBFS mount points, you can use cloudFiles.pathRewrites to change path prefixes.

See File notification options.

Connector upgrades

• The Snowflake Spark connector has been updated to v2.9.0.

• KMS encryption is now supported in the UNLOAD statement of the Redshift connector.

Bug fixes

• Fixed an issue for clusters enabled with table access control where select * from folder can show folder content even if the user has no file access permission.

• Non-admin database owners are now able to drop non-owned tables in databases. This fixes the issue where database owners could not drop a database if non-owned tables existed in the database.

Library upgrades

• Upgraded Python libraries:

  • certifi upgraded from 2020.12.5 to 2021.5.30

  • distill upgraded from 0.3.1 to 0.3.2

  • koalas upgraded from 1.8.0 to 1.8.1

  • protobuf upgraded from 3.17.0 to 3.17.3

• Upgraded R libraries:

  • base from 4.0.4 to 4.1.0

  • boot from 1.3-27 to 1.3-28

  • class from 7.3-18 to 7.3-19

  • cluster from 2.1.1 to 2.1.2

  • compiler from 4.0.4 to 4.1.0

  • datasets from 4.0.4 to 4.1.0

  • graphics from 4.0.4 to 4.1.0

  • grDevices from 4.0.4 to 4.1.0

  • grid from 4.0.4 to 4.1.0

  • KernSmooth from 2.23-18 to 2.23-20

  • lattice from 0.20-41 to 0.20-44

  • MASS from 7.3-53.1 to 7.3-54

  • Matrix from 1.3-2 to 1.3-3

  • methods from 4.0.4 to 4.1.0

  • mgcv from 1.8-33 to 1.8-35

  • nnet from 7.3-15 to 7.3-16

  • parallel from 4.0.4 to 4.1.0

  • Rserve from 1.8-7 to 1.8-8

  • SparkR from 3.1.1 to 3.1.2

  • splines from 4.0.4 to 4.1.0

  • stats from 4.0.4 to 4.1.0

  • stats4 from 4.0.4 to 4.1.0

  • survival from 3.2-7 to 3.2-11

  • tcltk from 4.0.4 to 4.1.0

  • tools from 4.0.4 to 4.1.0

  • utils from 4.0.4 to 4.1.0

• Upgraded Java libraries:

  • snowflake-jdbc from 3.12.8 to 3.13.3

  • spark-snowflake_2.12 from 2.8.1-spark_3.0 to 2.9.0-spark_3.1

  • RoaringBitmap from 0.9.0 to 0.9.14

  • shims from 0.9.0 to 0.9.14

  • rocksdbjni fromm 6.2.2 to 6.20.3

Apache Spark

Databricks Runtime 8.4 includes Apache Spark 3.1.2. This release includes all Spark fixes and improvements included in Databricks Runtime 8.3 (EoS), as well as the following additional bug fixes and improvements made to Spark:

• [SPARK-35792] [SQL] View should not capture configs used in RelationConversions

• [SPARK-35700] [SQL] Read char/varchar orc table with created and written by external systems

• [SPARK-35636] [SQL] Lambda keys should not be referenced outside of the lambda function

• [SPARK-35800] [Cherry Pick] Improving GroupState testability by introducing TestGroupState

• [SPARK-35391] Fix memory leak in ExecutorAllocationListener

• [SPARK-35799] [CherryPick] Fix the allUpdatesTimeMs metric measuring in FlatMapGroupsWithStateExec

• [SPARK-35763] [SS] Remove the StateStoreCustomMetric subclass enumeration dependency

• [SPARK-35791 [SQL] Release on-going map properly for NULL-aware ANTI join

• [SPARK-35695] [SQL] Collect observed metrics from cached and adaptive execution sub-trees

• [SPARK-35767] [SQL] Avoid executing child plan twice in CoalesceExec

• [SPARK-35746] [UI] Fix taskid in the stage page task event timeline

• [SPARK-35673] [SQL] Fix user-defined hint and unrecognized hint in subquery.

• [SPARK-35714] [CORE] Bug fix for deadlock during the executor shutdown

• [SPARK-35689] [SS] Add log warn when keyWithIndexToValue returns null value

• [SPARK-35589] [CORE][3.1] BlockManagerMasterEndpoint should not ignore index-only shuffle file during updating

• [SPARK-35643] [PYTHON] Fix ambiguous reference in functions.py column()

• [SPARK-35652] [SQL] joinWith on two table generated from same one

• [SPARK-35679] [SQL] instantToMicros overflow

• [SPARK-35602] [SS] Update state schema to be able to accept long length JSON

• [SPARK-35653] [SQL] Fix CatalystToExternalMap interpreted path fails for Map with case classes as keys or values

• [SPARK-35296] [SQL] Allow Dataset.observe to work even if CollectMetricsExec in a task handles multiple partitions.

• [SPARK-35659] [SS] Avoid write null to StateStore

• [SPARK-35665] [SQL] Resolve UnresolvedAlias in CollectMetrics

• [SPARK-35558] Optimizes for multi-quantile retrieval

• [SPARK-35621] [SQL] Add rule id pruning to the TypeCoercion rule

• [SPARK-35077] [SQL] Migrate to transformWithPruning for leftover optimizer rules

• [SPARK-35610] [CORE] Fix the memory leak introduced by the Executor’s stop shutdown hook

• [SPARK-35544] [SQL] Add tree pattern pruning to Analyzer rules

• [SPARK-35566] [SS] Fix StateStoreRestoreExec output rows

• [SPARK-35454] [SQL][3.1] One LogicalPlan can match multiple dataset ids

• [SPARK-35538] [SQL] Migrate transformAllExpressions call sites to use transformAllExpressionsWithPruning

• [SPARK-35106] [Core][SQL] Avoid failing rename caused by destination directory not exist

• [SPARK-35287] [SQL] Allow RemoveRedundantProjects to preserve ProjectExec which generates UnsafeRow for DataSourceV2ScanRelation

• [SPARK-35495] [R] Change SparkR maintainer for CRAN

• [SPARK-27991] [CORE] Defer the fetch request on Netty OOM

• [SPARK-35171] [R] Declare the markdown package as a dependency of the SparkR package

• [SPARK-35454] [SQL] One LogicalPlan can match multiple dataset ids

• [SPARK-35298] [SQL] Migrate to transformWithPruning for rules in Optimizer.scala

• [SPARK-35480] [SQL] Make percentile_approx work with pivot

• [SPARK-35093] [SQL] AQE now uses newQueryStage plan as key for looking up cached exchanges for re-use

• [SPARK-35146] [SQL] Migrate to transformWithPruning or resolveWithPruning for rules in finishAnalysis.scala

• [SPARK-35411] [SQL] Add essential information while serializing TreeNode to json

• [SPARK-35294] [SQL] Add tree traversal pruning in rules with dedicated files under optimizer

• [SPARK-34897] [SQL][3.1] Support reconcile schemas based on index after nested column pruning

• [SPARK-35144] [SQL] Migrate to transformWithPruning for object rules

• [SPARK-35155] [SQL] Add rule id pruning to Analyzer rules

• [SPARK-35382] [PYTHON] Fix lambda variable name issues in nested DataFrame functions in Python APIs.

• [SPARK-35359] [SQL] Insert data with char/varchar datatype will fail when data length exceed length limitation

• [SPARK-35381] [R] Fix lambda variable name issues in nested higher order functions at R APIs

System environment

• Operating System: Ubuntu 18.04.5 LTS

• Java: Zulu 8.54.0.21-CA-linux64

• Scala: 2.12.10

• Python: 3.8.8

• R: 4.1.0 (2021-05-18)

• Delta Lake 1.0.0

Installed Python libraries

Library	Version	Library	Version	Library	Version
appdirs	1.4.4	asn1crypto	1.4.0	backcall	0.2.0
boto3	1.16.7	botocore	1.19.7	brotlipy	0.7.0
certifi	2021.5.30	cffi	1.14.3	chardet	3.0.4
cryptography	3.1.1	cycler	0.10.0	Cython	0.29.21
decorator	4.4.2	distlib	0.3.2	docutils	0.15.2
entrypoints	0.3	facets-overview	1.0.0	filelock	3.0.12
idna	2.10	ipykernel	5.3.4	ipython	7.19.0
ipython-genutils	0.2.0	jedi	0.17.2	jmespath	0.10.0
joblib	0.17.0	jupyter-client	6.1.7	jupyter-core	4.6.3
kiwisolver	1.3.0	koalas	1.8.1	matplotlib	3.2.2
numpy	1.19.2	pandas	1.1.5	parso	0.7.0
patsy	0.5.1	pexpect	4.8.0	pickleshare	0.7.5
pip	20.2.4	plotly	4.14.3	prompt-toolkit	3.0.8
protobuf	3.17.3	psycopg2	2.8.5	ptyprocess	0.6.0
pyarrow	1.0.1	pycparser	2.20	Pygments	2.7.2
pyOpenSSL	19.1.0	pyparsing	2.4.7	PySocks	1.7.1
python-dateutil	2.8.1	pytz	2020.5	pyzmq	19.0.2
requests	2.24.0	retrying	1.3.3	s3transfer	0.3.6
scikit-learn	0.23.2	scipy	1.5.2	seaborn	0.10.0
setuptools	50.3.1	six	1.15.0	statsmodels	0.12.0
threadpoolctl	2.1.0	tornado	6.0.4	traitlets	5.0.5
urllib3	1.25.11	virtualenv	20.2.1	wcwidth	0.2.5
wheel	0.35.1

Installed R libraries

R libraries are installed from the Microsoft CRAN snapshot on 2020-11-02.

Library	Version	Library	Version	Library	Version
askpass	1.1	assertthat	0.2.1	backports	1.2.1
base	4.1.0	base64enc	0.1-3	BH	1.72.0-3
bit	4.0.4	bit64	4.0.5	blob	1.2.1
boot	1.3-28	brew	1.0-6	brio	1.1.0
broom	0.7.2	callr	3.5.1	caret	6.0-86
cellranger	1.1.0	chron	2.3-56	class	7.3-19
cli	2.2.0	clipr	0.7.1	cluster	2.1.2
codetools	0.2-18	colorspace	2.0-0	commonmark	1.7
compiler	4.1.0	config	0.3	covr	3.5.1
cpp11	0.2.4	crayon	1.3.4	credentials	1.3.0
crosstalk	1.1.0.1	curl	4.3	data.table	1.13.4
datasets	4.1.0	DBI	1.1.0	dbplyr	2.0.0
desc	1.2.0	devtools	2.3.2	diffobj	0.3.2
digest	0.6.27	dplyr	1.0.2	DT	0.16
ellipsis	0.3.1	evaluate	0.14	fansi	0.4.1
farver	2.0.3	fastmap	1.0.1	forcats	0.5.0
foreach	1.5.1	foreign	0.8-81	forge	0.2.0
fs	1.5.0	future	1.21.0	generics	0.1.0
gert	1.0.2	ggplot2	3.3.2	gh	1.2.0
gitcreds	0.1.1	glmnet	4.0-2	globals	0.14.0
glue	1.4.2	gower	0.2.2	graphics	4.1.0
grDevices	4.1.0	grid	4.1.0	gridExtra	2.3
gsubfn	0.7	gtable	0.3.0	haven	2.3.1
highr	0.8	hms	0.5.3	htmltools	0.5.0
htmlwidgets	1.5.3	httpuv	1.5.4	httr	1.4.2
hwriter	1.3.2	hwriterPlus	1.0-3	ini	0.3.1
ipred	0.9-9	isoband	0.2.3	iterators	1.0.13
jsonlite	1.7.2	KernSmooth	2.23-20	knitr	1.30
labeling	0.4.2	later	1.1.0.1	lattice	0.20-44
lava	1.6.8.1	lazyeval	0.2.2	lifecycle	0.2.0
listenv	0.8.0	lubridate	1.7.9.2	magrittr	2.0.1
markdown	1.1	MASS	7.3-54	Matrix	1.3-3
memoise	1.1.0	methods	4.1.0	mgcv	1.8-35
mime	0.9	ModelMetrics	1.2.2.2	modelr	0.1.8
munsell	0.5.0	nlme	3.1-152	nnet	7.3-16
numDeriv	2016.8-1.1	openssl	1.4.3	parallel	4.1.0
parallelly	1.22.0	pillar	1.4.7	pkgbuild	1.1.0
pkgconfig	2.0.3	pkgload	1.1.0	plogr	0.2.0
plyr	1.8.6	praise	1.0.0	prettyunits	1.1.1
pROC	1.16.2	processx	3.4.5	prodlim	2019.11.13
progress	1.2.2	promises	1.1.1	proto	1.0.0
ps	1.5.0	purrr	0.3.4	r2d3	0.2.3
R6	2.5.0	randomForest	4.6-14	rappdirs	0.3.1
rcmdcheck	1.3.3	RColorBrewer	1.1-2	Rcpp	1.0.5
readr	1.4.0	readxl	1.3.1	recipes	0.1.15
rematch	1.0.1	rematch2	2.1.2	remotes	2.2.0
reprex	0.3.0	reshape2	1.4.4	rex	1.2.0
rlang	0.4.9	rmarkdown	2.6	RODBC	1.3-17
roxygen2	7.1.1	rpart	4.1-15	rprojroot	2.0.2
Rserve	1.8-8	RSQLite	2.2.1	rstudioapi	0.13
rversions	2.0.2	rvest	0.3.6	scales	1.1.1
selectr	0.4-2	sessioninfo	1.1.1	shape	1.4.5
shiny	1.5.0	sourcetools	0.1.7	sparklyr	1.5.2
SparkR	3.1.1	spatial	7.3-11	splines	4.1.0
sqldf	0.4-11	SQUAREM	2020.5	stats	4.1.0
stats4	4.1.0	stringi	1.5.3	stringr	1.4.0
survival	3.2-11	sys	3.4	tcltk	4.1.0
TeachingDemos	2.10	testthat	3.0.0	tibble	3.0.4
tidyr	1.1.2	tidyselect	1.1.0	tidyverse	1.3.0
timeDate	3043.102	tinytex	0.28	tools	4.1.0
usethis	2.0.0	utf8	1.1.4	utils	4.1.0
uuid	0.1-4	vctrs	0.3.5	viridisLite	0.3.0
waldo	0.2.3	whisker	0.4	withr	2.3.0
xfun	0.19	xml2	1.3.2	xopen	1.0.0
xtable	1.8-4	yaml	2.2.1	zip	2.1.1

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID	Artifact ID	Version
antlr	antlr	2.7.7
com.amazonaws	amazon-kinesis-client	1.12.0
com.amazonaws	aws-java-sdk-autoscaling	1.11.655
com.amazonaws	aws-java-sdk-cloudformation	1.11.655
com.amazonaws	aws-java-sdk-cloudfront	1.11.655
com.amazonaws	aws-java-sdk-cloudhsm	1.11.655
com.amazonaws	aws-java-sdk-cloudsearch	1.11.655
com.amazonaws	aws-java-sdk-cloudtrail	1.11.655
com.amazonaws	aws-java-sdk-cloudwatch	1.11.655
com.amazonaws	aws-java-sdk-cloudwatchmetrics	1.11.655
com.amazonaws	aws-java-sdk-codedeploy	1.11.655
com.amazonaws	aws-java-sdk-cognitoidentity	1.11.655
com.amazonaws	aws-java-sdk-cognitosync	1.11.655
com.amazonaws	aws-java-sdk-config	1.11.655
com.amazonaws	aws-java-sdk-core	1.11.655
com.amazonaws	aws-java-sdk-datapipeline	1.11.655
com.amazonaws	aws-java-sdk-directconnect	1.11.655
com.amazonaws	aws-java-sdk-directory	1.11.655
com.amazonaws	aws-java-sdk-dynamodb	1.11.655
com.amazonaws	aws-java-sdk-ec2	1.11.655
com.amazonaws	aws-java-sdk-ecs	1.11.655
com.amazonaws	aws-java-sdk-efs	1.11.655
com.amazonaws	aws-java-sdk-elasticache	1.11.655
com.amazonaws	aws-java-sdk-elasticbeanstalk	1.11.655
com.amazonaws	aws-java-sdk-elasticloadbalancing	1.11.655
com.amazonaws	aws-java-sdk-elastictranscoder	1.11.655
com.amazonaws	aws-java-sdk-emr	1.11.655
com.amazonaws	aws-java-sdk-glacier	1.11.655
com.amazonaws	aws-java-sdk-glue	1.11.655
com.amazonaws	aws-java-sdk-iam	1.11.655
com.amazonaws	aws-java-sdk-importexport	1.11.655
com.amazonaws	aws-java-sdk-kinesis	1.11.655
com.amazonaws	aws-java-sdk-kms	1.11.655
com.amazonaws	aws-java-sdk-lambda	1.11.655
com.amazonaws	aws-java-sdk-logs	1.11.655
com.amazonaws	aws-java-sdk-machinelearning	1.11.655
com.amazonaws	aws-java-sdk-marketplacemeteringservice	1.11.655
com.amazonaws	aws-java-sdk-opsworks	1.11.655
com.amazonaws	aws-java-sdk-rds	1.11.655
com.amazonaws	aws-java-sdk-redshift	1.11.655
com.amazonaws	aws-java-sdk-route53	1.11.655
com.amazonaws	aws-java-sdk-s3	1.11.655
com.amazonaws	aws-java-sdk-ses	1.11.655
com.amazonaws	aws-java-sdk-simpledb	1.11.655
com.amazonaws	aws-java-sdk-simpleworkflow	1.11.655
com.amazonaws	aws-java-sdk-sns	1.11.655
com.amazonaws	aws-java-sdk-sqs	1.11.655
com.amazonaws	aws-java-sdk-ssm	1.11.655
com.amazonaws	aws-java-sdk-storagegateway	1.11.655
com.amazonaws	aws-java-sdk-sts	1.11.655
com.amazonaws	aws-java-sdk-support	1.11.655
com.amazonaws	aws-java-sdk-swf-libraries	1.11.22
com.amazonaws	aws-java-sdk-workspaces	1.11.655
com.amazonaws	jmespath-java	1.11.655
com.chuusai	shapeless_2.12	2.3.3
com.clearspring.analytics	stream	2.9.6
com.databricks	Rserve	1.8-3
com.databricks	jets3t	0.7.1-0
com.databricks.scalapb	compilerplugin_2.12	0.4.15-10
com.databricks.scalapb	scalapb-runtime_2.12	0.4.15-10
com.esotericsoftware	kryo-shaded	4.0.2
com.esotericsoftware	minlog	1.3.0
com.fasterxml	classmate	1.3.4
com.fasterxml.jackson.core	jackson-annotations	2.10.0
com.fasterxml.jackson.core	jackson-core	2.10.0
com.fasterxml.jackson.core	jackson-databind	2.10.0
com.fasterxml.jackson.dataformat	jackson-dataformat-cbor	2.10.0
com.fasterxml.jackson.datatype	jackson-datatype-joda	2.10.0
com.fasterxml.jackson.module	jackson-module-paranamer	2.10.0
com.fasterxml.jackson.module	jackson-module-scala_2.12	2.10.0
com.github.ben-manes.caffeine	caffeine	2.3.4
com.github.fommil	jniloader	1.1
com.github.fommil.netlib	core	1.1.2
com.github.fommil.netlib	native_ref-java	1.1
com.github.fommil.netlib	native_ref-java-natives	1.1
com.github.fommil.netlib	native_system-java	1.1
com.github.fommil.netlib	native_system-java-natives	1.1
com.github.fommil.netlib	netlib-native_ref-linux-x86_64-natives	1.1
com.github.fommil.netlib	netlib-native_system-linux-x86_64-natives	1.1
com.github.joshelser	dropwizard-metrics-hadoop-metrics2-reporter	0.1.2
com.github.luben	zstd-jni	1.4.8-1
com.github.wendykierp	JTransforms	3.1
com.google.code.findbugs	jsr305	3.0.0
com.google.code.gson	gson	2.2.4
com.google.flatbuffers	flatbuffers-java	1.9.0
com.google.guava	guava	15.0
com.google.protobuf	protobuf-java	2.6.1
com.h2database	h2	1.4.195
com.helger	profiler	1.1.1
com.jcraft	jsch	0.1.50
com.jolbox	bonecp	0.8.0.RELEASE
com.lihaoyi	sourcecode_2.12	0.1.9
com.microsoft.azure	azure-data-lake-store-sdk	2.3.9
com.microsoft.sqlserver	mssql-jdbc	9.2.1.jre8
com.ning	compress-lzf	1.0.3
com.sun.mail	javax.mail	1.5.2
com.tdunning	json	1.8
com.thoughtworks.paranamer	paranamer	2.8
com.trueaccord.lenses	lenses_2.12	0.4.12
com.twitter	chill-java	0.9.5
com.twitter	chill_2.12	0.9.5
com.twitter	util-app_2.12	7.1.0
com.twitter	util-core_2.12	7.1.0
com.twitter	util-function_2.12	7.1.0
com.twitter	util-jvm_2.12	7.1.0
com.twitter	util-lint_2.12	7.1.0
com.twitter	util-registry_2.12	7.1.0
com.twitter	util-stats_2.12	7.1.0
com.typesafe	config	1.2.1
com.typesafe.scala-logging	scala-logging_2.12	3.7.2
com.univocity	univocity-parsers	2.9.1
com.zaxxer	HikariCP	3.1.0
commons-beanutils	commons-beanutils	1.9.4
commons-cli	commons-cli	1.2
commons-codec	commons-codec	1.10
commons-collections	commons-collections	3.2.2
commons-configuration	commons-configuration	1.6
commons-dbcp	commons-dbcp	1.4
commons-digester	commons-digester	1.8
commons-fileupload	commons-fileupload	1.3.3
commons-httpclient	commons-httpclient	3.1
commons-io	commons-io	2.4
commons-lang	commons-lang	2.6
commons-logging	commons-logging	1.1.3
commons-net	commons-net	3.1
commons-pool	commons-pool	1.5.4
hive-2.3__hadoop-2.7	jets3t-0.7	liball_deps_2.12
hive-2.3__hadoop-2.7	zookeeper-3.4	liball_deps_2.12
info.ganglia.gmetric4j	gmetric4j	1.0.10
io.airlift	aircompressor	0.10
io.delta	delta-sharing-spark_2.12	0.1.0
io.dropwizard.metrics	metrics-core	4.1.1
io.dropwizard.metrics	metrics-graphite	4.1.1
io.dropwizard.metrics	metrics-healthchecks	4.1.1
io.dropwizard.metrics	metrics-jetty9	4.1.1
io.dropwizard.metrics	metrics-jmx	4.1.1
io.dropwizard.metrics	metrics-json	4.1.1
io.dropwizard.metrics	metrics-jvm	4.1.1
io.dropwizard.metrics	metrics-servlets	4.1.1
io.netty	netty-all	4.1.51.Final
io.prometheus	simpleclient	0.7.0
io.prometheus	simpleclient_common	0.7.0
io.prometheus	simpleclient_dropwizard	0.7.0
io.prometheus	simpleclient_pushgateway	0.7.0
io.prometheus	simpleclient_servlet	0.7.0
io.prometheus.jmx	collector	0.12.0
jakarta.annotation	jakarta.annotation-api	1.3.5
jakarta.validation	jakarta.validation-api	2.0.2
jakarta.ws.rs	jakarta.ws.rs-api	2.1.6
javax.activation	activation	1.1.1
javax.el	javax.el-api	2.2.4
javax.jdo	jdo-api	3.0.1
javax.servlet	javax.servlet-api	3.1.0
javax.servlet.jsp	jsp-api	2.1
javax.transaction	jta	1.1
javax.transaction	transaction-api	1.1
javax.xml.bind	jaxb-api	2.2.2
javax.xml.stream	stax-api	1.0-2
javolution	javolution	5.5.1
jline	jline	2.14.6
joda-time	joda-time	2.10.5
log4j	apache-log4j-extras	1.2.17
log4j	log4j	1.2.17
maven-trees	hive-2.3__hadoop-2.7	liball_deps_2.12
net.java.dev.jna	jna	5.8.0
net.razorvine	pyrolite	4.30
net.sf.jpam	jpam	1.1
net.sf.opencsv	opencsv	2.3
net.sf.supercsv	super-csv	2.2.0
net.snowflake	snowflake-ingest-sdk	0.9.6
net.snowflake	snowflake-jdbc	3.13.3
net.snowflake	spark-snowflake_2.12	2.9.0-spark_3.1
net.sourceforge.f2j	arpack_combined_all	0.1
org.acplt.remotetea	remotetea-oncrpc	1.1.2
org.antlr	ST4	4.0.4
org.antlr	antlr-runtime	3.5.2
org.antlr	antlr4-runtime	4.8-1
org.antlr	stringtemplate	3.2.1
org.apache.ant	ant	1.9.2
org.apache.ant	ant-jsch	1.9.2
org.apache.ant	ant-launcher	1.9.2
org.apache.arrow	arrow-format	2.0.0
org.apache.arrow	arrow-memory-core	2.0.0
org.apache.arrow	arrow-memory-netty	2.0.0
org.apache.arrow	arrow-vector	2.0.0
org.apache.avro	avro	1.8.2
org.apache.avro	avro-ipc	1.8.2
org.apache.avro	avro-mapred-hadoop2	1.8.2
org.apache.commons	commons-compress	1.20
org.apache.commons	commons-crypto	1.1.0
org.apache.commons	commons-lang3	3.10
org.apache.commons	commons-math3	3.4.1
org.apache.commons	commons-text	1.6
org.apache.curator	curator-client	2.7.1
org.apache.curator	curator-framework	2.7.1
org.apache.curator	curator-recipes	2.7.1
org.apache.derby	derby	10.12.1.1
org.apache.directory.api	api-asn1-api	1.0.0-M20
org.apache.directory.api	api-util	1.0.0-M20
org.apache.directory.server	apacheds-i18n	2.0.0-M15
org.apache.directory.server	apacheds-kerberos-codec	2.0.0-M15
org.apache.hadoop	hadoop-annotations	2.7.4
org.apache.hadoop	hadoop-auth	2.7.4
org.apache.hadoop	hadoop-client	2.7.4
org.apache.hadoop	hadoop-common	2.7.4
org.apache.hadoop	hadoop-hdfs	2.7.4
org.apache.hadoop	hadoop-mapreduce-client-app	2.7.4
org.apache.hadoop	hadoop-mapreduce-client-common	2.7.4
org.apache.hadoop	hadoop-mapreduce-client-core	2.7.4
org.apache.hadoop	hadoop-mapreduce-client-jobclient	2.7.4
org.apache.hadoop	hadoop-mapreduce-client-shuffle	2.7.4
org.apache.hadoop	hadoop-yarn-api	2.7.4
org.apache.hadoop	hadoop-yarn-client	2.7.4
org.apache.hadoop	hadoop-yarn-common	2.7.4
org.apache.hadoop	hadoop-yarn-server-common	2.7.4
org.apache.hive	hive-beeline	2.3.7
org.apache.hive	hive-cli	2.3.7
org.apache.hive	hive-jdbc	2.3.7
org.apache.hive	hive-llap-client	2.3.7
org.apache.hive	hive-llap-common	2.3.7
org.apache.hive	hive-serde	2.3.7
org.apache.hive	hive-shims	2.3.7
org.apache.hive	hive-storage-api	2.7.2
org.apache.hive.shims	hive-shims-0.23	2.3.7
org.apache.hive.shims	hive-shims-common	2.3.7
org.apache.hive.shims	hive-shims-scheduler	2.3.7
org.apache.htrace	htrace-core	3.1.0-incubating
org.apache.httpcomponents	httpclient	4.5.6
org.apache.httpcomponents	httpcore	4.4.12
org.apache.ivy	ivy	2.4.0
org.apache.mesos	mesos-shaded-protobuf	1.4.0
org.apache.orc	orc-core	1.5.12
org.apache.orc	orc-mapreduce	1.5.12
org.apache.orc	orc-shims	1.5.12
org.apache.parquet	parquet-column	1.10.1-databricks9
org.apache.parquet	parquet-common	1.10.1-databricks9
org.apache.parquet	parquet-encoding	1.10.1-databricks9
org.apache.parquet	parquet-format	2.4.0
org.apache.parquet	parquet-hadoop	1.10.1-databricks9
org.apache.parquet	parquet-jackson	1.10.1-databricks9
org.apache.thrift	libfb303	0.9.3
org.apache.thrift	libthrift	0.12.0
org.apache.xbean	xbean-asm7-shaded	4.15
org.apache.yetus	audience-annotations	0.5.0
org.apache.zookeeper	zookeeper	3.4.14
org.codehaus.jackson	jackson-core-asl	1.9.13
org.codehaus.jackson	jackson-jaxrs	1.9.13
org.codehaus.jackson	jackson-mapper-asl	1.9.13
org.codehaus.jackson	jackson-xc	1.9.13
org.codehaus.janino	commons-compiler	3.0.16
org.codehaus.janino	janino	3.0.16
org.datanucleus	datanucleus-api-jdo	4.2.4
org.datanucleus	datanucleus-core	4.1.17
org.datanucleus	datanucleus-rdbms	4.1.19
org.datanucleus	javax.jdo	3.2.0-m3
org.eclipse.jetty	jetty-client	9.4.36.v20210114
org.eclipse.jetty	jetty-continuation	9.4.36.v20210114
org.eclipse.jetty	jetty-http	9.4.36.v20210114
org.eclipse.jetty	jetty-io	9.4.36.v20210114
org.eclipse.jetty	jetty-jndi	9.4.36.v20210114
org.eclipse.jetty	jetty-plus	9.4.36.v20210114
org.eclipse.jetty	jetty-proxy	9.4.36.v20210114
org.eclipse.jetty	jetty-security	9.4.36.v20210114
org.eclipse.jetty	jetty-server	9.4.36.v20210114
org.eclipse.jetty	jetty-servlet	9.4.36.v20210114
org.eclipse.jetty	jetty-servlets	9.4.36.v20210114
org.eclipse.jetty	jetty-util	9.4.36.v20210114
org.eclipse.jetty	jetty-util-ajax	9.4.36.v20210114
org.eclipse.jetty	jetty-webapp	9.4.36.v20210114
org.eclipse.jetty	jetty-xml	9.4.36.v20210114
org.fusesource.leveldbjni	leveldbjni-all	1.8
org.glassfish.hk2	hk2-api	2.6.1
org.glassfish.hk2	hk2-locator	2.6.1
org.glassfish.hk2	hk2-utils	2.6.1
org.glassfish.hk2	osgi-resource-locator	1.0.3
org.glassfish.hk2.external	aopalliance-repackaged	2.6.1
org.glassfish.hk2.external	jakarta.inject	2.6.1
org.glassfish.jersey.containers	jersey-container-servlet	2.30
org.glassfish.jersey.containers	jersey-container-servlet-core	2.30
org.glassfish.jersey.core	jersey-client	2.30
org.glassfish.jersey.core	jersey-common	2.30
org.glassfish.jersey.core	jersey-server	2.30
org.glassfish.jersey.inject	jersey-hk2	2.30
org.glassfish.jersey.media	jersey-media-jaxb	2.30
org.hibernate.validator	hibernate-validator	6.1.0.Final
org.javassist	javassist	3.25.0-GA
org.jboss.logging	jboss-logging	3.3.2.Final
org.jdbi	jdbi	2.63.1
org.joda	joda-convert	1.7
org.jodd	jodd-core	3.5.2
org.json4s	json4s-ast_2.12	3.7.0-M5
org.json4s	json4s-core_2.12	3.7.0-M5
org.json4s	json4s-jackson_2.12	3.7.0-M5
org.json4s	json4s-scalap_2.12	3.7.0-M5
org.lz4	lz4-java	1.7.1
org.mariadb.jdbc	mariadb-java-client	2.2.5
org.objenesis	objenesis	2.5.1
org.postgresql	postgresql	42.1.4
org.roaringbitmap	RoaringBitmap	0.9.14
org.roaringbitmap	shims	0.9.14
org.rocksdb	rocksdbjni	6.20.3
org.rosuda.REngine	REngine	2.1.0
org.scala-lang	scala-compiler_2.12	2.12.10
org.scala-lang	scala-library_2.12	2.12.10
org.scala-lang	scala-reflect_2.12	2.12.10
org.scala-lang.modules	scala-collection-compat_2.12	2.1.1
org.scala-lang.modules	scala-parser-combinators_2.12	1.1.2
org.scala-lang.modules	scala-xml_2.12	1.2.0
org.scala-sbt	test-interface	1.0
org.scalacheck	scalacheck_2.12	1.14.2
org.scalactic	scalactic_2.12	3.0.8
org.scalanlp	breeze-macros_2.12	1.0
org.scalanlp	breeze_2.12	1.0
org.scalatest	scalatest_2.12	3.0.8
org.slf4j	jcl-over-slf4j	1.7.30
org.slf4j	jul-to-slf4j	1.7.30
org.slf4j	slf4j-api	1.7.30
org.slf4j	slf4j-log4j12	1.7.30
org.spark-project.spark	unused	1.0.0
org.springframework	spring-core	4.1.4.RELEASE
org.springframework	spring-test	4.1.4.RELEASE
org.threeten	threeten-extra	1.5.0
org.tukaani	xz	1.5
org.typelevel	algebra_2.12	2.0.0-M2
org.typelevel	cats-kernel_2.12	2.0.0-M4
org.typelevel	machinist_2.12	0.6.8
org.typelevel	macro-compat_2.12	1.1.1
org.typelevel	spire-macros_2.12	0.17.0-M1
org.typelevel	spire-platform_2.12	0.17.0-M1
org.typelevel	spire-util_2.12	0.17.0-M1
org.typelevel	spire_2.12	0.17.0-M1
org.wildfly.openssl	wildfly-openssl	1.0.7.Final
org.xerial	sqlite-jdbc	3.8.11.2
org.xerial.snappy	snappy-java	1.1.8.2
org.yaml	snakeyaml	1.24
oro	oro	2.0.8
pl.edu.icm	JLargeArrays	1.5
software.amazon.ion	ion-java	1.0.2
stax	stax-api	1.0.1
xmlenc	xmlenc	0.52