Databricks Runtime 9.1 LTS

The following release notes provide information about Databricks Runtime 9.1 LTS and Databricks Runtime 9.1 LTS Photon, powered by Apache Spark 3.1.2. Databricks released these images in September 2021. Photon is in Public Preview.

Note

LTS means this version is under long-term support. See Databricks Runtime LTS version lifecycle.

Note

These release notes may include references to features that are not available on Google Cloud as of this release.

New features and improvements

Auto Loader schema hints now work with array and map types

Array and map types are supported in Override schema inference with schema hints for Auto Loader.

Examples of schema hints for arrays include:

  1. arr Array<TYPE> changes the array type.

  2. arr.element TYPE changes the array type (by using the element keyword).

  3. arr.element.x TYPE changes a nested field type in an array of structures.

The first two examples are hints to change the type of array arr to TYPE, but they use different syntax. In the third example, arr is an array of structures with a field x. This example shows how to change the type of x to TYPE.

Examples of schema hints for maps include:

  1. m Map<KEY-TYPE, VALUE-TYPE> changes map key and value types.

  2. m.key TYPE changes the type of map keys.

  3. m.value TYPE changes the type of map values.

  4. m.key.x TYPE changes the field type in a map key.

  5. m.value.x TYPE changes the field type of a map value.

The first example changes both the key and value types of map m to KEY_TYPE and VALUE_TYPE respectively. The second and third examples can be used if only the key type or only the value type needs to be changed. In the fourth and fifth examples, m is a map with key and value of structure types with a field x. This example shows how to change the type of x to TYPE.

Avro file support for merge schema

The Avro file format now supports the mergeSchema option when reading files. Setting mergeSchema to true when reading Avro files will infer a schema from a set of Avro files rather than from a single file. This improves usability by inferring a schema that may be able to read all files even if their individual schemas differ. See Configuration.

Auto Loader incremental listing support (Public Preview)

In the case of lexicographically generated files, What is Auto Loader? now leverages lexical file ordering and existing optimized APIs to make the directory listing more efficient by listing from previously-ingested files rather than by listing the entire directory. Auto Loader automatically detects whether a given directory is suitable for incremental listing by default. To control this behavior explicitly, set the new cloudFiles.useIncrementalListing option to on (true), off (false), or automatic (auto). If you set this behavior to true, you can also set the cloudFiles.backfillInterval option to schedule regular backfills over your data, to make sure all of your data is completely ingested.

Delta now supports arbitrary replaceWhere

In combination with overwrite mode, the replaceWhere option can be used to simultaneously overwrite data that matches a predicate defined in the option. Previously, replaceWhere supported a predicate only over partition columns, but it can now be an arbitrary expression. See Write to a table.

Auto Loader for Google Cloud now supports file notifications (Public Preview)

Auto Loader now supports file notification mode on Google Cloud. Set .option("cloudFiles.useNotifications", "true") to allow Auto Loader to automatically set up Google Cloud Pub/Sub resources for you. With file notification mode, new files are detected and ingested as they arrive without listing the input directory. See What is Auto Loader file notification mode?.

CREATE FUNCTION now supports creating table functions

In addition to creating a scalar function that returns a scalar value, you can now create a table function that returns a set of rows. See CREATE FUNCTION (SQL and Python).

Kafka Streaming Source now reports estimatedTotalBytesBehindLatest metric

The Kafka streaming source now reports an estimate of how many bytes the consumer is behind the latest available byte after every batch. You can use this metric to track stream progress. See Retrieve Kafka metrics.

Example metric output:

StreamingQueryProgress {
  "batchId": 0,
  .....
  "sources": [ {
    "description" : "KafkaV2[Subscribe[topic-0]]",
    "metrics":{
      "avgOffsetsBehindLatest" : "1.0",
      "estimatedTotalBytesBehindLatest" : "80.0", // new
      "maxOffsetsBehindLatest" : "1",
      "minOffsetsBehindLatest" : "1"
    } ],
  ....
 }

For structs inside of arrays, Delta MERGE INTO now resolves struct fields by name and evolves struct schemas

Delta MERGE INTO now supports resolution of struct fields by name and automatic schema evolution for arrays of structs. When automatic schema evolution is enabled by setting spark.databricks.delta.schema.autoMerge.enabled to true, UPDATE and INSERT clauses will resolve struct fields inside of an array by name, casting to the corresponding data type that is defined in the target array and filling additional or missing fields in the source or target with null values. When automatic schema evolution is disabled, UPDATE and INSERT clauses will resolve struct fields inside of an array by name but will not be able to evolve the additional fields. See Automatic schema evolution for arrays of structs.

Bug fixes

  • Fixed a memory leak in the Amazon S3 connector that could happen in long running jobs or services, which was caused by JVM DeleteOnExit functionality.

Library upgrades

  • Upgraded Python libraries:

    • plotly from 4.14.3 to 5.1.0

  • Upgraded R libraries:

    • base from 4.1.0 to 4.1.1

    • bslib from 0.2.5.1 to 0.3.0

    • cachem from 1.0.5 to 1.0.6

    • compiler from 4.1.0 to 4.1.1

    • datasets from 4.1.0 to 4.1.1

    • future from 1.21.0 to 1.22.1

    • gert from 1.3.1 to 1.3.2

    • graphics from 4.1.0 to 4.1.1

    • grDevices from 4.1.0 to 4.1.1

    • grid from 4.1.0 to 4.1.1

  • Upgraded Java libraries:

    • org.eclipse.jetty from 9.4.36.v20210114 to 9.4.42.v20210604

Apache Spark

Databricks Runtime 9.1 LTS includes Apache Spark 3.1.2. This release includes all Spark fixes and improvements included in Databricks Runtime 9.0 (unsupported), as well as the following additional bug fixes and improvements made to Spark:

  • [SPARK-36674][SQL][CHERRY-PICK] Support ILIKE - case insensitive LIKE

  • [SPARK-36353][SQL][3.1] RemoveNoopOperators should keep output schema

  • [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer

  • [SPARK-36398][SQL] Redact sensitive information in Spark Thrift Server log

  • [SPARK-36498][SQL] Reorder inner fields of the input query in byName V2 write

  • [SPARK-36614][CORE][UI] Correct executor loss reason caused by decommission in UI

  • [SPARK-36012][SQL] Add null flag in SHOW CREATE TABLE

  • [SPARK-36509][CORE] Fix the issue that executors are never re-scheduled if the worker stops with standalone cluster

  • [SPARK-36603][CORE] Use WeakReference not SoftReference in LevelDB

  • [SPARK-36564][CORE] Fix NullPointerException in LiveRDDDistribution.toApi

  • [SPARK-36086][SQL][3.1] CollapseProject project replace alias should use origin column name

  • [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases

  • [SPARK-36400][SPARK-36398][SQL][WEBUI] Make ThriftServer recognize spark.sql.redaction.string.regex

  • [SPARK-34054][CORE] BlockManagerDecommissioner code cleanup

  • [SPARK-36500][CORE] Fix temp_shuffle file leaking when a task is interrupted

  • [SPARK-36489][SQL] Aggregate functions over no grouping keys, on tables with a single bucket, return multiple rows

  • [SPARK-36464][CORE] Fix Underlying Size Variable Initialization in ChunkedByteBufferOutputStream for Writing Over 2GB Data

  • [SPARK-36339][SQL][3.0] References to grouping that not part of aggregation should be replaced

  • [SPARK-36354][CORE] EventLogFileReader should skip rolling event log directories with no logs

  • [SPARK-36242][CORE][3.1] Ensure spill file closed before set success = true in ExternalSorter.spillMemoryIteratorToDisk method

  • [SPARK-36211][PYTHON] Correct typing of udf return value

  • [SPARK-34222][SQL] Enhance boolean simplification rule

  • [SPARK-35027][CORE] Close the inputStream in FileAppender when writin…

  • [SPARK-36269][SQL] Fix only set data columns to Hive column names config

  • [SPARK-36213][SQL] Normalize PartitionSpec for Describe Table Command with PartitionSpec

  • [SPARK-36210][SQL] Preserve column insertion order in Dataset.withColumns

  • [SPARK-36079][SQL] Null-based filter estimate should always be in the range [0, 1]

  • [SPARK-28266][SQL] convertToLogicalRelation should not interpret path property when reading Hive tables

System environment

  • Operating System: Ubuntu 20.04.4 LTS

  • Java: Zulu 8.56.0.21-CA-linux64

  • Scala: 2.12.10

  • Python: 3.8.10

  • R: 4.1.1

  • Delta Lake: 1.0.0

Installed Python libraries

Library

Version

Library

Version

Library

Version

Antergos Linux

2015.10 (ISO-Rolling)

appdirs

1.4.4

backcall

0.2.0

boto3

1.16.7

botocore

1.19.7

certifi

2020.12.5

chardet

4.0.0

cycler

0.10.0

Cython

0.29.23

dbus-python

1.2.16

decorator

5.0.6

distlib

0.3.2

distro-info

0.23ubuntu1

facets-overview

1.0.0

filelock

3.0.12

idna

2.10

ipykernel

5.3.4

ipython

7.22.0

ipython-genutils

0.2.0

jedi

0.17.2

jmespath

0.10.0

joblib

1.0.1

jupyter-client

6.1.12

jupyter-core

4.7.1

kiwisolver

1.3.1

koalas

1.8.1

matplotlib

3.4.2

numpy

1.19.2

pandas

1.2.4

parso

0.7.0

patsy

0.5.1

pexpect

4.8.0

pickleshare

0.7.5

Pillow

8.2.0

pip

21.0.1

plotly

5.1.0

prompt-toolkit

3.0.17

protobuf

3.17.2

psycopg2

2.8.5

ptyprocess

0.7.0

pyarrow

4.0.0

Pygments

2.8.1

PyGObject

3.36.0

pyparsing

2.4.7

python-apt

2.0.0+ubuntu0.20.4.5

python-dateutil

2.8.1

pytz

2020.5

pyzmq

20.0.0

requests

2.25.1

requests-unixsocket

0.2.0

s3transfer

0.3.7

scikit-learn

0.24.1

scipy

1.6.2

seaborn

0.11.1

setuptools

52.0.0

six

1.15.0

ssh-import-id

5.10

statsmodels

0.12.2

tenacity

8.0.1

threadpoolctl

2.1.0

tornado

6.1

traitlets

5.0.5

unattended-upgrades

0.1

urllib3

1.25.11

virtualenv

20.4.1

wcwidth

0.2.5

wheel

0.36.2

Installed R libraries

R libraries are installed from the Microsoft CRAN snapshot on 2021-09-08.

Library

Version

Library

Version

Library

Version

askpass

1.1

assertthat

0.2.1

backports

1.2.1

base

4.1.1

base64enc

0.1-3

bit

4.0.4

bit64

4.0.5

blob

1.2.2

boot

1.3-28

brew

1.0-6

brio

1.1.2

broom

0.7.9

bslib

0.3.0

cachem

1.0.6

callr

3.7.0

caret

6.0-88

cellranger

1.1.0

chron

2.3-56

class

7.3-19

cli

3.0.1

clipr

0.7.1

cluster

2.1.2

codetools

0.2-18

colorspace

2.0-2

commonmark

1.7

compiler

4.1.1

config

0.3.1

cpp11

0.3.1

crayon

1.4.1

credentials

1.3.1

curl

4.3.2

data.table

1.14.0

datasets

4.1.1

DBI

1.1.1

dbplyr

2.1.1

desc

1.3.0

devtools

2.4.2

diffobj

0.3.4

digest

0.6.27

dplyr

1.0.7

dtplyr

1.1.0

ellipsis

0.3.2

evaluate

0.14

fansi

0.5.0

farver

2.1.0

fastmap

1.1.0

forcats

0.5.1

foreach

1.5.1

foreign

0.8-81

forge

0.2.0

fs

1.5.0

future

1.22.1

future.apply

1.8.1

gargle

1.2.0

generics

0.1.0

gert

1.3.2

ggplot2

3.3.5

gh

1.3.0

gitcreds

0.1.1

glmnet

4.1-2

globals

0.14.0

glue

1.4.2

googledrive

2.0.0

googlesheets4

1.0.0

gower

0.2.2

graphics

4.1.1

grDevices

4.1.1

grid

4.1.1

gridExtra

2.3

gsubfn

0.7

gtable

0.3.0

haven

2.4.3

highr

0.9

hms

1.1.0

htmltools

0.5.2

htmlwidgets

1.5.3

httpuv

1.6.2

httr

1.4.2

hwriter

1.3.2

hwriterPlus

1.0-3

ids

1.0.1

ini

0.3.1

ipred

0.9-11

isoband

0.2.5

iterators

1.0.13

jquerylib

0.1.4

jsonlite

1.7.2

KernSmooth

2.23-20

knitr

1.33

labeling

0.4.2

later

1.3.0

lattice

0.20-44

lava

1.6.10

lifecycle

1.0.0

listenv

0.8.0

lubridate

1.7.10

magrittr

2.0.1

markdown

1.1

MASS

7.3-54

Matrix

1.3-4

memoise

2.0.0

methods

4.1.1

mgcv

1.8-36

mime

0.11

ModelMetrics

1.2.2.2

modelr

0.1.8

munsell

0.5.0

nlme

3.1-152

nnet

7.3-16

numDeriv

2016.8-1.1

openssl

1.4.5

parallel

4.1.1

parallelly

1.27.0

pillar

1.6.2

pkgbuild

1.2.0

pkgconfig

2.0.3

pkgload

1.2.1

plogr

0.2.0

plyr

1.8.6

praise

1.0.0

prettyunits

1.1.1

pROC

1.18.0

processx

3.5.2

prodlim

2019.11.13

progress

1.2.2

progressr

0.8.0

promises

1.2.0.1

proto

1.0.0

ps

1.6.0

purrr

0.3.4

r2d3

0.2.5

R6

2.5.1

randomForest

4.6-14

rappdirs

0.3.3

rcmdcheck

1.3.3

RColorBrewer

1.1-2

Rcpp

1.0.7

readr

2.0.1

readxl

1.3.1

recipes

0.1.16

rematch

1.0.1

rematch2

2.1.2

remotes

2.4.0

reprex

2.0.1

reshape2

1.4.4

rlang

0.4.11

rmarkdown

2.10

RODBC

1.3-18

roxygen2

7.1.1

rpart

4.1-15

rprojroot

2.0.2

Rserve

1.8-8

RSQLite

2.2.8

rstudioapi

0.13

rversions

2.1.1

rvest

1.0.1

sass

0.4.0

scales

1.1.1

selectr

0.4-2

sessioninfo

1.1.1

shape

1.4.6

shiny

1.6.0

sourcetools

0.1.7

sparklyr

1.7.1

SparkR

3.1.1

spatial

7.3-11

splines

4.1.1

sqldf

0.4-11

SQUAREM

2021.1

stats

4.1.1

stats4

4.1.1

stringi

1.7.4

stringr

1.4.0

survival

3.2-13

sys

3.4

tcltk

4.1.1

TeachingDemos

2.10

testthat

3.0.4

tibble

3.1.4

tidyr

1.1.3

tidyselect

1.1.1

tidyverse

1.3.1

timeDate

3043.102

tinytex

0.33

tools

4.1.1

tzdb

0.1.2

usethis

2.0.1

utf8

1.2.2

utils

4.1.1

uuid

0.1-4

vctrs

0.3.8

viridisLite

0.4.0

vroom

1.5.4

waldo

0.3.0

whisker

0.4

withr

2.4.2

xfun

0.25

xml2

1.3.2

xopen

1.0.0

xtable

1.8-4

yaml

2.2.1

zip

2.2.0

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID

Artifact ID

Version

antlr

antlr

2.7.7

com.amazonaws

amazon-kinesis-client

1.12.0

com.amazonaws

aws-java-sdk-autoscaling

1.11.655

com.amazonaws

aws-java-sdk-cloudformation

1.11.655

com.amazonaws

aws-java-sdk-cloudfront

1.11.655

com.amazonaws

aws-java-sdk-cloudhsm

1.11.655

com.amazonaws

aws-java-sdk-cloudsearch

1.11.655

com.amazonaws

aws-java-sdk-cloudtrail

1.11.655

com.amazonaws

aws-java-sdk-cloudwatch

1.11.655

com.amazonaws

aws-java-sdk-cloudwatchmetrics

1.11.655

com.amazonaws

aws-java-sdk-codedeploy

1.11.655

com.amazonaws

aws-java-sdk-cognitoidentity

1.11.655

com.amazonaws

aws-java-sdk-cognitosync

1.11.655

com.amazonaws

aws-java-sdk-config

1.11.655

com.amazonaws

aws-java-sdk-core

1.11.655

com.amazonaws

aws-java-sdk-datapipeline

1.11.655

com.amazonaws

aws-java-sdk-directconnect

1.11.655

com.amazonaws

aws-java-sdk-directory

1.11.655

com.amazonaws

aws-java-sdk-dynamodb

1.11.655

com.amazonaws

aws-java-sdk-ec2

1.11.655

com.amazonaws

aws-java-sdk-ecs

1.11.655

com.amazonaws

aws-java-sdk-efs

1.11.655

com.amazonaws

aws-java-sdk-elasticache

1.11.655

com.amazonaws

aws-java-sdk-elasticbeanstalk

1.11.655

com.amazonaws

aws-java-sdk-elasticloadbalancing

1.11.655

com.amazonaws

aws-java-sdk-elastictranscoder

1.11.655

com.amazonaws

aws-java-sdk-emr

1.11.655

com.amazonaws

aws-java-sdk-glacier

1.11.655

com.amazonaws

aws-java-sdk-glue

1.11.655

com.amazonaws

aws-java-sdk-iam

1.11.655

com.amazonaws

aws-java-sdk-importexport

1.11.655

com.amazonaws

aws-java-sdk-kinesis

1.11.655

com.amazonaws

aws-java-sdk-kms

1.11.655

com.amazonaws

aws-java-sdk-lambda

1.11.655

com.amazonaws

aws-java-sdk-logs

1.11.655

com.amazonaws

aws-java-sdk-machinelearning

1.11.655

com.amazonaws

aws-java-sdk-marketplacecommerceanalytics

1.11.655

com.amazonaws

aws-java-sdk-marketplacemeteringservice

1.11.655

com.amazonaws

aws-java-sdk-opsworks

1.11.655

com.amazonaws

aws-java-sdk-rds

1.11.655

com.amazonaws

aws-java-sdk-redshift

1.11.655

com.amazonaws

aws-java-sdk-route53

1.11.655

com.amazonaws

aws-java-sdk-s3

1.11.655

com.amazonaws

aws-java-sdk-ses

1.11.655

com.amazonaws

aws-java-sdk-simpledb

1.11.655

com.amazonaws

aws-java-sdk-simpleworkflow

1.11.655

com.amazonaws

aws-java-sdk-sns

1.11.655

com.amazonaws

aws-java-sdk-sqs

1.11.655

com.amazonaws

aws-java-sdk-ssm

1.11.655

com.amazonaws

aws-java-sdk-storagegateway

1.11.655

com.amazonaws

aws-java-sdk-sts

1.11.655

com.amazonaws

aws-java-sdk-support

1.11.655

com.amazonaws

aws-java-sdk-swf-libraries

1.11.22

com.amazonaws

aws-java-sdk-workspaces

1.11.655

com.amazonaws

jmespath-java

1.11.655

com.chuusai

shapeless_2.12

2.3.3

com.clearspring.analytics

stream

2.9.6

com.databricks

Rserve

1.8-3

com.databricks

jets3t

0.7.1-0

com.databricks.scalapb

compilerplugin_2.12

0.4.15-10

com.databricks.scalapb

scalapb-runtime_2.12

0.4.15-10

com.esotericsoftware

kryo-shaded

4.0.2

com.esotericsoftware

minlog

1.3.0

com.fasterxml

classmate

1.3.4

com.fasterxml.jackson.core

jackson-annotations

2.10.0

com.fasterxml.jackson.core

jackson-core

2.10.0

com.fasterxml.jackson.core

jackson-databind

2.10.0

com.fasterxml.jackson.dataformat

jackson-dataformat-cbor

2.10.0

com.fasterxml.jackson.datatype

jackson-datatype-joda

2.10.0

com.fasterxml.jackson.module

jackson-module-paranamer

2.10.0

com.fasterxml.jackson.module

jackson-module-scala_2.12

2.10.0

com.github.ben-manes.caffeine

caffeine

2.3.4

com.github.fommil

jniloader

1.1

com.github.fommil.netlib

core

1.1.2

com.github.fommil.netlib

native_ref-java

1.1

com.github.fommil.netlib

native_ref-java-natives

1.1

com.github.fommil.netlib

native_system-java

1.1

com.github.fommil.netlib

native_system-java-natives

1.1

com.github.fommil.netlib

netlib-native_ref-linux-x86_64-natives

1.1

com.github.fommil.netlib

netlib-native_system-linux-x86_64-natives

1.1

com.github.joshelser

dropwizard-metrics-hadoop-metrics2-reporter

0.1.2

com.github.luben

zstd-jni

1.4.8-1

com.github.wendykierp

JTransforms

3.1

com.google.code.findbugs

jsr305

3.0.0

com.google.code.gson

gson

2.2.4

com.google.crypto.tink

tink

1.6.0

com.google.flatbuffers

flatbuffers-java

1.9.0

com.google.guava

guava

15.0

com.google.protobuf

protobuf-java

2.6.1

com.h2database

h2

1.4.195

com.helger

profiler

1.1.1

com.jcraft

jsch

0.1.50

com.jolbox

bonecp

0.8.0.RELEASE

com.lihaoyi

sourcecode_2.12

0.1.9

com.microsoft.azure

azure-data-lake-store-sdk

2.3.9

com.microsoft.sqlserver

mssql-jdbc

9.2.1.jre8

com.ning

compress-lzf

1.0.3

com.sun.mail

javax.mail

1.5.2

com.tdunning

json

1.8

com.thoughtworks.paranamer

paranamer

2.8

com.trueaccord.lenses

lenses_2.12

0.4.12

com.twitter

chill-java

0.9.5

com.twitter

chill_2.12

0.9.5

com.twitter

util-app_2.12

7.1.0

com.twitter

util-core_2.12

7.1.0

com.twitter

util-function_2.12

7.1.0

com.twitter

util-jvm_2.12

7.1.0

com.twitter

util-lint_2.12

7.1.0

com.twitter

util-registry_2.12

7.1.0

com.twitter

util-stats_2.12

7.1.0

com.typesafe

config

1.2.1

com.typesafe.scala-logging

scala-logging_2.12

3.7.2

com.univocity

univocity-parsers

2.9.1

com.zaxxer

HikariCP

3.1.0

commons-beanutils

commons-beanutils

1.9.4

commons-cli

commons-cli

1.2

commons-codec

commons-codec

1.10

commons-collections

commons-collections

3.2.2

commons-configuration

commons-configuration

1.6

commons-dbcp

commons-dbcp

1.4

commons-digester

commons-digester

1.8

commons-fileupload

commons-fileupload

1.3.3

commons-httpclient

commons-httpclient

3.1

commons-io

commons-io

2.4

commons-lang

commons-lang

2.6

commons-logging

commons-logging

1.1.3

commons-net

commons-net

3.1

commons-pool

commons-pool

1.5.4

hive-2.3__hadoop-2.7

jets3t-0.7

liball_deps_2.12

hive-2.3__hadoop-2.7

zookeeper-3.4

liball_deps_2.12

info.ganglia.gmetric4j

gmetric4j

1.0.10

io.airlift

aircompressor

0.10

io.delta

delta-sharing-spark_2.12

0.1.0

io.dropwizard.metrics

metrics-core

4.1.1

io.dropwizard.metrics

metrics-graphite

4.1.1

io.dropwizard.metrics

metrics-healthchecks

4.1.1

io.dropwizard.metrics

metrics-jetty9

4.1.1

io.dropwizard.metrics

metrics-jmx

4.1.1

io.dropwizard.metrics

metrics-json

4.1.1

io.dropwizard.metrics

metrics-jvm

4.1.1

io.dropwizard.metrics

metrics-servlets

4.1.1

io.netty

netty-all

4.1.51.Final

io.prometheus

simpleclient

0.7.0

io.prometheus

simpleclient_common

0.7.0

io.prometheus

simpleclient_dropwizard

0.7.0

io.prometheus

simpleclient_pushgateway

0.7.0

io.prometheus

simpleclient_servlet

0.7.0

io.prometheus.jmx

collector

0.12.0

jakarta.annotation

jakarta.annotation-api

1.3.5

jakarta.validation

jakarta.validation-api

2.0.2

jakarta.ws.rs

jakarta.ws.rs-api

2.1.6

javax.activation

activation

1.1.1

javax.el

javax.el-api

2.2.4

javax.jdo

jdo-api

3.0.1

javax.servlet

javax.servlet-api

3.1.0

javax.servlet.jsp

jsp-api

2.1

javax.transaction

jta

1.1

javax.transaction

transaction-api

1.1

javax.xml.bind

jaxb-api

2.2.2

javax.xml.stream

stax-api

1.0-2

javolution

javolution

5.5.1

jline

jline

2.14.6

joda-time

joda-time

2.10.5

log4j

apache-log4j-extras

1.2.17

log4j

log4j

1.2.17

maven-trees

hive-2.3__hadoop-2.7

liball_deps_2.12

net.java.dev.jna

jna

5.8.0

net.razorvine

pyrolite

4.30

net.sf.jpam

jpam

1.1

net.sf.opencsv

opencsv

2.3

net.sf.supercsv

super-csv

2.2.0

net.snowflake

snowflake-ingest-sdk

0.9.6

net.snowflake

snowflake-jdbc

3.13.3

net.snowflake

spark-snowflake_2.12

2.9.0-spark_3.1

net.sourceforge.f2j

arpack_combined_all

0.1

org.acplt.remotetea

remotetea-oncrpc

1.1.2

org.antlr

ST4

4.0.4

org.antlr

antlr-runtime

3.5.2

org.antlr

antlr4-runtime

4.8-1

org.antlr

stringtemplate

3.2.1

org.apache.ant

ant

1.9.2

org.apache.ant

ant-jsch

1.9.2

org.apache.ant

ant-launcher

1.9.2

org.apache.arrow

arrow-format

2.0.0

org.apache.arrow

arrow-memory-core

2.0.0

org.apache.arrow

arrow-memory-netty

2.0.0

org.apache.arrow

arrow-vector

2.0.0

org.apache.avro

avro

1.8.2

org.apache.avro

avro-ipc

1.8.2

org.apache.avro

avro-mapred-hadoop2

1.8.2

org.apache.commons

commons-compress

1.20

org.apache.commons

commons-crypto

1.1.0

org.apache.commons

commons-lang3

3.10

org.apache.commons

commons-math3

3.4.1

org.apache.commons

commons-text

1.6

org.apache.curator

curator-client

2.7.1

org.apache.curator

curator-framework

2.7.1

org.apache.curator

curator-recipes

2.7.1

org.apache.derby

derby

10.12.1.1

org.apache.directory.api

api-asn1-api

1.0.0-M20

org.apache.directory.api

api-util

1.0.0-M20

org.apache.directory.server

apacheds-i18n

2.0.0-M15

org.apache.directory.server

apacheds-kerberos-codec

2.0.0-M15

org.apache.hadoop

hadoop-annotations

2.7.4

org.apache.hadoop

hadoop-auth

2.7.4

org.apache.hadoop

hadoop-client

2.7.4

org.apache.hadoop

hadoop-common

2.7.4

org.apache.hadoop

hadoop-hdfs

2.7.4

org.apache.hadoop

hadoop-mapreduce-client-app

2.7.4

org.apache.hadoop

hadoop-mapreduce-client-common

2.7.4

org.apache.hadoop

hadoop-mapreduce-client-core

2.7.4

org.apache.hadoop

hadoop-mapreduce-client-jobclient

2.7.4

org.apache.hadoop

hadoop-mapreduce-client-shuffle

2.7.4

org.apache.hadoop

hadoop-yarn-api

2.7.4

org.apache.hadoop

hadoop-yarn-client

2.7.4

org.apache.hadoop

hadoop-yarn-common

2.7.4

org.apache.hadoop

hadoop-yarn-server-common

2.7.4

org.apache.hive

hive-beeline

2.3.7

org.apache.hive

hive-cli

2.3.7

org.apache.hive

hive-jdbc

2.3.7

org.apache.hive

hive-llap-client

2.3.7

org.apache.hive

hive-llap-common

2.3.7

org.apache.hive

hive-serde

2.3.7

org.apache.hive

hive-shims

2.3.7

org.apache.hive

hive-storage-api

2.7.2

org.apache.hive.shims

hive-shims-0.23

2.3.7

org.apache.hive.shims

hive-shims-common

2.3.7

org.apache.hive.shims

hive-shims-scheduler

2.3.7

org.apache.htrace

htrace-core

3.1.0-incubating

org.apache.httpcomponents

httpclient

4.5.6

org.apache.httpcomponents

httpcore

4.4.12

org.apache.ivy

ivy

2.4.0

org.apache.mesos

mesos-shaded-protobuf

1.4.0

org.apache.orc

orc-core

1.5.12

org.apache.orc

orc-mapreduce

1.5.12

org.apache.orc

orc-shims

1.5.12

org.apache.parquet

parquet-column

1.10.1-databricks9

org.apache.parquet

parquet-common

1.10.1-databricks9

org.apache.parquet

parquet-encoding

1.10.1-databricks9

org.apache.parquet

parquet-format

2.4.0

org.apache.parquet

parquet-hadoop

1.10.1-databricks9

org.apache.parquet

parquet-jackson

1.10.1-databricks9

org.apache.thrift

libfb303

0.9.3

org.apache.thrift

libthrift

0.12.0

org.apache.xbean

xbean-asm7-shaded

4.15

org.apache.yetus

audience-annotations

0.5.0

org.apache.zookeeper

zookeeper

3.4.14

org.codehaus.jackson

jackson-core-asl

1.9.13

org.codehaus.jackson

jackson-jaxrs

1.9.13

org.codehaus.jackson

jackson-mapper-asl

1.9.13

org.codehaus.jackson

jackson-xc

1.9.13

org.codehaus.janino

commons-compiler

3.0.16

org.codehaus.janino

janino

3.0.16

org.datanucleus

datanucleus-api-jdo

4.2.4

org.datanucleus

datanucleus-core

4.1.17

org.datanucleus

datanucleus-rdbms

4.1.19

org.datanucleus

javax.jdo

3.2.0-m3

org.eclipse.jetty

jetty-client

9.4.42.v20210604

org.eclipse.jetty

jetty-continuation

9.4.42.v20210604

org.eclipse.jetty

jetty-http

9.4.42.v20210604

org.eclipse.jetty

jetty-io

9.4.42.v20210604

org.eclipse.jetty

jetty-jndi

9.4.42.v20210604

org.eclipse.jetty

jetty-plus

9.4.42.v20210604

org.eclipse.jetty

jetty-proxy

9.4.42.v20210604

org.eclipse.jetty

jetty-security

9.4.42.v20210604

org.eclipse.jetty

jetty-server

9.4.42.v20210604

org.eclipse.jetty

jetty-servlet

9.4.42.v20210604

org.eclipse.jetty

jetty-servlets

9.4.42.v20210604

org.eclipse.jetty

jetty-util

9.4.42.v20210604

org.eclipse.jetty

jetty-util-ajax

9.4.42.v20210604

org.eclipse.jetty

jetty-webapp

9.4.42.v20210604

org.eclipse.jetty

jetty-xml

9.4.42.v20210604

org.eclipse.jetty.websocket

websocket-api

9.4.42.v20210604

org.eclipse.jetty.websocket

websocket-client

9.4.42.v20210604

org.eclipse.jetty.websocket

websocket-common

9.4.42.v20210604

org.eclipse.jetty.websocket

websocket-server

9.4.42.v20210604

org.eclipse.jetty.websocket

websocket-servlet

9.4.42.v20210604

org.fusesource.leveldbjni

leveldbjni-all

1.8

org.glassfish.hk2

hk2-api

2.6.1

org.glassfish.hk2

hk2-locator

2.6.1

org.glassfish.hk2

hk2-utils

2.6.1

org.glassfish.hk2

osgi-resource-locator

1.0.3

org.glassfish.hk2.external

aopalliance-repackaged

2.6.1

org.glassfish.hk2.external

jakarta.inject

2.6.1

org.glassfish.jersey.containers

jersey-container-servlet

2.30

org.glassfish.jersey.containers

jersey-container-servlet-core

2.30

org.glassfish.jersey.core

jersey-client

2.30

org.glassfish.jersey.core

jersey-common

2.30

org.glassfish.jersey.core

jersey-server

2.30

org.glassfish.jersey.inject

jersey-hk2

2.30

org.glassfish.jersey.media

jersey-media-jaxb

2.30

org.hibernate.validator

hibernate-validator

6.1.0.Final

org.javassist

javassist

3.25.0-GA

org.jboss.logging

jboss-logging

3.3.2.Final

org.jdbi

jdbi

2.63.1

org.joda

joda-convert

1.7

org.jodd

jodd-core

3.5.2

org.json4s

json4s-ast_2.12

3.7.0-M5

org.json4s

json4s-core_2.12

3.7.0-M5

org.json4s

json4s-jackson_2.12

3.7.0-M5

org.json4s

json4s-scalap_2.12

3.7.0-M5

org.lz4

lz4-java

1.7.1

org.mariadb.jdbc

mariadb-java-client

2.2.5

org.objenesis

objenesis

2.5.1

org.postgresql

postgresql

42.1.4

org.roaringbitmap

RoaringBitmap

0.9.14

org.roaringbitmap

shims

0.9.14

org.rocksdb

rocksdbjni

6.20.3

org.rosuda.REngine

REngine

2.1.0

org.scala-lang

scala-compiler_2.12

2.12.10

org.scala-lang

scala-library_2.12

2.12.10

org.scala-lang

scala-reflect_2.12

2.12.10

org.scala-lang.modules

scala-collection-compat_2.12

2.1.1

org.scala-lang.modules

scala-parser-combinators_2.12

1.1.2

org.scala-lang.modules

scala-xml_2.12

1.2.0

org.scala-sbt

test-interface

1.0

org.scalacheck

scalacheck_2.12

1.14.2

org.scalactic

scalactic_2.12

3.0.8

org.scalanlp

breeze-macros_2.12

1.0

org.scalanlp

breeze_2.12

1.0

org.scalatest

scalatest_2.12

3.0.8

org.slf4j

jcl-over-slf4j

1.7.30

org.slf4j

jul-to-slf4j

1.7.30

org.slf4j

slf4j-api

1.7.30

org.slf4j

slf4j-log4j12

1.7.30

org.spark-project.spark

unused

1.0.0

org.springframework

spring-core

4.1.4.RELEASE

org.springframework

spring-test

4.1.4.RELEASE

org.threeten

threeten-extra

1.5.0

org.tukaani

xz

1.5

org.typelevel

algebra_2.12

2.0.0-M2

org.typelevel

cats-kernel_2.12

2.0.0-M4

org.typelevel

machinist_2.12

0.6.8

org.typelevel

macro-compat_2.12

1.1.1

org.typelevel

spire-macros_2.12

0.17.0-M1

org.typelevel

spire-platform_2.12

0.17.0-M1

org.typelevel

spire-util_2.12

0.17.0-M1

org.typelevel

spire_2.12

0.17.0-M1

org.wildfly.openssl

wildfly-openssl

1.0.7.Final

org.xerial

sqlite-jdbc

3.8.11.2

org.xerial.snappy

snappy-java

1.1.8.2

org.yaml

snakeyaml

1.24

oro

oro

2.0.8

pl.edu.icm

JLargeArrays

1.5

software.amazon.ion

ion-java

1.0.2

stax

stax-api

1.0.1

xmlenc

xmlenc

0.52