Databricks Runtime 14.1

The following release notes provide information about Databricks Runtime 14.1, powered by Apache Spark 3.5.0.

Databricks released these images in October 2023.

Note

These release notes may include references to features that are not available on Google Cloud as of this release.

New features and improvements

array_insert() is 1-based for negative indexes

The array_insert function is 1-based for both positive and negative indexes. It now inserts new element at the end of input arrays for the index -1. To restore the previous behavior, set spark.sql.legacy.negativeIndexInArrayInsert to true.

Delta v2 checkpoints enabled by default with liquid clustering

Newly created Delta tables with liquid clustering use v2 checkpoints by default. See Compatibility for tables with liquid clustering.

Drop Delta table feature in Public Preview

You can now drop some table features for Delta tables. Current support includes dropping deletionVectors and v2Checkpoint. See Drop Delta table features.

Stream from Unity Catalog views in Public Preview

You can now use Structured Streaming to perform streaming reads from views registered with Unity Catalog. Databricks only supports streaming reads from views defined against Delta tables. See Stream from Unity Catalog views.

Apache Pulsar connector in Public Preview

You can now use Structured Streaming to stream data from Apache Pulsar on Databricks. See Stream from Apache Pulsar.

Upgraded Snowflake driver

The Snowflake JDBC driver now uses version 3.13.33.

SQL Session variables

This release introduces the ability to declare temporary variables in a session which can be set and then referred to from within queries. See Variables.

Table arguments to functions support partitioning and ordering.

You can now use PARTITION BY and ORDER BY clauses to control how table arguments are passed to a function during function invocation.

New and enhanced builtin SQL functions

The following builtin functions have been added:

The following builtin functions have been enhanced:

  • mode: Support for an optional parameter forcing a deterministic result.

  • to_char: New support for DATE, TIMESTAMP, and BINARY.

  • to_varchar: New support for DATE, TIMESTAMP, and BINARY.

Improved handling of correlated subqueries

The ability to process correlation in subqueries has been extended:

  • Handle limit and order by in correlated scalar (lateral) subqueries.

  • Support window functions in correlated scalar subqueries.

  • Support correlated references in join predicates for scalar and lateral subqueries

Behavior change

Strict type checking in Photon Parquet reader

Photon fails when trying to read a decimal value from a Parquet column that is not a decimal type. Photon also fails when reading a fixed-length byte array from Parquet as a string.

Library upgrades

  • Upgraded Python libraries:

    • filelock from 3.12.2 to 3.12.3

    • s3transfer from 0.6.1 to 0.6.2

  • Upgraded Java libraries:

    • com.uber.h3 from 3.7.0 to 3.7.3

    • io.airlift.aircompressor from 0.24 to 0.25

    • io.delta.delta-sharing-spark_2.12 from 0.7.1 to 0.7.5

    • io.netty.netty-all from 4.1.93.Final to 4.1.96.Final

    • io.netty.netty-buffer from 4.1.93.Final to 4.1.96.Final

    • io.netty.netty-codec from 4.1.93.Final to 4.1.96.Final

    • io.netty.netty-codec-http from 4.1.93.Final to 4.1.96.Final

    • io.netty.netty-codec-http2 from 4.1.93.Final to 4.1.96.Final

    • io.netty.netty-codec-socks from 4.1.93.Final to 4.1.96.Final

    • io.netty.netty-common from 4.1.93.Final to 4.1.96.Final

    • io.netty.netty-handler from 4.1.93.Final to 4.1.96.Final

    • io.netty.netty-handler-proxy from 4.1.93.Final to 4.1.96.Final

    • io.netty.netty-resolver from 4.1.93.Final to 4.1.96.Final

    • io.netty.netty-transport from 4.1.93.Final to 4.1.96.Final

    • io.netty.netty-transport-classes-epoll from 4.1.93.Final to 4.1.96.Final

    • io.netty.netty-transport-classes-kqueue from 4.1.93.Final to 4.1.96.Final

    • io.netty.netty-transport-native-epoll from 4.1.93.Final-linux-x86_64 to 4.1.96.Final-linux-x86_64

    • io.netty.netty-transport-native-kqueue from 4.1.93.Final-osx-x86_64 to 4.1.96.Final-osx-x86_64

    • io.netty.netty-transport-native-unix-common from 4.1.93.Final to 4.1.96.Final

    • net.snowflake.snowflake-jdbc from 3.13.29 to 3.13.33

    • org.apache.orc.orc-core from 1.9.0-shaded-protobuf to 1.9.1-shaded-protobuf

    • org.apache.orc.orc-mapreduce from 1.9.0-shaded-protobuf to 1.9.1-shaded-protobuf

    • org.apache.orc.orc-shims from 1.9.0 to 1.9.1

Apache Spark

Databricks Runtime 14.1 includes Apache Spark 3.5.0. This release includes all Spark fixes and improvements included in Databricks Runtime 14.0 (unsupported), as well as the following additional bug fixes and improvements made to Spark:

  • [SPARK-45088] [DBRRM-557] Revert “[SC-142785][PYTHON][CONNECT] Make getitem work with duplicated column”

  • [SPARK-43453] [DBRRM-557]Revert “[SC-143135][PS] Ignore the names of MultiIndex when axis=1 for concat”

  • [SPARK-45225] [SC-143207][SQL] XML: XSD file URL support

  • [SPARK-45156] [SC-142782][SQL] Wrap inputName by backticks in the NON_FOLDABLE_INPUT error class

  • [SPARK-44910] [SC-143082][SQL] Encoders.bean does not support superclasses with generic type arguments

  • [SPARK-43453] [SC-143135][PS] Ignore the names of MultiIndex when axis=1 for concat

  • [SPARK-44463] [SS][CONNECT] Improve error handling for Connect steaming Python worker

  • [SPARK-44960] [SC-141023][UI] Unescape and consist error summary across UI pages

  • [SPARK-44788] [SC-142980][CONNECT][PYTHON][SQL] Add from_xml and schema_of_xml to pyspark, spark connect and sql function

  • [SPARK-44614] [SC-138460][PYTHON][CONNECT][3.5] Add missing packages in setup.py

  • [SPARK-45151] [SC-142861][CORE][UI] Task Level Thread Dump Support

  • [SPARK-45056] [SC-142779][PYTHON][SS][CONNECT] Termination tests for streamingQueryListener and foreachBatch

  • [SPARK-45084] [SC-142828][SS] StateOperatorProgress to use accurate effective shuffle partition number

  • [SPARK-44872] [SC-142405][CONNECT] Server testing infra and ReattachableExecuteSuite

  • [SPARK-45197] [SC-142984][CORE] Make StandaloneRestServer add JavaModuleOptions to drivers

  • [SPARK-44404] [SC-139601][SQL] Assign names to the error class LEGACYERROR_TEMP_[1009,1010,1013,1015,1016,1278]

  • [SPARK-44647] [SC-142297][SQL] Support SPJ where join keys are less than cluster keys

  • [SPARK-45088] [SC-142785][PYTHON][CONNECT] Make getitem work with duplicated columns

  • [SPARK-45128] [SC-142851][SQL] Support CalendarIntervalType in Arrow

  • [SPARK-45130] [SC-142976][CONNECT][ML][PYTHON] Avoid Spark connect ML model to change input pandas dataframe

  • [SPARK-45034] [SC-142959][SQL] Support deterministic mode function

  • [SPARK-45173] [SC-142931][UI] Remove some unnecessary sourceMapping files in UI

  • [SPARK-45162] [SC-142781][SQL] Support maps and array parameters constructed via call_function

  • [SPARK-45143] [SC-142840][PYTHON][CONNECT] Make PySpark compatible with PyArrow 13.0.0

  • [SPARK-45174] [SC-142837][CORE] Support spark.deploy.maxDrivers

  • [SPARK-45167] [SC-142956][CONNECT][PYTHON] Python client must call release_all

  • [SPARK-36191] [SC-142777][SQL] Handle limit and order by in correlated scalar (lateral) subqueries

  • [SPARK-45159] [SC-142829][PYTHON] Handle named arguments only when necessary

  • [SPARK-45133] [SC-142512][CONNECT] Make Spark Connect queries be FINISHED when last result task is finished

  • [SPARK-44801] [SC-140802][SQL][UI] Capture analyzing failed queries in Listener and UI

  • [SPARK-45139] [SC-142527][SQL] Add DatabricksDialect to handle SQL type conversion

  • [SPARK-45157] [SC-142546][SQL] Avoid repeated if checks in [On|Off|HeapColumnVector

  • [SPARK-45077] Revert “[SC-142069][UI] Upgrade dagre-d3.js from 04.3 to 0.6.4”

  • [SPARK-45145] [SC-142521][EXAMPLE] Add JavaSparkSQLCli example

  • [SPARK-43295] Revert “[SC-142254][PS] Support string type columns for DataFrameGroupBy.sum”

  • [SPARK-44915] [SC-142383][CORE] Validate checksum of remounted PVC’s shuffle data before recovery

  • [SPARK-45147] [SC-142524][CORE] Remove System.setSecurityManager usage

  • [SPARK-45104] [SC-142206][UI] Upgrade graphlib-dot.min.js to 1.0.2

  • [SPARK-44238] [SC-141606][CORE][SQL] Introduce a new readFrom method with byte array input for BloomFilter

  • [SPARK-45060] [SC-141742][SQL] Fix an internal error from to_char()on NULL format

  • [SPARK-43252] [SC-142381][SQL] Replace the error class _LEGACY_ERROR_TEMP_2016 with an internal error

  • [SPARK-45069] [SC-142279][SQL] SQL variable should always be resolved after outer reference

  • [SPARK-44911] [SC-142388][SQL] Create hive table with invalid column should return error class

  • [SPARK-42754] [SC-125567][SQL][UI] Fix backward compatibility issue in nested SQL execution

  • [SPARK-45121] [SC-142375][CONNECT][PS] Support Series.empty for Spark Connect.

  • [SPARK-44805] [SC-142203][SQL] getBytes/getShorts/getInts/etc. should work in a column vector that has a dictionary

  • [SPARK-45027] [SC-142248][PYTHON] Hide internal functions/variables in pyspark.sql.functions from auto-completion

  • [SPARK-45073] [SC-141791][PS][CONNECT] Replace LastNotNull with Last(ignoreNulls=True)

  • [SPARK-44901] [SC-141676][SQL] Manual backport: Add API in Python UDTF ‘analyze’ method to return partitioning/ordering expressions

  • [SPARK-45076] [SC-141795][PS] Switch to built-in repeat function

  • [SPARK-44162] [SC-141605][CORE] Support G1GC in spark metrics

  • [SPARK-45053] [SC-141733][PYTHON][MINOR] Log improvement in python version mismatch

  • [SPARK-44866] [SC-142221][SQL] Add SnowflakeDialect to handle BOOLEAN type correctly

  • [SPARK-45064] [SC-141775][PYTHON][CONNECT] Add the missing scale parameter in ceil/ceiling

  • [SPARK-45059] [SC-141757][CONNECT][PYTHON] Add try_reflect functions to Scala and Python

  • [SPARK-43251] [SC-142280][SQL] Replace the error class _LEGACY_ERROR_TEMP_2015 with an internal error

  • [SPARK-45052] [SC-141736][SQL][PYTHON][CONNECT] Make function aliases output column name consistent with SQL

  • [SPARK-44239] [SC-141502][SQL] Free memory allocated by large vectors when vectors are reset

  • [SPARK-43295] [SC-142254][PS] Support string type columns for DataFrameGroupBy.sum

  • [SPARK-45080] [SC-142062][SS] Explicitly call out support for columnar in DSv2 streaming data sources

  • [SPARK-45036] [SC-141768][SQL] SPJ: Simplify the logic to handle partially clustered distribution

  • [SPARK-45077] [SC-142069][UI] Upgrade dagre-d3.js from 0.4.3 to 0.6.4

  • [SPARK-45091] [SC-142020][PYTHON][CONNECT][SQL] Function floor/round/bround accept Column type scale

  • [SPARK-45090] [SC-142019][PYTHON][CONNECT] DataFrame.{cube, rollup} support column ordinals

  • [SPARK-44743] [SC-141625][SQL] Add try_reflect function

  • [SPARK-45086] [SC-142052][UI] Display hexadecimal for thread lock hash code

  • [SPARK-44952] [SC-141644][SQL][PYTHON] Support named arguments in aggregate Pandas UDFs

  • [SPARK-44987] [SC-141552][SQL] Assign a name to the error class _LEGACY_ERROR_TEMP_1100

  • [SPARK-45032] [SC-141730][CONNECT] Fix compilation warnings related to Top-level wildcard is not allowed and will error under -Xsource:3

  • [SPARK-45048] [SC-141629][CONNECT] Add additional tests for Python client and attachable execution

  • [SPARK-45072] [SC-141807][CONNECT] Fix outer scopes for ammonite classes

  • [SPARK-45033] [SC-141759][SQL] Support maps by parameterized sql()

  • [SPARK-45066] [SC-141772][SQL][PYTHON][CONNECT] Make function repeat accept column-type n

  • [SPARK-44860] [SC-141103][SQL] Add SESSION_USER function

  • [SPARK-45074] [SC-141796][PYTHON][CONNECT] DataFrame.{sort, sortWithinPartitions} support column ordinals

  • [SPARK-45047] [SC-141774][PYTHON][CONNECT] DataFrame.groupBy support ordinals

  • [SPARK-44863] [SC-140798][UI] Add a button to download thread dump as a txt in Spark UI

  • [SPARK-45026] [SC-141604][CONNECT] spark.sql should support datatypes not compatible with arrow

  • [SPARK-44999] [SC-141145][CORE] Refactor ExternalSorter to reduce checks on shouldPartition when calling getPartition

  • [SPARK-42304] [SC-141501][SQL] Rename _LEGACY_ERROR_TEMP_2189 to GET_TABLES_BY_TYPE_UNSUPPORTED_BY_HIVE_VERSION

  • [SPARK-43781] [SC-139450][SQL] Fix IllegalStateException when cogrouping two datasets derived from the same source

  • [SPARK-45018] [SC-141516][PYTHON][CONNECT] Add CalendarIntervalType to Python Client

  • [SPARK-45024] [SC-141513][PYTHON][CONNECT] Filter out some configurations in Session Creation

  • [SPARK-45017] [SC-141508][PYTHON] Add CalendarIntervalType to PySpark

  • [SPARK-44720] [SC-139375][CONNECT] Make Dataset use Encoder instead of AgnosticEncoder

  • [SPARK-44982] [SC-141027][CONNECT] Mark Spark Connect server configurations as static

  • [SPARK-44839] [SC-140900][SS][CONNECT] Better Error Logging when user tries to serialize spark session

  • [SPARK-44865] [SC-140905][SS] Make StreamingRelationV2 support metadata column

  • [SPARK-45001] [SC-141141][PYTHON][CONNECT] Implement DataFrame.foreachPartition

  • [SPARK-44497] [SC-141017][WEBUI] Show task partition id in Task table

  • [SPARK-45006] [SC-141143][UI] Use the same date format of other UI date elements for the x-axis of timelines

  • [SPARK-45000] [SC-141135][PYTHON][CONNECT] Implement DataFrame.foreach

  • [SPARK-44967] [SC-141137][SQL][CONNECT] Unit should be considered first before using Boolean for TreeNodeTag

  • [SPARK-44993] [SC-141088][CORE] Add ShuffleChecksumUtils.compareChecksums by reusing ShuffleChecksumTestHelp.compareChecksums

  • [SPARK-44807] [SC-140176][CONNECT] Add Dataset.metadataColumn to Scala Client

  • [SPARK-44965] [SC-141098][PYTHON] Hide internal functions/variables from pyspark.sql.functions

  • [SPARK-44983] [SC-141030][SQL] Convert binary to string by to_char for the formats: hex, base64, utf-8

  • [SPARK-44980] [DBRRM-462][SC-141024][PYTHON][CONNECT] Fix inherited namedtuples to work in createDataFrame

  • [SPARK-44985] [SC-141033][CORE] Use toString instead of stacktrace for task reaper threadDump

  • [SPARK-44984] [SC-141028][PYTHON][CONNECT] Remove _get_alias from DataFrame

  • [SPARK-44975] [SC-141013][SQL] Remove BinaryArithmetic useless override resolved

  • [SPARK-44969] [SC-140957][SQL] Reuse ArrayInsert in ArrayAppend

  • [SPARK-44549] [SC-140714][SQL] Support window functions in correlated scalar subqueries

  • [SPARK-44938] [SC-140811][SQL] Change default value of spark.sql.maxSinglePartitionBytes to 128m

  • [SPARK-44918] [SC-140816][SQL][PYTHON] Support named arguments in scalar Python/Pandas UDFs

  • [SPARK-44966] [SC-140907][CORE][CONNECT] Change the never changed var to val

  • [SPARK-41471] [SC-140804][SQL] Reduce Spark shuffle when only one side of a join is KeyGroupedPartitioning

  • [SPARK-44214] [SC-140528][CORE] Support Spark Driver Live Log UI

  • [SPARK-44861] [SC-140716][CONNECT] jsonignore SparkListenerConnectOperationStarted.planRequest

  • [SPARK-44776] [SC-140519][CONNECT] Add ProducedRowCount to SparkListenerConnectOperationFinished

  • [SPARK-42017] [SC-140765][PYTHON][CONNECT] df['col_name'] should validate the column name

  • [SPARK-40178] [SC-140433][SQL][COONECT] Support coalesce hints with ease for PySpark and R

  • [SPARK-44840] [SC-140593][SQL] Make array_insert() 1-based for negative indexes

  • [SPARK-44939] [SC-140778][R] Support Java 21 in SparkR SystemRequirements

  • [SPARK-44936] [SC-140770][CORE] Simplify the log when Spark HybridStore hits the memory limit

  • [SPARK-44908] [SC-140712][ML][CONNECT] Fix cross validator foldCol param functionality

  • [SPARK-44816] [SC-140717][CONNECT] Improve error message when UDF class is not found

  • [SPARK-44909] [SC-140710][ML] Skip starting torch distributor log streaming server when it is not available

  • [SPARK-44920] [SC-140707][CORE] Use await() instead of awaitUninterruptibly() in TransportClientFactory.createClient()

  • [SPARK-44905] [SC-140703][SQL] Stateful lastRegex causes NullPointerException on eval for regexp_replace

  • [SPARK-43987] [SC-139594][Shuffle] Separate finalizeShuffleMerge Processing to Dedicated Thread Pools

  • [SPARK-42768] [SC-140549][SQL] Enable cached plan apply AQE by default

  • [SPARK-44741] [SC-139447][CORE] Support regex-based MetricFilter in StatsdSink

  • [SPARK-44751] [SC-140532][SQL] XML FileFormat Interface implementation

  • [SPARK-44868] [SC-140438][SQL] Convert datetime to string by to_char/to_varchar

  • [SPARK-44748] [SC-140504][SQL] Query execution for the PARTITION BY clause in UDTF TABLE arguments

  • [SPARK-44873] [SC-140427] Support alter view with nested columns in Hive client

  • [SPARK-44876] [SC-140431][PYTHON] Fix Arrow-optimized Python UDF on Spark Connect

  • [SPARK-44520] [SC-137845][SQL] Replace the term UNSUPPORTED_DATA_SOURCE_FOR_DIRECT_QUERY with UNSUPPORTED_DATASOURCE_FOR_DIRECT_QUERY and disclosure root AE

  • [SPARK-42664] [SC-139769][CONNECT] Support bloomFilter function for DataFrameStatFunctions

  • [SPARK-43567] [SC-139227][PS] Support use_na_sentinel for factorize

  • [SPARK-44859] [SC-140254][SS] Fix incorrect property name in structured streaming doc

  • [SPARK-44822] [SC-140182][PYTHON][SQL] Make Python UDTFs by default non-deterministic

  • [SPARK-44731] [SC-139524][PYTHON][CONNECT] Make TimestampNTZ works with literals in Python Spark Connect

  • [SPARK-44836] [SC-140180][PYTHON] Refactor Arrow Python UDTF

  • [SPARK-44714] [SC-139238] Ease restriction of LCA resolution regarding queries with having

  • [SPARK-44749] [SC-139664][SQL][PYTHON] Support named arguments in Python UDTF

  • [SPARK-44737] [SC-139512][SQL][UI] Should not display json format errors on SQL page for non-SparkThrowables on SQL Tab

  • [SPARK-44665] [SC-139307][PYTHON] Add support for pandas DataFrame assertDataFrameEqual

  • [SPARK-44736] [SC-139622][CONNECT] Add Dataset.explode to Spark Connect Scala Client

  • [SPARK-44732] [SC-139422][SQL] Built-in XML data source support

  • [SPARK-44694] [SC-139213][PYTHON][CONNECT] Refactor active sessions and expose them as an API

  • [SPARK-44652] [SC-138881] Raise error when only one df is None

  • [SPARK-44562] [SC-138824][SQL] Add OptimizeOneRowRelationSubquery in batch of Subquery

  • [SPARK-44717] [SC-139319][PYTHON][PS] Respect TimestampNTZ in resampling

  • [SPARK-42849] [SC-139365] [SQL] Session Variables

  • [SPARK-44236] [SC-139239][SQL] Disable WholeStageCodegen when set spark.sql.codegen.factoryMode to NO_CODEGEN

  • [SPARK-44695] [SC-139316][PYTHON] Improve error message for DataFrame.toDF

  • [SPARK-44680] [SC-139234][SQL] Improve the error for parameters in DEFAULT

  • [SPARK-43402] [SC-138321][SQL] FileSourceScanExec supports push down data filter with scalar subquery

  • [SPARK-44641] [SC-139216][SQL] Incorrect result in certain scenarios when SPJ is not triggered

  • [SPARK-44689] [SC-139219][CONNECT] Make the exception handling of function SparkConnectPlanner#unpackScalarScalaUDF more universal

  • [SPARK-41636] [SC-139061][SQL] Make sure selectFilters returns predicates in deterministic order

  • [SPARK-44132] [SC-139197][SQL] Materialize Stream of join column names to avoid codegen failure

  • [SPARK-42330] [SC-138838][SQL] Assign the name RULE_ID_NOT_FOUND to the error class _LEGACY_ERROR_TEMP_2175

  • [SPARK-44683] [SC-139214][SS] Logging level isn’t passed to RocksDB state store provider correctly

  • [SPARK-44658] [SC-138868][CORE] ShuffleStatus.getMapStatus should return None instead of Some(null)

  • [SPARK-44603] [SC-138353] Add pyspark.testing to setup.py

  • [SPARK-44252] [SC-137505][SS] Define a new error class and apply for the case where loading state from DFS fails

  • [SPARK-29497] [DBRRM-396][SC-138477][CONNECT] Throw error when UDF is not deserializable.

  • [SPARK-44624] [DBRRM-396][SC-138900][CONNECT] Retry ExecutePlan in case initial request didn’t reach server

  • [SPARK-41400] [DBRRM-396][SC-138287][CONNECT] Remove Connect Client Catalyst Dependency

  • [SPARK-44059] [SC-138833][SQL] Add better error messages for SQL named argumnts

  • [SPARK-44620] [SC-138831][SQL][PS][CONNECT] Make ResolvePivot retain the Plan_ID_TAG

  • [SPARK-43838] [SC-137413][SQL] Fix subquery on single table with having clause can’t be optimized

  • [SPARK-44555] [SC-138820][SQL] Use checkError() to check Exception in command Suite & assign some error class names

  • [SPARK-44280] [SC-138821][SQL] Add convertJavaTimestampToTimestamp in JDBCDialect API

  • [SPARK-44602] [SC-138337][SQL][CONNECT][PS] Make WidenSetOperationTypes retain the Plan_ID_TAG

  • [SPARK-42941] [SC-138389][SS][CONNECT] Python StreamingQueryListener

  • [SPARK-43838] Revert “[SC-137413][SQL] Fix subquery on single ta…

  • [SPARK-44538] [SC-138178][CONNECT][SQL] Reinstate Row.jsonValue and friends

  • [SPARK-44421] [SC-138434][SPARK-44423][CONNECT] Reattachable execution in Spark Connect

  • [SPARK-43838] [SC-137413][SQL] Fix subquery on single table with having clause can’t be optimized

  • [SPARK-44587] [SC-138315][SQL][CONNECT] Increase protobuf marshaller recursion limit

  • [SPARK-44605] [SC-138338][CORE] Refine internal ShuffleWriteProcessor API

  • [SPARK-44394] [SC-138291][CONNECT][WEBUI] Add a Spark UI page for Spark Connect

  • [SPARK-44585] [SC-138286][MLLIB] Fix warning condition in MLLib RankingMetrics ndcgAk

  • [SPARK-44198] [SC-138137][CORE] Support propagation of the log level to the executors

  • [SPARK-44454] [SC-138071][SQL][HIVE] HiveShim getTablesByType support fallback

  • [SPARK-44425] [SC-138135][CONNECT] Validate that user provided sessionId is an UUID

  • [SPARK-43611] [SC-138051][SQL][PS][CONNCECT] Make ExtractWindowExpressions retain the PLAN_ID_TAG

  • [SPARK-44560] [SC-138117][PYTHON][CONNECT] Improve tests and documentation for Arrow Python UDF

  • [SPARK-44482] [SC-138067][CONNECT] Connect server should can specify the bind address

  • [SPARK-44528] [SC-138047][CONNECT] Support proper usage of hasattr() for Connect dataframe

  • [SPARK-44525] [SC-138043][SQL] Improve error message when Invoke method is not found

  • [SPARK-44355] [SC-137878][SQL] Move WithCTE into command queries

Databricks ODBC/JDBC driver support

Databricks supports ODBC/JDBC drivers released in the past 2 years. Please download the recently released drivers and upgrade (download ODBC, download JDBC).

System environment

  • Operating System: Ubuntu 22.04.3 LTS

  • Java: Zulu 8.72.0.17-CA-linux64

  • Scala: 2.12.15

  • Python: 3.10.12

  • R: 4.3.1

  • Delta Lake: 3.0.0

Installed Python libraries

Library

Version

Library

Version

Library

Version

anyio

3.5.0

argon2-cffi

21.3.0

argon2-cffi-bindings

21.2.0

asttokens

2.0.5

attrs

22.1.0

backcall

0.2.0

beautifulsoup4

4.11.1

black

22.6.0

bleach

4.1.0

blinker

1.4

boto3

1.24.28

botocore

1.27.96

certifi

2022.12.7

cffi

1.15.1

chardet

4.0.0

charset-normalizer

2.0.4

click

8.0.4

comm

0.1.2

contourpy

1.0.5

cryptography

39.0.1

cycler

0.11.0

Cython

0.29.32

databricks-sdk

0.1.6

dbus-python

1.2.18

debugpy

1.6.7

decorator

5.1.1

defusedxml

0.7.1

distlib

0.3.7

docstring-to-markdown

0.11

entrypoints

0.4

executing

0.8.3

facets-overview

1.1.1

fastjsonschema

2.18.0

filelock

3.12.3

fonttools

4.25.0

GCC runtime library

1.10.0

googleapis-common-protos

1.60.0

grpcio

1.48.2

grpcio-status

1.48.1

httplib2

0.20.2

idna

3.4

importlib-metadata

4.6.4

ipykernel

6.25.0

ipython

8.14.0

ipython-genutils

0.2.0

ipywidgets

7.7.2

jedi

0.18.1

jeepney

0.7.1

Jinja2

3.1.2

jmespath

0.10.0

joblib

1.2.0

jsonschema

4.17.3

jupyter-client

7.3.4

jupyter-server

1.23.4

jupyter_core

5.2.0

jupyterlab-pygments

0.1.2

jupyterlab-widgets

1.0.0

keyring

23.5.0

kiwisolver

1.4.4

launchpadlib

1.10.16

lazr.restfulclient

0.14.4

lazr.uri

1.0.6

lxml

4.9.1

MarkupSafe

2.1.1

matplotlib

3.7.0

matplotlib-inline

0.1.6

mccabe

0.7.0

mistune

0.8.4

more-itertools

8.10.0

mypy-extensions

0.4.3

nbclassic

0.5.2

nbclient

0.5.13

nbconvert

6.5.4

nbformat

5.7.0

nest-asyncio

1.5.6

nodeenv

1.8.0

notebook

6.5.2

notebook_shim

0.2.2

numpy

1.23.5

oauthlib

3.2.0

packaging

22.0

pandas

1.5.3

pandocfilters

1.5.0

parso

0.8.3

pathspec

0.10.3

patsy

0.5.3

pexpect

4.8.0

pickleshare

0.7.5

Pillow

9.4.0

pip

22.3.1

platformdirs

2.5.2

plotly

5.9.0

pluggy

1.0.0

prometheus-client

0.14.1

prompt-toolkit

3.0.36

protobuf

4.24.0

psutil

5.9.0

psycopg2

2.9.3

ptyprocess

0.7.0

pure-eval

0.2.2

pyarrow

8.0.0

pycparser

2.21

pydantic

1.10.6

pyflakes

3.0.1

Pygments

2.11.2

PyGObject

3.42.1

PyJWT

2.3.0

pyodbc

4.0.32

pyparsing

3.0.9

pyright

1.1.294

pyrsistent

0.18.0

python-dateutil

2.8.2

python-lsp-jsonrpc

1.0.0

python-lsp-server

1.7.1

pytoolconfig

1.2.5

pytz

2022.7

pyzmq

23.2.0

requests

2.28.1

rope

1.7.0

s3transfer

0.6.2

scikit-learn

1.1.1

seaborn

0.12.2

SecretStorage

3.3.1

Send2Trash

1.8.0

setuptools

65.6.3

six

1.16.0

sniffio

1.2.0

soupsieve

2.3.2.post1

ssh-import-id

5.11

stack-data

0.2.0

statsmodels

0.13.5

tenacity

8.1.0

terminado

0.17.1

threadpoolctl

2.2.0

tinycss2

1.2.1

tokenize-rt

4.2.1

tomli

2.0.1

tornado

6.1

traitlets

5.7.1

typing_extensions

4.4.0

ujson

5.4.0

unattended-upgrades

0.1

urllib3

1.26.14

virtualenv

20.16.7

wadllib

1.3.6

wcwidth

0.2.5

webencodings

0.5.1

websocket-client

0.58.0

whatthepatch

1.0.2

wheel

0.38.4

widgetsnbextension

3.6.1

yapf

0.31.0

zipp

1.0.0

SciPy

1.10.1

Installed R libraries

R libraries are installed from the Posit Package Manager CRAN snapshot on 2023-02-10.

Library

Version

Library

Version

Library

Version

arrow

12.0.1

askpass

1.1

assertthat

0.2.1

backports

1.4.1

base

4.3.1

base64enc

0.1-3

bit

4.0.5

bit64

4.0.5

blob

1.2.4

boot

1.3-28

brew

1.0-8

brio

1.1.3

broom

1.0.5

bslib

0.5.0

cachem

1.0.8

callr

3.7.3

caret

6.0-94

cellranger

1.1.0

chron

2.3-61

class

7.3-22

cli

3.6.1

clipr

0.8.0

clock

0.7.0

cluster

2.1.4

codetools

0.2-19

colorspace

2.1-0

commonmark

1.9.0

compiler

4.3.1

config

0.3.1

conflicted

1.2.0

cpp11

0.4.4

crayon

1.5.2

credentials

1.3.2

curl

5.0.1

data.table

1.14.8

datasets

4.3.1

DBI

1.1.3

dbplyr

2.3.3

desc

1.4.2

devtools

2.4.5

diagram

1.6.5

diffobj

0.3.5

digest

0.6.33

downlit

0.4.3

dplyr

1.1.2

dtplyr

1.3.1

e1071

1.7-13

ellipsis

0.3.2

evaluate

0.21

fansi

1.0.4

farver

2.1.1

fastmap

1.1.1

fontawesome

0.5.1

forcats

1.0.0

foreach

1.5.2

foreign

0.8-82

forge

0.2.0

fs

1.6.2

future

1.33.0

future.apply

1.11.0

gargle

1.5.1

generics

0.1.3

gert

1.9.2

ggplot2

3.4.2

gh

1.4.0

gitcreds

0.1.2

glmnet

4.1-7

globals

0.16.2

glue

1.6.2

googledrive

2.1.1

googlesheets4

1.1.1

gower

1.0.1

graphics

4.3.1

grDevices

4.3.1

grid

4.3.1

gridExtra

2.3

gsubfn

0.7

gtable

0.3.3

hardhat

1.3.0

haven

2.5.3

highr

0.10

hms

1.1.3

htmltools

0.5.5

htmlwidgets

1.6.2

httpuv

1.6.11

httr

1.4.6

httr2

0.2.3

ids

1.0.1

ini

0.3.1

ipred

0.9-14

isoband

0.2.7

iterators

1.0.14

jquerylib

0.1.4

jsonlite

1.8.7

KernSmooth

2.23-21

knitr

1.43

labeling

0.4.2

later

1.3.1

lattice

0.21-8

lava

1.7.2.1

lifecycle

1.0.3

listenv

0.9.0

lubridate

1.9.2

magrittr

2.0.3

markdown

1.7

MASS

7.3-60

Matrix

1.5-4.1

memoise

2.0.1

methods

4.3.1

mgcv

1.8-42

mime

0.12

miniUI

0.1.1.1

ModelMetrics

1.2.2.2

modelr

0.1.11

munsell

0.5.0

nlme

3.1-162

nnet

7.3-19

numDeriv

2016.8-1.1

openssl

2.0.6

parallel

4.3.1

parallelly

1.36.0

pillar

1.9.0

pkgbuild

1.4.2

pkgconfig

2.0.3

pkgdown

2.0.7

pkgload

1.3.2.1

plogr

0.2.0

plyr

1.8.8

praise

1.0.0

prettyunits

1.1.1

pROC

1.18.4

processx

3.8.2

prodlim

2023.03.31

profvis

0.3.8

progress

1.2.2

progressr

0.13.0

promises

1.2.0.1

proto

1.0.0

proxy

0.4-27

ps

1.7.5

purrr

1.0.1

r2d3

0.2.6

R6

2.5.1

ragg

1.2.5

randomForest

4.7-1.1

rappdirs

0.3.3

rcmdcheck

1.4.0

RColorBrewer

1.1-3

Rcpp

1.0.11

RcppEigen

0.3.3.9.3

readr

2.1.4

readxl

1.4.3

recipes

1.0.6

rematch

1.0.1

rematch2

2.1.2

remotes

2.4.2

reprex

2.0.2

reshape2

1.4.4

rlang

1.1.1

rmarkdown

2.23

RODBC

1.3-20

roxygen2

7.2.3

rpart

4.1.19

rprojroot

2.0.3

Rserve

1.8-11

RSQLite

2.3.1

rstudioapi

0.15.0

rversions

2.1.2

rvest

1.0.3

sass

0.4.6

scales

1.2.1

selectr

0.4-2

sessioninfo

1.2.2

shape

1.4.6

shiny

1.7.4.1

sourcetools

0.1.7-1

sparklyr

1.8.1

SparkR

3.5.0

spatial

7.3-15

splines

4.3.1

sqldf

0.4-11

SQUAREM

2021.1

stats

4.3.1

stats4

4.3.1

stringi

1.7.12

stringr

1.5.0

survival

3.5-5

sys

3.4.2

systemfonts

1.0.4

tcltk

4.3.1

testthat

3.1.10

textshaping

0.3.6

tibble

3.2.1

tidyr

1.3.0

tidyselect

1.2.0

tidyverse

2.0.0

timechange

0.2.0

timeDate

4022.108

tinytex

0.45

tools

4.3.1

tzdb

0.4.0

urlchecker

1.0.1

usethis

2.2.2

utf8

1.2.3

utils

4.3.1

uuid

1.1-0

vctrs

0.6.3

viridisLite

0.4.2

vroom

1.6.3

waldo

0.5.1

whisker

0.4.1

withr

2.5.0

xfun

0.39

xml2

1.3.5

xopen

1.0.0

xtable

1.8-4

yaml

2.3.7

zip

2.3.0

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID

Artifact ID

Version

antlr

antlr

2.7.7

com.amazonaws

amazon-kinesis-client

1.12.0

com.amazonaws

aws-java-sdk-autoscaling

1.12.390

com.amazonaws

aws-java-sdk-cloudformation

1.12.390

com.amazonaws

aws-java-sdk-cloudfront

1.12.390

com.amazonaws

aws-java-sdk-cloudhsm

1.12.390

com.amazonaws

aws-java-sdk-cloudsearch

1.12.390

com.amazonaws

aws-java-sdk-cloudtrail

1.12.390

com.amazonaws

aws-java-sdk-cloudwatch

1.12.390

com.amazonaws

aws-java-sdk-cloudwatchmetrics

1.12.390

com.amazonaws

aws-java-sdk-codedeploy

1.12.390

com.amazonaws

aws-java-sdk-cognitoidentity

1.12.390

com.amazonaws

aws-java-sdk-cognitosync

1.12.390

com.amazonaws

aws-java-sdk-config

1.12.390

com.amazonaws

aws-java-sdk-core

1.12.390

com.amazonaws

aws-java-sdk-datapipeline

1.12.390

com.amazonaws

aws-java-sdk-directconnect

1.12.390

com.amazonaws

aws-java-sdk-directory

1.12.390

com.amazonaws

aws-java-sdk-dynamodb

1.12.390

com.amazonaws

aws-java-sdk-ec2

1.12.390

com.amazonaws

aws-java-sdk-ecs

1.12.390

com.amazonaws

aws-java-sdk-efs

1.12.390

com.amazonaws

aws-java-sdk-elasticache

1.12.390

com.amazonaws

aws-java-sdk-elasticbeanstalk

1.12.390

com.amazonaws

aws-java-sdk-elasticloadbalancing

1.12.390

com.amazonaws

aws-java-sdk-elastictranscoder

1.12.390

com.amazonaws

aws-java-sdk-emr

1.12.390

com.amazonaws

aws-java-sdk-glacier

1.12.390

com.amazonaws

aws-java-sdk-glue

1.12.390

com.amazonaws

aws-java-sdk-iam

1.12.390

com.amazonaws

aws-java-sdk-importexport

1.12.390

com.amazonaws

aws-java-sdk-kinesis

1.12.390

com.amazonaws

aws-java-sdk-kms

1.12.390

com.amazonaws

aws-java-sdk-lambda

1.12.390

com.amazonaws

aws-java-sdk-logs

1.12.390

com.amazonaws

aws-java-sdk-machinelearning

1.12.390

com.amazonaws

aws-java-sdk-opsworks

1.12.390

com.amazonaws

aws-java-sdk-rds

1.12.390

com.amazonaws

aws-java-sdk-redshift

1.12.390

com.amazonaws

aws-java-sdk-route53

1.12.390

com.amazonaws

aws-java-sdk-s3

1.12.390

com.amazonaws

aws-java-sdk-ses

1.12.390

com.amazonaws

aws-java-sdk-simpledb

1.12.390

com.amazonaws

aws-java-sdk-simpleworkflow

1.12.390

com.amazonaws

aws-java-sdk-sns

1.12.390

com.amazonaws

aws-java-sdk-sqs

1.12.390

com.amazonaws

aws-java-sdk-ssm

1.12.390

com.amazonaws

aws-java-sdk-storagegateway

1.12.390

com.amazonaws

aws-java-sdk-sts

1.12.390

com.amazonaws

aws-java-sdk-support

1.12.390

com.amazonaws

aws-java-sdk-swf-libraries

1.11.22

com.amazonaws

aws-java-sdk-workspaces

1.12.390

com.amazonaws

jmespath-java

1.12.390

com.clearspring.analytics

stream

2.9.6

com.databricks

Rserve

1.8-3

com.databricks

databricks-sdk-java

0.7.0

com.databricks

jets3t

0.7.1-0

com.databricks.scalapb

compilerplugin_2.12

0.4.15-10

com.databricks.scalapb

scalapb-runtime_2.12

0.4.15-10

com.esotericsoftware

kryo-shaded

4.0.2

com.esotericsoftware

minlog

1.3.0

com.fasterxml

classmate

1.3.4

com.fasterxml.jackson.core

jackson-annotations

2.15.2

com.fasterxml.jackson.core

jackson-core

2.15.2

com.fasterxml.jackson.core

jackson-databind

2.15.2

com.fasterxml.jackson.dataformat

jackson-dataformat-cbor

2.15.2

com.fasterxml.jackson.datatype

jackson-datatype-joda

2.15.2

com.fasterxml.jackson.datatype

jackson-datatype-jsr310

2.15.1

com.fasterxml.jackson.module

jackson-module-paranamer

2.15.2

com.fasterxml.jackson.module

jackson-module-scala_2.12

2.15.2

com.github.ben-manes.caffeine

caffeine

2.9.3

com.github.fommil

jniloader

1.1

com.github.fommil.netlib

native_ref-java

1.1

com.github.fommil.netlib

native_ref-java

1.1-natives

com.github.fommil.netlib

native_system-java

1.1

com.github.fommil.netlib

native_system-java

1.1-natives

com.github.fommil.netlib

netlib-native_ref-linux-x86_64

1.1-natives

com.github.fommil.netlib

netlib-native_system-linux-x86_64

1.1-natives

com.github.luben

zstd-jni

1.5.5-4

com.github.wendykierp

JTransforms

3.1

com.google.code.findbugs

jsr305

3.0.0

com.google.code.gson

gson

2.10.1

com.google.crypto.tink

tink

1.9.0

com.google.errorprone

error_prone_annotations

2.10.0

com.google.flatbuffers

flatbuffers-java

1.12.0

com.google.guava

guava

15.0

com.google.protobuf

protobuf-java

2.6.1

com.helger

profiler

1.1.1

com.jcraft

jsch

0.1.55

com.jolbox

bonecp

0.8.0.RELEASE

com.lihaoyi

sourcecode_2.12

0.1.9

com.microsoft.azure

azure-data-lake-store-sdk

2.3.9

com.microsoft.sqlserver

mssql-jdbc

11.2.2.jre8

com.ning

compress-lzf

1.1.2

com.sun.mail

javax.mail

1.5.2

com.sun.xml.bind

jaxb-core

2.2.11

com.sun.xml.bind

jaxb-impl

2.2.11

com.tdunning

json

1.8

com.thoughtworks.paranamer

paranamer

2.8

com.trueaccord.lenses

lenses_2.12

0.4.12

com.twitter

chill-java

0.10.0

com.twitter

chill_2.12

0.10.0

com.twitter

util-app_2.12

7.1.0

com.twitter

util-core_2.12

7.1.0

com.twitter

util-function_2.12

7.1.0

com.twitter

util-jvm_2.12

7.1.0

com.twitter

util-lint_2.12

7.1.0

com.twitter

util-registry_2.12

7.1.0

com.twitter

util-stats_2.12

7.1.0

com.typesafe

config

1.2.1

com.typesafe.scala-logging

scala-logging_2.12

3.7.2

com.uber

h3

3.7.3

com.univocity

univocity-parsers

2.9.1

com.zaxxer

HikariCP

4.0.3

commons-cli

commons-cli

1.5.0

commons-codec

commons-codec

1.16.0

commons-collections

commons-collections

3.2.2

commons-dbcp

commons-dbcp

1.4

commons-fileupload

commons-fileupload

1.5

commons-httpclient

commons-httpclient

3.1

commons-io

commons-io

2.13.0

commons-lang

commons-lang

2.6

commons-logging

commons-logging

1.1.3

commons-pool

commons-pool

1.5.4

dev.ludovic.netlib

arpack

3.0.3

dev.ludovic.netlib

blas

3.0.3

dev.ludovic.netlib

lapack

3.0.3

info.ganglia.gmetric4j

gmetric4j

1.0.10

io.airlift

aircompressor

0.25

io.delta

delta-sharing-spark_2.12

0.7.5

io.dropwizard.metrics

metrics-annotation

4.2.19

io.dropwizard.metrics

metrics-core

4.2.19

io.dropwizard.metrics

metrics-graphite

4.2.19

io.dropwizard.metrics

metrics-healthchecks

4.2.19

io.dropwizard.metrics

metrics-jetty9

4.2.19

io.dropwizard.metrics

metrics-jmx

4.2.19

io.dropwizard.metrics

metrics-json

4.2.19

io.dropwizard.metrics

metrics-jvm

4.2.19

io.dropwizard.metrics

metrics-servlets

4.2.19

io.netty

netty-all

4.1.96.Final

io.netty

netty-buffer

4.1.96.Final

io.netty

netty-codec

4.1.96.Final

io.netty

netty-codec-http

4.1.96.Final

io.netty

netty-codec-http2

4.1.96.Final

io.netty

netty-codec-socks

4.1.96.Final

io.netty

netty-common

4.1.96.Final

io.netty

netty-handler

4.1.96.Final

io.netty

netty-handler-proxy

4.1.96.Final

io.netty

netty-resolver

4.1.96.Final

io.netty

netty-tcnative-boringssl-static

2.0.61.Final-linux-aarch_64

io.netty

netty-tcnative-boringssl-static

2.0.61.Final-linux-x86_64

io.netty

netty-tcnative-boringssl-static

2.0.61.Final-osx-aarch_64

io.netty

netty-tcnative-boringssl-static

2.0.61.Final-osx-x86_64

io.netty

netty-tcnative-boringssl-static

2.0.61.Final-windows-x86_64

io.netty

netty-tcnative-classes

2.0.61.Final

io.netty

netty-transport

4.1.96.Final

io.netty

netty-transport-classes-epoll

4.1.96.Final

io.netty

netty-transport-classes-kqueue

4.1.96.Final

io.netty

netty-transport-native-epoll

4.1.96.Final

io.netty

netty-transport-native-epoll

4.1.96.Final-linux-aarch_64

io.netty

netty-transport-native-epoll

4.1.96.Final-linux-x86_64

io.netty

netty-transport-native-kqueue

4.1.96.Final-osx-aarch_64

io.netty

netty-transport-native-kqueue

4.1.96.Final-osx-x86_64

io.netty

netty-transport-native-unix-common

4.1.96.Final

io.prometheus

simpleclient

0.7.0

io.prometheus

simpleclient_common

0.7.0

io.prometheus

simpleclient_dropwizard

0.7.0

io.prometheus

simpleclient_pushgateway

0.7.0

io.prometheus

simpleclient_servlet

0.7.0

io.prometheus.jmx

collector

0.12.0

jakarta.annotation

jakarta.annotation-api

1.3.5

jakarta.servlet

jakarta.servlet-api

4.0.3

jakarta.validation

jakarta.validation-api

2.0.2

jakarta.ws.rs

jakarta.ws.rs-api

2.1.6

javax.activation

activation

1.1.1

javax.el

javax.el-api

2.2.4

javax.jdo

jdo-api

3.0.1

javax.transaction

jta

1.1

javax.transaction

transaction-api

1.1

javax.xml.bind

jaxb-api

2.2.11

javolution

javolution

5.5.1

jline

jline

2.14.6

joda-time

joda-time

2.12.1

net.java.dev.jna

jna

5.8.0

net.razorvine

pickle

1.3

net.sf.jpam

jpam

1.1

net.sf.opencsv

opencsv

2.3

net.sf.supercsv

super-csv

2.2.0

net.snowflake

snowflake-ingest-sdk

0.9.6

net.snowflake

snowflake-jdbc

3.13.33

net.sourceforge.f2j

arpack_combined_all

0.1

org.acplt.remotetea

remotetea-oncrpc

1.1.2

org.antlr

ST4

4.0.4

org.antlr

antlr-runtime

3.5.2

org.antlr

antlr4-runtime

4.9.3

org.antlr

stringtemplate

3.2.1

org.apache.ant

ant

1.9.16

org.apache.ant

ant-jsch

1.9.16

org.apache.ant

ant-launcher

1.9.16

org.apache.arrow

arrow-format

12.0.1

org.apache.arrow

arrow-memory-core

12.0.1

org.apache.arrow

arrow-memory-netty

12.0.1

org.apache.arrow

arrow-vector

12.0.1

org.apache.avro

avro

1.11.2

org.apache.avro

avro-ipc

1.11.2

org.apache.avro

avro-mapred

1.11.2

org.apache.commons

commons-collections4

4.4

org.apache.commons

commons-compress

1.23.0

org.apache.commons

commons-crypto

1.1.0

org.apache.commons

commons-lang3

3.12.0

org.apache.commons

commons-math3

3.6.1

org.apache.commons

commons-text

1.10.0

org.apache.curator

curator-client

2.13.0

org.apache.curator

curator-framework

2.13.0

org.apache.curator

curator-recipes

2.13.0

org.apache.datasketches

datasketches-java

3.1.0

org.apache.datasketches

datasketches-memory

2.0.0

org.apache.derby

derby

10.14.2.0

org.apache.hadoop

hadoop-client-runtime

3.3.6

org.apache.hive

hive-beeline

2.3.9

org.apache.hive

hive-cli

2.3.9

org.apache.hive

hive-jdbc

2.3.9

org.apache.hive

hive-llap-client

2.3.9

org.apache.hive

hive-llap-common

2.3.9

org.apache.hive

hive-serde

2.3.9

org.apache.hive

hive-shims

2.3.9

org.apache.hive

hive-storage-api

2.8.1

org.apache.hive.shims

hive-shims-0.23

2.3.9

org.apache.hive.shims

hive-shims-common

2.3.9

org.apache.hive.shims

hive-shims-scheduler

2.3.9

org.apache.httpcomponents

httpclient

4.5.14

org.apache.httpcomponents

httpcore

4.4.16

org.apache.ivy

ivy

2.5.1

org.apache.logging.log4j

log4j-1.2-api

2.20.0

org.apache.logging.log4j

log4j-api

2.20.0

org.apache.logging.log4j

log4j-core

2.20.0

org.apache.logging.log4j

log4j-slf4j2-impl

2.20.0

org.apache.mesos

mesos

1.11.0-shaded-protobuf

org.apache.orc

orc-core

1.9.1-shaded-protobuf

org.apache.orc

orc-mapreduce

1.9.1-shaded-protobuf

org.apache.orc

orc-shims

1.9.1

org.apache.thrift

libfb303

0.9.3

org.apache.thrift

libthrift

0.12.0

org.apache.ws.xmlschema

xmlschema-core

2.3.0

org.apache.xbean

xbean-asm9-shaded

4.23

org.apache.yetus

audience-annotations

0.13.0

org.apache.zookeeper

zookeeper

3.6.3

org.apache.zookeeper

zookeeper-jute

3.6.3

org.checkerframework

checker-qual

3.31.0

org.codehaus.jackson

jackson-core-asl

1.9.13

org.codehaus.jackson

jackson-mapper-asl

1.9.13

org.codehaus.janino

commons-compiler

3.0.16

org.codehaus.janino

janino

3.0.16

org.datanucleus

datanucleus-api-jdo

4.2.4

org.datanucleus

datanucleus-core

4.1.17

org.datanucleus

datanucleus-rdbms

4.1.19

org.datanucleus

javax.jdo

3.2.0-m3

org.eclipse.jetty

jetty-client

9.4.52.v20230823

org.eclipse.jetty

jetty-continuation

9.4.52.v20230823

org.eclipse.jetty

jetty-http

9.4.52.v20230823

org.eclipse.jetty

jetty-io

9.4.52.v20230823

org.eclipse.jetty

jetty-jndi

9.4.52.v20230823

org.eclipse.jetty

jetty-plus

9.4.52.v20230823

org.eclipse.jetty

jetty-proxy

9.4.52.v20230823

org.eclipse.jetty

jetty-security

9.4.52.v20230823

org.eclipse.jetty

jetty-server

9.4.52.v20230823

org.eclipse.jetty

jetty-servlet

9.4.52.v20230823

org.eclipse.jetty

jetty-servlets

9.4.52.v20230823

org.eclipse.jetty

jetty-util

9.4.52.v20230823

org.eclipse.jetty

jetty-util-ajax

9.4.52.v20230823

org.eclipse.jetty

jetty-webapp

9.4.52.v20230823

org.eclipse.jetty

jetty-xml

9.4.52.v20230823

org.eclipse.jetty.websocket

websocket-api

9.4.52.v20230823

org.eclipse.jetty.websocket

websocket-client

9.4.52.v20230823

org.eclipse.jetty.websocket

websocket-common

9.4.52.v20230823

org.eclipse.jetty.websocket

websocket-server

9.4.52.v20230823

org.eclipse.jetty.websocket

websocket-servlet

9.4.52.v20230823

org.fusesource.leveldbjni

leveldbjni-all

1.8

org.glassfish.hk2

hk2-api

2.6.1

org.glassfish.hk2

hk2-locator

2.6.1

org.glassfish.hk2

hk2-utils

2.6.1

org.glassfish.hk2

osgi-resource-locator

1.0.3

org.glassfish.hk2.external

aopalliance-repackaged

2.6.1

org.glassfish.hk2.external

jakarta.inject

2.6.1

org.glassfish.jersey.containers

jersey-container-servlet

2.40

org.glassfish.jersey.containers

jersey-container-servlet-core

2.40

org.glassfish.jersey.core

jersey-client

2.40

org.glassfish.jersey.core

jersey-common

2.40

org.glassfish.jersey.core

jersey-server

2.40

org.glassfish.jersey.inject

jersey-hk2

2.40

org.hibernate.validator

hibernate-validator

6.1.7.Final

org.ini4j

ini4j

0.5.4

org.javassist

javassist

3.29.2-GA

org.jboss.logging

jboss-logging

3.3.2.Final

org.jdbi

jdbi

2.63.1

org.jetbrains

annotations

17.0.0

org.joda

joda-convert

1.7

org.jodd

jodd-core

3.5.2

org.json4s

json4s-ast_2.12

3.7.0-M11

org.json4s

json4s-core_2.12

3.7.0-M11

org.json4s

json4s-jackson_2.12

3.7.0-M11

org.json4s

json4s-scalap_2.12

3.7.0-M11

org.lz4

lz4-java

1.8.0

org.mariadb.jdbc

mariadb-java-client

2.7.9

org.mlflow

mlflow-spark

2.2.0

org.objenesis

objenesis

2.5.1

org.postgresql

postgresql

42.6.0

org.roaringbitmap

RoaringBitmap

0.9.45

org.roaringbitmap

shims

0.9.45

org.rocksdb

rocksdbjni

8.3.2

org.rosuda.REngine

REngine

2.1.0

org.scala-lang

scala-compiler_2.12

2.12.15

org.scala-lang

scala-library_2.12

2.12.15

org.scala-lang

scala-reflect_2.12

2.12.15

org.scala-lang.modules

scala-collection-compat_2.12

2.9.0

org.scala-lang.modules

scala-parser-combinators_2.12

1.1.2

org.scala-lang.modules

scala-xml_2.12

1.2.0

org.scala-sbt

test-interface

1.0

org.scalacheck

scalacheck_2.12

1.14.2

org.scalactic

scalactic_2.12

3.2.15

org.scalanlp

breeze-macros_2.12

2.1.0

org.scalanlp

breeze_2.12

2.1.0

org.scalatest

scalatest-compatible

3.2.15

org.scalatest

scalatest-core_2.12

3.2.15

org.scalatest

scalatest-diagrams_2.12

3.2.15

org.scalatest

scalatest-featurespec_2.12

3.2.15

org.scalatest

scalatest-flatspec_2.12

3.2.15

org.scalatest

scalatest-freespec_2.12

3.2.15

org.scalatest

scalatest-funspec_2.12

3.2.15

org.scalatest

scalatest-funsuite_2.12

3.2.15

org.scalatest

scalatest-matchers-core_2.12

3.2.15

org.scalatest

scalatest-mustmatchers_2.12

3.2.15

org.scalatest

scalatest-propspec_2.12

3.2.15

org.scalatest

scalatest-refspec_2.12

3.2.15

org.scalatest

scalatest-shouldmatchers_2.12

3.2.15

org.scalatest

scalatest-wordspec_2.12

3.2.15

org.scalatest

scalatest_2.12

3.2.15

org.slf4j

jcl-over-slf4j

2.0.7

org.slf4j

jul-to-slf4j

2.0.7

org.slf4j

slf4j-api

2.0.7

org.threeten

threeten-extra

1.7.1

org.tukaani

xz

1.9

org.typelevel

algebra_2.12

2.0.1

org.typelevel

cats-kernel_2.12

2.1.1

org.typelevel

spire-macros_2.12

0.17.0

org.typelevel

spire-platform_2.12

0.17.0

org.typelevel

spire-util_2.12

0.17.0

org.typelevel

spire_2.12

0.17.0

org.wildfly.openssl

wildfly-openssl

1.1.3.Final

org.xerial

sqlite-jdbc

3.42.0.0

org.xerial.snappy

snappy-java

1.1.10.3

org.yaml

snakeyaml

2.0

oro

oro

2.0.8

pl.edu.icm

JLargeArrays

1.5

software.amazon.cryptools

AmazonCorrettoCryptoProvider

1.6.1-linux-x86_64

software.amazon.ion

ion-java

1.0.2

stax

stax-api

1.0.1