Databricks Runtime 14.2 (EoS)

Note

Support for this Databricks Runtime version has ended. For the end-of-support date, see End-of-support history. For all supported Databricks Runtime versions, see Databricks Runtime release notes versions and compatibility.

The following release notes provide information about Databricks Runtime 14.2, powered by Apache Spark 3.5.0.

Databricks released this version in November 2023.

Note

These release notes may include references to features that are not available on Google Cloud as of this release.

Tip

To see release notes for Databricks Runtime versions that have reached end-of-support (EoS), see End-of-support Databricks Runtime release notes. The EoS Databricks Runtime versions have been retired and might not be updated.

New features and improvements

CACHE SELECT is now ignored

The SQL command CACHE SELECT is now ignored in favor of an improved caching algorithm. See Optimize performance with caching on Databricks.

Improved caching behavior

DiskCache placement by the Spark scheduler now has faster partition assignment, better balancing during cluster autoscaling, and more consistent query execution.

Fixed corrupt file handling in DML commands

The DML commands DELETE, UPDATE, and MERGE INTO no longer respect the read options ignoreCorruptFiles and ignoreMissingFiles. When encountering an unreadable file in a table, these commands now fail even if these options are specified.

foreachBatch and StreamingListener support

You can now use the foreachBatch() and StreamingListener APIs with Structured Streaming in shared clusters. See Use foreachBatch to write to arbitrary data sinks and Monitoring Structured Streaming queries on Databricks.

Row-level concurrency is Generally Available and on by default

Row-level concurrency reduces conflicts between concurrent write operations by detecting changes at the row-level. Row-level concurrency is only supported on tables without partitioning, which includes tables with liquid clustering. Row-level concurrency is enabled by default on Delta tables with deletion vectors enabled. See Write conflicts with row-level concurrency.

Shallow clone for Unity Catalog external tables (Public Preview)

You can now use shallow clone with Unity Catalog external tables. See Shallow clone for Unity Catalog tables.

New disk caching assignment algorithm

The Spark scheduler now uses a new disk caching algorithm. The algorithm improves disk usage and partition assignment across nodes, with faster assignment both initially and after cluster scaling events. Stickier cache assignment improves consistency across runs and reduces data moved during rebalancing operations.

Support for Python scalar user-defined functions on shared clusters (Public Preview)

You can now use Python scalar user-defined functions on clusters configured with shared access mode. See User-defined scalar functions - Python.

from_avro with schema registry connector adds support for schema evolution

You can now enable your pipelines to restart when evolved records are detected. Previously, if schema evolution happened with the from_avro connector, the new columns would return null. See Read and write streaming Avro data.

Faster multi-threaded statistics collection

Statistics collection is up to 10 times faster on small clusters when running CONVERT TO DELTA or cloning from Iceberg and Parquet tables. See Convert to Delta Lake and Clone Parquet and Iceberg tables.

Pushdown filters in the DeltaSource on Delta files

For better utilization, partition filters on Delta tables streaming queries are now pushed down to Delta before rate limiting.

Support for Scala scalar user-defined functions on shared clusters (Public Preview)

You can now use Scala scalar user-defined functions on Unity Catalog-enabled clusters configured with shared access mode. See User-defined scalar functions - Scala.

Library upgrades

  • Upgraded Python libraries:

    • fastjsonschema from 2.18.0 to 2.19.0

    • filelock from 3.12.3 to 3.12.4

    • googleapis-common-protos from 1.60.0 to 1.61.0

  • Upgraded R libraries:

  • Upgraded Java libraries:

Apache Spark

Databricks Runtime 14.2 includes Apache Spark 3.5.0. This release includes all Spark fixes and improvements included in Databricks Runtime 14.1, as well as the following additional bug fixes and improvements made to Spark:

  • [SPARK-45592] [DBRRM-624] Revert “[SC-146977][SQL] Correctness issue in AQE with InMemoryTableScanExec”

  • [SPARK-45524] [DBRRM-625] Revert “[SC-146319][PYTHON][SQL] Initial support f…

  • [SPARK-45433] [DBRRM-621] Revert “[SC-145163][SQL] Fix CSV/JSON schema infer…

  • [SPARK-45592] [SC-146977][SQL] Correctness issue in AQE with InMemoryTableScanExec

  • [SPARK-45354] Revert “[SC-143991][SQL] Resolve functions bottom-up”

  • [SPARK-45680] [TEST-ONLY][SASP-2347][CONNECT] Fix tests

  • [SPARK-45680] [CONNECT] Release session

  • [SPARK-43380] [SC-146726][ES-897115][SQL] Fix slowdown in Avro read

  • [SPARK-45649] [SC-146742][SQL] Unify the prepare framework for OffsetWindowFunctionFrame

  • [SPARK-40820] [SC-146727][PYTHON][SQL] Creating StructType from Json

  • [SPARK-45620] [SC-146225][PYTHON] Fix user-facing APIs related to Python UDTF to use camelCase

  • [SPARK-45727] [SC-146978][SS] Remove unused map in watermark propagation simulation

  • [SPARK-45723] [SC-146890][PYTHON][CONNECT] Catalog methods avoid pandas conversion

  • [SPARK-45574] [SC-146106][SQL] Add :: syntax as a shorthand for casting

  • [SPARK-45670] [SC-146725][CORE][3.5] SparkSubmit does not support --total-executor-cores when deploying on K8s

  • [SPARK-45661] [SC-146594][SQL][PYTHON] Add toNullable in StructType, MapType and ArrayType

  • [SPARK-45524] [SC-146319][PYTHON][SQL] Initial support for Python data source read API

  • [SPARK-45542] [SC-145892][CORE] Replace setSafeMode(HdfsConstants.SafeModeAction, boolean) with setSafeMode(SafeModeAction, boolean)

  • [SPARK-45652] [SC-146641][SQL] SPJ: Handle empty input partitions after dynamic filtering

  • [SPARK-45454] [SC-144846][SQL] Set the table’s default owner to current_user

  • [SPARK-45554] [SC-146875][PYTHON] Introduce flexible parameter to assertSchemaEqual

  • [SPARK-45242] [SC-143410][SQL] Use DataFrame ID to semantically validate CollectMetrics

  • [SPARK-45674] [SC-146862][CONNECT][PYTHON] Improve error message for JVM-dependent attributes on Spark Connect.

  • [SPARK-45646] [SC-146276][SQL] Remove hardcoding time variables prior to Hive 2.0

  • [SPARK-43380] [SC-146726][ES-897115][SQL] Fix slowdown in Avro read

  • [SPARK-45545] [SC-146484][CORE] Pass SSLOptions wherever we create a SparkTransportConf

  • [SPARK-45641] [SC-146260][UI] Display the application start time on AllJobsPage

  • [SPARK-40154] [SC-146491][Python][Docs] Correct storage level in Dataframe.cache docstring

  • [SPARK-45626] [SC-146259][SQL] Convert LEGACYERROR_TEMP_1055 to REQUIRES_SINGLE_PART_NAMESPACE

  • [SPARK-45507] [SC-145900][SQL] Correctness fix for nested correlated scalar subqueries with COUNT aggregates

  • [SPARK-45619] [SC-146226][CONNECT][PYTHON] Apply the observed metrics to Observation object

  • [SPARK-45588] [SC-145909][PROTOBUF][CONNECT][MINOR] Scaladoc improvement for StreamingForeachBatchHelper

  • [SPARK-45616] [SC-146120][CORE] Avoid ParVector, which does not propagate ThreadLocals or SparkSession

  • [SPARK-45604] [SC-146105][SQL] Add LogicalType checking on INT64 -> DateTime conversion on Parquet Vectorized Reader

  • [SPARK-44649] [SC-145891][SQL] Runtime Filter supports passing equivalent creation side expressions

  • [SPARK-41674] [SC-128408][SQL] Runtime filter should supports multi level shuffle join side as filter creation side

  • [SPARK-45547] [SC-146228][ML] Validate Vectors with built-in function

  • [SPARK-45558] [SC-145890][SS] Introduce a metadata file for streaming stateful operator

  • [SPARK-45618] [SC-146252][CORE] Remove BaseErrorHandler

  • [SPARK-45638] [SC-146251][SQL][Avro] Improve test coverage of decimal conversion

  • [SPARK-44837] [SC-146020][SQL] Change ALTER TABLE ALTER PARTITION column error message

  • [SPARK-45561] [SC-146264][SQL] Add proper conversions for TINYINT in MySQLDialect

  • [SPARK-45632] [SC-146255][SQL] Table cache should avoid unnecessary ColumnarToRow when enable AQE

  • [SPARK-45569] [SC-145932][SQL] Assign name to the error LEGACYERROR_TEMP_2152

  • [SPARK-44784] [SC-141015][CONNECT] Make SBT testing hermetic.

  • [SPARK-45452] [SC-144836][SQL] Improve InMemoryFileIndex to use FileSystem.listFiles API

  • [SPARK-44735] [SC-146186][SQL] Add warning msg when inserting columns with the same name by row that don’t match up

  • [SPARK-45351] [SC-144791][CORE] Change spark.shuffle.service.db.backend default value to ROCKSDB

  • [SPARK-45553] [SC-145906][PS] Deprecate assertPandasOnSparkEqual

  • [SPARK-45613] [SC-146022][CORE] Expose DeterministicLevel as a DeveloperApi

  • [SPARK-45549] [SC-145895][CORE] Remove unused numExistingExecutors in CoarseGrainedSchedulerBackend

  • [SPARK-45609] [SC-146030][CONNECT] Include SqlState in SparkThrowable proto message

  • [SPARK-45595] [SC-146037] expose sqlstate in error message

  • [SPARK-45628] [SC-146123][ML] Improve vector_to_array for .mllib.linalg.SparseVector

  • [SPARK-45009] [SC-145884][SQL][FOLLOW UP] Turn off decorrelation in join conditions for AQE InSubquery test

  • [SPARK-45576] [SC-145883][CORE] Remove unnecessary debug logs in ReloadingX509TrustManagerSuite

  • [SPARK-45485] [SC-145719][CONNECT] User agent improvements: Use SPARK_CONNECT_USER_AGENT env variable and include environment specific attributes

  • [SPARK-45508] [SC-145513][CORE] Add “–add-opens=java.base/jdk.internal.ref=ALL-UNNAMED” so Platform can access Cleaner on Java 9+

  • [SPARK-45595] Revert “[SC-146037] Expose SQLSTATE in error message”

  • [SPARK-45595] [SC-146037] Expose SQLSTATE in error message

  • [SPARK-45112] Revert “[SC-143259][SQL] Use UnresolvedFunction ba…

  • [SPARK-45257] [SC-143411][CORE] Enable spark.eventLog.compress by default

  • [SPARK-45586] [SC-145899][SQL] Reduce compiler latency for plans with large expression trees

  • [SPARK-45517] [SC-145691][CONNECT][14.x] Expand more exception constructors to support error framework parameters

  • [SPARK-45581] [SC-145896] Make SQLSTATE mandatory.

  • [SPARK-45427] [SC-145590][CORE] Add RPC SSL settings to SSLOptions and SparkTransportConf

  • [SPARK-45261] [SC-143357][CORE] Fix EventLogFileWriters to handle none as a codec

  • [SPARK-45582] [SC-145897][SS] Ensure that store instance is not used after calling commit within output mode streaming aggregation

  • [SPARK-45562] [SC-145720][SC-144650][SQL] XML: Make ‘rowTag’ a required option

  • [SPARK-45392] [SC-144362][CORE][SQL][SS] Replace Class.newInstance() with Class.getDeclaredConstructor().newInstance()

  • [SPARK-45433] [SC-145163][SQL] Fix CSV/JSON schema inference when timestamps do not match specified timestampFormat

  • [SPARK-45458] [SC-145011][SQL] Convert IllegalArgumentException to SparkIllegalArgumentException in bitwiseExpressions

  • [SPARK-45564] [SC-145735][SQL] Simplify ‘DataFrameStatFunctions.bloomFilter’ with ‘BloomFilterAggregate’ expression

  • [SPARK-45566] [SC-145777][PS] Support Pandas-like testing utils for Pandas API on Spark

  • [SPARK-45498] [SC-145509][CORE] Followup: Ignore task completion from old stage a…

  • [SPARK-45483] [SC-145152][CONNECT] Correct the function groups in connect.functions

  • [SPARK-45577] [SC-145781][PYTHON] Fix UserDefinedPythonTableFunctionAnalyzeRunner to pass folded values from named arguments

  • [SPARK-45009] [SC-138223][FOLLOW UP] Turn off decorrelation in join conditions for AetherSQLQuerySuite

  • [SPARK-45415] [SC-145344] Allow selective disabling of “fallocate” in RocksDB statestore

  • [SPARK-45500] [SC-145134][CORE][WEBUI] Show the number of abnormally completed drivers in MasterPage

  • [SPARK-44120] [SC-144349][PYTHON] Support Python 3.12

  • [SPARK-45439] [SC-145589][SQL][UI] Reduce memory usage of LiveStageMetrics.accumIdsToMetricType

  • [SPARK-45565] [SC-145717][UI] Unnecessary JSON.stringify and JSON.parse loop for task list on stage detail

  • [SPARK-45567] [SC-145718][CONNECT] Remove redundant if in org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender#run

  • [SPARK-45112] [SC-143259][SQL] Use UnresolvedFunction based resolution in SQL Dataset functions

  • [SPARK-45486] [SASP-2457][SC-145612][CONNECT] Make add_artifact request idempotent

  • [SPARK-44913] [SC-144326][SQL] DS V2 supports push down V2 UDF that has magic method

  • [SPARK-45538] [SC-145635][PYTHON][CONNECT]pyspark connect overwrite_partitions bug

  • [SPARK-45491] [SC-145622] Add missing SQLSTATES 2/2

  • [SPARK-45009] [SC-145620][SC-138223][SQL] Decorrelate predicate subqueries in join condition

  • [SPARK-45377] [SC-144377][CORE] Handle InputStream in NettyLogger

  • [SPARK-45248] [SC-143759][CORE]Set the timeout for spark ui server

  • [SPARK-44594] [SC-145611][SS] Remove redundant method parameter in kafka connector

  • [SPARK-45539] [SC-145621][SS] Add assert and log to indicate watermark definition is required for streaming aggregation queries in append mode

  • [SPARK-44262] [SC-145639][SQL] Add dropTable and getInsertStatement to JdbcDialect

  • [SPARK-45516] [SC-145480][CONNECT] Include QueryContext in SparkThrowable proto message

  • [SPARK-45310] [SC-145127][CORE] Report shuffle block status should respect shuffle service during decommission migration

  • [SPARK-45521] [SC-145487][ML] Avoid re-computation of nnz in VectorAssembler

  • [SPARK-45418] [SC-145485][SQL][PYTHON][CONNECT] Change current_database() column alias to current_schema()

  • [SPARK-45116] [SC-145142][SQL] Add some comment for param of JdbcDialect createTable

  • [SPARK-45495] [SC-145529][core] Support stage level task resource profile for k8s cluster when dynamic allocation disabled

  • [SPARK-45487] [SC-145559] Fix SQLSTATEs and temp errors

  • [SPARK-45505] [SC-145550][PYTHON] Refactor analyzeInPython to make it reusable

  • [SPARK-45132] [SC-145425][SQL] Fix IDENTIFIER for function invocation

  • [SPARK-45213] [SC-145020][SQL] Assign name to the error LEGACYERROR_TEMP_2151

  • [SPARK-45416] [SC-145126][CONNECT] Sanity check consistency of Arrow results

  • [SPARK-45163] [SC-143386][SQL] Merge UNSUPPORTED_VIEW_OPERATION & UNSUPPORTED_TABLE_OPERATION & fix some issue

  • [SPARK-45421] [SC-144814][SQL] Catch AnalysisException over InlineCTE

  • [SPARK-43664] [SC-145404][CONNECT][PS] Raise exception for ps.sql with Pandas-on-Spark object on Spark Connect.

  • [SPARK-45402] [SC-145371][SQL][PYTHON] Add UDTF API for ‘eval’ and ‘terminate’ methods to consume previous ‘analyze’ result

  • [SPARK-36112] [SC-67885] [SQL] Support correlated EXISTS and IN subqueries using DecorrelateInnerQuery framework

  • [SPARK-45383] [SC-144929][SQL] Fix error message for time travel with non-existing table

  • [SPARK-43254] [SC-143318][SQL] Assign a name to the error LEGACYERROR_TEMP_2018

  • [SPARK-45204] [SC-145167][CONNECT] Add optional ExecuteHolder to SparkConnectPlanner

  • [SPARK-45451] [SC-145144][SQL] Make the default storage level of dataset cache configurable

  • [SPARK-45192] [SC-143194][UI] Fix overdue lineInterpolate parameter for graphviz edge

  • [SPARK-45467] [SC-145146][CORE] Replace Proxy.getProxyClass() with Proxy.newProxyInstance().getClass

  • [SPARK-45397] [SC-145139][ML][CONNECT] Add array assembler feature transformer

  • [SPARK-44855] [SC-145226][CONNECT] Small tweaks to attaching ExecuteGrpcResponseSender to ExecuteResponseObserver

  • [SPARK-45398] [SC-144796][SQL] Append ESCAPE in sql() of the Like expression

  • [SPARK-45494] [SC-145129][CORE][PYTHON] Introduce util functions to read/write a byte array for PythonWorkerUtils

  • [SPARK-45464] [SC-145125][CORE] Fix network-yarn distribution build

  • [SPARK-45461] [SC-144851][CORE][SQL][MLLIB] Introduce a mapper for StorageLevel

  • [SPARK-45450] [SC-145009][PYTHON] Fix imports according to PEP8: pyspark.pandas and pyspark (core)

  • [SPARK-45475] [SC-145018][SQL] Uses DataFrame.foreachPartition instead of RDD.foreachPartition in JdbcUtils

  • [SPARK-45271] [SC-143778][SQL] Merge LEGACYERROR_TEMP_1113 into TABLE_OPERATION & delete some unused method in QueryCompilationErrors

  • [SPARK-45449] [SC-145013][SQL] Cache Invalidation Issue with JDBC Table

  • [SPARK-45473] [SC-145017][SQL] Fix incorrect error message for RoundBase

  • [SPARK-43299] [SC-145022][SS][CONNECT] Convert StreamingQueryException in Scala Client

  • [SPARK-45474] [SC-145014][CORE][WEBUI] Support top-level filtering in MasterPage JSON API

  • [SPARK-45205] [SC-145012][SQL] CommandResultExec to override iterator methods to avoid triggering multiple jobs.

  • [SPARK-45472] [SC-145004][SS] RocksDB State Store Doesn’t Need to Recheck checkpoint path existence

  • [SPARK-45470] [SC-145010][SQL] Avoid paste string value of hive orc compression kind

  • [SPARK-45448] [SC-144829][PYTHON] Fix imports according to PEP8: pyspark.testing, pyspark.mllib, pyspark.resource and pyspark.streaming

  • [SPARK-45446] [SC-144828][PYTHON] Fix imports according to PEP8: pyspark.errors and pyspark.ml

  • [SPARK-45239] [SC-144832][CONNECT] Reduce default spark.connect.jvmStacktrace.maxSize

  • [SPARK-45436] [SC-144831][PYTHON][CONNECT] DataFrame methods check same session

  • [SPARK-45413] [SC-144847][CORE] Add warning for prepare drop LevelDB support

  • [SPARK-45462] [SC-144848][CORE][WEBUI] Show Duration in ApplicationPage

  • [SPARK-44527] [SC-144855][SQL] Replace ScalarSubquery with null if its maxRows is 0

  • [SPARK-45401] [SC-144854][PYTHON] Add a new method cleanup in the UDTF interface

  • [SPARK-43704] [SC-144849][CONNECT][PS] Support MultiIndex for to_series()

  • [SPARK-45424] [SC-144888][SQL] Fix TimestampFormatter return optional parse results when only prefix match

  • [SPARK-45441] [SC-144833][PYTHON] Introduce more util functions for PythonWorkerUtils

  • [SPARK-45412] [SC-144803][PYTHON][CONNECT] Validate the plan and session in DataFrame.__init__

  • [SPARK-45408] [SC-144810][CORE] Add RPC SSL settings to TransportConf

  • [SPARK-45432] [SC-144818][CORE] Remove deprecated Hadoop-2 LocatedFileStatus constructor

  • [SPARK-45434] [SC-144819][ML][CONNECT] LogisticRegression checks the training labels

  • [SPARK-45420] [SC-144808][SQL][PYTHON][CONNECT] Add DataType.fromDDL into PySpark

  • [SPARK-45406] [SC-144793][PYTHON][CONNECT] Delete schema from DataFrame constructor

  • [SPARK-45404] [SC-144799][CORE] Support AWS_ENDPOINT_URL env variable

  • [SPARK-43620] [SC-144792][CONNECT][PS] Fix Pandas APIs depends on unsupported features

  • [SPARK-45354] [SC-143991][SQL] Resolve functions bottom-up

  • [SPARK-45394] [SASP-1480][SC-144712][PYTHON][CONNECT] Add retries for artifact API. Improve error handling (follow-up to [SPARK-45093]).

  • [SPARK-45120] [SC-142780][SPARK-45150][UI] Upgrade d3 from v3 to v7(v7.8.5) and apply api changes in UI

  • [SPARK-44838] [SC-143983][SQL] raise_error improvement

  • [SPARK-45312] [SC-143754][SQL][UI] Support toggle display/hide plan svg on execution page

  • [SPARK-45378] [SC-144448][CORE] Add convertToNettyForSsl to ManagedBuffer

  • [SPARK-44762] [SC-144338][CONNECT][CORE] Doc for SparkConnect.addJobTag and Connect SparkSession.addTag

  • [SPARK-45383] Handle unresolved RelationTimeTravel gracefully

  • [SPARK-45347] [SC-144512][SQL][CONNECT] Include SparkThrowable in FetchErrorDetailsResponse

  • [SPARK-45012] [SC-141512][SQL] CheckAnalysis should throw inlined plan in AnalysisException

  • [SPARK-45359] [SC-144336][PYTHON][CONNECT] DataFrame.{columns, colRegex, explain} should raise exceptions when plan is invalid

  • [SPARK-45227] [SC-144233][CORE] Fix a subtle thread-safety issue with CoarseGrainedExecutorBackend

  • [SPARK-45266] [SC-144171][PYTHON] Refactor ResolveFunctions analyzer rule to delay making lateral join when table arguments are used

  • [SPARK-45266] Revert “[SC-144171][PYTHON] Refactor ResolveFunctions analyzer rule to delay making lateral join when table arguments are used”

  • [SPARK-45371] [SC-144389][CONNECT] Fix shading issues in the Spark Connect Scala Client

  • [SPARK-45057] [SC-144214][CORE] Avoid acquire read lock when keepReadLock is false

  • [SPARK-45385] [SC-144357][SQL] Deprecate spark.sql.parser.escapedStringLiterals

  • [SPARK-45340] [SC-143917][SQL] Remove the SQL config spark.sql.hive.verifyPartitionPath

  • [SPARK-45266] [SC-144171][PYTHON] Refactor ResolveFunctions analyzer rule to delay making lateral join when table arguments are used

  • [SPARK-45362] [SC-144198][PYTHON] Project out PARTITION BY expressions before Python UDTF ‘eval’ method consumes them

  • [SPARK-45346] [SC-143925][SQL] Parquet schema inference should respect case sensitive flag when merging schema

  • [SPARK-45341] [SC-143933][CORE] Correct the title level in the comments of KVStore.java to make sbt doc run successfully with Java 17

  • [SPARK-45334] [SC-143914][SQL] Remove misleading comment in parquetSchemaConverter

  • [SPARK-45337] [SC-143932][CORE] Refactor AbstractCommandBuilder#getScalaVersion to remove the check for Scala 2.12

  • [SPARK-45329] [SC-143918][PYTHON][CONNECT] DataFrame methods skip pandas conversion

  • [SPARK-43662] [SC-143923][PS][CONNECT] Support merge_asof in Spark Connect

  • [SPARK-44126] [SC-143751][CORE] Shuffle migration failure count should not increase when target executor decommissioned

  • [SPARK-44550] [SC-119768][SQL] Enable correctness fixes for null IN (empty list) under ANSI

  • [SPARK-45316] [SC-143783][CORE][SQL] Add new parameters ignoreCorruptFiles/ignoreMissingFiles to HadoopRDD and NewHadoopRDD

  • [SPARK-45093] [SC-143414][CONNECT][PYTHON] Error reporting for addArtifacts query

  • [SPARK-44756] [SC-143787][CORE] Executor hangs when RetryingBlockTransferor fails to initiate retry

  • [SPARK-45333] [SC-143768][CORE] Fix one unit mistake related to spark.eventLog.buffer.kb

  • [SPARK-45317] [SC-143750][SQL][CONNECT] Handle null filename in stack traces of exceptions

  • [SPARK-45191] [SC-143407][SQL] InMemoryTableScanExec simpleStringWithNodeId adds columnar info

  • [SPARK-45235] [SC-143314][CONNECT][PYTHON] Support map and array parameters by sql()

  • [SPARK-45138] [SC-143309][SS] Define a new error class and apply it when checkpointing state to DFS fails

  • [SPARK-45297] [SC-143644][SQL] Remove workaround for dateformatter added in SPARK-31827

  • [SPARK-44345] [SC-143593][CORE] Lower unknown shuffle map output log level to WARN if shuffle migration is enabled

  • [SPARK-44463] [SC-143251][SS][CONNECT] Improve error handling for Connect steaming Python worker

  • [SPARK-45207] [SC-143603][SQL][CONNECT] Implement Error Enrichment for Scala Client

  • [SPARK-45240] [SC-143602][SQL][CONNECT] Implement Error Enrichment for Python Client

  • [SPARK-45216] [SC-143288][SQL] Fix non-deterministic seeded Dataset APIs

  • [SPARK-45251] [SC-143419][CONNECT] Add client_type field for FetchErrorDetails

  • [SPARK-45137] [SC-143001][CONNECT] Support map/array parameters in parameterized sql()

  • [SPARK-45224] [SC-143257][PYTHON] Add examples w/ map and array as parameters of sql()

  • [SPARK-44622] [SC-143205][SQL][CONNECT] Implement FetchErrorDetails RPC

  • [SPARK-45178] [SC-136089] Fallback to execute a single batch for Trigger.AvailableNow with unsupported sources rather than using wrapper

  • [SPARK-44823] [14.x][SC-142979][PYTHON] Update black to 23.9.1 and fixerroneous check

  • [SPARK-45078] [SC-142954][SQL] Fix array_insert ImplicitCastInputTypes not work

  • [SPARK-44579] [SC-138428][SQL] Support Interrupt On Cancel in SQLExecution

  • [SPARK-45252] [SC-143371][CORE] Escape the greater/less than symbols in the comments to make sbt doc execute successfully

  • [SPARK-45189] [SC-142973][SQL] Creating UnresolvedRelation from TableIdentifier should include the catalog field

  • [SPARK-45229] [SC-143254][CORE][UI] Show the number of drivers waiting in SUBMITTED status in MasterPage

  • [SPARK-43453] Revert “[SC-143135][PS] Ignore the names of MultiIndex when axis=1 for concat”

Databricks ODBC/JDBC driver support

Databricks supports ODBC/JDBC drivers released in the past 2 years. Please download the recently released drivers and upgrade (download ODBC, download JDBC).

See Databricks Runtime 14.2 maintenance updates.

System environment

  • Operating System: Ubuntu 22.04.3 LTS

  • Java: Zulu 8.72.0.17-CA-linux64

  • Scala: 2.12.15

  • Python: 3.10.12

  • R: 4.3.1

  • Delta Lake: 3.0.0

Installed Python libraries

Library

Version

Library

Version

Library

Version

anyio

3.5.0

argon2-cffi

21.3.0

argon2-cffi-bindings

21.2.0

asttokens

2.0.5

attrs

22.1.0

backcall

0.2.0

beautifulsoup4

4.11.1

black

22.6.0

bleach

4.1.0

blinker

1.4

boto3

1.24.28

botocore

1.27.96

certifi

2022.12.7

cffi

1.15.1

chardet

4.0.0

charset-normalizer

2.0.4

click

8.0.4

comm

0.1.2

contourpy

1.0.5

cryptography

39.0.1

cycler

0.11.0

Cython

0.29.32

databricks-sdk

0.1.6

dbus-python

1.2.18

debugpy

1.6.7

decorator

5.1.1

defusedxml

0.7.1

distlib

0.3.7

docstring-to-markdown

0.11

entrypoints

0.4

executing

0.8.3

facets-overview

1.1.1

fastjsonschema

2.19.0

filelock

3.12.4

fonttools

4.25.0

googleapis-common-protos

1.61.0

grpcio

1.48.2

grpcio-status

1.48.1

httplib2

0.20.2

idna

3.4

importlib-metadata

4.6.4

ipykernel

6.25.0

ipython

8.14.0

ipython-genutils

0.2.0

ipywidgets

7.7.2

jedi

0.18.1

jeepney

0.7.1

Jinja2

3.1.2

jmespath

0.10.0

joblib

1.2.0

jsonschema

4.17.3

jupyter-client

7.3.4

jupyter-server

1.23.4

jupyter_core

5.2.0

jupyterlab-pygments

0.1.2

jupyterlab-widgets

1.0.0

keyring

23.5.0

kiwisolver

1.4.4

launchpadlib

1.10.16

lazr.restfulclient

0.14.4

lazr.uri

1.0.6

lxml

4.9.1

MarkupSafe

2.1.1

matplotlib

3.7.0

matplotlib-inline

0.1.6

mccabe

0.7.0

mistune

0.8.4

more-itertools

8.10.0

mypy-extensions

0.4.3

nbclassic

0.5.2

nbclient

0.5.13

nbconvert

6.5.4

nbformat

5.7.0

nest-asyncio

1.5.6

nodeenv

1.8.0

notebook

6.5.2

notebook_shim

0.2.2

numpy

1.23.5

oauthlib

3.2.0

packaging

22.0

pandas

1.5.3

pandocfilters

1.5.0

parso

0.8.3

pathspec

0.10.3

patsy

0.5.3

pexpect

4.8.0

pickleshare

0.7.5

Pillow

9.4.0

pip

22.3.1

platformdirs

2.5.2

plotly

5.9.0

pluggy

1.0.0

prometheus-client

0.14.1

prompt-toolkit

3.0.36

protobuf

4.24.0

psutil

5.9.0

psycopg2

2.9.3

ptyprocess

0.7.0

pure-eval

0.2.2

pyarrow

8.0.0

pyarrow-hotfix

0.4

pycparser

2.21

pydantic

1.10.6

pyflakes

3.1.0

Pygments

2.11.2

PyGObject

3.42.1

PyJWT

2.3.0

pyodbc

4.0.32

pyparsing

3.0.9

pyright

1.1.294

pyrsistent

0.18.0

python-dateutil

2.8.2

python-lsp-jsonrpc

1.1.1

python-lsp-server

1.8.0

pytoolconfig

1.2.5

pytz

2022.7

pyzmq

23.2.0

requests

2.28.1

rope

1.7.0

s3transfer

0.6.2

scikit-learn

1.1.1

scipy

1.10.0

seaborn

0.12.2

SecretStorage

3.3.1

Send2Trash

1.8.0

setuptools

65.6.3

six

1.16.0

sniffio

1.2.0

soupsieve

2.3.2.post1

ssh-import-id

5.11

stack-data

0.2.0

statsmodels

0.13.5

tenacity

8.1.0

terminado

0.17.1

threadpoolctl

2.2.0

tinycss2

1.2.1

tokenize-rt

4.2.1

tomli

2.0.1

tornado

6.1

traitlets

5.7.1

typing_extensions

4.4.0

ujson

5.4.0

unattended-upgrades

0.1

urllib3

1.26.14

virtualenv

20.16.7

wadllib

1.3.6

wcwidth

0.2.5

webencodings

0.5.1

websocket-client

0.58.0

whatthepatch

1.0.2

wheel

0.38.4

widgetsnbextension

3.6.1

yapf

0.33.0

zipp

1.0.0

Installed R libraries

R libraries are installed from the Posit Package Manager CRAN snapshot on 2023-02-10.

Library

Version

Library

Version

Library

Version

arrow

12.0.1

askpass

1.1

assertthat

0.2.1

backports

1.4.1

base

4.3.1

base64enc

0.1-3

bit

4.0.5

bit64

4.0.5

blob

1.2.4

boot

1.3-28

brew

1.0-8

brio

1.1.3

broom

1.0.5

bslib

0.5.0

cachem

1.0.8

callr

3.7.3

caret

6.0-94

cellranger

1.1.0

chron

2.3-61

class

7.3-22

cli

3.6.1

clipr

0.8.0

clock

0.7.0

cluster

2.1.4

codetools

0.2-19

colorspace

2.1-0

commonmark

1.9.0

compiler

4.3.1

config

0.3.1

conflicted

1.2.0

cpp11

0.4.4

crayon

1.5.2

credentials

1.3.2

curl

5.0.1

data.table

1.14.8

datasets

4.3.1

DBI

1.1.3

dbplyr

2.3.3

desc

1.4.2

devtools

2.4.5

diagram

1.6.5

diffobj

0.3.5

digest

0.6.33

downlit

0.4.3

dplyr

1.1.2

dtplyr

1.3.1

e1071

1.7-13

ellipsis

0.3.2

evaluate

0.21

fansi

1.0.4

farver

2.1.1

fastmap

1.1.1

fontawesome

0.5.1

forcats

1.0.0

foreach

1.5.2

foreign

0.8-82

forge

0.2.0

fs

1.6.2

future

1.33.0

future.apply

1.11.0

gargle

1.5.1

generics

0.1.3

gert

1.9.2

ggplot2

3.4.2

gh

1.4.0

gitcreds

0.1.2

glmnet

4.1-7

globals

0.16.2

glue

1.6.2

googledrive

2.1.1

googlesheets4

1.1.1

gower

1.0.1

graphics

4.3.1

grDevices

4.3.1

grid

4.3.1

gridExtra

2.3

gsubfn

0.7

gtable

0.3.3

hardhat

1.3.0

haven

2.5.3

highr

0.10

hms

1.1.3

htmltools

0.5.5

htmlwidgets

1.6.2

httpuv

1.6.11

httr

1.4.6

httr2

0.2.3

ids

1.0.1

ini

0.3.1

ipred

0.9-14

isoband

0.2.7

iterators

1.0.14

jquerylib

0.1.4

jsonlite

1.8.7

KernSmooth

2.23-21

knitr

1.43

labeling

0.4.2

later

1.3.1

lattice

0.21-8

lava

1.7.2.1

lifecycle

1.0.3

listenv

0.9.0

lubridate

1.9.2

magrittr

2.0.3

markdown

1.7

MASS

7.3-60

Matrix

1.5-4.1

memoise

2.0.1

methods

4.3.1

mgcv

1.8-42

mime

0.12

miniUI

0.1.1.1

ModelMetrics

1.2.2.2

modelr

0.1.11

munsell

0.5.0

nlme

3.1-162

nnet

7.3-19

numDeriv

2016.8-1.1

openssl

2.0.6

parallel

4.3.1

parallelly

1.36.0

pillar

1.9.0

pkgbuild

1.4.2

pkgconfig

2.0.3

pkgdown

2.0.7

pkgload

1.3.2.1

plogr

0.2.0

plyr

1.8.8

praise

1.0.0

prettyunits

1.1.1

pROC

1.18.4

processx

3.8.2

prodlim

2023.03.31

profvis

0.3.8

progress

1.2.2

progressr

0.13.0

promises

1.2.0.1

proto

1.0.0

proxy

0.4-27

ps

1.7.5

purrr

1.0.1

r2d3

0.2.6

R6

2.5.1

ragg

1.2.5

randomForest

4.7-1.1

rappdirs

0.3.3

rcmdcheck

1.4.0

RColorBrewer

1.1-3

Rcpp

1.0.11

RcppEigen

0.3.3.9.3

readr

2.1.4

readxl

1.4.3

recipes

1.0.6

rematch

1.0.1

rematch2

2.1.2

remotes

2.4.2

reprex

2.0.2

reshape2

1.4.4

rlang

1.1.1

rmarkdown

2.23

RODBC

1.3-20

roxygen2

7.2.3

rpart

4.1.19

rprojroot

2.0.3

Rserve

1.8-11

RSQLite

2.3.1

rstudioapi

0.15.0

rversions

2.1.2

rvest

1.0.3

sass

0.4.6

scales

1.2.1

selectr

0.4-2

sessioninfo

1.2.2

shape

1.4.6

shiny

1.7.4.1

sourcetools

0.1.7-1

sparklyr

1.8.1

spatial

7.3-15

splines

4.3.1

sqldf

0.4-11

SQUAREM

2021.1

stats

4.3.1

stats4

4.3.1

stringi

1.7.12

stringr

1.5.0

survival

3.5-5

sys

3.4.2

systemfonts

1.0.4

tcltk

4.3.1

testthat

3.1.10

textshaping

0.3.6

tibble

3.2.1

tidyr

1.3.0

tidyselect

1.2.0

tidyverse

2.0.0

timechange

0.2.0

timeDate

4022.108

tinytex

0.45

tools

4.3.1

tzdb

0.4.0

urlchecker

1.0.1

usethis

2.2.2

utf8

1.2.3

utils

4.3.1

uuid

1.1-0

vctrs

0.6.3

viridisLite

0.4.2

vroom

1.6.3

waldo

0.5.1

whisker

0.4.1

withr

2.5.0

xfun

0.39

xml2

1.3.5

xopen

1.0.0

xtable

1.8-4

yaml

2.3.7

zip

2.3.0

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID

Artifact ID

Version

antlr

antlr

2.7.7

com.amazonaws

amazon-kinesis-client

1.12.0

com.amazonaws

aws-java-sdk-autoscaling

1.12.390

com.amazonaws

aws-java-sdk-cloudformation

1.12.390

com.amazonaws

aws-java-sdk-cloudfront

1.12.390

com.amazonaws

aws-java-sdk-cloudhsm

1.12.390

com.amazonaws

aws-java-sdk-cloudsearch

1.12.390

com.amazonaws

aws-java-sdk-cloudtrail

1.12.390

com.amazonaws

aws-java-sdk-cloudwatch

1.12.390

com.amazonaws

aws-java-sdk-cloudwatchmetrics

1.12.390

com.amazonaws

aws-java-sdk-codedeploy

1.12.390

com.amazonaws

aws-java-sdk-cognitoidentity

1.12.390

com.amazonaws

aws-java-sdk-cognitosync

1.12.390

com.amazonaws

aws-java-sdk-config

1.12.390

com.amazonaws

aws-java-sdk-core

1.12.390

com.amazonaws

aws-java-sdk-datapipeline

1.12.390

com.amazonaws

aws-java-sdk-directconnect

1.12.390

com.amazonaws

aws-java-sdk-directory

1.12.390

com.amazonaws

aws-java-sdk-dynamodb

1.12.390

com.amazonaws

aws-java-sdk-ec2

1.12.390

com.amazonaws

aws-java-sdk-ecs

1.12.390

com.amazonaws

aws-java-sdk-efs

1.12.390

com.amazonaws

aws-java-sdk-elasticache

1.12.390

com.amazonaws

aws-java-sdk-elasticbeanstalk

1.12.390

com.amazonaws

aws-java-sdk-elasticloadbalancing

1.12.390

com.amazonaws

aws-java-sdk-elastictranscoder

1.12.390

com.amazonaws

aws-java-sdk-emr

1.12.390

com.amazonaws

aws-java-sdk-glacier

1.12.390

com.amazonaws

aws-java-sdk-glue

1.12.390

com.amazonaws

aws-java-sdk-iam

1.12.390

com.amazonaws

aws-java-sdk-importexport

1.12.390

com.amazonaws

aws-java-sdk-kinesis

1.12.390

com.amazonaws

aws-java-sdk-kms

1.12.390

com.amazonaws

aws-java-sdk-lambda

1.12.390

com.amazonaws

aws-java-sdk-logs

1.12.390

com.amazonaws

aws-java-sdk-machinelearning

1.12.390

com.amazonaws

aws-java-sdk-opsworks

1.12.390

com.amazonaws

aws-java-sdk-rds

1.12.390

com.amazonaws

aws-java-sdk-redshift

1.12.390

com.amazonaws

aws-java-sdk-route53

1.12.390

com.amazonaws

aws-java-sdk-s3

1.12.390

com.amazonaws

aws-java-sdk-ses

1.12.390

com.amazonaws

aws-java-sdk-simpledb

1.12.390

com.amazonaws

aws-java-sdk-simpleworkflow

1.12.390

com.amazonaws

aws-java-sdk-sns

1.12.390

com.amazonaws

aws-java-sdk-sqs

1.12.390

com.amazonaws

aws-java-sdk-ssm

1.12.390

com.amazonaws

aws-java-sdk-storagegateway

1.12.390

com.amazonaws

aws-java-sdk-sts

1.12.390

com.amazonaws

aws-java-sdk-support

1.12.390

com.amazonaws

aws-java-sdk-swf-libraries

1.11.22

com.amazonaws

aws-java-sdk-workspaces

1.12.390

com.amazonaws

jmespath-java

1.12.390

com.clearspring.analytics

stream

2.9.6

com.databricks

Rserve

1.8-3

com.databricks

databricks-sdk-java

0.7.0

com.databricks

jets3t

0.7.1-0

com.databricks.scalapb

compilerplugin_2.12

0.4.15-10

com.databricks.scalapb

scalapb-runtime_2.12

0.4.15-10

com.esotericsoftware

kryo-shaded

4.0.2

com.esotericsoftware

minlog

1.3.0

com.fasterxml

classmate

1.3.4

com.fasterxml.jackson.core

jackson-annotations

2.15.2

com.fasterxml.jackson.core

jackson-core

2.15.2

com.fasterxml.jackson.core

jackson-databind

2.15.2

com.fasterxml.jackson.dataformat

jackson-dataformat-cbor

2.15.2

com.fasterxml.jackson.datatype

jackson-datatype-joda

2.15.2

com.fasterxml.jackson.datatype

jackson-datatype-jsr310

2.15.1

com.fasterxml.jackson.module

jackson-module-paranamer

2.15.2

com.fasterxml.jackson.module

jackson-module-scala_2.12

2.15.2

com.github.ben-manes.caffeine

caffeine

2.9.3

com.github.fommil

jniloader

1.1

com.github.fommil.netlib

native_ref-java

1.1

com.github.fommil.netlib

native_ref-java

1.1-natives

com.github.fommil.netlib

native_system-java

1.1

com.github.fommil.netlib

native_system-java

1.1-natives

com.github.fommil.netlib

netlib-native_ref-linux-x86_64

1.1-natives

com.github.fommil.netlib

netlib-native_system-linux-x86_64

1.1-natives

com.github.luben

zstd-jni

1.5.5-4

com.github.wendykierp

JTransforms

3.1

com.google.code.findbugs

jsr305

3.0.0

com.google.code.gson

gson

2.10.1

com.google.crypto.tink

tink

1.9.0

com.google.errorprone

error_prone_annotations

2.10.0

com.google.flatbuffers

flatbuffers-java

1.12.0

com.google.guava

guava

15.0

com.google.protobuf

protobuf-java

2.6.1

com.helger

profiler

1.1.1

com.jcraft

jsch

0.1.55

com.jolbox

bonecp

0.8.0.RELEASE

com.lihaoyi

sourcecode_2.12

0.1.9

com.microsoft.azure

azure-data-lake-store-sdk

2.3.9

com.microsoft.sqlserver

mssql-jdbc

11.2.2.jre8

com.ning

compress-lzf

1.1.2

com.sun.mail

javax.mail

1.5.2

com.sun.xml.bind

jaxb-core

2.2.11

com.sun.xml.bind

jaxb-impl

2.2.11

com.tdunning

json

1.8

com.thoughtworks.paranamer

paranamer

2.8

com.trueaccord.lenses

lenses_2.12

0.4.12

com.twitter

chill-java

0.10.0

com.twitter

chill_2.12

0.10.0

com.twitter

util-app_2.12

7.1.0

com.twitter

util-core_2.12

7.1.0

com.twitter

util-function_2.12

7.1.0

com.twitter

util-jvm_2.12

7.1.0

com.twitter

util-lint_2.12

7.1.0

com.twitter

util-registry_2.12

7.1.0

com.twitter

util-stats_2.12

7.1.0

com.typesafe

config

1.2.1

com.typesafe.scala-logging

scala-logging_2.12

3.7.2

com.uber

h3

3.7.3

com.univocity

univocity-parsers

2.9.1

com.zaxxer

HikariCP

4.0.3

commons-cli

commons-cli

1.5.0

commons-codec

commons-codec

1.16.0

commons-collections

commons-collections

3.2.2

commons-dbcp

commons-dbcp

1.4

commons-fileupload

commons-fileupload

1.5

commons-httpclient

commons-httpclient

3.1

commons-io

commons-io

2.13.0

commons-lang

commons-lang

2.6

commons-logging

commons-logging

1.1.3

commons-pool

commons-pool

1.5.4

dev.ludovic.netlib

arpack

3.0.3

dev.ludovic.netlib

blas

3.0.3

dev.ludovic.netlib

lapack

3.0.3

info.ganglia.gmetric4j

gmetric4j

1.0.10

io.airlift

aircompressor

0.25

io.delta

delta-sharing-client_2.12

1.0.2

io.dropwizard.metrics

metrics-annotation

4.2.19

io.dropwizard.metrics

metrics-core

4.2.19

io.dropwizard.metrics

metrics-graphite

4.2.19

io.dropwizard.metrics

metrics-healthchecks

4.2.19

io.dropwizard.metrics

metrics-jetty9

4.2.19

io.dropwizard.metrics

metrics-jmx

4.2.19

io.dropwizard.metrics

metrics-json

4.2.19

io.dropwizard.metrics

metrics-jvm

4.2.19

io.dropwizard.metrics

metrics-servlets

4.2.19

io.netty

netty-all

4.1.96.Final

io.netty

netty-buffer

4.1.96.Final

io.netty

netty-codec

4.1.96.Final

io.netty

netty-codec-http

4.1.96.Final

io.netty

netty-codec-http2

4.1.96.Final

io.netty

netty-codec-socks

4.1.96.Final

io.netty

netty-common

4.1.96.Final

io.netty

netty-handler

4.1.96.Final

io.netty

netty-handler-proxy

4.1.96.Final

io.netty

netty-resolver

4.1.96.Final

io.netty

netty-tcnative-boringssl-static

2.0.61.Final-linux-aarch_64

io.netty

netty-tcnative-boringssl-static

2.0.61.Final-linux-x86_64

io.netty

netty-tcnative-boringssl-static

2.0.61.Final-osx-aarch_64

io.netty

netty-tcnative-boringssl-static

2.0.61.Final-osx-x86_64

io.netty

netty-tcnative-boringssl-static

2.0.61.Final-windows-x86_64

io.netty

netty-tcnative-classes

2.0.61.Final

io.netty

netty-transport

4.1.96.Final

io.netty

netty-transport-classes-epoll

4.1.96.Final

io.netty

netty-transport-classes-kqueue

4.1.96.Final

io.netty

netty-transport-native-epoll

4.1.96.Final

io.netty

netty-transport-native-epoll

4.1.96.Final-linux-aarch_64

io.netty

netty-transport-native-epoll

4.1.96.Final-linux-x86_64

io.netty

netty-transport-native-kqueue

4.1.96.Final-osx-aarch_64

io.netty

netty-transport-native-kqueue

4.1.96.Final-osx-x86_64

io.netty

netty-transport-native-unix-common

4.1.96.Final

io.prometheus

simpleclient

0.7.0

io.prometheus

simpleclient_common

0.7.0

io.prometheus

simpleclient_dropwizard

0.7.0

io.prometheus

simpleclient_pushgateway

0.7.0

io.prometheus

simpleclient_servlet

0.7.0

io.prometheus.jmx

collector

0.12.0

jakarta.annotation

jakarta.annotation-api

1.3.5

jakarta.servlet

jakarta.servlet-api

4.0.3

jakarta.validation

jakarta.validation-api

2.0.2

jakarta.ws.rs

jakarta.ws.rs-api

2.1.6

javax.activation

activation

1.1.1

javax.el

javax.el-api

2.2.4

javax.jdo

jdo-api

3.0.1

javax.transaction

jta

1.1

javax.transaction

transaction-api

1.1

javax.xml.bind

jaxb-api

2.2.11

javolution

javolution

5.5.1

jline

jline

2.14.6

joda-time

joda-time

2.12.1

net.java.dev.jna

jna

5.8.0

net.razorvine

pickle

1.3

net.sf.jpam

jpam

1.1

net.sf.opencsv

opencsv

2.3

net.sf.supercsv

super-csv

2.2.0

net.snowflake

snowflake-ingest-sdk

0.9.6

net.sourceforge.f2j

arpack_combined_all

0.1

org.acplt.remotetea

remotetea-oncrpc

1.1.2

org.antlr

ST4

4.0.4

org.antlr

antlr-runtime

3.5.2

org.antlr

antlr4-runtime

4.9.3

org.antlr

stringtemplate

3.2.1

org.apache.ant

ant

1.9.16

org.apache.ant

ant-jsch

1.9.16

org.apache.ant

ant-launcher

1.9.16

org.apache.arrow

arrow-format

12.0.1

org.apache.arrow

arrow-memory-core

12.0.1

org.apache.arrow

arrow-memory-netty

12.0.1

org.apache.arrow

arrow-vector

12.0.1

org.apache.avro

avro

1.11.2

org.apache.avro

avro-ipc

1.11.2

org.apache.avro

avro-mapred

1.11.2

org.apache.commons

commons-collections4

4.4

org.apache.commons

commons-compress

1.23.0

org.apache.commons

commons-crypto

1.1.0

org.apache.commons

commons-lang3

3.12.0

org.apache.commons

commons-math3

3.6.1

org.apache.commons

commons-text

1.10.0

org.apache.curator

curator-client

2.13.0

org.apache.curator

curator-framework

2.13.0

org.apache.curator

curator-recipes

2.13.0

org.apache.datasketches

datasketches-java

3.1.0

org.apache.datasketches

datasketches-memory

2.0.0

org.apache.derby

derby

10.14.2.0

org.apache.hadoop

hadoop-client-runtime

3.3.6

org.apache.hive

hive-beeline

2.3.9

org.apache.hive

hive-cli

2.3.9

org.apache.hive

hive-jdbc

2.3.9

org.apache.hive

hive-llap-client

2.3.9

org.apache.hive

hive-llap-common

2.3.9

org.apache.hive

hive-serde

2.3.9

org.apache.hive

hive-shims

2.3.9

org.apache.hive

hive-storage-api

2.8.1

org.apache.hive.shims

hive-shims-0.23

2.3.9

org.apache.hive.shims

hive-shims-common

2.3.9

org.apache.hive.shims

hive-shims-scheduler

2.3.9

org.apache.httpcomponents

httpclient

4.5.14

org.apache.httpcomponents

httpcore

4.4.16

org.apache.ivy

ivy

2.5.1

org.apache.logging.log4j

log4j-1.2-api

2.20.0

org.apache.logging.log4j

log4j-api

2.20.0

org.apache.logging.log4j

log4j-core

2.20.0

org.apache.logging.log4j

log4j-slf4j2-impl

2.20.0

org.apache.mesos

mesos

1.11.0-shaded-protobuf

org.apache.orc

orc-core

1.9.1-shaded-protobuf

org.apache.orc

orc-mapreduce

1.9.1-shaded-protobuf

org.apache.orc

orc-shims

1.9.1

org.apache.thrift

libfb303

0.9.3

org.apache.thrift

libthrift

0.12.0

org.apache.ws.xmlschema

xmlschema-core

2.3.0

org.apache.xbean

xbean-asm9-shaded

4.23

org.apache.yetus

audience-annotations

0.13.0

org.apache.zookeeper

zookeeper

3.6.3

org.apache.zookeeper

zookeeper-jute

3.6.3

org.checkerframework

checker-qual

3.31.0

org.codehaus.jackson

jackson-core-asl

1.9.13

org.codehaus.jackson

jackson-mapper-asl

1.9.13

org.codehaus.janino

commons-compiler

3.0.16

org.codehaus.janino

janino

3.0.16

org.datanucleus

datanucleus-api-jdo

4.2.4

org.datanucleus

datanucleus-core

4.1.17

org.datanucleus

datanucleus-rdbms

4.1.19

org.datanucleus

javax.jdo

3.2.0-m3

org.eclipse.jetty

jetty-client

9.4.52.v20230823

org.eclipse.jetty

jetty-continuation

9.4.52.v20230823

org.eclipse.jetty

jetty-http

9.4.52.v20230823

org.eclipse.jetty

jetty-io

9.4.52.v20230823

org.eclipse.jetty

jetty-jndi

9.4.52.v20230823

org.eclipse.jetty

jetty-plus

9.4.52.v20230823

org.eclipse.jetty

jetty-proxy

9.4.52.v20230823

org.eclipse.jetty

jetty-security

9.4.52.v20230823

org.eclipse.jetty

jetty-server

9.4.52.v20230823

org.eclipse.jetty

jetty-servlet

9.4.52.v20230823

org.eclipse.jetty

jetty-servlets

9.4.52.v20230823

org.eclipse.jetty

jetty-util

9.4.52.v20230823

org.eclipse.jetty

jetty-util-ajax

9.4.52.v20230823

org.eclipse.jetty

jetty-webapp

9.4.52.v20230823

org.eclipse.jetty

jetty-xml

9.4.52.v20230823

org.eclipse.jetty.websocket

websocket-api

9.4.52.v20230823

org.eclipse.jetty.websocket

websocket-client

9.4.52.v20230823

org.eclipse.jetty.websocket

websocket-common

9.4.52.v20230823

org.eclipse.jetty.websocket

websocket-server

9.4.52.v20230823

org.eclipse.jetty.websocket

websocket-servlet

9.4.52.v20230823

org.fusesource.leveldbjni

leveldbjni-all

1.8

org.glassfish.hk2

hk2-api

2.6.1

org.glassfish.hk2

hk2-locator

2.6.1

org.glassfish.hk2

hk2-utils

2.6.1

org.glassfish.hk2

osgi-resource-locator

1.0.3

org.glassfish.hk2.external

aopalliance-repackaged

2.6.1

org.glassfish.hk2.external

jakarta.inject

2.6.1

org.glassfish.jersey.containers

jersey-container-servlet

2.40

org.glassfish.jersey.containers

jersey-container-servlet-core

2.40

org.glassfish.jersey.core

jersey-client

2.40

org.glassfish.jersey.core

jersey-common

2.40

org.glassfish.jersey.core

jersey-server

2.40

org.glassfish.jersey.inject

jersey-hk2

2.40

org.hibernate.validator

hibernate-validator

6.1.7.Final

org.ini4j

ini4j

0.5.4

org.javassist

javassist

3.29.2-GA

org.jboss.logging

jboss-logging

3.3.2.Final

org.jdbi

jdbi

2.63.1

org.jetbrains

annotations

17.0.0

org.joda

joda-convert

1.7

org.jodd

jodd-core

3.5.2

org.json4s

json4s-ast_2.12

3.7.0-M11

org.json4s

json4s-core_2.12

3.7.0-M11

org.json4s

json4s-jackson_2.12

3.7.0-M11

org.json4s

json4s-scalap_2.12

3.7.0-M11

org.lz4

lz4-java

1.8.0

org.mariadb.jdbc

mariadb-java-client

2.7.9

org.mlflow

mlflow-spark

2.2.0

org.objenesis

objenesis

2.5.1

org.postgresql

postgresql

42.6.0

org.roaringbitmap

RoaringBitmap

0.9.45

org.roaringbitmap

shims

0.9.45

org.rocksdb

rocksdbjni

8.3.2

org.rosuda.REngine

REngine

2.1.0

org.scala-lang

scala-compiler_2.12

2.12.15

org.scala-lang

scala-library_2.12

2.12.15

org.scala-lang

scala-reflect_2.12

2.12.15

org.scala-lang.modules

scala-collection-compat_2.12

2.9.0

org.scala-lang.modules

scala-parser-combinators_2.12

1.1.2

org.scala-lang.modules

scala-xml_2.12

1.2.0

org.scala-sbt

test-interface

1.0

org.scalacheck

scalacheck_2.12

1.14.2

org.scalactic

scalactic_2.12

3.2.15

org.scalanlp

breeze-macros_2.12

2.1.0

org.scalanlp

breeze_2.12

2.1.0

org.scalatest

scalatest-compatible

3.2.15

org.scalatest

scalatest-core_2.12

3.2.15

org.scalatest

scalatest-diagrams_2.12

3.2.15

org.scalatest

scalatest-featurespec_2.12

3.2.15

org.scalatest

scalatest-flatspec_2.12

3.2.15

org.scalatest

scalatest-freespec_2.12

3.2.15

org.scalatest

scalatest-funspec_2.12

3.2.15

org.scalatest

scalatest-funsuite_2.12

3.2.15

org.scalatest

scalatest-matchers-core_2.12

3.2.15

org.scalatest

scalatest-mustmatchers_2.12

3.2.15

org.scalatest

scalatest-propspec_2.12

3.2.15

org.scalatest

scalatest-refspec_2.12

3.2.15

org.scalatest

scalatest-shouldmatchers_2.12

3.2.15

org.scalatest

scalatest-wordspec_2.12

3.2.15

org.scalatest

scalatest_2.12

3.2.15

org.slf4j

jcl-over-slf4j

2.0.7

org.slf4j

jul-to-slf4j

2.0.7

org.slf4j

slf4j-api

2.0.7

org.threeten

threeten-extra

1.7.1

org.tukaani

xz

1.9

org.typelevel

algebra_2.12

2.0.1

org.typelevel

cats-kernel_2.12

2.1.1

org.typelevel

spire-macros_2.12

0.17.0

org.typelevel

spire-platform_2.12

0.17.0

org.typelevel

spire-util_2.12

0.17.0

org.typelevel

spire_2.12

0.17.0

org.wildfly.openssl

wildfly-openssl

1.1.3.Final

org.xerial

sqlite-jdbc

3.42.0.0

org.xerial.snappy

snappy-java

1.1.10.3

org.yaml

snakeyaml

2.0

oro

oro

2.0.8

pl.edu.icm

JLargeArrays

1.5

software.amazon.cryptools

AmazonCorrettoCryptoProvider

1.6.1-linux-x86_64

software.amazon.ion

ion-java

1.0.2

stax

stax-api

1.0.1