Databricks Runtime 13.3 LTS

The following release notes provide information about Databricks Runtime 13.3 LTS, powered by Apache Spark 3.4.1.

Databricks released this version in August 2023.

Note

LTS means this version is under long-term support. See Databricks Runtime LTS version lifecycle.

Note

These release notes may include references to features that are not available on Google Cloud as of this release.

Tip

To see release notes for Databricks Runtime versions that have reached end-of-support (EoS), see End-of-support Databricks Runtime release notes. The EoS Databricks Runtime versions have been retired and might not be updated.

New features and improvements

Scala support for shared clusters

Unity Catalog-enabled shared clusters now support Scala, when using Databricks Runtime 13.3 and above.

Allowlist for JARs and Maven coordinates on Unity Catalog shared clusters is in Public Preview

You can now use the Unity Catalog allowlist to control installation of JARs and Maven coordinates on compute with shared access mode running Databricks Runtime 13.3 and above. See Allowlist libraries and init scripts on shared compute.

Volumes support for JARs is in Public Preview

You can now use Unity Catalog volumes to store JARs on compute with assigned or shared access modes running Databricks Runtime 13.3 and above. See Install libraries from a volume.

Safe casts enabled by default for Delta Lake operations

Delta UPDATE and MERGE operations now result in an error when values cannot be safely cast to the type in the target table schema. All implicit casts in Delta now follow spark.sql.storeAssignmentPolicy instead of spark.sql.ansi.enabled. See ANSI compliance in Databricks Runtime.

Introducing tags with Unity Catalog

With Databricks Runtime 13.3 LTS and later, you can apply tags to your securable objects. Tags help you organize securable objects and simplify search and discovery of your data assets. See Apply tags to Unity Catalog securable objects.

Databricks Runtime returns an error if a file is modified between query planning and invocation

Databricks Runtime queries now return an error if a file is updated between query planning and invocation. Before this change, Databricks Runtime would read a file between these stages, which occasionally lead to unpredictable results.

Block schema overwrite when using dynamic partition overwrites

You cannot set overwriteSchema to true in combination with dynamic partition overwrites in Delta Lake. This prevents possible table corruption due to schema mismatch. See Dynamic partition overwrites.

Integrate copyFile API into dbutils.fs.cp

The dbutils copy command, dbutils.fs.cp is now optimized for faster copying. With this improvement, copy operations can be up to 100x faster, depending on the file size. The feature is available across all file systems accessible in Databricks, including for Unity Catalog Volumes and Databricks Filesystem mounts.

Databricks ODBC/JDBC driver support

Databricks supports ODBC and JDBC drivers released in the past 2 years. Download the latest driver at Download ODBC Driver and Download JDBC Driver.

Support for unnamed (positional) parameter markers

The SparkSession.sql() API now supports parameterized queries using the ? syntax in addition to parameterized queries using the :parm syntax. See parameter markers.

Enable time series column labeling

You can now label time series columns when creating primary key constraints.

New bitmap SQL functions

Databricks now provided a set of functions which can be used to map integral numeric values onto a bitmap and aggregate them. This can be used to pre-compute number of distinct values.

Improved encryption functions

The aes_encrypt, aes_decrypt, and try_aes_decrypt functions have been improved with support for initialization vectors (IV) and authenticated additional data (AAD).

Unity Catalog support for REFRESH FOREIGN

You can now update foreign catalogs, schemas, and tables in Unity Catalog using REFRESH FOREIGN CATALOG, REFRESH FOREIGN SCHEMA, and REFRESH FOREIGN TABLE. See REFRESH FOREIGN (CATALOG, SCHEMA, and TABLE).

INSERT BY NAME

When inserting rows into a table you can now let Databricks map columns and fields in the source query automatically to the table’s columns by their name. See INSERT INTO.

Share materialized views with Delta Sharing

The ALTER SHARE statement now supports materialized views.

Breaking changes

Changes to implicit casting when storing rows in a table with MERGE INTO and UPDATE

Databricks now follows the configuration spark.sql.storeAssignmentPolicy for implicit casting when storing rows in a table. The default value ANSI throws an error when storing values that overflow. Previously, values would be stored as NULL by default.

As in the following example, you can address overflows by rewriting the table to use a broader type for the column causing the error.

-- Enable column mapping
ALTER TABLE MyTable SET TBLPROPERTIES (
    'delta.minReaderVersion' = '2',
    'delta.minWriterVersion' = '5',
    'delta.columnMapping.mode' = 'name'
  )

-- Rename the old column and create a new column with the new type using the old name
ALTER TABLE MyTable RENAME ID to ID_old
ALTER TABLE MyTable ADD COLUMN ID BIGINT

-- Rewrite the entire table
UPDATE MyTable SET ID = ID_old

-- Drop the old column
ALTER TABLE MyTable DROP COLUMN ID_old

Parquet Schema Inference Changes

When inferring schemas from Parquet files not written by Spark, int64 timestamp columns annotated with isAdjustedToUTC=false will now default to TIMESTAMP_NTZ type. Previously, these were inferred as TIMESTAMP type. This update enhances the accuracy of mapping Parquet types to Spark SQL types.

As a result, reading data from external Parquet files into a Delta table may lead to an error if the timestampNtz feature is not enabled. The error will display as follows:

Fatal exception of the update:
com.databricks.sql.transaction.tahoe.DeltaTableFeatureException
Your table schema requires manual enablement of the following table feature(s): timestampNtz.

To maintain the previous behavior and prevent this error, you can set the Spark configuration parameter spark.sql.parquet.inferTimestampNTZ.enabled to false.

Library upgrades

  • Upgraded Python libraries:

    • debugpy from 1.5.1 to 1.6.0

    • distlib from 0.3.6 to 0.3.7

    • fastjsonschema from 2.16.3 to 2.18.0

    • filelock from 3.12.0 to 3.12.2

  • Upgraded R libraries:

    • SparkR from 3.4.0 to 3.4.1

  • Upgraded Java libraries:

    • org.apache.orc.orc-core from 1.8.3-shaded-protobuf to 1.8.4-shaded-protobuf

    • org.apache.orc.orc-mapreduce from 1.8.3-shaded-protobuf to 1.8.4-shaded-protobuf

    • org.apache.orc.orc-shims from 1.8.3 to 1.8.4

    • org.eclipse.jetty.jetty-client from 9.4.50.v20221201 to 9.4.51.v20230217

    • org.eclipse.jetty.jetty-continuation from 9.4.50.v20221201 to 9.4.51.v20230217

    • org.eclipse.jetty.jetty-http from 9.4.50.v20221201 to 9.4.51.v20230217

    • org.eclipse.jetty.jetty-io from 9.4.50.v20221201 to 9.4.51.v20230217

    • org.eclipse.jetty.jetty-jndi from 9.4.50.v20221201 to 9.4.51.v20230217

    • org.eclipse.jetty.jetty-plus from 9.4.50.v20221201 to 9.4.51.v20230217

    • org.eclipse.jetty.jetty-proxy from 9.4.50.v20221201 to 9.4.51.v20230217

    • org.eclipse.jetty.jetty-security from 9.4.50.v20221201 to 9.4.51.v20230217

    • org.eclipse.jetty.jetty-server from 9.4.50.v20221201 to 9.4.51.v20230217

    • org.eclipse.jetty.jetty-servlet from 9.4.50.v20221201 to 9.4.51.v20230217

    • org.eclipse.jetty.jetty-servlets from 9.4.50.v20221201 to 9.4.51.v20230217

    • org.eclipse.jetty.jetty-util from 9.4.50.v20221201 to 9.4.51.v20230217

    • org.eclipse.jetty.jetty-util-ajax from 9.4.50.v20221201 to 9.4.51.v20230217

    • org.eclipse.jetty.jetty-webapp from 9.4.50.v20221201 to 9.4.51.v20230217

    • org.eclipse.jetty.jetty-xml from 9.4.50.v20221201 to 9.4.51.v20230217

    • org.eclipse.jetty.websocket.websocket-api from 9.4.50.v20221201 to 9.4.51.v20230217

    • org.eclipse.jetty.websocket.websocket-client from 9.4.50.v20221201 to 9.4.51.v20230217

    • org.eclipse.jetty.websocket.websocket-common from 9.4.50.v20221201 to 9.4.51.v20230217

    • org.eclipse.jetty.websocket.websocket-server from 9.4.50.v20221201 to 9.4.51.v20230217

    • org.eclipse.jetty.websocket.websocket-servlet from 9.4.50.v20221201 to 9.4.51.v20230217

    • org.xerial.snappy.snappy-java from 1.1.8.4 to 1.1.10.1

Apache Spark

Databricks Runtime 13.3 includes Apache Spark 3.4.1. This release includes all Spark fixes and improvements included in Databricks Runtime 13.2 (EoS), as well as the following additional bug fixes and improvements made to Spark:

  • You can now set cluster environment variable SNOWFLAKE_SPARK_CONNECTOR_VERSION=2.12 to use Spark-snowflake connector v2.12.0.

  • [SPARK-43380] [DBRRM-383] Revert “[SC-133100][SQL] Fix Avro data type conver…

  • [SPARK-44504] [Backport][13.2][13.x][13.3] Unload provider thereby forcing DB instance close and releasing resources on maintenance task error

  • [SPARK-44059] [SC-137023] Add analyzer support of named arguments for built-in functions

  • [SPARK-44485] [SC-137653][SQL] Optimize TreeNode.generateTreeString

  • [SPARK-42944] [SC-137164][SS][PYTHON] Streaming ForeachBatch in Python

  • [SPARK-44446] [SC-136994][PYTHON] Add checks for expected list type special cases

  • [SPARK-44439] [SC-136973][CONNECT][SS]Fixed listListeners to only send ids back to client

  • [SPARK-44216] [SC-136941] [PYTHON] Make assertSchemaEqual API public

  • [SPARK-44201] [SC-136778][CONNECT][SS]Add support for Streaming Listener in Scala for Spark Connect

  • [SPARK-43915] [SC-134766][SQL] Assign names to the error class LEGACYERROR_TEMP_[2438-2445]

  • [SPARK-43798] Revert “[SC-133990][SQL][PYTHON] Support Python user-defined table functions”

  • [SPARK-44398] [SC-136720][CONNECT] Scala foreachBatch API

  • [SPARK-44476] [SC-137169][CORE][CONNECT] Fix population of artifacts for a JobArtifactState with no associated artifacts

  • [SPARK-44269] [SC-135817][SQL] Assign names to the error class LEGACYERROR_TEMP_[2310-2314]

  • [SPARK-44395] [SC-136744][SQL] Update TVF arguments to require parentheses around identifier after TABLE keyword

  • [SPARK-43915] [SC-134766][SQL] Assign names to the error class LEGACYERROR_TEMP_[2438-2445]

  • [SPARK-43995] [SC-136794][SPARK-43996][CONNECT] Add support for UDFRegistration to the Connect Scala Client

  • [SPARK-44109] [SC-134755][CORE] Remove duplicate preferred locations of each RDD partition

  • [SPARK-44169] [SC-135497][SQL] Assign names to the error class LEGACYERROR_TEMP_[2300-2304]

  • [SPARK-41487] [SC-135636][SQL] Assign name to LEGACYERROR_TEMP_1020

  • [SPARK-44398] [SC-136720][CONNECT] Scala foreachBatch API

  • [SPARK-44153] [SC-134977][CORE][UI] Support Heap Histogram column in Executors tab

  • [SPARK-44044] [SC-135695][SS] Improve Error message for Window functions with streaming

  • [SPARK-43914] [SC-135414][SQL] Assign names to the error class LEGACYERROR_TEMP_[2433-2437]

  • [SPARK-44217] [SC-136805][PYTHON] Allow custom precision for fp approx equality

  • [SPARK-44370] [SC-136575][CONNECT] Migrate Buf remote generation alpha to remote plugins

  • [SPARK-44410] [SC-136930][PYTHON][CONNECT] Set active session in create, not just getOrCreate

  • [SPARK-44348] [SAS-1910][SC-136644][13.X][CORE][CONNECT][PYTHON] Reenable test_artifact with relevant changes

  • [SPARK-44145] [SC-136698][SQL] Callback when ready for execution

  • [SPARK-44264] [SC-136870][PYTHON][ML] FunctionPickler Class

  • [SPARK-43986] [SC-135623][SQL] Create error classes for HyperLogLog function call failures

  • [SPARK-44139] [SC-134967][SQL] Discard completely pushed down filters in group-based MERGE operations

  • [SPARK-44338] [SC-136718][SQL] Fix view schema mismatch error message

  • [SPARK-44208] [SC-135505][CORE][SQL] Assign clear error class names for some logic that directly uses exceptions

  • [SPARK-44364] [SC-136773] [PYTHON] Add support for List[Row] data type for expected

  • [SPARK-44180] [SC-136792][SQL] DistributionAndOrderingUtils should apply ResolveTimeZone

  • [SPARK-43389] [SC-136772][SQL] Added a null check for lineSep option

  • [SPARK-44388] [SC-136695][CONNECT] Fix protobuf cast issue when UDF instance is updated

  • [SPARK-44352] [SC-136373][CONNECT] Put back sameType and friends in DataType.

  • [SPARK-43975] [SC-134265][SQL] DataSource V2: Handle UPDATE commands for group-based sources

  • [SPARK-44360] [SC-136461][SQL] Support schema pruning in delta-based MERGE operations

  • [SPARK-44138] [SC-134907][SQL] Prohibit non-deterministic expressions, subqueries and aggregates in MERGE conditions

  • [SPARK-43321] [SC-136678][Connect][Followup] Better names for APIs used in Scala Client joinWith

  • [SPARK-44263] [SC-136447][CONNECT] Custom Interceptors Support

  • [SPARK-39851] [SC-123383][SQL] Improve join stats estimation if one side can keep uniqueness

  • [SPARK-44390] [SC-136677][CORE][SQL] Rename SparkSerDerseUtils to SparkSerDeUtils

  • [SPARK-44004] [SC-134751][SQL] Assign name & improve error message for frequent LEGACY errors.

  • [SPARK-44322] [SC-136204][CONNECT] Make parser use SqlApiConf instead of SQLConf.

  • [SPARK-44271] [SC-136347][SQL] Move default values functions from StructType to ResolveDefaultColumns

  • [SPARK-44332] [SC-136413][CORE][WEBUI] Fix the sorting error of Executor ID Column on Executors UI Page

  • [SPARK-44282] [SC-135948][CONNECT] Prepare DataType parsing for use in Spark Connect Scala Client

  • [SPARK-44349] [SC-136581][R] Add math functions to SparkR

  • [SPARK-44397] [SC-136647][PYTHON] Expose assertDataFrameEqual in pyspark.testing.utils

  • [SPARK-42981] [SC-136620][CONNECT] Add direct arrow serialization

  • [SPARK-44373] [SC-136577][SQL] Wrap withActive for Dataset API w/ parse logic to make parser related configuration work

  • [SPARK-44340] [SC-136574][SQL] Define the computing logic through PartitionEvaluator API and use it in WindowGroupLimitExec

  • [SPARK-43665] [SC-136573][CONNECT][PS] Enable PandasSQLStringFormatter.vformat to work with Spark Connect

  • [SPARK-44325] [SC-136572][SQL] Use PartitionEvaluator API in SortMergeJoinExec

  • [SPARK-44315] [SC-136171][SQL][CONNECT] Move DefinedByConstructorParams to sql/api

  • [SPARK-44275] [SC-136184][CONNECT] Add configurable retry mechanism to Scala Spark Connect

  • [SPARK-44326] [SC-136208][SQL][CONNECT] Move utils that are used from Scala client to the common modules

  • [SPARK-43321] [SC-136156][Connect] Dataset#Joinwith

  • [SPARK-44343] [SC-136372][CONNECT] Prepare ScalaReflection to the move to SQL/API

  • [SPARK-44313] [SC-136081][SQL] Fix generated column expression validation when there is a char/varchar column in the schema

  • [SPARK-43963] [SC-134145][SQL] DataSource V2: Handle MERGE commands for group-based sources

  • [SPARK-44363] [SC-136433] [PYTHON] Display percent of unequal rows in DataFrame comparison

  • [SPARK-44251] [SC-136432][SQL] Set nullable correctly on coalesced join key in full outer USING join

  • [SPARK-43948] [SC-133366][SQL] Assign names to the error class LEGACYERROR_TEMP_[0050|0057|0058|0059]

  • [SPARK-44329] [SC-136326][CONNECT][PYTHON] Add hll_sketch_agg, hll_union_agg, to_varchar, try_aes_decrypt to Scala and Python

  • [SPARK-44351] [SC-136353][SQL] Make some syntactic simplification

  • [SPARK-44281] [SC-135963][SQL] Move QueryCompilation error that used by DataType to sql/api as DataTypeErrors

  • [SPARK-44283] [SC-136109][CONNECT] Move Origin to SQL/API

  • [SPARK-43926] [SC-135590][CONNECT][PYTHON] Add array_agg, array_size, cardinality, count_min_sketch,mask,named_struct,json_* to Scala and Python

  • [SPARK-44327] [SC-136187][SQL][CONNECT] Add functions any and len to Scala

  • [SPARK-44290] [SC-136300][CONNECT] Session-based files and archives in Spark Connect

  • [SPARK-44061] [SC-136299][PYTHON] Add assertDataFrameEqual util function

  • [SPARK-44331] [SC-136324][CONNECT][PYTHON] Add bitmap functions to Scala and Python

  • [SPARK-44342] [SC-136334][SQL] Replace SQLContext with SparkSession for GenTPCDSData

  • [SPARK-42583] [SC-124190][SQL] Remove the outer join if they are all distinct aggregate functions

  • [SPARK-44333] [SC-136206][CONNECT][SQL] Move EnhancedLogicalPlan out of ParserUtils

  • [SPARK-43824] [SC-132655][SPARK-43825] [SQL] Assign names to the error class LEGACYERROR_TEMP_128[1-2]

  • [SPARK-43939] [SC-134602][CONNECT][PYTHON] Add try_* functions to Scala and Python

  • [SPARK-44193] [SC-135886][CONNECT] Implement GRPC exceptions interception for conversion

  • [SPARK-44284] [SC-136107][CONNECT] Create simple conf system for sql/api

  • [SPARK-43913] [SC-133368][SQL] Assign names to the error class LEGACYERROR_TEMP_[2426-2432]

  • [SPARK-44291] Revert “[SC-135924][SPARK-43416][CONNECT] Fix incorrect schema for range query”

  • [SPARK-44312] [SC-136185][CONNECT][PYTHON] Allow to set a user agent with an environment variable

  • [SPARK-44154] [SC-136060] Implement bitmap functions

  • [SPARK-43885] [SC-133996][SQL] DataSource V2: Handle MERGE commands for delta-based sources

  • [SPARK-43924] [SC-135161][CONNECT][PYTHON] Add misc functions to Scala and Python

  • [SPARK-43969] [SC-134556][SQL] Refactor & Assign names to the error class LEGACYERROR_TEMP_1170

  • [SPARK-44185] [SC-135730][SQL] Fix inconsistent path qualifying between catalog and data operations

  • [SPARK-44073] [SC-134562][SQL][PYTHON][CONNECT] Add date time functions to Scala, Python and Connect - part 2

  • [SPARK-43942] [SC-134527][CONNECT][PYTHON] Add string functions to Scala and Python - part 1

  • [SPARK-44266] [SC-135813][SQL] Move Util.truncatedString to sql/api

  • [SPARK-44195] [SC-135722][R] Add JobTag APIs to SparkR SparkContext

  • [SPARK-44200] [SC-135736][SQL] Support TABLE argument parser rule for TableValuedFunction

  • [SPARK-44254] [SC-135709][SQL] Move QueryExecutionErrors that used by DataType to sql/api as DataTypeErrors

  • [SPARK-44291] [SC-135924][SPARK-43416][CONNECT] Fix incorrect schema for range query

  • [SPARK-43922] [SC-135612][SQL] Add named parameter support in parser for function calls

  • [SPARK-44300] [SC-135882][CONNECT] Fix artifact cleanup to limit deletion scope to session specific artifacts

  • [SPARK-44293] [SC-135849][CONNECT] Fix invalid URI for custom JARs in Spark Connect

  • [SPARK-44215] [SC-135850][SHUFFLE] If num chunks are 0, then server should throw a RuntimeException

  • [SPARK-44245] [SC-135851][PYTHON] pyspark.sql.dataframe sample() doctests should be illustrative-only

  • [SPARK-44274] [SC-135814][CONNECT] Move out util functions used by ArtifactManager to common/utils

  • [SPARK-44194] [SC-135815][PYTHON][CORE] Add JobTag APIs to PySpark SparkContext

  • [SPARK-42828] [SC-135733][PYTHON][SQL] More explicit Python type annotations for GroupedData

  • [SPARK-44079] [SC-135486][SQL] Fix ArrayIndexOutOfBoundsException when parse array as struct using PERMISSIVE mode with corrupt record

  • [SPARK-44255] [SC-135809][SQL] Relocate StorageLevel to common/utils

  • [SPARK-42169] [SC-135795] [SQL] Implement code generation for to_csv function (StructsToCsv)

  • [SPARK-44249] [SC-135719][SQL][PYTHON] Refactor PythonUDTFRunner to send its return type separately

  • [SPARK-43353] [SC-132734][PYTHON] Migrate remaining session errors into error class

  • [SPARK-44211] [SC-135718][PYTHON][CONNECT] Implement SparkSession.is_stopped

  • [SPARK-42784] [SC-135691] should still create subDir when the number of subDir in merge dir is less than conf

  • [SPARK-41599] [SC-135616] Memory leak in FileSystem.CACHE when submitting apps to secure cluster using InProcessLauncher

  • [SPARK-44241] [SC-135613][Core] Mistakenly set io.connectionTimeout/connectionCreationTimeout to zero or negative will cause incessant executor cons/destructions

  • [SPARK-44133] [SC-134795][13.X][PYTHON] Upgrade MyPy from 0.920 to 0.982

  • [SPARK-42941] [SC-134707][SS][CONNECT][1/2] StreamingQueryListener - Event Serde in JSON format

  • [SPARK-44188] [SC-135173][CORE] Remove useless resetAllPartitions method in ActiveJob

  • [SPARK-43757] [SC-135418][CONNECT] Change client compatibility from allow list to deny list

  • [SPARK-43474] [SC-135521] [SS] [CONNECT] Add a spark connect access to runtime Dataframes by ID.

  • [SPARK-44183] [SC-135209][PYTHON] Increate PyArrow minimum version to 4.0.0

  • [SPARK-44248] [SC-135554][SS][SQL][Kafka] Add preferred location in kafka source v2

  • [SPARK-43929] [SC-134510][SQL][PYTHON][CONNECT] Add date time functions to Scala, Python and Connect API - part 1

  • [SPARK-44161] [SC-135326][Connect] Handle Row input for UDFs

  • [SPARK-44227] [SC-135520][SQL] Extract SchemaUtils from StructField

  • [SPARK-44064] [SC-135116][CORE][SQL] Add a new apply function to NonFateSharingCache

  • [SPARK-44171] [SC-135269][SQL] Assign names to the error class LEGACYERROR_TEMP_[2279-2282] & delete some unused error classes

  • [SPARK-44056] [SC-134758][SQL] Include UDF name in UDF execution failure error message when available

  • [SPARK-44205] [SC-135422][SQL] Extract Catalyst Code from DecimalType

  • [SPARK-44220] [SC-135484][SQL] Move StringConcat to sql/api

  • [SPARK-44146] [SC-135230][CONNECT] Isolate Spark Connect Session jars and classfiles

  • [SPARK-44206] [SC-135411][SQL] DataSet.selectExpr scope Session.active

  • [SPARK-40850] [SC-135417][SQL] Fix test case interpreted queries may execute Codegen

  • [SPARK-44204] [SC-135253][SQL][HIVE] Add missing recordHiveCall for getPartitionNames

  • [SPARK-44237] [SC-135485][CORE] Simplify DirectByteBuffer constructor lookup logic

  • [SPARK-44189] [SC-135170][CONNECT][PYTHON] Support positional parameters by sql()

  • [SPARK-44140] [SC-134893][SQL][PYTHON] Support positional parameters in Python sql()

  • [SPARK-44164] [SC-135158][SQL] Extract toAttribute method from StructField to Util class

  • [SPARK-44178] [SC-135113][CONNECT] Support positional parameters in sql()

  • [SPARK-43208] [SC-128955][SQL][HIVE] IsolatedClassLoader should close barrier class InputStream after reading

  • [SPARK-43063] [SC-128382][SQL] df.show handle null should print NULL instead of null

  • [SPARK-42539] [SC-124464][SQL][HIVE] Eliminate separate classloader when using ‘builtin’ Hive version for metadata client

  • [SPARK-43992] [SC-133645][SQL][PYTHON][CONNECT] Add optional pattern for Catalog.listFunctions

  • [SPARK-44066] [SC-134878][SQL] Support positional parameters in Scala/Java sql()

  • [SPARK-44078] [SC-134869][CONNECT][CORE] Add support for classloader/resource isolation

  • [SPARK-43470] [SC-135187][CORE] Add OS, Java, Python version information to application log

  • [SPARK-43136] [SC-135238][CONNECT][Followup] Adding tests for KeyAs

  • [SPARK-39740] [SC-135093][UI]: Upgrade vis timeline to 7.7.2 to fix CVE-2020-28487

  • [SPARK-42298] [SC-133952][SQL] Assign name to LEGACYERROR_TEMP_2132

  • [SPARK-43961] [SC-133459][SQL][PYTHON][CONNECT] Add optional pattern for Catalog.listTables

  • [SPARK-44026] [SC-135181] Allow providing initial value for SQLMetrics

  • [SPARK-44202] [SC-135244][CORE] Add JobTag APIs to JavaSparkContext

  • [SPARK-44030] [SC-135237][SQL] Implement DataTypeExpression to offer Unapply for expression

  • [SPARK-43876] Revert “[SC-134603][SQL] Enable fast hashmap for distinct queries”

  • [SPARK-40082] [SC-126437] Schedule mergeFinalize when push merge shuffleMapStage retry but no running tasks

  • [SPARK-43888] [SC-132893][CORE] Relocate Logging to common/utils

  • [SPARK-43474] [SC-134400] [SS] [CONNECT] Add SessionHolder to SparkConnectPlanner

  • [SPARK-43944] [SC-134490][CONNECT][PYTHON] Add string functions to Scala and Python - part 2

  • [SPARK-44134] [SC-134954][CORE] Fix setting resources (GPU/FPGA) to 0 when they are set in spark-defaults.conf

  • [SPARK-44142] [SC-134947][PYTHON] Replace type with tpe in utility to convert python types to spark types

  • [SPARK-43952] [SC-134695][CORE][CONNECT][SQL] Add SparkContext APIs for query cancellation by tag

  • [SPARK-44000] [SC-133634][SQL] Add hint to disable broadcasting and replicating one side of join

  • [SPARK-43937] [SC-134362][CONNECT][PYTHON] Add ifnull,isnotnull,equal_null,nullif,nvl,nvl2 to Scala and Python

  • [SPARK-43624] [13.x][SC-134557][PS][CONNECT] Add EWM to SparkConnectPlanner.

  • [SPARK-44040] [SC-134366][SQL] Fix compute stats when AggregateExec node above QueryStageExec

  • [SPARK-43485] [SC-131264][SQL] Fix the error message for the unit argument of the datetime add/diff functions

  • [SPARK-43794] [SC-132561][SQL] Assign a name to the error class LEGACYERROR_TEMP_1335

  • [SPARK-43511] [SC-134749][CONNECT][SS]Implemented MapGroupsWithState and FlatMapGroupsWithState APIs for Spark Connect

  • [SPARK-43529] [SQL] Support CREATE/REPLACE TABLE OPTIONS expressions + code cleanup

  • [SPARK-44106] [SC-134750][PYTHON][CONNECT] Add __repr__ for GroupedData

  • [SPARK-42299] [SC-133190] Assign name to LEGACYERROR_TEMP_2206

  • [SPARK-43290] [SC-134388][SQL] Adds support for aes_encrypt IVs and AAD

  • [SPARK-43932] [SC-134138][SQL][PYTHON][CONNECT] Add current like functions to Scala and Python

  • [SPARK-43934] [SC-134108][SQL][PYTHON][CONNECT] Add regexp_* functions to Scala and Python

  • [SPARK-44107] [SC-134746][CONNECT][PYTHON] Hide unsupported Column methods from auto-completion

  • [SPARK-42941] [SC-134707][SS][CONNECT][1/2] StreamingQueryListener - Event Serde in JSON format

  • [SPARK-43773] [SC-132659][CONNECT][PYTHON] Implement ‘levenshtein(str1, str2[, threshold])’ functions in python client

  • [SPARK-44125] [SC-134745][R] Support Java 21 in SparkR

  • [SPARK-43919] [SC-133374][SQL] Extract JSON functionality out of Row

  • [SPARK-43769] [SC-132520][CONNECT] Implement ‘levenshtein(str1, str2[, threshold])’ functions

  • [SPARK-44012] [SC-134662][SS] KafkaDataConsumer to print some read status

  • [SPARK-43876] [SC-134603][SQL] Enable fast hashmap for distinct queries

  • [SPARK-44024] [SC-134497][SQL] Change to use map when unzip only used to extract a single element

  • [SPARK-43928] [SC-134467][SQL][PYTHON][CONNECT] Add bit operations to Scala, Python and Connect API

  • [SPARK-44092] [SC-134506][CORE] Add Utils.isJavaVersionAtLeast21 and make core module pass with Java 21

  • [SPARK-44018] [SC-134512][SQL] Improve the hashCode and toString for some DS V2 Expression

  • [SPARK-44105] [SC-134582][SQL] LastNonNull should be lazily resolved

  • [SPARK-44081] [SC-134434] Simplify PartitionedFileUtil API a little

  • [SPARK-43791] [SC-132531][SQL] Assign a name to the error class LEGACYERROR_TEMP_1336

  • [SPARK-44071] [SC-134435] Define and use Unresolved[Leaf|Unary]Node traits.

  • [SPARK-43852] [SC-132890][SPARK-43853][SPARK-43854][SPARK-43855][SPARK-43856] Assign names to the error class LEGACYERROR_TEMP_2418-2425

  • [SPARK-43742] [SC-132684][SQL] Refactor default column value resolution

  • [SPARK-43493] [SC-132063][SQL] Add a max distance argument to the levenshtein() function

  • [SPARK-44075] [SC-134382][CONNECT] Make transformStatCorr lazy

  • [SPARK-43925] [SC-134381][SQL][PYTHON][CONNECT] Add some, bool_or,bool_and,every to Scala, Python and Connect

  • [SPARK-43931] [SC-134158][SQL][PYTHON][CONNECT] Add make_* functions to Scala and Python

  • [SPARK-43762] [SC-132602][SPARK-43763][SPARK-43764][SPARK-43765][SPARK-43766][SQL] Assign names to the error class LEGACYERROR_TEMP_24[06-10]

  • [SPARK-43962] [SC-133367][SQL] Improve error messages: CANNOT_DECODE_URL, CANNOT_MERGE_INCOMPATIBLE_DATA_TYPE, CANNOT_PARSE_DECIMAL, CANNOT_READ_FILE_FOOTER, CANNOT_RECOGNIZE_HIVE_TYPE.

  • [SPARK-43938] [SC-133887][CONNECT][PYTHON] Add to_* functions to Scala and Python

  • [SPARK-44055] [SC-134321][CORE] Remove redundant override functions from CheckpointRDD

  • [SPARK-43802] [SC-132597][SQL] Fix codegen for unhex and unbase64 with failOnError=true

  • [SPARK-43798] [SC-133990][SQL][PYTHON] Support Python user-defined table functions

  • [SPARK-43941] [SC-134286][SQL][PYTHON][CONNECT] Add any_value, approx_percentile, count_if, first_value, histogram_numeric, last_value, reduce to Scala, Python and Connect API

  • [SPARK-43380] [SC-133100][SQL] Fix Avro data type conversion issues to avoid producing incorrect results

  • [SPARK-43803] [SC-133729] [SS] [CONNECT] Improve awaitTermination() to handle client disconnects

  • [SPARK-43179] [SC-129070][SHUFFLE] Allowing apps to control whether their metadata gets saved in the db by the External Shuffle Service

Databricks ODBC/JDBC driver support

Databricks supports ODBC/JDBC drivers released in the past 2 years. Please download the recently released drivers and upgrade (download ODBC, download JDBC).

See Databricks Runtime 13.3 maintenance updates.

System environment

  • Operating System: Ubuntu 22.04.2 LTS

  • Java: Zulu 8.70.0.23-CA-linux64

  • Scala: 2.12.15

  • Python: 3.10.12

  • R: 4.2.2

  • Delta Lake: 2.4.0

Installed Python libraries

Library

Version

Library

Version

Library

Version

appdirs

1.4.4

argon2-cffi

21.3.0

argon2-cffi-bindings

21.2.0

asttokens

2.2.1

attrs

21.4.0

backcall

0.2.0

beautifulsoup4

4.11.1

black

22.6.0

bleach

4.1.0

blinker

1.4

boto3

1.24.28

botocore

1.27.28

certifi

2022.9.14

cffi

1.15.1

chardet

4.0.0

charset-normalizer

2.0.4

click

8.0.4

cryptography

37.0.1

cycler

0.11.0

Cython

0.29.32

databricks-sdk

0.1.6

dbus-python

1.2.18

debugpy

1.6.0

decorator

5.1.1

defusedxml

0.7.1

distlib

0.3.7

docstring-to-markdown

0.12

entrypoints

0.4

executing

1.2.0

facets-overview

1.0.3

fastjsonschema

2.18.0

filelock

3.12.2

fonttools

4.25.0

googleapis-common-protos

1.56.4

grpcio

1.48.1

grpcio-status

1.48.1

httplib2

0.20.2

idna

3.3

importlib-metadata

4.6.4

ipykernel

6.17.1

ipython

8.10.0

ipython-genutils

0.2.0

ipywidgets

7.7.2

jedi

0.18.1

jeepney

0.7.1

Jinja2

2.11.3

jmespath

0.10.0

joblib

1.2.0

jsonschema

4.16.0

jupyter-client

7.3.4

jupyter_core

4.11.2

jupyterlab-pygments

0.1.2

jupyterlab-widgets

1.0.0

keyring

23.5.0

kiwisolver

1.4.2

launchpadlib

1.10.16

lazr.restfulclient

0.14.4

lazr.uri

1.0.6

MarkupSafe

2.0.1

matplotlib

3.5.2

matplotlib-inline

0.1.6

mccabe

0.7.0

mistune

0.8.4

more-itertools

8.10.0

mypy-extensions

0.4.3

nbclient

0.5.13

nbconvert

6.4.4

nbformat

5.5.0

nest-asyncio

1.5.5

nodeenv

1.8.0

notebook

6.4.12

numpy

1.21.5

oauthlib

3.2.0

packaging

21.3

pandas

1.4.4

pandocfilters

1.5.0

parso

0.8.3

pathspec

0.9.0

patsy

0.5.2

pexpect

4.8.0

pickleshare

0.7.5

Pillow

9.2.0

pip

22.2.2

platformdirs

2.5.2

plotly

5.9.0

pluggy

1.0.0

prometheus-client

0.14.1

prompt-toolkit

3.0.36

protobuf

3.19.4

psutil

5.9.0

psycopg2

2.9.3

ptyprocess

0.7.0

pure-eval

0.2.2

pyarrow

8.0.0

pycparser

2.21

pydantic

1.10.6

pyflakes

3.0.1

Pygments

2.11.2

PyGObject

3.42.1

PyJWT

2.3.0

pyodbc

4.0.32

pyparsing

3.0.9

pyright

1.1.294

pyrsistent

0.18.0

python-dateutil

2.8.2

python-lsp-jsonrpc

1.0.0

python-lsp-server

1.7.1

pytoolconfig

1.2.2

pytz

2022.1

pyzmq

23.2.0

requests

2.28.1

rope

1.7.0

s3transfer

0.6.0

scikit-learn

1.1.1

scipy

1.9.1

seaborn

0.11.2

SecretStorage

3.3.1

Send2Trash

1.8.0

setuptools

63.4.1

six

1.16.0

soupsieve

2.3.1

ssh-import-id

5.11

stack-data

0.6.2

statsmodels

0.13.2

tenacity

8.1.0

terminado

0.13.1

testpath

0.6.0

threadpoolctl

2.2.0

tokenize-rt

4.2.1

tomli

2.0.1

tornado

6.1

traitlets

5.1.1

typing_extensions

4.3.0

ujson

5.4.0

unattended-upgrades

0.1

urllib3

1.26.11

virtualenv

20.16.3

wadllib

1.3.6

wcwidth

0.2.5

webencodings

0.5.1

whatthepatch

1.0.2

wheel

0.37.1

widgetsnbextension

3.6.1

yapf

0.31.0

zipp

1.0.0

Installed R libraries

R libraries are installed from the Microsoft CRAN snapshot on 2023-02-10.

Library

Version

Library

Version

Library

Version

arrow

10.0.1

askpass

1.1

assertthat

0.2.1

backports

1.4.1

base

4.2.2

base64enc

0.1-3

bit

4.0.5

bit64

4.0.5

blob

1.2.3

boot

1.3-28

brew

1.0-8

brio

1.1.3

broom

1.0.3

bslib

0.4.2

cachem

1.0.6

callr

3.7.3

caret

6.0-93

cellranger

1.1.0

chron

2.3-59

class

7.3-21

cli

3.6.0

clipr

0.8.0

clock

0.6.1

cluster

2.1.4

codetools

0.2-19

colorspace

2.1-0

commonmark

1.8.1

compiler

4.2.2

config

0.3.1

cpp11

0.4.3

crayon

1.5.2

credentials

1.3.2

curl

5.0.0

data.table

1.14.6

datasets

4.2.2

DBI

1.1.3

dbplyr

2.3.0

desc

1.4.2

devtools

2.4.5

diffobj

0.3.5

digest

0.6.31

downlit

0.4.2

dplyr

1.1.0

dtplyr

1.2.2

e1071

1.7-13

ellipsis

0.3.2

evaluate

0.20

fansi

1.0.4

farver

2.1.1

fastmap

1.1.0

fontawesome

0.5.0

forcats

1.0.0

foreach

1.5.2

foreign

0.8-82

forge

0.2.0

fs

1.6.1

future

1.31.0

future.apply

1.10.0

gargle

1.3.0

generics

0.1.3

gert

1.9.2

ggplot2

3.4.0

gh

1.3.1

gitcreds

0.1.2

glmnet

4.1-6

globals

0.16.2

glue

1.6.2

googledrive

2.0.0

googlesheets4

1.0.1

gower

1.0.1

graphics

4.2.2

grDevices

4.2.2

grid

4.2.2

gridExtra

2.3

gsubfn

0.7

gtable

0.3.1

hardhat

1.2.0

haven

2.5.1

highr

0.10

hms

1.1.2

htmltools

0.5.4

htmlwidgets

1.6.1

httpuv

1.6.8

httr

1.4.4

ids

1.0.1

ini

0.3.1

ipred

0.9-13

isoband

0.2.7

iterators

1.0.14

jquerylib

0.1.4

jsonlite

1.8.4

KernSmooth

2.23-20

knitr

1.42

labeling

0.4.2

later

1.3.0

lattice

0.20-45

lava

1.7.1

lifecycle

1.0.3

listenv

0.9.0

lubridate

1.9.1

magrittr

2.0.3

markdown

1.5

MASS

7.3-58.2

Matrix

1.5-1

memoise

2.0.1

methods

4.2.2

mgcv

1.8-41

mime

0.12

miniUI

0.1.1.1

ModelMetrics

1.2.2.2

modelr

0.1.10

munsell

0.5.0

nlme

3.1-162

nnet

7.3-18

numDeriv

2016.8-1.1

openssl

2.0.5

parallel

4.2.2

parallelly

1.34.0

pillar

1.8.1

pkgbuild

1.4.0

pkgconfig

2.0.3

pkgdown

2.0.7

pkgload

1.3.2

plogr

0.2.0

plyr

1.8.8

praise

1.0.0

prettyunits

1.1.1

pROC

1.18.0

processx

3.8.0

prodlim

2019.11.13

profvis

0.3.7

progress

1.2.2

progressr

0.13.0

promises

1.2.0.1

proto

1.0.0

proxy

0.4-27

ps

1.7.2

purrr

1.0.1

r2d3

0.2.6

R6

2.5.1

ragg

1.2.5

randomForest

4.7-1.1

rappdirs

0.3.3

rcmdcheck

1.4.0

RColorBrewer

1.1-3

Rcpp

1.0.10

RcppEigen

0.3.3.9.3

readr

2.1.3

readxl

1.4.2

recipes

1.0.4

rematch

1.0.1

rematch2

2.1.2

remotes

2.4.2

reprex

2.0.2

reshape2

1.4.4

rlang

1.0.6

rmarkdown

2.20

RODBC

1.3-20

roxygen2

7.2.3

rpart

4.1.19

rprojroot

2.0.3

Rserve

1.8-12

RSQLite

2.2.20

rstudioapi

0.14

rversions

2.1.2

rvest

1.0.3

sass

0.4.5

scales

1.2.1

selectr

0.4-2

sessioninfo

1.2.2

shape

1.4.6

shiny

1.7.4

sourcetools

0.1.7-1

sparklyr

1.7.9

SparkR

3.4.1

spatial

7.3-15

splines

4.2.2

sqldf

0.4-11

SQUAREM

2021.1

stats

4.2.2

stats4

4.2.2

stringi

1.7.12

stringr

1.5.0

survival

3.5-3

sys

3.4.1

systemfonts

1.0.4

tcltk

4.2.2

testthat

3.1.6

textshaping

0.3.6

tibble

3.1.8

tidyr

1.3.0

tidyselect

1.2.0

tidyverse

1.3.2

timechange

0.2.0

timeDate

4022.108

tinytex

0.44

tools

4.2.2

tzdb

0.3.0

urlchecker

1.0.1

usethis

2.1.6

utf8

1.2.3

utils

4.2.2

uuid

1.1-0

vctrs

0.5.2

viridisLite

0.4.1

vroom

1.6.1

waldo

0.4.0

whisker

0.4.1

withr

2.5.0

xfun

0.37

xml2

1.3.3

xopen

1.0.0

xtable

1.8-4

yaml

2.3.7

zip

2.2.2

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID

Artifact ID

Version

antlr

antlr

2.7.7

com.amazonaws

amazon-kinesis-client

1.12.0

com.amazonaws

aws-java-sdk-autoscaling

1.12.390

com.amazonaws

aws-java-sdk-cloudformation

1.12.390

com.amazonaws

aws-java-sdk-cloudfront

1.12.390

com.amazonaws

aws-java-sdk-cloudhsm

1.12.390

com.amazonaws

aws-java-sdk-cloudsearch

1.12.390

com.amazonaws

aws-java-sdk-cloudtrail

1.12.390

com.amazonaws

aws-java-sdk-cloudwatch

1.12.390

com.amazonaws

aws-java-sdk-cloudwatchmetrics

1.12.390

com.amazonaws

aws-java-sdk-codedeploy

1.12.390

com.amazonaws

aws-java-sdk-cognitoidentity

1.12.390

com.amazonaws

aws-java-sdk-cognitosync

1.12.390

com.amazonaws

aws-java-sdk-config

1.12.390

com.amazonaws

aws-java-sdk-core

1.12.390

com.amazonaws

aws-java-sdk-datapipeline

1.12.390

com.amazonaws

aws-java-sdk-directconnect

1.12.390

com.amazonaws

aws-java-sdk-directory

1.12.390

com.amazonaws

aws-java-sdk-dynamodb

1.12.390

com.amazonaws

aws-java-sdk-ec2

1.12.390

com.amazonaws

aws-java-sdk-ecs

1.12.390

com.amazonaws

aws-java-sdk-efs

1.12.390

com.amazonaws

aws-java-sdk-elasticache

1.12.390

com.amazonaws

aws-java-sdk-elasticbeanstalk

1.12.390

com.amazonaws

aws-java-sdk-elasticloadbalancing

1.12.390

com.amazonaws

aws-java-sdk-elastictranscoder

1.12.390

com.amazonaws

aws-java-sdk-emr

1.12.390

com.amazonaws

aws-java-sdk-glacier

1.12.390

com.amazonaws

aws-java-sdk-glue

1.12.390

com.amazonaws

aws-java-sdk-iam

1.12.390

com.amazonaws

aws-java-sdk-importexport

1.12.390

com.amazonaws

aws-java-sdk-kinesis

1.12.390

com.amazonaws

aws-java-sdk-kms

1.12.390

com.amazonaws

aws-java-sdk-lambda

1.12.390

com.amazonaws

aws-java-sdk-logs

1.12.390

com.amazonaws

aws-java-sdk-machinelearning

1.12.390

com.amazonaws

aws-java-sdk-opsworks

1.12.390

com.amazonaws

aws-java-sdk-rds

1.12.390

com.amazonaws

aws-java-sdk-redshift

1.12.390

com.amazonaws

aws-java-sdk-route53

1.12.390

com.amazonaws

aws-java-sdk-s3

1.12.390

com.amazonaws

aws-java-sdk-ses

1.12.390

com.amazonaws

aws-java-sdk-simpledb

1.12.390

com.amazonaws

aws-java-sdk-simpleworkflow

1.12.390

com.amazonaws

aws-java-sdk-sns

1.12.390

com.amazonaws

aws-java-sdk-sqs

1.12.390

com.amazonaws

aws-java-sdk-ssm

1.12.390

com.amazonaws

aws-java-sdk-storagegateway

1.12.390

com.amazonaws

aws-java-sdk-sts

1.12.390

com.amazonaws

aws-java-sdk-support

1.12.390

com.amazonaws

aws-java-sdk-swf-libraries

1.11.22

com.amazonaws

aws-java-sdk-workspaces

1.12.390

com.amazonaws

jmespath-java

1.12.390

com.clearspring.analytics

stream

2.9.6

com.databricks

Rserve

1.8-3

com.databricks

databricks-sdk-java

0.2.0

com.databricks

jets3t

0.7.1-0

com.databricks.scalapb

compilerplugin_2.12

0.4.15-10

com.databricks.scalapb

scalapb-runtime_2.12

0.4.15-10

com.esotericsoftware

kryo-shaded

4.0.2

com.esotericsoftware

minlog

1.3.0

com.fasterxml

classmate

1.3.4

com.fasterxml.jackson.core

jackson-annotations

2.14.2

com.fasterxml.jackson.core

jackson-core

2.14.2

com.fasterxml.jackson.core

jackson-databind

2.14.2

com.fasterxml.jackson.dataformat

jackson-dataformat-cbor

2.14.2

com.fasterxml.jackson.datatype

jackson-datatype-joda

2.14.2

com.fasterxml.jackson.datatype

jackson-datatype-jsr310

2.13.4

com.fasterxml.jackson.module

jackson-module-paranamer

2.14.2

com.fasterxml.jackson.module

jackson-module-scala_2.12

2.14.2

com.github.ben-manes.caffeine

caffeine

2.9.3

com.github.fommil

jniloader

1.1

com.github.fommil.netlib

native_ref-java

1.1

com.github.fommil.netlib

native_ref-java

1.1-natives

com.github.fommil.netlib

native_system-java

1.1

com.github.fommil.netlib

native_system-java

1.1-natives

com.github.fommil.netlib

netlib-native_ref-linux-x86_64

1.1-natives

com.github.fommil.netlib

netlib-native_system-linux-x86_64

1.1-natives

com.github.luben

zstd-jni

1.5.2-5

com.github.wendykierp

JTransforms

3.1

com.google.code.findbugs

jsr305

3.0.0

com.google.code.gson

gson

2.8.9

com.google.crypto.tink

tink

1.7.0

com.google.errorprone

error_prone_annotations

2.10.0

com.google.flatbuffers

flatbuffers-java

1.12.0

com.google.guava

guava

15.0

com.google.protobuf

protobuf-java

2.6.1

com.h2database

h2

2.1.214

com.helger

profiler

1.1.1

com.jcraft

jsch

0.1.55

com.jolbox

bonecp

0.8.0.RELEASE

com.lihaoyi

sourcecode_2.12

0.1.9

com.microsoft.azure

azure-data-lake-store-sdk

2.3.9

com.microsoft.sqlserver

mssql-jdbc

11.2.2.jre8

com.ning

compress-lzf

1.1.2

com.sun.mail

javax.mail

1.5.2

com.sun.xml.bind

jaxb-core

2.2.11

com.sun.xml.bind

jaxb-impl

2.2.11

com.tdunning

json

1.8

com.thoughtworks.paranamer

paranamer

2.8

com.trueaccord.lenses

lenses_2.12

0.4.12

com.twitter

chill-java

0.10.0

com.twitter

chill_2.12

0.10.0

com.twitter

util-app_2.12

7.1.0

com.twitter

util-core_2.12

7.1.0

com.twitter

util-function_2.12

7.1.0

com.twitter

util-jvm_2.12

7.1.0

com.twitter

util-lint_2.12

7.1.0

com.twitter

util-registry_2.12

7.1.0

com.twitter

util-stats_2.12

7.1.0

com.typesafe

config

1.2.1

com.typesafe.scala-logging

scala-logging_2.12

3.7.2

com.uber

h3

3.7.0

com.univocity

univocity-parsers

2.9.1

com.zaxxer

HikariCP

4.0.3

commons-cli

commons-cli

1.5.0

commons-codec

commons-codec

1.15

commons-collections

commons-collections

3.2.2

commons-dbcp

commons-dbcp

1.4

commons-fileupload

commons-fileupload

1.5

commons-httpclient

commons-httpclient

3.1

commons-io

commons-io

2.11.0

commons-lang

commons-lang

2.6

commons-logging

commons-logging

1.1.3

commons-pool

commons-pool

1.5.4

dev.ludovic.netlib

arpack

3.0.3

dev.ludovic.netlib

blas

3.0.3

dev.ludovic.netlib

lapack

3.0.3

info.ganglia.gmetric4j

gmetric4j

1.0.10

io.airlift

aircompressor

0.21

io.delta

delta-sharing-spark_2.12

0.7.1

io.dropwizard.metrics

metrics-core

4.2.10

io.dropwizard.metrics

metrics-graphite

4.2.10

io.dropwizard.metrics

metrics-healthchecks

4.2.10

io.dropwizard.metrics

metrics-jetty9

4.2.10

io.dropwizard.metrics

metrics-jmx

4.2.10

io.dropwizard.metrics

metrics-json

4.2.10

io.dropwizard.metrics

metrics-jvm

4.2.10

io.dropwizard.metrics

metrics-servlets

4.2.10

io.netty

netty-all

4.1.87.Final

io.netty

netty-buffer

4.1.87.Final

io.netty

netty-codec

4.1.87.Final

io.netty

netty-codec-http

4.1.87.Final

io.netty

netty-codec-http2

4.1.87.Final

io.netty

netty-codec-socks

4.1.87.Final

io.netty

netty-common

4.1.87.Final

io.netty

netty-handler

4.1.87.Final

io.netty

netty-handler-proxy

4.1.87.Final

io.netty

netty-resolver

4.1.87.Final

io.netty

netty-transport

4.1.87.Final

io.netty

netty-transport-classes-epoll

4.1.87.Final

io.netty

netty-transport-classes-kqueue

4.1.87.Final

io.netty

netty-transport-native-epoll

4.1.87.Final

io.netty

netty-transport-native-epoll

4.1.87.Final-linux-aarch_64

io.netty

netty-transport-native-epoll

4.1.87.Final-linux-x86_64

io.netty

netty-transport-native-kqueue

4.1.87.Final-osx-aarch_64

io.netty

netty-transport-native-kqueue

4.1.87.Final-osx-x86_64

io.netty

netty-transport-native-unix-common

4.1.87.Final

io.prometheus

simpleclient

0.7.0

io.prometheus

simpleclient_common

0.7.0

io.prometheus

simpleclient_dropwizard

0.7.0

io.prometheus

simpleclient_pushgateway

0.7.0

io.prometheus

simpleclient_servlet

0.7.0

io.prometheus.jmx

collector

0.12.0

jakarta.annotation

jakarta.annotation-api

1.3.5

jakarta.servlet

jakarta.servlet-api

4.0.3

jakarta.validation

jakarta.validation-api

2.0.2

jakarta.ws.rs

jakarta.ws.rs-api

2.1.6

javax.activation

activation

1.1.1

javax.el

javax.el-api

2.2.4

javax.jdo

jdo-api

3.0.1

javax.transaction

jta

1.1

javax.transaction

transaction-api

1.1

javax.xml.bind

jaxb-api

2.2.11

javolution

javolution

5.5.1

jline

jline

2.14.6

joda-time

joda-time

2.12.1

ml.combust.mleap

mleap-databricks-runtime_2.12

v0.20.0-db2

net.java.dev.jna

jna

5.8.0

net.razorvine

pickle

1.3

net.sf.jpam

jpam

1.1

net.sf.opencsv

opencsv

2.3

net.sf.supercsv

super-csv

2.2.0

net.snowflake

snowflake-ingest-sdk

0.9.6

net.snowflake

snowflake-jdbc

3.13.33

net.sourceforge.f2j

arpack_combined_all

0.1

org.acplt.remotetea

remotetea-oncrpc

1.1.2

org.antlr

ST4

4.0.4

org.antlr

antlr-runtime

3.5.2

org.antlr

antlr4-runtime

4.9.3

org.antlr

stringtemplate

3.2.1

org.apache.ant

ant

1.9.16

org.apache.ant

ant-jsch

1.9.16

org.apache.ant

ant-launcher

1.9.16

org.apache.arrow

arrow-format

11.0.0

org.apache.arrow

arrow-memory-core

11.0.0

org.apache.arrow

arrow-memory-netty

11.0.0

org.apache.arrow

arrow-vector

11.0.0

org.apache.avro

avro

1.11.1

org.apache.avro

avro-ipc

1.11.1

org.apache.avro

avro-mapred

1.11.1

org.apache.commons

commons-collections4

4.4

org.apache.commons

commons-compress

1.21

org.apache.commons

commons-crypto

1.1.0

org.apache.commons

commons-lang3

3.12.0

org.apache.commons

commons-math3

3.6.1

org.apache.commons

commons-text

1.10.0

org.apache.curator

curator-client

2.13.0

org.apache.curator

curator-framework

2.13.0

org.apache.curator

curator-recipes

2.13.0

org.apache.datasketches

datasketches-java

3.1.0

org.apache.datasketches

datasketches-memory

2.0.0

org.apache.derby

derby

10.14.2.0

org.apache.hadoop

hadoop-client-runtime

3.3.4

org.apache.hive

hive-beeline

2.3.9

org.apache.hive

hive-cli

2.3.9

org.apache.hive

hive-jdbc

2.3.9

org.apache.hive

hive-llap-client

2.3.9

org.apache.hive

hive-llap-common

2.3.9

org.apache.hive

hive-serde

2.3.9

org.apache.hive

hive-shims

2.3.9

org.apache.hive

hive-storage-api

2.8.1

org.apache.hive.shims

hive-shims-0.23

2.3.9

org.apache.hive.shims

hive-shims-common

2.3.9

org.apache.hive.shims

hive-shims-scheduler

2.3.9

org.apache.httpcomponents

httpclient

4.5.14

org.apache.httpcomponents

httpcore

4.4.16

org.apache.ivy

ivy

2.5.1

org.apache.logging.log4j

log4j-1.2-api

2.19.0

org.apache.logging.log4j

log4j-api

2.19.0

org.apache.logging.log4j

log4j-core

2.19.0

org.apache.logging.log4j

log4j-slf4j2-impl

2.19.0

org.apache.mesos

mesos

1.11.0-shaded-protobuf

org.apache.orc

orc-core

1.8.4-shaded-protobuf

org.apache.orc

orc-mapreduce

1.8.4-shaded-protobuf

org.apache.orc

orc-shims

1.8.4

org.apache.thrift

libfb303

0.9.3

org.apache.thrift

libthrift

0.12.0

org.apache.xbean

xbean-asm9-shaded

4.22

org.apache.yetus

audience-annotations

0.13.0

org.apache.zookeeper

zookeeper

3.6.3

org.apache.zookeeper

zookeeper-jute

3.6.3

org.checkerframework

checker-qual

3.19.0

org.codehaus.jackson

jackson-core-asl

1.9.13

org.codehaus.jackson

jackson-mapper-asl

1.9.13

org.codehaus.janino

commons-compiler

3.0.16

org.codehaus.janino

janino

3.0.16

org.datanucleus

datanucleus-api-jdo

4.2.4

org.datanucleus

datanucleus-core

4.1.17

org.datanucleus

datanucleus-rdbms

4.1.19

org.datanucleus

javax.jdo

3.2.0-m3

org.eclipse.jetty

jetty-client

9.4.51.v20230217

org.eclipse.jetty

jetty-continuation

9.4.51.v20230217

org.eclipse.jetty

jetty-http

9.4.51.v20230217

org.eclipse.jetty

jetty-io

9.4.51.v20230217

org.eclipse.jetty

jetty-jndi

9.4.51.v20230217

org.eclipse.jetty

jetty-plus

9.4.51.v20230217

org.eclipse.jetty

jetty-proxy

9.4.51.v20230217

org.eclipse.jetty

jetty-security

9.4.51.v20230217

org.eclipse.jetty

jetty-server

9.4.51.v20230217

org.eclipse.jetty

jetty-servlet

9.4.51.v20230217

org.eclipse.jetty

jetty-servlets

9.4.51.v20230217

org.eclipse.jetty

jetty-util

9.4.51.v20230217

org.eclipse.jetty

jetty-util-ajax

9.4.51.v20230217

org.eclipse.jetty

jetty-webapp

9.4.51.v20230217

org.eclipse.jetty

jetty-xml

9.4.51.v20230217

org.eclipse.jetty.websocket

websocket-api

9.4.51.v20230217

org.eclipse.jetty.websocket

websocket-client

9.4.51.v20230217

org.eclipse.jetty.websocket

websocket-common

9.4.51.v20230217

org.eclipse.jetty.websocket

websocket-server

9.4.51.v20230217

org.eclipse.jetty.websocket

websocket-servlet

9.4.51.v20230217

org.fusesource.leveldbjni

leveldbjni-all

1.8

org.glassfish.hk2

hk2-api

2.6.1

org.glassfish.hk2

hk2-locator

2.6.1

org.glassfish.hk2

hk2-utils

2.6.1

org.glassfish.hk2

osgi-resource-locator

1.0.3

org.glassfish.hk2.external

aopalliance-repackaged

2.6.1

org.glassfish.hk2.external

jakarta.inject

2.6.1

org.glassfish.jersey.containers

jersey-container-servlet

2.36

org.glassfish.jersey.containers

jersey-container-servlet-core

2.36

org.glassfish.jersey.core

jersey-client

2.36

org.glassfish.jersey.core

jersey-common

2.36

org.glassfish.jersey.core

jersey-server

2.36

org.glassfish.jersey.inject

jersey-hk2

2.36

org.hibernate.validator

hibernate-validator

6.1.7.Final

org.ini4j

ini4j

0.5.4

org.javassist

javassist

3.25.0-GA

org.jboss.logging

jboss-logging

3.3.2.Final

org.jdbi

jdbi

2.63.1

org.jetbrains

annotations

17.0.0

org.joda

joda-convert

1.7

org.jodd

jodd-core

3.5.2

org.json4s

json4s-ast_2.12

3.7.0-M11

org.json4s

json4s-core_2.12

3.7.0-M11

org.json4s

json4s-jackson_2.12

3.7.0-M11

org.json4s

json4s-scalap_2.12

3.7.0-M11

org.lz4

lz4-java

1.8.0

org.mariadb.jdbc

mariadb-java-client

2.7.4

org.mlflow

mlflow-spark

2.2.0

org.objenesis

objenesis

2.5.1

org.postgresql

postgresql

42.3.8

org.roaringbitmap

RoaringBitmap

0.9.39

org.roaringbitmap

shims

0.9.39

org.rocksdb

rocksdbjni

7.8.3

org.rosuda.REngine

REngine

2.1.0

org.scala-lang

scala-compiler_2.12

2.12.15

org.scala-lang

scala-library_2.12

2.12.15

org.scala-lang

scala-reflect_2.12

2.12.15

org.scala-lang.modules

scala-collection-compat_2.12

2.4.3

org.scala-lang.modules

scala-parser-combinators_2.12

1.1.2

org.scala-lang.modules

scala-xml_2.12

1.2.0

org.scala-sbt

test-interface

1.0

org.scalacheck

scalacheck_2.12

1.14.2

org.scalactic

scalactic_2.12

3.2.15

org.scalanlp

breeze-macros_2.12

2.1.0

org.scalanlp

breeze_2.12

2.1.0

org.scalatest

scalatest-compatible

3.2.15

org.scalatest

scalatest-core_2.12

3.2.15

org.scalatest

scalatest-diagrams_2.12

3.2.15

org.scalatest

scalatest-featurespec_2.12

3.2.15

org.scalatest

scalatest-flatspec_2.12

3.2.15

org.scalatest

scalatest-freespec_2.12

3.2.15

org.scalatest

scalatest-funspec_2.12

3.2.15

org.scalatest

scalatest-funsuite_2.12

3.2.15

org.scalatest

scalatest-matchers-core_2.12

3.2.15

org.scalatest

scalatest-mustmatchers_2.12

3.2.15

org.scalatest

scalatest-propspec_2.12

3.2.15

org.scalatest

scalatest-refspec_2.12

3.2.15

org.scalatest

scalatest-shouldmatchers_2.12

3.2.15

org.scalatest

scalatest-wordspec_2.12

3.2.15

org.scalatest

scalatest_2.12

3.2.15

org.slf4j

jcl-over-slf4j

2.0.6

org.slf4j

jul-to-slf4j

2.0.6

org.slf4j

slf4j-api

2.0.6

org.threeten

threeten-extra

1.7.1

org.tukaani

xz

1.9

org.typelevel

algebra_2.12

2.0.1

org.typelevel

cats-kernel_2.12

2.1.1

org.typelevel

spire-macros_2.12

0.17.0

org.typelevel

spire-platform_2.12

0.17.0

org.typelevel

spire-util_2.12

0.17.0

org.typelevel

spire_2.12

0.17.0

org.wildfly.openssl

wildfly-openssl

1.1.3.Final

org.xerial

sqlite-jdbc

3.42.0.0

org.xerial.snappy

snappy-java

1.1.10.1

org.yaml

snakeyaml

1.33

oro

oro

2.0.8

pl.edu.icm

JLargeArrays

1.5

software.amazon.cryptools

AmazonCorrettoCryptoProvider

1.6.1-linux-x86_64

software.amazon.ion

ion-java

1.0.2

stax

stax-api

1.0.1