Databricks Runtime 15.4 LTS

The following release notes provide information about Databricks Runtime 15.4 LTS, powered by Apache Spark 3.5.0.

Databricks released this version in August 2024.

Note

LTS means this version is under long-term support. See Databricks Runtime LTS version lifecycle.

Note

These release notes might include references to features that are not available on Google Cloud as of this release.

Tip

To see release notes for Databricks Runtime versions that have reached end-of-support (EoS), see End-of-support Databricks Runtime release notes. The EoS Databricks Runtime versions have been retired and might not be updated.

Behavioral changes

Using a VARIANT as the input or output type with a Python UDF, UDAF, or UDTF throws an exception

[Breaking change] In Databricks Runtime 15.3 and above, calling any Python user-defined function (UDF), user-defined aggregate function (UDAF), or user-defined table function (UDTF) that uses a VARIANT type as an argument or return value throws an exception. This change is made to prevent issues that might occur because of an invalid value returned by one of these functions. To learn more about the VARIANT type, see use VARIANTs to store semi-structured data.

Change to the default schema binding mode for views

Views now adapt to schema changes in the underlying query by using schema compensation with regular casting rules. This is a change from the previous default of BINDING mode, which raised errors when a safe cast could not be performed when referencing the view.

See CREATE VIEW and cast function.

Disallow using the undocumented ! syntax instead of NOT outside boolean expressions

With this release, the use of ! as a synonym for NOT outside of boolean expressions is no longer allowed. For example, statements such as the following: CREATE ... IF ! EXISTS, IS ! NULL, a ! NULL column or field property, ! IN and ! BETWEEN, must be replaced with: CREATE ... IF NOT EXISTS, IS NOT NULL, a NOT NULL column or field property, NOT IN and NOT BETWEEN.

This change ensures consistency, aligns with the SQL standard, and makes your SQL more portable.

The boolean prefix operator ! (for example, !is_mgr or !(true AND false)) is unaffected by this change.

Disallow undocumented column definition syntax in views

Databricks supports CREATE VIEW with named columns and column comments. Previously, the specification of column types, NOT NULL constraints, or DEFAULT has been allowed. With this release, you can no longer use this syntax.

This change ensures consistency, aligns with the SQL standard, and supports future enhancements.

Consistent error handling for Base64 decoding in Spark and Photon

This release changes how Photon handles Base64 decoding errors to match the Spark handling of these errors. Before these changes, the Photon and Spark code generation path sometimes failed to raise parsing exceptions, while the Spark interpreted execution correctly raised IllegalArgumentException or ConversionInvalidInputError. This update ensures that Photon consistently raises the same exceptions as Spark during Base64 decoding errors, providing more predictable and reliable error handling.

Adding a CHECK constraint on an invalid column now returns the UNRESOLVED_COLUMN.WITH_SUGGESTION error class

To provide more useful error messaging, in Databricks Runtime 15.3 and above, an ALTER TABLE ADD CONSTRAINT statement that includes a CHECK constraint referencing an invalid column name returns the UNRESOLVED_COLUMN.WITH_SUGGESTION error class. Previously, an INTERNAL_ERROR was returned.

New features and improvements

UTF-8 validation functions

This release introduces the following functions for validating UTF-8 strings:

• is_valid_utf8 verified whether a string is a valid UTF-8 string.

• make_valid_utf8 converts a potentially invalid UTF-8 string to a valid UTF-8 string using substitution characters

• validate_utf8 raises an error if the input is not a valid UTF-8 string.

• try_validate_utf8 returns NULL if the input is not a valid UTF-8 string.

to_avro and from_avro functions

The to_avro and from_avro functions allow conversion of SQL types to Avro binary data and back.

Typed Dataset APIs with Scala UDFs

This release includes added support for typed Dataset APIs with Scala user-defined functions (excluding user-defined aggregate functions) on Unity Catalog-enabled compute with shared access mode. See Typed Dataset APIs.

Enable UniForm Iceberg using ALTER TABLE

You can now enable UniForm Iceberg on existing tables without rewriting data files. See Enable by altering an existing table.

try_url_decode function

This release introduces the try_url_decode function, which decodes a URL-encoded string. If the string is not in the correct format, the function returns NULL instead of raising an error.

Optionally allow the optimizer to rely on unenforced foreign key constraints

To improve query performance, you can now specify the RELY keyword on FOREIGN KEY constraints when you CREATE or ALTER a table.

Parallelized job runs for selective overwrites

Selective overwrites using replaceWhere now run jobs that delete data and insert new data in parallel, improving query performance and cluster utilization.

Improved performance for change data feed with selective overwrites

Selective overwrites using replaceWhere on tables with change data feed no longer write separate change data files for inserted data. These operations use a hidden _change_type column present in the underlying Parquet data files to record changes without write amplification.

Improved query latency for the COPY INTO command

This release includes a change that improves the query latency for the COPY INTO command. This improvement is implemented by making the loading of state by the RocksDB state store asynchronous. With this change, you should see an improvement in start times for queries with large states, such as queries with a large number of already ingested files.

Support for dropping the check constraints table feature

You can now drop the checkConstraints table feature from a Delta table using ALTER TABLE table_name DROP FEATURE checkConstraints. See Disable check constraints.

Expanded support for Java and Scala libraries

Starting in Databricks Runtime 15.4 LTS, all Java and Scala libraries bundled with Databricks Runtime are available in all compute access modes when you use Unity Catalog. To learn more about language support on Unity Catalog-enabled compute, see Compute access mode limitations for Unity Catalog.

Expanded support for Scala Dataset operations

With this release, Unity Catalog-enabled compute resources that use shared access mode support the following Scala Dataset operations: map, mapPartitions, foreachPartition, flatMap, reduce and filter.

Scala is GA on Unity Catalog shared compute

With this release, Scala is generally available on shared access mode Unity Catalog-enabled compute, including support for scalar user-defined functions (UDFs). Structured Streaming, Hive UDFs, and Hive user-defined aggregate functions are not supported. For a complete list of limitations, see Compute access mode limitations for Unity Catalog.

Bug fixes

Library upgrades

• Upgraded Python libraries:

  • azure-core from 1.30.1 to 1.30.2

  • google-auth from 2.29.0 to 2.31.0

  • google-cloud-storage from 2.16.0 to 2.17.0

  • google-resumable-media from 2.7.0 to 2.7.1

  • googleapis-common-protos from 1.63.0 to 1.63.2

  • mlflow-skinny from 2.11.3 to 2.11.4

  • proto-plus from 1.23.0 to 1.24.0

  • s3transfer from 0.10.1 to 0.10.2

• Upgraded R libraries:

• Upgraded Java libraries:

  • com.databricks.databricks-sdk-java from 0.17.1 to 0.27.0

  • com.ibm.icu.icu4j from 72.1 to 75.1

  • software.amazon.cryptools.AmazonCorrettoCryptoProvider from 1.6.1-linux-x86_64 to 1.6.2-linux-x86_64

Apache Spark

Databricks Runtime 15.4 LTS includes Apache Spark 3.5.0. This release includes all Spark fixes and improvements included in Databricks Runtime 15.3, and the following additional bug fixes and improvements made to Spark:

• [SPARK-48503] [DBRRM-1150][SC-172196][SQL] Allow grouping on expressions in scalar subqueries, if they are bound to outer rows

• [SPARK-48834] [BEHAVE-79][SC-170972][SQL] Disable variant input/output to python scalar UDFs, UDTFs, UDAFs during query compilation

• [SPARK-48441] [SC-170980][SQL][WARMFIX] Fix StringTrim behaviour for non-UTF8_BINARY collations

• [SPARK-48440] [SC-170895][SQL][WARMFIX] Fix StringTranslate behaviour for non-UTF8_BINARY collations

• [SPARK-48872] [SC-170866][PYTHON] Reduce the overhead of capturecall_site

• [SPARK-48862] [SC-170845][PYTHON][CONNECT] Avoid calling _proto_to_string when INFO level is not enabled

• [SPARK-48852] [SC-170837][CONNECT] Fix string trim function in connect

• [SPARK-48791] [SC-170658][CORE] Fix perf regression caused by the accumulators registration overhead using CopyOnWriteArrayList

• [SPARK-48118] [SQL] Support SPARK_SQL_LEGACY_CREATE_HIVE_TABLE env variable

• [SPARK-48241] [SC-165811][SQL] CSV parsing failure with char/varchar type columns

• [SPARK-48168] [SC-166900][SQL] Add bitwise shifting operators support

• [SPARK-48148] [SC-165630][CORE] JSON objects should not be modified when read as STRING

• [SPARK-46625] [SC-170561] CTE with Identifier clause as reference

• [SPARK-48771] [SC-170546][SQL] Speed up LogicalPlanIntegrity.validateExprIdUniqueness for large query plans

• [SPARK-48831] [BEHAVE-76][SC-170554][CONNECT] Make default column name of cast compatible with Spark Classic

• [SPARK-48623] [SC-170544][CORE] Structured logging migrations [Part 2]

• [SPARK-48296] [SC-166138][SQL] Codegen Support for to_xml

• [SPARK-48027] [SC-165154][SQL] InjectRuntimeFilter for multi-level join should check child join type

• [SPARK-48686] [SC-170365][SQL] Improve performance of ParserUtils.unescapeSQLString

• [SPARK-48798] [SC-170588][PYTHON] Introduce spark.profile.render for SparkSession-based profiling

• [SPARK-48048] [SC-169099] Revert “[SC-164846][CONNECT][SS] Added client side listener support for Scala”

• [SPARK-47910] [SC-168929][CORE] close stream when DiskBlockObjectWriter closeResources to avoid memory leak

• [SPARK-48816] [SC-170547][SQL] Shorthand for interval converters in UnivocityParser

• [SPARK-48589] [SC-170132][SQL][SS] Add option snapshotStartBatchId and snapshotPartitionId to state data source

• [SPARK-48280] [SC-170293][SQL] Improve collation testing surface area using expression walking

• [SPARK-48837] [SC-170540][ML] In CountVectorizer, only read binary parameter once per transform, not once per row

• [SPARK-48803] [SC-170541][SQL] Throw internal error in Orc(De)serializer to align with ParquetWriteSupport

• [SPARK-48764] [SC-170129][PYTHON] Filtering out IPython-related frames from user stack

• [SPARK-48818] [SC-170414][PYTHON] Simplify percentile functions

• [SPARK-48479] [SC-169079][SQL] Support creating scalar and table SQL UDFs in parser

• [SPARK-48697] [SC-170122][LC-4703][SQL] Add collation aware string filters

• [SPARK-48800] [SC-170409][CONNECT][SS] Deflake ClientStreamingQuerySuite

• [SPARK-48738] [SC-169814][SQL] Correct since version for built-in func alias random, position, mod, cardinality, current_schema, user, session_user,char_length,character_length

• [SPARK-48638] [SC-169575][CONNECT] Add ExecutionInfo support for DataFrame

• [SPARK-48064] [SC-164697][SQL] Update error messages for routine related error classes

• [SPARK-48810] [CONNECT] Session stop() API should be idempotent and not fail if the session is already closed by the server

• [SPARK-48650] [15.x][PYTHON] Display correct call site from IPython Notebook

• [SPARK-48271] [SC-166076][SQL] Turn match error in RowEncoder into UNSUPPORTED_DATA_TYPE_FOR_ENCODER

• [SPARK-48709] [SC-169603][SQL] Fix varchar type resolution mismatch for DataSourceV2 CTAS

• [SPARK-48792] [SC-170335][SQL] Fix regression for INSERT with partial column list to a table with char/varchar

• [SPARK-48767] [SC-170330][SQL] Fix some error prompts when variant type data is invalid

• [SPARK-48719] [SC-170339][SQL] Fix the calculation bug of RegrSlope & RegrIntercept when the first parameter is null

• [SPARK-48815] [SC-170334][CONNECT] Update environment when stoping connect session

• [SPARK-48646] [SC-169020][PYTHON] Refine Python data source API docstring and type hints

• [SPARK-48806] [SC-170310][SQL] Pass actual exception when url_decode fails

• [SPARK-47777] [SC-168818] fix python streaming data source connect test

• [SPARK-48732] [SC-169793][SQL] Cleanup deprecated api usage related to JdbcDialect.compileAggregate

• [SPARK-48675] [SC-169538][SQL] Fix cache table with collated column

• [SPARK-48623] [SC-169034][CORE] Structured Logging Migrations

• [SPARK-48655] [SC-169542][SQL] SPJ: Add tests for shuffle skipping for aggregate queries

• [SPARK-48586] [SC-169808][SS] Remove lock acquisition in doMaintenance() by making a deep copy of file mappings in RocksDBFileManager in load()

• [SPARK-48799] [Backport][15.x][SC-170283][SS] Refactor versioning for operator metadata read/write and callers

• [SPARK-48808] [SC-170309][SQL] Fix NPE when connecting thriftserver through Hive 1.2.1 and the result schema is empty

• [SPARK-48715] [SC-170291][SQL] Integrate UTF8String validation into collation-aware string function implementations

• [SPARK-48747] [SC-170120][SQL] Add code point iterator to UTF8String

• [SPARK-48748] [SC-170115][SQL] Cache numChars in UTF8String

• [SPARK-48744] [SC-169817][Core] Log entry should be constructed only once

• [SPARK-46122] [SC-164313][SQL] Set spark.sql.legacy.createHiveTableByDefault to false by default

• [SPARK-48765] [SC-170119][DEPLOY] Enhance default value evaluation for SPARK_IDENT_STRING

• [SPARK-48759] [SC-170128][SQL] Add migration doc for CREATE TABLE AS SELECT behavior change behavior change since Spark 3.4

• [SPARK-48598] [SC-169484][PYTHON][CONNECT] Propagate cached schema in dataframe operations

• [SPARK-48766] [SC-170126][PYTHON] Document the behavior difference of extraction between element_at and try_element_at

• [SPARK-48768] [SC-170124][PYTHON][CONNECT] Should not cache explain

• [SPARK-48770] [Backport][15.x][SC-170133][SS] Change to read operator metadata once on driver to check if we can find info for numColsPrefixKey used for session window agg queries

• [SPARK-48656] [SC-169529][CORE] Do a length check and throw COLLECTION_SIZE_LIMIT_EXCEEDED error in CartesianRDD.getPartitions

• [SPARK-48597] [SC-168817][SQL] Introduce a marker for isStreaming property in text representation of logical plan

• [SPARK-48472] [SC-169044][SQL] Enable reflect expressions with collated strings

• [SPARK-48699] [SC-169597][SQL] Refine collation API

• [SPARK-48682] [SC-169812][SQL][BEHAVE-58] Use ICU in InitCap expression for UTF8_BINARY strings

• [SPARK-48282] [SC-169813][SQL] Alter string search logic for UTF8_BINARY_LCASE collation (StringReplace, FindInSet)

• [SPARK-47353] [SC-169599][SQL] Enable collation support for the Mode expression

• [SPARK-48320] [SPARK-48490] Sync the latest Logging trait and test cases from OSS Spark

• [SPARK-48629] [SC-169479] Migrate the residual code to structured logging framework

• [SPARK-48681] [SC-169469][SQL][BEHAVE-58] Use ICU in Lower/Upper expressions for UTF8_BINARY strings

• [SPARK-48573] [15.x][SC-169582][SQL] Upgrade ICU version

• [SPARK-48687] [Backport][15.x][SS] Add change to perform state schema validation and update on driver for stateful queries

• [SPARK-47579] [15.x][SC-167310][CORE][PART4] Migrate logInfo with variables to structured logging framework

• [SPARK-48008] [SC-167363][1/2] Support UDAFs in Spark Connect

• [SPARK-48578] [SC-169505][SQL] add UTF8 string validation related functions

• [SPARK-48670] [SC-169598][SQL] Providing suggestion as part of error message when invalid collation name is given

• [SPARK-48059] [SPARK-48145][SPARK-48134][SPARK-48182][SPARK-48209][SPA… …RK-48291] Structured log framework on the java side

• [SPARK-47599] [15.x][SC-166000][MLLIB] MLLib: Migrate logWarn with variables to structured logging framework

• [SPARK-48706] [SC-169589][PYTHON] Python UDF in higher order functions should not throw internal error

• [SPARK-48498] [BEHAVE-38][SC-168060][SQL] Always do char padding in predicates

• [SPARK-48662] [SC-169533][SQL] Fix StructsToXml expression with collations

• [SPARK-48482] [SC-167702][PYTHON][15.x] dropDuplicates and dropDuplicatesWIthinWatermark should accept variable length args

• [SPARK-48678] [SC-169463][CORE] Performance optimizations for SparkConf.get(ConfigEntry)

• [SPARK-48576] [SQL] Rename UTF8_BINARY_LCASE to UTF8_LCASE

• [SPARK-47927] [SC-164123][SQL]: Fix nullability attribute in UDF decoder

• [SPARK-47579] [SC-165297][CORE][PART1] Migrate logInfo with variables to structured logging framework (new)

• [SPARK-48695] [SC-169473][PYTHON] TimestampNTZType.fromInternal not use the deprecated methods

• [SPARK-48431] [SC-167290][LC-4066][SQL] Do not forward predicates on collated columns to file readers

• [SPARK-47579] Revert “[SC-165297][CORE][PART1] Migrate logInfo with variables to structured logging framework”

• [SPARK-47585] [SC-164306][SQL] SQL core: Migrate logInfo with variables to structured logging framework

• [SPARK-48466] [SC-169042][SQL] Create dedicated node for EmptyRelation in AQE

• [SPARK-47579] [SC-165297][CORE][PART1] Migrate logInfo with variables to structured logging framework

• [SPARK-48410] [SC-168320][SQL] Fix InitCap expression for UTF8_BINARY_LCASE & ICU collations

• [SPARK-48318] [SC-167709][SQL] Enable hash join support for all collations (complex types)

• [SPARK-48435] [SC-168128][SQL] UNICODE collation should not support binary equality

• [SPARK-48555] [SC-169041][SQL][PYTHON][CONNECT] Support using Columns as parameters for several functions in pyspark/scala

• [SPARK-48591] [SC-169081][PYTHON] Add a helper function to simplify Column.py

• [SPARK-48574] [SC-169043][SQL] Fix support for StructTypes with collations

• [SPARK-48305] [SC-166390][SQL] Add collation support for CurrentLike expressions

• [SPARK-48342] [SC-168941][SQL] Introduction of SQL Scripting Parser

• [SPARK-48649] [SC-169024][SQL] Add “ignoreInvalidPartitionPaths” and “spark.sql.files.ignoreInvalidPartitionPaths” configs to allow ignoring invalid partition paths

• [SPARK-48000] [SC-167194][SQL] Enable hash join support for all collations (StringType)

• [SPARK-48459] [SC-168947][CONNECT][PYTHON] Implement DataFrameQueryContext in Spark Connect

• [SPARK-48602] [SC-168692][SQL] Make csv generator support different output style with spark.sql.binaryOutputStyle

• [SPARK-48283] [SC-168129][SQL] Modify string comparison for UTF8_BINARY_LCASE

• [SPARK-48610] [SC-168830][SQL] refactor: use auxiliary idMap instead of OP_ID_TAG

• [SPARK-48634] [SC-169021][PYTHON][CONNECT] Avoid statically initialize threadpool at ExecutePlanResponseReattachableIterator

• [SPARK-47911] [SC-164658][SQL] Introduces a universal BinaryFormatter to make binary output consistent

• [SPARK-48642] [SC-168889][CORE] False SparkOutOfMemoryError caused by killing task on spilling

• [SPARK-48572] [SC-168844][SQL] Fix DateSub, DateAdd, WindowTime, TimeWindow and SessionWindow expressions

• [SPARK-48600] [SC-168841][SQL] Fix FrameLessOffsetWindowFunction expressions implicit casting

• [SPARK-48644] [SC-168933][SQL] Do a length check and throw COLLECTION_SIZE_LIMIT_EXCEEDED error in Hex.hex

• [SPARK-48587] [SC-168824][VARIANT] Avoid storage amplification when accessing a sub-Variant

• [SPARK-48647] [SC-168936][PYTHON][CONNECT] Refine the error message for YearMonthIntervalType in df.collect

• [SPARK-48307] [SC-167802][SQL] InlineCTE should keep not-inlined relations in the original WithCTE node

• [SPARK-48596] [SC-168581][SQL] Perf improvement for calculating hex string for long

• [SPARK-48621] [SC-168726][SQL] Fix Like simplification in Optimizer for collated strings

• [SPARK-47148] [SC-164179][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation

• [SPARK-48584] [SC-168579][SQL] Perf improvement for unescapePathName

• [SPARK-48281] [SC-167260][SQL] Alter string search logic for UTF8_BINARY_LCASE collation (StringInStr, SubstringIndex)

• [SPARK-48577] [SC-168826][SQL] Invalid UTF-8 byte sequence replacement

• [SPARK-48595] [SC-168580][CORE] Cleanup deprecated api usage related to commons-compress

• [SPARK-48030] [SC-164303][SQL] SPJ: cache rowOrdering and structType for InternalRowComparableWrapper

• [SPARK-48004] [SC-164005][SQL] Add WriteFilesExecBase trait for v1 write

• [SPARK-48551] [SC-168438][SQL] Perf improvement for escapePathName

• [SPARK-48565] [SC-168437][UI] Fix thread dump display in UI

• [SPARK-48364] [SC-166782][SQL] Add AbstractMapType type casting and fix RaiseError parameter map to work with collated strings

• [SPARK-48421] [SC-168689][SQL] SPJ: Add documentation

• [SPARK-48604] [SC-168698][SQL] Replace deprecated new ArrowType.Decimal(precision, scale) method call

• [SPARK-46947] [SC-157561][CORE] Delay memory manager initialization until Driver plugin is loaded

• [SPARK-48411] [SC-168576][SS][PYTHON] Add E2E test for DropDuplicateWithinWatermark

• [SPARK-48543] [SC-168697][SS] Track state row validation failures using explicit error class

• [SPARK-48221] [SC-167143][SQL] Alter string search logic for UTF8_BINARY_LCASE collation (Contains, StartsWith, EndsWith, StringLocate)

• [SPARK-47415] [SC-168441][SQL] Add collation support for Levenshtein expression

• [SPARK-48593] [SC-168719][PYTHON][CONNECT] Fix the string representation of lambda function

• [SPARK-48622] [SC-168710][SQL] get SQLConf once when resolving column names

• [SPARK-48594] [SC-168685][PYTHON][CONNECT] Rename parent field to child in ColumnAlias

• [SPARK-48403] [SC-168319][SQL] Fix Lower & Upper expressions for UTF8_BINARY_LCASE & ICU collations

• [SPARK-48162] [SC-166062][SQL] Add collation support for MISC expressions

• [SPARK-48518] [SC-167718][CORE] Make LZF compression be able to run in parallel

• [SPARK-48474] [SC-167447][CORE] Fix the class name of the log in SparkSubmitArguments & SparkSubmit

• [SPARK-48012] [SC-168267][SQL] SPJ: Support Transfrom Expressions for One Side Shuffle

• [SPARK-48552] [SC-168212][SQL] multi-line CSV schema inference should also throw FAILED_READ_FILE

• [SPARK-48560] [SC-168268][SS][PYTHON] Make StreamingQueryListener.spark settable

• [SPARK-48569] [SC-168321][SS][CONNECT] Handle edge cases in query.name

• [SPARK-47260] [SC-167323][SQL] Assign name to error class LEGACYERROR_TEMP_3250

• [SPARK-48564] [SC-168327][PYTHON][CONNECT] Propagate cached schema in set operations

• [SPARK-48155] [SC-165910][SQL] AQEPropagateEmptyRelation for join should check if remain child is just BroadcastQueryStageExec

• [SPARK-48506] [SC-167720][CORE] Compression codec short names are case insensitive except for event logging

• [SPARK-48447] [SC-167607][SS] Check state store provider class before invoking the constructor

• [SPARK-47977] [SC-167650] DateTimeUtils.timestampDiff and DateTimeUtils.timestampAdd should not throw INTERNAL_ERROR exception

• [SPARK-48513] [Backport][15.x][SC-168085][SS] Add error class for state schema compatibility and minor refactoring

• [SPARK-48413] [SC-167669][SQL] ALTER COLUMN with collation

• [SPARK-48561] [SC-168250][PS][CONNECT] Throw PandasNotImplementedError for unsupported plotting functions

• [SPARK-48465] [SC-167531][SQL] Avoid no-op empty relation propagation

• [SPARK-48553] [SC-168166][PYTHON][CONNECT] Cache more properties

• [SPARK-48540] [SC-168069][CORE] Avoid ivy output loading settings to stdout

• [SPARK-48535] [SC-168057][SS] Update config docs to indicate possibility of data loss/corruption issue if skip nulls for stream-stream joins config is enabled

• [SPARK-48536] [SC-168059][PYTHON][CONNECT] Cache user specified schema in applyInPandas and applyInArrow

• [SPARK-47873] [SC-163473][SQL] Write collated strings to Hive metastore using the regular string type

• [SPARK-48461] [SC-167442][SQL] Replace NullPointerExceptions with error class in AssertNotNull expression

• [SPARK-47833] [SC-163191][SQL][CORE] Supply caller stackstrace for checkAndGlobPathIfNecessary AnalysisException

• [SPARK-47898] [SC-163146][SQL] Port HIVE-12270: Add DBTokenStore support to HS2 delegation token

• [SPARK-47578] [SC-167497][R] Migrate RPackageUtils with variables to structured logging framework

• [SPARK-47875] [SC-162935][CORE] Remove spark.deploy.recoverySerializer

• [SPARK-47552] [SC-160880][CORE] Set spark.hadoop.fs.s3a.connection.establish.timeout to 30s if missing

• [SPARK-47972] [SC-167692][SQL] Restrict CAST expression for collations

• [SPARK-48430] [SC-167489][SQL] Fix map value extraction when map contains collated strings

• [SPARK-47318] [SC-162712][CORE][3.5] Adds HKDF round to AuthEngine key derivation to follow standard KEX practices

• [SPARK-48503] [BEHAVE-29][ES-1135236][SQL] Fix invalid scalar subqueries with group-by on non-equivalent columns that were incorrectly allowed

• [SPARK-48508] [SC-167695][CONNECT][PYTHON] Cache user specified schema in DataFrame.{to, mapInPandas, mapInArrow}

• [SPARK-23015] [SC-167188][WINDOWS] Fix bug in Windows where starting multiple Spark instances within the same second causes a failure

• [SPARK-45891] [SC-167608]Revert ” Describe shredding scheme for Variant”

• [SPARK-48391] [SC-167554][CORE]Using addAll instead of add function in fromAccumulatorInfos method of TaskMetrics Class

• [SPARK-48496] [SC-167600][CORE] Use static regex Pattern instances in JavaUtils.timeStringAs and JavaUtils.byteStringAs

• [SPARK-48476] [SC-167488][SQL] fix NPE error message for null delmiter csv

• [SPARK-48489] [SC-167598][SQL] Throw a better, user-facing error when reading illegal schema from text Data Source

• [SPARK-48471] [SC-167324][CORE] Improve documentation and usage guide for history server

• [SPARK-45891] [SC-167597] Describe shredding scheme for Variant

• [SPARK-47333] [SC-159043][SQL] Use checkInputDataTypes to check the parameter types of the function to_xml

• [SPARK-47387] [SC-159310][SQL] Remove some unused error classes

• [SPARK-48265] [ES-1131440][SQL] Infer window group limit batch should do constant folding

• [SPARK-47716] [SC-167444][SQL] Avoid view name conflict in SQLQueryTestSuite semantic sort test case

• [SPARK-48159] [SC-167141][SQL] Extending support for collated strings on datetime expressions

• [SPARK-48462] [SC-167271][SQL][Tests] Use withSQLConf in tests: Refactor HiveQuerySuite and HiveTableScanSuite

• [SPARK-48392] [SC-167195][CORE] Also load spark-defaults.conf when provided --properties-file

• [SPARK-48468] [SC-167417] Add LogicalQueryStage interface in catalyst

• [SPARK-47578] [SC-164988][CORE] Manual backport for Spark PR #46309: Migrate logWarning with variables to structured logging framework

• [SPARK-48415] [SC-167321]Revert “[PYTHON] Refactor TypeName to support parameterized datatypes”

• [SPARK-46544] [SC-151361][SQL] Support v2 DESCRIBE TABLE EXTENDED with table stats

• [SPARK-48325] [SC-166963][CORE] Always specify messages in ExecutorRunner.killProcess

• [SPARK-46841] [SC-167139][SQL] Add collation support for ICU locales and collation specifiers

• [SPARK-47221] [SC-157870][SQL] Uses signatures from CsvParser to AbstractParser

• [SPARK-47246] [SC-158138][SQL] Replace InternalRow.fromSeq with new GenericInternalRow to save a collection conversion

• [SPARK-47597] [SC-163932][STREAMING] Manual backport for Spark PR #46192: Streaming: Migrate logInfo with variables to structured logging framework

• [SPARK-48415] [SC-167130][PYTHON] Refactor TypeName to support parameterized datatypes

• [SPARK-48434] [SC-167132][PYTHON][CONNECT] Make printSchema use the cached schema

• [SPARK-48432] [ES-1097114][SQL] Avoid unboxing integers in UnivocityParser

• [SPARK-47463] [SC-162840][SQL] Use V2Predicate to wrap expression with return type of boolean

• [SPARK-47781] [SC-162293][SQL] Handle negative scale decimals for JDBC data sources

• [SPARK-48394] [SC-166966][CORE] Cleanup mapIdToMapIndex on mapoutput unregister

• [SPARK-47072] [SC-156933][SQL] Fix supported interval formats in error messages

• [SPARK-47001] [SC-162487][SQL] Pushdown verification in optimizer

• [SPARK-48335] [SC-166387][PYTHON][CONNECT] Make _parse_datatype_string compatible with Spark Connect

• [SPARK-48329] [SC-166518][SQL] Enable spark.sql.sources.v2.bucketing.pushPartValues.enabled by default

• [SPARK-48412] [SC-166898][PYTHON] Refactor data type json parse

• [SPARK-48215] [SC-166781][SQL] Extending support for collated strings on date_format expression

• [SPARK-45009] [SC-166873][SQL][FOLLOW UP] Add error class and tests for decorrelation of predicate subqueries in join condition which reference both join child

• [SPARK-47960] [SC-165295][SS][15.x] Allow chaining other stateful operators after transformWithState operator.

• [SPARK-48340] [SC-166468][PYTHON] Support TimestampNTZ infer schema miss prefer_timestamp_ntz

• [SPARK-48157] [SC-165902][SQL] Add collation support for CSV expressions

• [SPARK-48158] [SC-165652][SQL] Add collation support for XML expressions

• [SPARK-48160] [SC-166064][SQL] Add collation support for XPATH expressions

• [SPARK-48229] [SC-165901][SQL] Add collation support for inputFile expressions

• [SPARK-48367] [SC-166487][CONNECT] Fix lint-scala for scalafmt to detect files to format properly

• [SPARK-47858] [SC-163095][SPARK-47852][PYTHON][SQL] Refactoring the structure for DataFrame error context

• [SPARK-48370] [SC-166787][CONNECT] Checkpoint and localCheckpoint in Scala Spark Connect client

• [SPARK-48247] [SC-166028][PYTHON] Use all values in a dict when inferring MapType schema

• [SPARK-48395] [SC-166794][PYTHON] Fix StructType.treeString for parameterized types

• [SPARK-48393] [SC-166784][PYTHON] Move a group of constants to pyspark.util

• [SPARK-48372] [SC-166776][SPARK-45716][PYTHON] Implement StructType.treeString

• [SPARK-48258] [SC-166467][PYTHON][CONNECT] Checkpoint and localCheckpoint in Spark Connect

See Databricks Runtime 15.4 LTS maintenance updates.

System environment

• Operating System: Ubuntu 22.04.4 LTS

• Java: Zulu 8.78.0.19-CA-linux64

• Scala: 2.12.18

• Python: 3.11.0

• R: 4.3.2

• Delta Lake: 3.2.0

Installed Python libraries

Library	Version	Library	Version	Library	Version
asttokens	2.0.5	astunparse	1.6.3	azure-core	1.30.2
azure-storage-blob	12.19.1	azure-storage-file-datalake	12.14.0	backcall	0.2.0
black	23.3.0	blinker	1.4	boto3	1.34.39
botocore	1.34.39	cachetools	5.3.3	certifi	2023.7.22
cffi	1.15.1	chardet	4.0.0	charset-normalizer	2.0.4
click	8.0.4	cloudpickle	2.2.1	comm	0.1.2
contourpy	1.0.5	cryptography	41.0.3	cycler	0.11.0
Cython	0.29.32	databricks-sdk	0.20.0	dbus-python	1.2.18
debugpy	1.6.7	decorator	5.1.1	distlib	0.3.8
entrypoints	0.4	executing	0.8.3	facets-overview	1.1.1
filelock	3.13.4	fonttools	4.25.0	gitdb	4.0.11
GitPython	3.1.43	google-api-core	2.18.0	google-auth	2.31.0
google-cloud-core	2.4.1	google-cloud-storage	2.17.0	google-crc32c	1.5.0
google-resumable-media	2.7.1	googleapis-common-protos	1.63.2	grpcio	1.60.0
grpcio-status	1.60.0	httplib2	0.20.2	idna	3.4
importlib-metadata	6.0.0	ipyflow-core	0.0.198	ipykernel	6.25.1
ipython	8.15.0	ipython-genutils	0.2.0	ipywidgets	7.7.2
isodate	0.6.1	jedi	0.18.1	jeepney	0.7.1
jmespath	0.10.0	joblib	1.2.0	jupyter_client	7.4.9
jupyter_core	5.3.0	keyring	23.5.0	kiwisolver	1.4.4
launchpadlib	1.10.16	lazr.restfulclient	0.14.4	lazr.uri	1.0.6
matplotlib	3.7.2	matplotlib-inline	0.1.6	mlflow-skinny	2.11.4
more-itertools	8.10.0	mypy-extensions	0.4.3	nest-asyncio	1.5.6
numpy	1.23.5	oauthlib	3.2.0	packaging	23.2
pandas	1.5.3	parso	0.8.3	pathspec	0.10.3
patsy	0.5.3	pexpect	4.8.0	pickleshare	0.7.5
Pillow	9.4.0	pip	23.2.1	platformdirs	3.10.0
plotly	5.9.0	prompt-toolkit	3.0.36	proto-plus	1.24.0
protobuf	4.24.1	psutil	5.9.0	psycopg2	2.9.3
ptyprocess	0.7.0	pure-eval	0.2.2	pyarrow	14.0.1
pyasn1	0.4.8	pyasn1-modules	0.2.8	pyccolo	0.0.52
pycparser	2.21	pydantic	1.10.6	Pygments	2.15.1
PyGObject	3.42.1	PyJWT	2.3.0	pyodbc	4.0.38
pyparsing	3.0.9	python-dateutil	2.8.2	python-lsp-jsonrpc	1.1.1
pytz	2022.7	PyYAML	6.0	pyzmq	23.2.0
requests	2.31.0	rsa	4.9	s3transfer	0.10.2
scikit-learn	1.3.0	scipy	1.11.1	seaborn	0.12.2
SecretStorage	3.3.1	setuptools	68.0.0	six	1.16.0
smmap	5.0.1	sqlparse	0.5.0	ssh-import-id	5.11
stack-data	0.2.0	statsmodels	0.14.0	tenacity	8.2.2
threadpoolctl	2.2.0	tokenize-rt	4.2.1	tornado	6.3.2
traitlets	5.7.1	typing_extensions	4.10.0	tzdata	2022.1
ujson	5.4.0	unattended-upgrades	0.1	urllib3	1.26.16
virtualenv	20.24.2	wadllib	1.3.6	wcwidth	0.2.5
wheel	0.38.4	zipp	3.11.0

Installed R libraries

R libraries are installed from the Posit Package Manager CRAN snapshot.

Library	Version	Library	Version	Library	Version
arrow	14.0.0.2	askpass	1.2.0	assertthat	0.2.1
backports	1.4.1	base	4.3.2	base64enc	0.1-3
bigD	0.2.0	bit	4.0.5	bit64	4.0.5
bitops	1.0-7	blob	1.2.4	boot	1.3-28
brew	1.0-10	brio	1.1.4	broom	1.0.5
bslib	0.6.1	cachem	1.0.8	callr	3.7.3
caret	6.0-94	cellranger	1.1.0	chron	2.3-61
class	7.3-22	cli	3.6.2	clipr	0.8.0
clock	0.7.0	cluster	2.1.4	codetools	0.2-19
colorspace	2.1-0	commonmark	1.9.1	compiler	4.3.2
config	0.3.2	conflicted	1.2.0	cpp11	0.4.7
crayon	1.5.2	credentials	2.0.1	curl	5.2.0
data.table	1.15.0	datasets	4.3.2	DBI	1.2.1
dbplyr	2.4.0	desc	1.4.3	devtools	2.4.5
diagram	1.6.5	diffobj	0.3.5	digest	0.6.34
downlit	0.4.3	dplyr	1.1.4	dtplyr	1.3.1
e1071	1.7-14	ellipsis	0.3.2	evaluate	0.23
fansi	1.0.6	farver	2.1.1	fastmap	1.1.1
fontawesome	0.5.2	forcats	1.0.0	foreach	1.5.2
foreign	0.8-85	forge	0.2.0	fs	1.6.3
future	1.33.1	future.apply	1.11.1	gargle	1.5.2
generics	0.1.3	gert	2.0.1	ggplot2	3.4.4
gh	1.4.0	git2r	0.33.0	gitcreds	0.1.2
glmnet	4.1-8	globals	0.16.2	glue	1.7.0
googledrive	2.1.1	googlesheets4	1.1.1	gower	1.0.1
graphics	4.3.2	grDevices	4.3.2	grid	4.3.2
gridExtra	2.3	gsubfn	0.7	gt	0.10.1
gtable	0.3.4	hardhat	1.3.1	haven	2.5.4
highr	0.10	hms	1.1.3	htmltools	0.5.7
htmlwidgets	1.6.4	httpuv	1.6.14	httr	1.4.7
httr2	1.0.0	ids	1.0.1	ini	0.3.1
ipred	0.9-14	isoband	0.2.7	iterators	1.0.14
jquerylib	0.1.4	jsonlite	1.8.8	juicyjuice	0.1.0
KernSmooth	2.23-21	knitr	1.45	labeling	0.4.3
later	1.3.2	lattice	0.21-8	lava	1.7.3
lifecycle	1.0.4	listenv	0.9.1	lubridate	1.9.3
magrittr	2.0.3	markdown	1.12	MASS	7.3-60
Matrix	1.5-4.1	memoise	2.0.1	methods	4.3.2
mgcv	1.8-42	mime	0.12	miniUI	0.1.1.1
mlflow	2.10.0	ModelMetrics	1.2.2.2	modelr	0.1.11
munsell	0.5.0	nlme	3.1-163	nnet	7.3-19
numDeriv	2016.8-1.1	openssl	2.1.1	parallel	4.3.2
parallelly	1.36.0	pillar	1.9.0	pkgbuild	1.4.3
pkgconfig	2.0.3	pkgdown	2.0.7	pkgload	1.3.4
plogr	0.2.0	plyr	1.8.9	praise	1.0.0
prettyunits	1.2.0	pROC	1.18.5	processx	3.8.3
prodlim	2023.08.28	profvis	0.3.8	progress	1.2.3
progressr	0.14.0	promises	1.2.1	proto	1.0.0
proxy	0.4-27	ps	1.7.6	purrr	1.0.2
R6	2.5.1	ragg	1.2.7	randomForest	4.7-1.1
rappdirs	0.3.3	rcmdcheck	1.4.0	RColorBrewer	1.1-3
Rcpp	1.0.12	RcppEigen	0.3.3.9.4	reactable	0.4.4
reactR	0.5.0	readr	2.1.5	readxl	1.4.3
recipes	1.0.9	rematch	2.0.0	rematch2	2.1.2
remotes	2.4.2.1	reprex	2.1.0	reshape2	1.4.4
rlang	1.1.3	rmarkdown	2.25	RODBC	1.3-23
roxygen2	7.3.1	rpart	4.1.21	rprojroot	2.0.4
Rserve	1.8-13	RSQLite	2.3.5	rstudioapi	0.15.0
rversions	2.1.2	rvest	1.0.3	sass	0.4.8
scales	1.3.0	selectr	0.4-2	sessioninfo	1.2.2
shape	1.4.6	shiny	1.8.0	sourcetools	0.1.7-1
sparklyr	1.8.4	spatial	7.3-15	splines	4.3.2
sqldf	0.4-11	SQUAREM	2021.1	stats	4.3.2
stats4	4.3.2	stringi	1.8.3	stringr	1.5.1
survival	3.5-5	swagger	3.33.1	sys	3.4.2
systemfonts	1.0.5	tcltk	4.3.2	testthat	3.2.1
textshaping	0.3.7	tibble	3.2.1	tidyr	1.3.1
tidyselect	1.2.0	tidyverse	2.0.0	timechange	0.3.0
timeDate	4032.109	tinytex	0.49	tools	4.3.2
tzdb	0.4.0	urlchecker	1.0.1	usethis	2.2.2
utf8	1.2.4	utils	4.3.2	uuid	1.2-0
V8	4.4.1	vctrs	0.6.5	viridisLite	0.4.2
vroom	1.6.5	waldo	0.5.2	whisker	0.4.1
withr	3.0.0	xfun	0.41	xml2	1.3.6
xopen	1.0.0	xtable	1.8-4	yaml	2.3.8
zeallot	0.1.0	zip	2.3.1

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID	Artifact ID	Version
antlr	antlr	2.7.7
com.amazonaws	amazon-kinesis-client	1.12.0
com.amazonaws	aws-java-sdk-autoscaling	1.12.610
com.amazonaws	aws-java-sdk-cloudformation	1.12.610
com.amazonaws	aws-java-sdk-cloudfront	1.12.610
com.amazonaws	aws-java-sdk-cloudhsm	1.12.610
com.amazonaws	aws-java-sdk-cloudsearch	1.12.610
com.amazonaws	aws-java-sdk-cloudtrail	1.12.610
com.amazonaws	aws-java-sdk-cloudwatch	1.12.610
com.amazonaws	aws-java-sdk-cloudwatchmetrics	1.12.610
com.amazonaws	aws-java-sdk-codedeploy	1.12.610
com.amazonaws	aws-java-sdk-cognitoidentity	1.12.610
com.amazonaws	aws-java-sdk-cognitosync	1.12.610
com.amazonaws	aws-java-sdk-config	1.12.610
com.amazonaws	aws-java-sdk-core	1.12.610
com.amazonaws	aws-java-sdk-datapipeline	1.12.610
com.amazonaws	aws-java-sdk-directconnect	1.12.610
com.amazonaws	aws-java-sdk-directory	1.12.610
com.amazonaws	aws-java-sdk-dynamodb	1.12.610
com.amazonaws	aws-java-sdk-ec2	1.12.610
com.amazonaws	aws-java-sdk-ecs	1.12.610
com.amazonaws	aws-java-sdk-efs	1.12.610
com.amazonaws	aws-java-sdk-elasticache	1.12.610
com.amazonaws	aws-java-sdk-elasticbeanstalk	1.12.610
com.amazonaws	aws-java-sdk-elasticloadbalancing	1.12.610
com.amazonaws	aws-java-sdk-elastictranscoder	1.12.610
com.amazonaws	aws-java-sdk-emr	1.12.610
com.amazonaws	aws-java-sdk-glacier	1.12.610
com.amazonaws	aws-java-sdk-glue	1.12.610
com.amazonaws	aws-java-sdk-iam	1.12.610
com.amazonaws	aws-java-sdk-importexport	1.12.610
com.amazonaws	aws-java-sdk-kinesis	1.12.610
com.amazonaws	aws-java-sdk-kms	1.12.610
com.amazonaws	aws-java-sdk-lambda	1.12.610
com.amazonaws	aws-java-sdk-logs	1.12.610
com.amazonaws	aws-java-sdk-machinelearning	1.12.610
com.amazonaws	aws-java-sdk-opsworks	1.12.610
com.amazonaws	aws-java-sdk-rds	1.12.610
com.amazonaws	aws-java-sdk-redshift	1.12.610
com.amazonaws	aws-java-sdk-route53	1.12.610
com.amazonaws	aws-java-sdk-s3	1.12.610
com.amazonaws	aws-java-sdk-ses	1.12.610
com.amazonaws	aws-java-sdk-simpledb	1.12.610
com.amazonaws	aws-java-sdk-simpleworkflow	1.12.610
com.amazonaws	aws-java-sdk-sns	1.12.610
com.amazonaws	aws-java-sdk-sqs	1.12.610
com.amazonaws	aws-java-sdk-ssm	1.12.610
com.amazonaws	aws-java-sdk-storagegateway	1.12.610
com.amazonaws	aws-java-sdk-sts	1.12.610
com.amazonaws	aws-java-sdk-support	1.12.610
com.amazonaws	aws-java-sdk-swf-libraries	1.11.22
com.amazonaws	aws-java-sdk-workspaces	1.12.610
com.amazonaws	jmespath-java	1.12.610
com.clearspring.analytics	stream	2.9.6
com.databricks	Rserve	1.8-3
com.databricks	databricks-sdk-java	0.27.0
com.databricks	jets3t	0.7.1-0
com.databricks.scalapb	compilerplugin_2.12	0.4.15-10
com.databricks.scalapb	scalapb-runtime_2.12	0.4.15-10
com.esotericsoftware	kryo-shaded	4.0.2
com.esotericsoftware	minlog	1.3.0
com.fasterxml	classmate	1.3.4
com.fasterxml.jackson.core	jackson-annotations	2.15.2
com.fasterxml.jackson.core	jackson-core	2.15.2
com.fasterxml.jackson.core	jackson-databind	2.15.2
com.fasterxml.jackson.dataformat	jackson-dataformat-cbor	2.15.2
com.fasterxml.jackson.dataformat	jackson-dataformat-yaml	2.15.2
com.fasterxml.jackson.datatype	jackson-datatype-joda	2.15.2
com.fasterxml.jackson.datatype	jackson-datatype-jsr310	2.16.0
com.fasterxml.jackson.module	jackson-module-paranamer	2.15.2
com.fasterxml.jackson.module	jackson-module-scala_2.12	2.15.2
com.github.ben-manes.caffeine	caffeine	2.9.3
com.github.fommil	jniloader	1.1
com.github.fommil.netlib	native_ref-java	1.1
com.github.fommil.netlib	native_ref-java	1.1-natives
com.github.fommil.netlib	native_system-java	1.1
com.github.fommil.netlib	native_system-java	1.1-natives
com.github.fommil.netlib	netlib-native_ref-linux-x86_64	1.1-natives
com.github.fommil.netlib	netlib-native_system-linux-x86_64	1.1-natives
com.github.luben	zstd-jni	1.5.5-4
com.github.wendykierp	JTransforms	3.1
com.google.code.findbugs	jsr305	3.0.0
com.google.code.gson	gson	2.10.1
com.google.crypto.tink	tink	1.9.0
com.google.errorprone	error_prone_annotations	2.10.0
com.google.flatbuffers	flatbuffers-java	23.5.26
com.google.guava	guava	15.0
com.google.protobuf	protobuf-java	2.6.1
com.helger	profiler	1.1.1
com.ibm.icu	icu4j	75.1
com.jcraft	jsch	0.1.55
com.jolbox	bonecp	0.8.0.RELEASE
com.lihaoyi	sourcecode_2.12	0.1.9
com.microsoft.azure	azure-data-lake-store-sdk	2.3.9
com.microsoft.sqlserver	mssql-jdbc	11.2.2.jre8
com.ning	compress-lzf	1.1.2
com.sun.mail	javax.mail	1.5.2
com.sun.xml.bind	jaxb-core	2.2.11
com.sun.xml.bind	jaxb-impl	2.2.11
com.tdunning	json	1.8
com.thoughtworks.paranamer	paranamer	2.8
com.trueaccord.lenses	lenses_2.12	0.4.12
com.twitter	chill-java	0.10.0
com.twitter	chill_2.12	0.10.0
com.twitter	util-app_2.12	7.1.0
com.twitter	util-core_2.12	7.1.0
com.twitter	util-function_2.12	7.1.0
com.twitter	util-jvm_2.12	7.1.0
com.twitter	util-lint_2.12	7.1.0
com.twitter	util-registry_2.12	7.1.0
com.twitter	util-stats_2.12	7.1.0
com.typesafe	config	1.4.3
com.typesafe.scala-logging	scala-logging_2.12	3.7.2
com.uber	h3	3.7.3
com.univocity	univocity-parsers	2.9.1
com.zaxxer	HikariCP	4.0.3
commons-cli	commons-cli	1.5.0
commons-codec	commons-codec	1.16.0
commons-collections	commons-collections	3.2.2
commons-dbcp	commons-dbcp	1.4
commons-fileupload	commons-fileupload	1.5
commons-httpclient	commons-httpclient	3.1
commons-io	commons-io	2.13.0
commons-lang	commons-lang	2.6
commons-logging	commons-logging	1.1.3
commons-pool	commons-pool	1.5.4
dev.ludovic.netlib	arpack	3.0.3
dev.ludovic.netlib	blas	3.0.3
dev.ludovic.netlib	lapack	3.0.3
info.ganglia.gmetric4j	gmetric4j	1.0.10
io.airlift	aircompressor	0.25
io.delta	delta-sharing-client_2.12	1.1.1
io.dropwizard.metrics	metrics-annotation	4.2.19
io.dropwizard.metrics	metrics-core	4.2.19
io.dropwizard.metrics	metrics-graphite	4.2.19
io.dropwizard.metrics	metrics-healthchecks	4.2.19
io.dropwizard.metrics	metrics-jetty9	4.2.19
io.dropwizard.metrics	metrics-jmx	4.2.19
io.dropwizard.metrics	metrics-json	4.2.19
io.dropwizard.metrics	metrics-jvm	4.2.19
io.dropwizard.metrics	metrics-servlets	4.2.19
io.netty	netty-all	4.1.96.Final
io.netty	netty-buffer	4.1.96.Final
io.netty	netty-codec	4.1.96.Final
io.netty	netty-codec-http	4.1.96.Final
io.netty	netty-codec-http2	4.1.96.Final
io.netty	netty-codec-socks	4.1.96.Final
io.netty	netty-common	4.1.96.Final
io.netty	netty-handler	4.1.96.Final
io.netty	netty-handler-proxy	4.1.96.Final
io.netty	netty-resolver	4.1.96.Final
io.netty	netty-tcnative-boringssl-static	2.0.61.Final-linux-aarch_64
io.netty	netty-tcnative-boringssl-static	2.0.61.Final-linux-x86_64
io.netty	netty-tcnative-boringssl-static	2.0.61.Final-osx-aarch_64
io.netty	netty-tcnative-boringssl-static	2.0.61.Final-osx-x86_64
io.netty	netty-tcnative-boringssl-static	2.0.61.Final-windows-x86_64
io.netty	netty-tcnative-classes	2.0.61.Final
io.netty	netty-transport	4.1.96.Final
io.netty	netty-transport-classes-epoll	4.1.96.Final
io.netty	netty-transport-classes-kqueue	4.1.96.Final
io.netty	netty-transport-native-epoll	4.1.96.Final
io.netty	netty-transport-native-epoll	4.1.96.Final-linux-aarch_64
io.netty	netty-transport-native-epoll	4.1.96.Final-linux-x86_64
io.netty	netty-transport-native-kqueue	4.1.96.Final-osx-aarch_64
io.netty	netty-transport-native-kqueue	4.1.96.Final-osx-x86_64
io.netty	netty-transport-native-unix-common	4.1.96.Final
io.prometheus	simpleclient	0.7.0
io.prometheus	simpleclient_common	0.7.0
io.prometheus	simpleclient_dropwizard	0.7.0
io.prometheus	simpleclient_pushgateway	0.7.0
io.prometheus	simpleclient_servlet	0.7.0
io.prometheus.jmx	collector	0.12.0
jakarta.annotation	jakarta.annotation-api	1.3.5
jakarta.servlet	jakarta.servlet-api	4.0.3
jakarta.validation	jakarta.validation-api	2.0.2
jakarta.ws.rs	jakarta.ws.rs-api	2.1.6
javax.activation	activation	1.1.1
javax.el	javax.el-api	2.2.4
javax.jdo	jdo-api	3.0.1
javax.transaction	jta	1.1
javax.transaction	transaction-api	1.1
javax.xml.bind	jaxb-api	2.2.11
javolution	javolution	5.5.1
jline	jline	2.14.6
joda-time	joda-time	2.12.1
net.java.dev.jna	jna	5.8.0
net.razorvine	pickle	1.3
net.sf.jpam	jpam	1.1
net.sf.opencsv	opencsv	2.3
net.sf.supercsv	super-csv	2.2.0
net.snowflake	snowflake-ingest-sdk	0.9.6
net.sourceforge.f2j	arpack_combined_all	0.1
org.acplt.remotetea	remotetea-oncrpc	1.1.2
org.antlr	ST4	4.0.4
org.antlr	antlr-runtime	3.5.2
org.antlr	antlr4-runtime	4.9.3
org.antlr	stringtemplate	3.2.1
org.apache.ant	ant	1.10.11
org.apache.ant	ant-jsch	1.10.11
org.apache.ant	ant-launcher	1.10.11
org.apache.arrow	arrow-format	15.0.0
org.apache.arrow	arrow-memory-core	15.0.0
org.apache.arrow	arrow-memory-netty	15.0.0
org.apache.arrow	arrow-vector	15.0.0
org.apache.avro	avro	1.11.3
org.apache.avro	avro-ipc	1.11.3
org.apache.avro	avro-mapred	1.11.3
org.apache.commons	commons-collections4	4.4
org.apache.commons	commons-compress	1.23.0
org.apache.commons	commons-crypto	1.1.0
org.apache.commons	commons-lang3	3.12.0
org.apache.commons	commons-math3	3.6.1
org.apache.commons	commons-text	1.10.0
org.apache.curator	curator-client	2.13.0
org.apache.curator	curator-framework	2.13.0
org.apache.curator	curator-recipes	2.13.0
org.apache.datasketches	datasketches-java	3.1.0
org.apache.datasketches	datasketches-memory	2.0.0
org.apache.derby	derby	10.14.2.0
org.apache.hadoop	hadoop-client-runtime	3.3.6
org.apache.hive	hive-beeline	2.3.9
org.apache.hive	hive-cli	2.3.9
org.apache.hive	hive-jdbc	2.3.9
org.apache.hive	hive-llap-client	2.3.9
org.apache.hive	hive-llap-common	2.3.9
org.apache.hive	hive-serde	2.3.9
org.apache.hive	hive-shims	2.3.9
org.apache.hive	hive-storage-api	2.8.1
org.apache.hive.shims	hive-shims-0.23	2.3.9
org.apache.hive.shims	hive-shims-common	2.3.9
org.apache.hive.shims	hive-shims-scheduler	2.3.9
org.apache.httpcomponents	httpclient	4.5.14
org.apache.httpcomponents	httpcore	4.4.16
org.apache.ivy	ivy	2.5.1
org.apache.logging.log4j	log4j-1.2-api	2.22.1
org.apache.logging.log4j	log4j-api	2.22.1
org.apache.logging.log4j	log4j-core	2.22.1
org.apache.logging.log4j	log4j-layout-template-json	2.22.1
org.apache.logging.log4j	log4j-slf4j2-impl	2.22.1
org.apache.orc	orc-core	1.9.2-shaded-protobuf
org.apache.orc	orc-mapreduce	1.9.2-shaded-protobuf
org.apache.orc	orc-shims	1.9.2
org.apache.thrift	libfb303	0.9.3
org.apache.thrift	libthrift	0.12.0
org.apache.ws.xmlschema	xmlschema-core	2.3.0
org.apache.xbean	xbean-asm9-shaded	4.23
org.apache.yetus	audience-annotations	0.13.0
org.apache.zookeeper	zookeeper	3.6.3
org.apache.zookeeper	zookeeper-jute	3.6.3
org.checkerframework	checker-qual	3.31.0
org.codehaus.jackson	jackson-core-asl	1.9.13
org.codehaus.jackson	jackson-mapper-asl	1.9.13
org.codehaus.janino	commons-compiler	3.0.16
org.codehaus.janino	janino	3.0.16
org.datanucleus	datanucleus-api-jdo	4.2.4
org.datanucleus	datanucleus-core	4.1.17
org.datanucleus	datanucleus-rdbms	4.1.19
org.datanucleus	javax.jdo	3.2.0-m3
org.eclipse.collections	eclipse-collections	11.1.0
org.eclipse.collections	eclipse-collections-api	11.1.0
org.eclipse.jetty	jetty-client	9.4.52.v20230823
org.eclipse.jetty	jetty-continuation	9.4.52.v20230823
org.eclipse.jetty	jetty-http	9.4.52.v20230823
org.eclipse.jetty	jetty-io	9.4.52.v20230823
org.eclipse.jetty	jetty-jndi	9.4.52.v20230823
org.eclipse.jetty	jetty-plus	9.4.52.v20230823
org.eclipse.jetty	jetty-proxy	9.4.52.v20230823
org.eclipse.jetty	jetty-security	9.4.52.v20230823
org.eclipse.jetty	jetty-server	9.4.52.v20230823
org.eclipse.jetty	jetty-servlet	9.4.52.v20230823
org.eclipse.jetty	jetty-servlets	9.4.52.v20230823
org.eclipse.jetty	jetty-util	9.4.52.v20230823
org.eclipse.jetty	jetty-util-ajax	9.4.52.v20230823
org.eclipse.jetty	jetty-webapp	9.4.52.v20230823
org.eclipse.jetty	jetty-xml	9.4.52.v20230823
org.eclipse.jetty.websocket	websocket-api	9.4.52.v20230823
org.eclipse.jetty.websocket	websocket-client	9.4.52.v20230823
org.eclipse.jetty.websocket	websocket-common	9.4.52.v20230823
org.eclipse.jetty.websocket	websocket-server	9.4.52.v20230823
org.eclipse.jetty.websocket	websocket-servlet	9.4.52.v20230823
org.fusesource.leveldbjni	leveldbjni-all	1.8
org.glassfish.hk2	hk2-api	2.6.1
org.glassfish.hk2	hk2-locator	2.6.1
org.glassfish.hk2	hk2-utils	2.6.1
org.glassfish.hk2	osgi-resource-locator	1.0.3
org.glassfish.hk2.external	aopalliance-repackaged	2.6.1
org.glassfish.hk2.external	jakarta.inject	2.6.1
org.glassfish.jersey.containers	jersey-container-servlet	2.40
org.glassfish.jersey.containers	jersey-container-servlet-core	2.40
org.glassfish.jersey.core	jersey-client	2.40
org.glassfish.jersey.core	jersey-common	2.40
org.glassfish.jersey.core	jersey-server	2.40
org.glassfish.jersey.inject	jersey-hk2	2.40
org.hibernate.validator	hibernate-validator	6.1.7.Final
org.ini4j	ini4j	0.5.4
org.javassist	javassist	3.29.2-GA
org.jboss.logging	jboss-logging	3.3.2.Final
org.jdbi	jdbi	2.63.1
org.jetbrains	annotations	17.0.0
org.joda	joda-convert	1.7
org.jodd	jodd-core	3.5.2
org.json4s	json4s-ast_2.12	3.7.0-M11
org.json4s	json4s-core_2.12	3.7.0-M11
org.json4s	json4s-jackson_2.12	3.7.0-M11
org.json4s	json4s-scalap_2.12	3.7.0-M11
org.lz4	lz4-java	1.8.0
org.mlflow	mlflow-spark_2.12	2.9.1
org.objenesis	objenesis	2.5.1
org.postgresql	postgresql	42.6.1
org.roaringbitmap	RoaringBitmap	0.9.45-databricks
org.roaringbitmap	shims	0.9.45-databricks
org.rocksdb	rocksdbjni	8.11.4
org.rosuda.REngine	REngine	2.1.0
org.scala-lang	scala-compiler_2.12	2.12.15
org.scala-lang	scala-library_2.12	2.12.15
org.scala-lang	scala-reflect_2.12	2.12.15
org.scala-lang.modules	scala-collection-compat_2.12	2.11.0
org.scala-lang.modules	scala-parser-combinators_2.12	1.1.2
org.scala-lang.modules	scala-xml_2.12	1.2.0
org.scala-sbt	test-interface	1.0
org.scalacheck	scalacheck_2.12	1.14.2
org.scalactic	scalactic_2.12	3.2.15
org.scalanlp	breeze-macros_2.12	2.1.0
org.scalanlp	breeze_2.12	2.1.0
org.scalatest	scalatest-compatible	3.2.15
org.scalatest	scalatest-core_2.12	3.2.15
org.scalatest	scalatest-diagrams_2.12	3.2.15
org.scalatest	scalatest-featurespec_2.12	3.2.15
org.scalatest	scalatest-flatspec_2.12	3.2.15
org.scalatest	scalatest-freespec_2.12	3.2.15
org.scalatest	scalatest-funspec_2.12	3.2.15
org.scalatest	scalatest-funsuite_2.12	3.2.15
org.scalatest	scalatest-matchers-core_2.12	3.2.15
org.scalatest	scalatest-mustmatchers_2.12	3.2.15
org.scalatest	scalatest-propspec_2.12	3.2.15
org.scalatest	scalatest-refspec_2.12	3.2.15
org.scalatest	scalatest-shouldmatchers_2.12	3.2.15
org.scalatest	scalatest-wordspec_2.12	3.2.15
org.scalatest	scalatest_2.12	3.2.15
org.slf4j	jcl-over-slf4j	2.0.7
org.slf4j	jul-to-slf4j	2.0.7
org.slf4j	slf4j-api	2.0.7
org.slf4j	slf4j-simple	1.7.25
org.threeten	threeten-extra	1.7.1
org.tukaani	xz	1.9
org.typelevel	algebra_2.12	2.0.1
org.typelevel	cats-kernel_2.12	2.1.1
org.typelevel	spire-macros_2.12	0.17.0
org.typelevel	spire-platform_2.12	0.17.0
org.typelevel	spire-util_2.12	0.17.0
org.typelevel	spire_2.12	0.17.0
org.wildfly.openssl	wildfly-openssl	1.1.3.Final
org.xerial	sqlite-jdbc	3.42.0.0
org.xerial.snappy	snappy-java	1.1.10.3
org.yaml	snakeyaml	2.0
oro	oro	2.0.8
pl.edu.icm	JLargeArrays	1.5
software.amazon.cryptools	AmazonCorrettoCryptoProvider	1.6.2-linux-x86_64
software.amazon.ion	ion-java	1.0.2
stax	stax-api	1.0.1