Databricks Runtime 13.1 (EoS)

Note

Support for this Databricks Runtime version has ended. For the end-of-support date, see End-of-support history. For all supported Databricks Runtime versions, see Databricks Runtime release notes versions and compatibility.

The following release notes provide information about Databricks Runtime 13.1, powered by Apache Spark 3.4.0.

Databricks released this version in May 2023.

Note

These release notes may include references to features that are not available on Google Cloud as of this release.

New features and improvements

Cluster support for JDK 17 (Public Preview)

Databricks now provides cluster support for Java Development Kit (JDK) 17. See Databricks SDK for Java.

Add, change, or delete data in streaming tables

You can now use DML statements to modify streaming tables published to Unity Catalog by Delta Live Tables pipelines. See Add, change, or delete data in a streaming table and Add, change, or delete data in a target streaming table. You can also use DML statements to modify streaming tables created in Databricks SQL.

Read Kafka with SQL

You can now use the read_kafka SQL function for reading Kafka data. Streaming with SQL is supported only in Delta Live Tables or with streaming tables in Databricks SQL. See read_kafka table-valued function.

New SQL built-in functions

The following functions have been added:

Unity Catalog support for cluster-scoped Python libraries

Unity Catalog has some limitations on library usage. On Databricks Runtime 13.1 and above, cluster-scoped Python libraries are supported, including Python wheel files that are uploaded as workspace files. Libraries that are referenced using DBFS filepaths are not supported, whether in the DBFS root or an external location mounted to DBFS. Non-Python libraries are not supported. See Cluster libraries.

On Databricks Runtime 13.0 and below, cluster-scoped libraries are not supported on clusters that use shared access mode in a Unity Catalog-enabled workspace.

Expanded default enablement for optimized writes in Unity Catalog

Default optimized write support for Delta tables registered in Unity Catalog has expanded to include CTAS statements and INSERT operations for partitioned tables. This behavior aligns to defaults on SQL warehouses. See Optimized writes for Delta Lake on Databricks.

Advanced support for stateful operators in Structured Streaming workloads

You can now chain multiple stateful operators together, meaning that you can feed the output of an operation such as a windowed aggregation to another stateful operation such as a join. See What is stateful streaming?.

Delta clone for Unity Catalog is in Public Preview

You can now use shallow clone to create new Unity Catalog managed tables from existing Unity Catalog managed tables. See Shallow clone for Unity Catalog tables.

Pub/Sub support for Structured Streaming

You can now use a built-in connector to subscribe to Google Pub/Sub with Structured Streaming. See Subscribe to Google Pub/Sub.

Drop duplicates within watermarks in Structured Streaming

You can now use dropDuplicatesWithinWatermark in combination with a specified watermark threshold to deduplicate records in Structured Streaming. See Drop duplicates within watermark.

Expanded support for Delta conversions from Iceberg tables with truncated partition columns

You can now use CLONE and CONVERT TO DELTA with Iceberg tables that have partitions defined on truncated columns of types int, long, and string. Truncated columns of type decimal are not supported.

Stream schema changes with column mapping in Delta Lake

You now can provide a schema tracking location to enable streaming from Delta tables with column mapping enabled. See Streaming with column mapping and schema changes.

Remove START VERSION

START VERSION is now deprecated for ALTER SHARE.

New H3 expressions available with Python

The h3_coverash3 and h3_coverash3string expressions are available with Python.

Bug fixes

Parquet failOnUnknownFields no longer silently drop data on type mismatch

If a Parquet file was read with just the failOnUnknownFields option or with Auto Loader in the failOnNewColumns schema evolution mode, columns that have different data types now fail and recommend using rescuedDataColumn. Auto Loader now correctly reads and no longer rescues Integer, Short, or Byte types if one of these data types is provided. The Parquet file suggests one of the other two types.

Breaking changes

Upgrade sqlite-jdbc version to 3.42.0.0 to address CVE-2023-32697

Upgrade sqlite-jdbc version from 3.8.11.2 to 3.42.0.0. The APIs of version 3.42.0.0 are not fully compatible with 3.8.11.2. If using sqlite-jdbc in your code, check the sqlite-jdbc compatibility report for details. If you migrate to 13.1 and use sqlite, confirm your methods and return type in version 3.42.0.0.

Library upgrades

  • Upgraded Python libraries:

    • facets-overview from 1.0.2 to 1.0.3

    • filelock from 3.10.7 to 3.12.0

    • pyarrow from 7.0.0 to 8.0.0

    • tenacity from 8.0.1 to 8.1.0

  • Upgraded R libraries:

  • Upgraded Java libraries:

    • com.github.ben-manes.caffeine.caffeine from 2.3.4 to 2.9.3

    • io.delta.delta-sharing-spark_2.12 from 0.6.8 to 0.6.4

    • net.snowflake.snowflake-jdbc from 3.13.29 to 3.13.22

    • org.checkerframework.checker-qual from 3.5.0 to 3.19.0

    • org.scalactic.scalactic_2.12 from 3.0.8 to 3.2.15

    • org.scalatest.scalatest_2.12 from 3.0.8 to 3.2.15

    • org.xerial.sqlite-jdbc from 3.8.11.2 to 3.42.0.0

Apache Spark

Databricks Runtime 13.1 includes Apache Spark 3.4.0. This release includes all Spark fixes and improvements included in Databricks Runtime 13.0 (EoS), as well as the following additional bug fixes and improvements made to Spark:

  • [SPARK-42719] [DBRRM-199][SC-131578] Revert “[SC-125225] `MapOutputTracker#getMap…

  • [SPARK-39696] [DBRRM-166][SC-130056][CORE] Revert [SC-127830]/

  • [SPARK-43331] [SC-130064][CONNECT] Add Spark Connect SparkSession.interruptAll

  • [SPARK-43332] [SC-130051][CONNECT][PYTHON] Make it possible to extend ChannelBuilder for SparkConnectClient

  • [SPARK-43323] [SC-129966][SQL][PYTHON] Fix DataFrame.toPandas with Arrow enabled to handle exceptions properly

  • [SPARK-42940] [SC-129896][SS][CONNECT] Improve session management for streaming queries

  • [SPARK-43032] [SC-125756] [CONNECT][SS] Add Streaming query manager

  • [SPARK-16484] [SC-129975][SQL] Add support for Datasketches HllSketch

  • [SPARK-43260] [SC-129281][PYTHON] Migrate the Spark SQL pandas arrow type errors into error class.

  • [SPARK-41766] [SC-129964][CORE] Handle decommission request sent before executor registration

  • [SPARK-43307] [SC-129971][PYTHON] Migrate PandasUDF value errors into error class

  • [SPARK-43206] [SC-129903] [SS] [CONNECT] StreamingQuery exception() include stack trace

  • [SPARK-43311] [SC-129905][SS] Add RocksDB state store provider memory management enhancements

  • [SPARK-43237] [SC-129898][CORE] Handle null exception message in event log

  • [SPARK-43320] [SC-129899][SQL][HIVE] Directly call Hive 2.3.9 API

  • [SPARK-43270] [SC-129897][PYTHON] Implement __dir__() in pyspark.sql.dataframe.DataFrame to include columns

  • [SPARK-43183] Revert “[SC-128938][SS] Introduce a new callback “…

  • [SPARK-43143] [SC-129902] [SS] [CONNECT] Scala StreamingQuery awaitTermination()

  • [SPARK-43257] [SC-129675][SQL] Replace the error class LEGACYERROR_TEMP_2022 by an internal error

  • [SPARK-43198] [SC-129470][CONNECT] Fix “Could not initialise class ammonite…” error when using filter

  • [SPARK-43165] [SC-129777][SQL] Move canWrite to DataTypeUtils

  • [SPARK-43298] [SC-129729][PYTHON][ML] predict_batch_udf with scalar input fails with batch size of one

  • [SPARK-43298] [SC-129700]Revert “[PYTHON][ML] predict_batch_udf with scalar input fails with batch size of one”

  • [SPARK-43052] [SC-129663][CORE] Handle stacktrace with null file name in event log

  • [SPARK-43183] [SC-128938][SS] Introduce a new callback “onQueryIdle” to StreamingQueryListener

  • [SPARK-43209] [SC-129190][CONNECT][PYTHON] Migrate Expression errors into error class

  • [SPARK-42151] [SC-128754][SQL] Align UPDATE assignments with table attributes

  • [SPARK-43134] [SC-129468] [CONNECT] [SS] JVM client StreamingQuery exception() API

  • [SPARK-43298] [SC-129699][PYTHON][ML] predict_batch_udf with scalar input fails with batch size of one

  • [SPARK-43248] [SC-129660][SQL] Unnecessary serialize/deserialize of Path on parallel gather partition stats

  • [SPARK-43274] [SC-129464][SPARK-43275][PYTHON][CONNECT] Introduce PySparkNotImplementedError

  • [SPARK-43146] [SC-128804][CONNECT][PYTHON] Implement eager evaluation for repr and reprhtml_

  • [SPARK-42953] [SC-129469][Connect][Followup] Fix maven test build for Scala client UDF tests

  • [SPARK-43144] [SC-129280] Scala Client DataStreamReader table() API

  • [SPARK-43136] [SC-129358][CONNECT] Adding groupByKey + mapGroup + coGroup functions

  • [SPARK-43156] [SC-129672][SC-128532][SQL] Fix COUNT(*) is null bug in correlated scalar subquery

  • [SPARK-43046] [SC-129110] [SS] [Connect] Implemented Python API dropDuplicatesWithinWatermark for Spark Connect

  • [SPARK-43199] [SC-129467][SQL] Make InlineCTE idempotent

  • [SPARK-43293] [SC-129657][SQL] __qualified_access_only should be ignored in normal columns

  • [SPARK-43276] [SC-129461][CONNECT][PYTHON] Migrate Spark Connect Window errors into error class

  • [SPARK-43174] [SC-129109][SQL] Fix SparkSQLCLIDriver completer

  • [SPARK-43084] [SC-128654] [SS] Add applyInPandasWithState support for spark connect

  • [SPARK-43119] [SC-129040][SQL] Support Get SQL Keywords Dynamically Thru JDBC API and TVF

  • [SPARK-43082] [SC-129112][CONNECT][PYTHON] Arrow-optimized Python UDFs in Spark Connect

  • [SPARK-43085] [SC-128432][SQL] Support column DEFAULT assignment for multi-part table names

  • [SPARK-43226] [LC-671] Define extractors for file-constant metadata

  • [SPARK-43210] [SC-129189][CONNECT][PYTHON] Introduce PySparkAssertionError

  • [SPARK-43214] [SC-129199][SQL] Post driver-side metrics for LocalTableScanExec/CommandResultExec

  • [SPARK-43285] [SC-129347] Fix ReplE2ESuite consistently failing with JDK 17

  • [SPARK-43268] [SC-129249][SQL] Use proper error classes when exceptions are constructed with a message

  • [SPARK-43142] [SC-129299] Fix DSL expressions on attributes with special characters

  • [SPARK-43129] [SC-128896] Scala core API for streaming Spark Connect

  • [SPARK-43233] [SC-129250] [SS] Add logging for Kafka Batch Reading for topic partition, offset range and task ID

  • [SPARK-43249] [SC-129195][CONNECT] Fix missing stats for SQL Command

  • [SPARK-42945] [SC-129188][CONNECT] Support PYSPARK_JVM_STACKTRACE_ENABLED in Spark Connect

  • [SPARK-43178] [SC-129197][CONNECT][PYTHON] Migrate UDF errors into PySpark error framework

  • [SPARK-43123] [SC-128494][SQL] Internal field metadata should not be leaked to catalogs

  • [SPARK-43217] [SC-129205] Correctly recurse in nested maps/arrays in findNestedField

  • [SPARK-43243] [SC-129294][PYTHON][CONNECT] Add level param to printSchema for Python

  • [SPARK-43230] [SC-129191][CONNECT] Simplify DataFrameNaFunctions.fillna

  • [SPARK-43088] [SC-128403][SQL] Respect RequiresDistributionAndOrdering in CTAS/RTAS

  • [SPARK-43234] [SC-129192][CONNECT][PYTHON] Migrate ValueError from Conect DataFrame into error class

  • [SPARK-43212] [SC-129187][SS][PYTHON] Migrate Structured Streaming errors into error class

  • [SPARK-43239] [SC-129186][PS] Remove null_counts from info()

  • [SPARK-43190] [SC-128930][SQL] ListQuery.childOutput should be consistent with child output

  • [SPARK-43191] [SC-128924][CORE] Replace reflection w/ direct calling for Hadoop CallerContext

  • [SPARK-43193] [SC-129042][SS] Remove workaround for HADOOP-12074

  • [SPARK-42657] [SC-128621][CONNECT] Support to find and transfer client-side REPL classfiles to server as artifacts

  • [SPARK-43098] [SC-77059][SQL] Fix correctness COUNT bug when scalar subquery has group by clause

  • [SPARK-43213] [SC-129062][PYTHON] Add DataFrame.offset to vanilla PySpark

  • [SPARK-42982] [SC-128400][CONNECT][PYTHON] Fix createDataFrame to respect the given schema ddl

  • [SPARK-43124] [SC-129011][SQL] Dataset.show projects CommandResults locally

  • [SPARK-42998] [SC-127422][CONNECT][PYTHON] Fix DataFrame.collect with null struct

  • [SPARK-41498] [SC-125343]Revert ” Propagate metadata through Union”

  • [SPARK-42960] [SC-129010] [CONNECT] [SS] Add await_termination() and exception() API for Streaming Query in Python

  • [SPARK-42552] [SC-128824][SQL] Correct the two-stage parsing strategy of antlr parser

  • [SPARK-43207] [SC-128937][CONNECT] Add helper functions to extract value from literal expression

  • [SPARK-43186] [SC-128841][SQL][HIVE] Remove workaround for FileSinkDesc

  • [SPARK-43107] [SC-128533][SQL] Coalesce buckets in join applied on broadcast join stream side

  • [SPARK-43195] [SC-128922][CORE] Remove unnecessary serializable wrapper in HadoopFSUtils

  • [SPARK-43137] [SC-128828][SQL] Improve ArrayInsert if the position is foldable and positive.

  • [SPARK-37829] [SC-128827][SQL] Dataframe.joinWith outer-join should return a null value for unmatched row

  • [SPARK-43042] [SC-128602] [SS] [Connect] Add table() API support for DataStreamReader

  • [SPARK-43153] [SC-128753][CONNECT] Skip Spark execution when the dataframe is local

  • [SPARK-43064] [SC-128496][SQL] Spark SQL CLI SQL tab should only show once statement once

  • [SPARK-43126] [SC-128447][SQL] Mark two Hive UDF expressions as stateful

  • [SPARK-43111] [SC-128750][PS][CONNECT][PYTHON] Merge nested if statements into single if statements

  • [SPARK-43113] [SC-128749][SQL] Evaluate stream-side variables when generating code for a bound condition

  • [SPARK-42895] [SC-127258][CONNECT] Improve error messages for stopped Spark sessions

  • [SPARK-42884] [SC-126662][CONNECT] Add Ammonite REPL integration

  • [SPARK-43168] [SC-128674][SQL] Remove get PhysicalDataType method from Datatype class

  • [SPARK-43121] [SC-128455][SQL] Use BytesWritable.copyBytes instead of manual copy in `HiveInspectors

  • [SPARK-42916] [SC-128389][SQL] JDBCTableCatalog Keeps Char/Varchar meta on the read-side

  • [SPARK-43050] [SC-128550][SQL] Fix construct aggregate expressions by replacing grouping functions

  • [SPARK-43095] [SC-128549][SQL] Avoid Once strategy’s idempotence is broken for batch: Infer Filters

  • [SPARK-43130] [SC-128597][SQL] Move InternalType to PhysicalDataType

  • [SPARK-43105] [SC-128456][CONNECT] Abbreviate Bytes and Strings in proto message

  • [SPARK-43099] [SC-128596][SQL] Use getName instead of getCanonicalName to get builder class name when registering udf to FunctionRegistry

  • [SPARK-42994] [SC-128586][ML][CONNECT] PyTorch Distributor support Local Mode

  • [SPARK-42859] Revert “[SC-127935][CONNECT][PS] Basic support for pandas API on Spark Connect”

  • [SPARK-43021] [SC-128472][SQL] CoalesceBucketsInJoin not work when using AQE

  • [SPARK-43125] [SC-128477][CONNECT] Fix Connect Server Can’t Handle Exception With Null Message

  • [SPARK-43147] [SC-128594] fix flake8 lint for local check

  • [SPARK-43031] [SC-128360] [SS] [Connect] Enable unit test and doctest for streaming

  • [SPARK-43039] [LC-67] Support custom fields in the file source _metadata column.

  • [SPARK-43120] [SC-128407][SS] Add support for tracking pinned blocks memory usage for RocksDB state store

  • [SPARK-43110] [SC-128381][SQL] Move asIntegral to PhysicalDataType

  • [SPARK-43118] [SC-128398][SS] Remove unnecessary assert for UninterruptibleThread in KafkaMicroBatchStream

  • [SPARK-43055] [SC-128331][CONNECT][PYTHON] Support duplicated nested field names

  • [SPARK-42437] [SC-128339][PYTHON][CONNECT] PySpark catalog.cacheTable will allow to specify storage level

  • [SPARK-42985] [SC-128332][CONNECT][PYTHON] Fix createDataFrame to respect the SQL configs

  • [SPARK-39696] [SC-127830][CORE] Fix data race in access to TaskMetrics.externalAccums

  • [SPARK-43103] [SC-128335][SQL] Moving Integral to PhysicalDataType

  • [SPARK-42741] [SC-125547][SQL] Do not unwrap casts in binary comparison when literal is null

  • [SPARK-43057] [SC-127948][CONNECT][PYTHON] Migrate Spark Connect Column errors into error class

  • [SPARK-42859] [SC-127935][CONNECT][PS] Basic support for pandas API on Spark Connect

  • [SPARK-43013] [SC-127773][PYTHON] Migrate ValueError from DataFrame into PySparkValueError.

  • [SPARK-43089] [SC-128051][CONNECT] Redact debug string in UI

  • [SPARK-43028] [SC-128070][SQL] Add error class SQL_CONF_NOT_FOUND

  • [SPARK-42999] [SC-127842][Connect] Dataset#foreach, foreachPartition

  • [SPARK-43066] [SC-127937][SQL] Add test for dropDuplicates in JavaDatasetSuite

  • [SPARK-43075] [SC-127939][CONNECT] Change gRPC to grpcio when it is not installed.

  • [SPARK-42953] [SC-127809][Connect] Typed filter, map, flatMap, mapPartitions

  • [SPARK-42597] [SC-125506][SQL] Support unwrap date type to timestamp type

  • [SPARK-42931] [SC-127933][SS] Introduce dropDuplicatesWithinWatermark

  • [SPARK-43073] [SC-127943][CONNECT] Add proto data types constants

  • [SPARK-43077] [SC-128050][SQL] Improve the error message of UNRECOGNIZED_SQL_TYPE

  • [SPARK-42951] [SC-128030][SS][Connect] DataStreamReader APIs

  • [SPARK-43049] [SC-127846][SQL] Use CLOB instead of VARCHAR(255) for StringType for Oracle JDBC

  • [SPARK-43018] [SC-127762][SQL] Fix bug for INSERT commands with timestamp literals

  • [SPARK-42855] [SC-127722][SQL] Use runtime null checks in TableOutputResolver

  • [SPARK-43030] [SC-127847][SQL] Deduplicate relations with metadata columns

  • [SPARK-42993] [SC-127829][ML][CONNECT] Make PyTorch Distributor compatible with Spark Connect

  • [SPARK-43058] [SC-128072][SQL] Move Numeric and Fractional to PhysicalDataType

  • [SPARK-43056] [SC-127946][SS] RocksDB state store commit should continue background work only if its paused

  • [SPARK-43059] [SC-127947][CONNECT][PYTHON] Migrate TypeError from DataFrame(Reader|Writer) into error class

  • [SPARK-43071] [SC-128018][SQL] Support SELECT DEFAULT with ORDER BY, LIMIT, OFFSET for INSERT source relation

  • [SPARK-43061] [SC-127956][CORE][SQL] Introduce PartitionEvaluator for SQL operator execution

  • [SPARK-43067] [SC-127938][SS] Correct the location of error class resource file in Kafka connector

  • [SPARK-43019] [SC-127844][SQL] Move Ordering to PhysicalDataType

  • [SPARK-43010] [SC-127759][PYTHON] Migrate Column errors into error class

  • [SPARK-42840] [SC-127782][SQL] Change _LEGACY_ERROR_TEMP_2004 error to internal error

  • [SPARK-43041] [SC-127765][SQL] Restore constructors of exceptions for compatibility in connector API

  • [SPARK-42939] [SC-127761][SS][CONNECT] Core streaming Python API for Spark Connect

  • [SPARK-42844] [SC-127766][SQL] Update the error class _LEGACY_ERROR_TEMP_2008 to INVALID_URL

  • [SPARK-42316] [SC-127720][SQL] Assign name to LEGACYERROR_TEMP_2044

  • [SPARK-42995] [SC-127723][CONNECT][PYTHON] Migrate Spark Connect DataFrame errors into error class

  • [SPARK-42983] [SC-127717][CONNECT][PYTHON] Fix createDataFrame to handle 0-dim numpy array properly

  • [SPARK-42955] [SC-127476][SQL] Skip classifyException and wrap AnalysisException for SparkThrowable

  • [SPARK-42949] [SC-127255][SQL] Simplify code for NAAJ

  • [SPARK-43011] [SC-127577][SQL] array_insert should fail with 0 index

  • [SPARK-42974] [SC-127487][CORE] Restore Utils.createTempDir to use the ShutdownHookManager and clean up JavaUtils.createTempDir method.

  • [SPARK-42964] [SC-127585][SQL] PosgresDialect ‘42P07’ also means table already exists

  • [SPARK-42978] [SC-127351][SQL] Derby&PG: RENAME cannot qualify a new-table-Name with a schema-Name

  • [SPARK-37980] [SC-127668][SQL] Access row_index via _metadata if possible in tests

  • [SPARK-42655] [SC-127591][SQL] Incorrect ambiguous column reference error

  • [SPARK-43009] [SC-127596][SQL] Parameterized sql() with Any constants

  • [SPARK-43026] [SC-127590][SQL] Apply AQE with non-exchange table cache

  • [SPARK-42963] [SC-127576][SQL] Extend SparkSessionExtensions to inject rules into AQE query stage optimizer

  • [SPARK-42918] [SC-127357] Generalize handling of metadata attributes in FileSourceStrategy

  • [SPARK-42806] [SC-127452][SPARK-42811][CONNECT] Add Catalog support

  • [SPARK-42997] [SC-127535][SQL] TableOutputResolver must use correct column paths in error messages for arrays and maps

  • [SPARK-43006] [SC-127486][PYSPARK] Fix typo in StorageLevel eq()

  • [SPARK-43005] [SC-127485][PYSPARK] Fix typo in pyspark/pandas/config.py

  • [SPARK-43004] [SC-127457][CORE] Fix typo in ResourceRequest.equals()

  • [SPARK-42907] [SC-126984][CONNECT][PYTHON] Implement Avro functions

  • [SPARK-42979] [SC-127272][SQL] Define literal constructors as keywords

  • [SPARK-42946] [SC-127252][SQL] Redact sensitive data which is nested by variable substitution

  • [SPARK-42952] [SC-127260][SQL] Simplify the parameter of analyzer rule PreprocessTableCreation and DataSourceAnalysis

  • [SPARK-42683] [LC-75] Automatically rename conflicting metadata columns

  • [SPARK-42853] [SC-126101][FollowUp] Fix conflicts

  • [SPARK-42929] [SC-126748][CONNECT] make mapInPandas / mapInArrow support “is_barrier”

  • [SPARK-42968] [SC-127271][SS] Add option to skip commit coordinator as part of StreamingWrite API for DSv2 sources/sinks

  • [SPARK-42954] [SC-127261][PYTHON][CONNECT] Add YearMonthIntervalType to PySpark and Spark Connect Python Client

  • [SPARK-41359] [SC-127256][SQL] Use PhysicalDataType instead of DataType in UnsafeRow

  • [SPARK-42873] [SC-127262][SQL] Define Spark SQL types as keywords

  • [SPARK-42808] [SC-126302][CORE] Avoid getting availableProcessors every time in MapOutputTrackerMaster#getStatistics

  • [SPARK-42937] [SC-126880][SQL] PlanSubqueries should set InSubqueryExec#shouldBroadcast to true

  • [SPARK-42896] [SC-126729][SQL][PYTHON] Make mapInPandas / mapInArrow support barrier mode execution

  • [SPARK-42874] [SC-126442][SQL] Enable new golden file test framework for analysis for all input files

  • [SPARK-42922] [SC-126850][SQL] Move from Random to SecureRandom

  • [SPARK-42753] [SC-126369] ReusedExchange refers to non-existent nodes

  • [SPARK-40822] [SC-126274][SQL] Stable derived column aliases

  • [SPARK-42908] [SC-126856][PYTHON] Raise RuntimeError when SparkContext is required but not initialized

  • [SPARK-42779] [SC-126042][SQL] Allow V2 writes to indicate advisory shuffle partition size

  • [SPARK-42914] [SC-126727][PYTHON] Reuse transformUnregisteredFunction for DistributedSequenceID.

  • [SPARK-42878] [SC-126882][CONNECT] The table API in DataFrameReader could also accept options

  • [SPARK-42927] [SC-126883][CORE] Change the access scope of o.a.spark.util.Iterators#size to private[util]

  • [SPARK-42943] [SC-126879][SQL] Use LONGTEXT instead of TEXT for StringType for effective length

  • [SPARK-37677] [SC-126855][CORE] Unzip could keep file permissions

  • [SPARK-42891] [13.x][SC-126458][CONNECT][PYTHON] Implement CoGrouped Map API

  • [SPARK-41876] [SC-126849][CONNECT][PYTHON] Implement DataFrame.toLocalIterator

  • [SPARK-42930] [SC-126761][CORE][SQL] Change the access scope of ProtobufSerDe related implementations to private[protobuf]

  • [SPARK-42819] [SC-125879][SS] Add support for setting max_write_buffer_number and write_buffer_size for RocksDB used in streaming

  • [SPARK-42924] [SC-126737][SQL][CONNECT][PYTHON] Clarify the comment of parameterized SQL args

  • [SPARK-42748] [SC-126455][CONNECT] Server-side Artifact Management

  • [SPARK-42816] [SC-126365][CONNECT] Support Max Message size up to 128MB

  • [SPARK-42850] [SC-126109][SQL] Remove duplicated rule CombineFilters in Optimizer

  • [SPARK-42662] [SC-126355][CONNECT][PS] Add proto message for pandas API on Spark default index

  • [SPARK-42720] [SC-126136][PS][SQL] Uses expression for distributed-sequence default index instead of plan

  • [SPARK-42790] [SC-126174][SQL] Abstract the excluded method for better test for JDBC docker tests.

  • [SPARK-42900] [SC-126473][CONNECT][PYTHON] Fix createDataFrame to respect inference and column names

  • [SPARK-42917] [SC-126657][SQL] Correct getUpdateColumnNullabilityQuery for DerbyDialect

  • [SPARK-42684] [SC-125157][SQL] v2 catalog should not allow column default value by default

  • [SPARK-42861] [SC-126635][SQL] Use private[sql] instead of protected[sql] to avoid generating API doc

  • [SPARK-42920] [SC-126728][CONNECT][PYTHON] Enable tests for UDF with UDT

  • [SPARK-42791] [SC-126617][SQL] Create a new golden file test framework for analysis

  • [SPARK-42911] [SC-126652][PYTHON] Introduce more basic exceptions

  • [SPARK-42904] [SC-126634][SQL] Char/Varchar Support for JDBC Catalog

  • [SPARK-42901] [SC-126459][CONNECT][PYTHON] Move StorageLevel into a separate file to avoid potential file recursively imports

  • [SPARK-42894] [SC-126451][CONNECT] Support cache/persist/unpersist/storageLevel for Spark connect jvm client

  • [SPARK-42792] [SC-125852][SS] Add support for WRITE_FLUSH_BYTES for RocksDB used in streaming stateful operators

  • [SPARK-41233] [SC-126441][CONNECT][PYTHON] Add array_prepend to Spark Connect Python client

  • [SPARK-42681] [SC-125149][SQL] Relax ordering constraint for ALTER TABLE ADD|REPLACE column descriptor

  • [SPARK-42889] [SC-126367][CONNECT][PYTHON] Implement cache, persist, unpersist, and storageLevel

  • [SPARK-42824] [SC-125985][CONNECT][PYTHON] Provide a clear error message for unsupported JVM attributes

  • [SPARK-42340] [SC-126131][CONNECT][PYTHON] Implement Grouped Map API

  • [SPARK-42892] [SC-126454][SQL] Move sameType and relevant methods out of DataType

  • [SPARK-42827] [SC-126126][CONNECT] Support functions#array_prepend for Scala connect client

  • [SPARK-42823] [SC-125987][SQL] spark-sql shell supports multipart namespaces for initialization

  • [SPARK-42817] [SC-125960][CORE] Logging the shuffle service name once in ApplicationMaster

  • [SPARK-42786] [SC-126438][Connect] Typed Select

  • [SPARK-42800] [SC-125868][CONNECT][PYTHON][ML] Implement ml function {array_to_vector, vector_to_array}

  • [SPARK-42052] [SC-126439][SQL] Codegen Support for HiveSimpleUDF

  • [SPARK-41233] [SC-126110][SQL][PYTHON] Add array_prepend function

  • [SPARK-42864] [SC-126268][ML][3.4] Make IsotonicRegression.PointsAccumulator private

  • [SPARK-42876] [SC-126281][SQL] DataType’s physicalDataType should be private[sql]

  • [SPARK-42101] [SC-125437][SQL] Make AQE support InMemoryTableScanExec

  • [SPARK-41290] [SC-124030][SQL] Support GENERATED ALWAYS AS expressions for columns in create/replace table statements

  • [SPARK-42870] [SC-126220][CONNECT] Move toCatalystValue to connect-common

  • [SPARK-42247] [SC-126107][CONNECT][PYTHON] Fix UserDefinedFunction to have returnType

  • [SPARK-42875] [SC-126258][CONNECT][PYTHON] Fix toPandas to handle timezone and map types properly

  • [SPARK-42757] [SC-125626][CONNECT] Implement textFile for DataFrameReader

  • [SPARK-42803] [SC-126081][CORE][SQL][ML] Use getParameterCount function instead of getParameterTypes.length

  • [SPARK-42833] [SC-126043][SQL] Refactor applyExtensions in SparkSession

  • [SPARK-41765] Revert “[SC-123550][SQL] Pull out v1 write metrics…

  • [SPARK-42848] [SC-126105][CONNECT][PYTHON] Implement DataFrame.registerTempTable

  • [SPARK-42020] [SC-126103][CONNECT][PYTHON] Support UserDefinedType in Spark Connect

  • [SPARK-42818] [SC-125861][CONNECT][PYTHON] Implement DataFrameReader/Writer.jdbc

  • [SPARK-42812] [SC-125867][CONNECT] Add client_type to AddArtifactsRequest protobuf message

  • [SPARK-42772] [SC-125860][SQL] Change the default value of JDBC options about push down to true

  • [SPARK-42771] [SC-125855][SQL] Refactor HiveGenericUDF

  • [SPARK-25050] [SC-123839][SQL] Avro: writing complex unions

  • [SPARK-42765] [SC-125850][CONNECT][PYTHON] Enable importing pandas_udf from pyspark.sql.connect.functions

  • [SPARK-42719] [SC-125225][CORE] MapOutputTracker#getMapLocation should respect spark.shuffle.reduceLocality.enabled

  • [SPARK-42480] [SC-125173][SQL] Improve the performance of drop partitions

  • [SPARK-42689] [SC-125195][CORE][SHUFFLE] Allow ShuffleDriverComponent to declare if shuffle data is reliably stored

  • [SPARK-42726] [SC-125279][CONNECT][PYTHON] Implement DataFrame.mapInArrow

  • [SPARK-41765] [SC-123550][SQL] Pull out v1 write metrics to WriteFiles

  • [SPARK-41171] [SC-124191][SQL] Infer and push down window limit through window if partitionSpec is empty

  • [SPARK-42686] [SC-125292][CORE] Defer formatting for debug messages in TaskMemoryManager

  • [SPARK-42756] [SC-125443][CONNECT][PYTHON] Helper function to convert proto literal to value in Python Client

  • [SPARK-42793] [SC-125627][CONNECT] connect module requires build_profile_flags

  • [SPARK-42701] [SC-125192][SQL] Add the try_aes_decrypt() function

  • [SPARK-42679] [SC-125438][CONNECT][PYTHON] createDataFrame doesn’t work with non-nullable schema

  • [SPARK-42733] [SC-125542][CONNECT][Followup] Write without path or table

  • [SPARK-42777] [SC-125525][SQL] Support converting TimestampNTZ catalog stats to plan stats

  • [SPARK-42770] [SC-125558][CONNECT] Add truncatedTo(ChronoUnit.MICROS) to make SQLImplicitsTestSuite in Java 17 daily test GA task pass

  • [SPARK-42752] [SC-125550][PYSPARK][SQL] Make PySpark exceptions printable during initialization

  • [SPARK-42732] [SC-125544][PYSPARK][CONNECT] Support spark connect session getActiveSession method

  • [SPARK-42755] [SC-125442][CONNECT] Factor literal value conversion out to connect-common

  • [SPARK-42747] [SC-125399][ML] Fix incorrect internal status of LoR and AFT

  • [SPARK-42740] [SC-125439][SQL] Fix the bug that pushdown offset or paging is invalid for some built-in dialect

  • [SPARK-42745] [SC-125332][SQL] Improved AliasAwareOutputExpression works with DSv2

  • [SPARK-42743] [SC-125330][SQL] Support analyze TimestampNTZ columns

  • [SPARK-42721] [SC-125371][CONNECT] RPC logging interceptor

  • [SPARK-42691] [SC-125397][CONNECT][PYTHON] Implement Dataset.semanticHash

  • [SPARK-42688] [SC-124922][CONNECT] Rename Connect proto Request client_id to session_id

  • [SPARK-42310] [SC-122792][SQL] Assign name to LEGACYERROR_TEMP_1289

  • [SPARK-42685] [SC-125339][CORE] Optimize Utils.bytesToString routines

  • [SPARK-42725] [SC-125296][CONNECT][PYTHON] Make LiteralExpression support array params

  • [SPARK-42702] [SC-125293][SPARK-42623][SQL] Support parameterized query in subquery and CTE

  • [SPARK-42697] [SC-125189][WEBUI] Fix /api/v1/applications to return total uptime instead of 0 for the duration field

  • [SPARK-42733] [SC-125278][CONNECT][PYTHON] Fix DataFrameWriter.save to work without path parameter

  • [SPARK-42376] [SC-124928][SS] Introduce watermark propagation among operators

  • [SPARK-42710] [SC-125205][CONNECT][PYTHON] Rename FrameMap proto to MapPartitions

  • [SPARK-37099] [SC-123542][SQL] Introduce the group limit of Window for rank-based filter to optimize top-k computation

  • [SPARK-42630] [SC-125207][CONNECT][PYTHON] Introduce UnparsedDataType and delay parsing DDL string until SparkConnectClient is available

  • [SPARK-42690] [SC-125193][CONNECT] Implement CSV/JSON parsing functions for Scala client

  • [SPARK-42709] [SC-125172][PYTHON] Remove the assumption of __file__ being available

  • [SPARK-42318] [SC-122648][SPARK-42319][SQL] Assign name to LEGACYERROR_TEMP_(2123|2125)

  • [SPARK-42723] [SC-125183][SQL] Support parser data type json “timestamp_ltz” as TimestampType

  • [SPARK-42722] [SC-125175][CONNECT][PYTHON] Python Connect def schema() should not cache the schema

  • [SPARK-42643] [SC-125152][CONNECT][PYTHON] Register Java (aggregate) user-defined functions

  • [SPARK-42656] [SC-125177][CONNECT][Followup] Fix the spark-connect script

  • [SPARK-41516] [SC-123899] [SQL] Allow jdbc dialects to override the query used to create a table

  • [SPARK-41725] [SC-124396][CONNECT] Eager Execution of DF.sql()

  • [SPARK-42687] [SC-124896][SS] Better error message for the unsupport pivot operation in Streaming

  • [SPARK-42676] [SC-124809][SS] Write temp checkpoints for streaming queries to local filesystem even if default FS is set differently

  • [SPARK-42303] [SC-122644][SQL] Assign name to LEGACYERROR_TEMP_1326

  • [SPARK-42553] [SC-124560][SQL] Ensure at least one time unit after “interval”

  • [SPARK-42649] [SC-124576][CORE] Remove the standard Apache License header from the top of third-party source files

  • [SPARK-42611] [SC-124395][SQL] Insert char/varchar length checks for inner fields during resolution

  • [SPARK-42419] [SC-124019][CONNECT][PYTHON] Migrate into error framework for Spark Connect Column API.

  • [SPARK-42637] [SC-124522][CONNECT] Add SparkSession.stop()

  • [SPARK-42647] [SC-124647][PYTHON] Change alias for numpy deprecated and removed types

  • [SPARK-42616] [SC-124389][SQL] SparkSQLCLIDriver shall only close started hive sessionState

  • [SPARK-42593] [SC-124405][PS] Deprecate & remove the APIs that will be removed in pandas 2.0.

  • [SPARK-41870] [SC-124402][CONNECT][PYTHON] Fix createDataFrame to handle duplicated column names

  • [SPARK-42569] [SC-124379][CONNECT] Throw exceptions for unsupported session API

  • [SPARK-42631] [SC-124526][CONNECT] Support custom extensions in Scala client

  • [SPARK-41868] [SC-124387][CONNECT][PYTHON] Fix createDataFrame to support durations

  • [SPARK-42572] [SC-124171][SQL][SS] Fix behavior for StateStoreProvider.validateStateRowFormat

System environment

  • Operating System: Ubuntu 22.04.2 LTS

  • Java: Zulu 8.70.0.23-CA-linux64

  • Scala: 2.12.15

  • Python: 3.10.12

  • R: 4.2.2

  • Delta Lake: 2.4.0

Installed Python libraries

Library

Version

Library

Version

Library

Version

appdirs

1.4.4

argon2-cffi

21.3.0

argon2-cffi-bindings

21.2.0

asttokens

2.2.1

attrs

21.4.0

backcall

0.2.0

beautifulsoup4

4.11.1

black

22.6.0

bleach

4.1.0

blinker

1.4

boto3

1.24.28

botocore

1.27.28

certifi

2022.9.14

cffi

1.15.1

chardet

4.0.0

charset-normalizer

2.0.4

click

8.0.4

cryptography

37.0.1

cycler

0.11.0

Cython

0.29.32

dbus-python

1.2.18

debugpy

1.5.1

decorator

5.1.1

defusedxml

0.7.1

distlib

0.3.6

docstring-to-markdown

0.12

entrypoints

0.4

executing

1.2.0

facets-overview

1.0.3

fastjsonschema

2.16.3

filelock

3.12.0

fonttools

4.25.0

googleapis-common-protos

1.56.4

grpcio

1.48.1

grpcio-status

1.48.1

httplib2

0.20.2

idna

3.3

importlib-metadata

4.6.4

ipykernel

6.17.1

ipython

8.10.0

ipython-genutils

0.2.0

ipywidgets

7.7.2

jedi

0.18.1

jeepney

0.7.1

Jinja2

2.11.3

jmespath

0.10.0

joblib

1.2.0

jsonschema

4.16.0

jupyter-client

7.3.4

jupyter_core

4.11.2

jupyterlab-pygments

0.1.2

jupyterlab-widgets

1.0.0

keyring

23.5.0

kiwisolver

1.4.2

launchpadlib

1.10.16

lazr.restfulclient

0.14.4

lazr.uri

1.0.6

MarkupSafe

2.0.1

matplotlib

3.5.2

matplotlib-inline

0.1.6

mccabe

0.7.0

mistune

0.8.4

more-itertools

8.10.0

mypy-extensions

0.4.3

nbclient

0.5.13

nbconvert

6.4.4

nbformat

5.5.0

nest-asyncio

1.5.5

nodeenv

1.7.0

notebook

6.4.12

numpy

1.21.5

oauthlib

3.2.0

packaging

21.3

pandas

1.4.4

pandocfilters

1.5.0

parso

0.8.3

pathspec

0.9.0

patsy

0.5.2

pexpect

4.8.0

pickleshare

0.7.5

Pillow

9.2.0

pip

22.2.2

platformdirs

2.5.2

plotly

5.9.0

pluggy

1.0.0

prometheus-client

0.14.1

prompt-toolkit

3.0.36

protobuf

3.19.4

psutil

5.9.0

psycopg2

2.9.3

ptyprocess

0.7.0

pure-eval

0.2.2

pyarrow

8.0.0

pycparser

2.21

pydantic

1.10.6

pyflakes

3.0.1

Pygments

2.11.2

PyGObject

3.42.1

PyJWT

2.3.0

pyodbc

4.0.32

pyparsing

3.0.9

pyright

1.1.294

pyrsistent

0.18.0

python-dateutil

2.8.2

python-lsp-jsonrpc

1.0.0

python-lsp-server

1.7.1

pytoolconfig

1.2.2

pytz

2022.1

pyzmq

23.2.0

requests

2.28.1

rope

1.7.0

s3transfer

0.6.0

scikit-learn

1.1.1

scipy

1.9.1

seaborn

0.11.2

SecretStorage

3.3.1

Send2Trash

1.8.0

setuptools

63.4.1

six

1.16.0

soupsieve

2.3.1

ssh-import-id

5.11

stack-data

0.6.2

statsmodels

0.13.2

tenacity

8.1.0

terminado

0.13.1

testpath

0.6.0

threadpoolctl

2.2.0

tokenize-rt

4.2.1

tomli

2.0.1

tornado

6.1

traitlets

5.1.1

typing_extensions

4.3.0

ujson

5.4.0

unattended-upgrades

0.1

urllib3

1.26.11

virtualenv

20.16.3

wadllib

1.3.6

wcwidth

0.2.5

webencodings

0.5.1

whatthepatch

1.0.2

wheel

0.37.1

widgetsnbextension

3.6.1

yapf

0.31.0

zipp

1.0.0

Installed R libraries

R libraries are installed from the Microsoft CRAN snapshot on 2023-02-10.

Library

Version

Library

Version

Library

Version

arrow

10.0.1

askpass

1.1

assertthat

0.2.1

backports

1.4.1

base

4.2.2

base64enc

0.1-3

bit

4.0.5

bit64

4.0.5

blob

1.2.3

boot

1.3-28

brew

1.0-8

brio

1.1.3

broom

1.0.3

bslib

0.4.2

cachem

1.0.6

callr

3.7.3

caret

6.0-93

cellranger

1.1.0

chron

2.3-59

class

7.3-21

cli

3.6.0

clipr

0.8.0

clock

0.6.1

cluster

2.1.4

codetools

0.2-19

colorspace

2.1-0

commonmark

1.8.1

compiler

4.2.2

config

0.3.1

cpp11

0.4.3

crayon

1.5.2

credentials

1.3.2

curl

5.0.0

data.table

1.14.6

datasets

4.2.2

DBI

1.1.3

dbplyr

2.3.0

desc

1.4.2

devtools

2.4.5

diffobj

0.3.5

digest

0.6.31

downlit

0.4.2

dplyr

1.1.0

dtplyr

1.2.2

e1071

1.7-13

ellipsis

0.3.2

evaluate

0.20

fansi

1.0.4

farver

2.1.1

fastmap

1.1.0

fontawesome

0.5.0

forcats

1.0.0

foreach

1.5.2

foreign

0.8-82

forge

0.2.0

fs

1.6.1

future

1.31.0

future.apply

1.10.0

gargle

1.3.0

generics

0.1.3

gert

1.9.2

ggplot2

3.4.0

gh

1.3.1

gitcreds

0.1.2

glmnet

4.1-6

globals

0.16.2

glue

1.6.2

googledrive

2.0.0

googlesheets4

1.0.1

gower

1.0.1

graphics

4.2.2

grDevices

4.2.2

grid

4.2.2

gridExtra

2.3

gsubfn

0.7

gtable

0.3.1

hardhat

1.2.0

haven

2.5.1

highr

0.10

hms

1.1.2

htmltools

0.5.4

htmlwidgets

1.6.1

httpuv

1.6.8

httr

1.4.4

ids

1.0.1

ini

0.3.1

ipred

0.9-13

isoband

0.2.7

iterators

1.0.14

jquerylib

0.1.4

jsonlite

1.8.4

KernSmooth

2.23-20

knitr

1.42

labeling

0.4.2

later

1.3.0

lattice

0.20-45

lava

1.7.1

lifecycle

1.0.3

listenv

0.9.0

lubridate

1.9.1

magrittr

2.0.3

markdown

1.5

MASS

7.3-58.2

Matrix

1.5-1

memoise

2.0.1

methods

4.2.2

mgcv

1.8-41

mime

0.12

miniUI

0.1.1.1

ModelMetrics

1.2.2.2

modelr

0.1.10

munsell

0.5.0

nlme

3.1-162

nnet

7.3-18

numDeriv

2016.8-1.1

openssl

2.0.5

parallel

4.2.2

parallelly

1.34.0

pillar

1.8.1

pkgbuild

1.4.0

pkgconfig

2.0.3

pkgdown

2.0.7

pkgload

1.3.2

plogr

0.2.0

plyr

1.8.8

praise

1.0.0

prettyunits

1.1.1

pROC

1.18.0

processx

3.8.0

prodlim

2019.11.13

profvis

0.3.7

progress

1.2.2

progressr

0.13.0

promises

1.2.0.1

proto

1.0.0

proxy

0.4-27

ps

1.7.2

purrr

1.0.1

r2d3

0.2.6

R6

2.5.1

ragg

1.2.5

randomForest

4.7-1.1

rappdirs

0.3.3

rcmdcheck

1.4.0

RColorBrewer

1.1-3

Rcpp

1.0.10

RcppEigen

0.3.3.9.3

readr

2.1.3

readxl

1.4.2

recipes

1.0.4

rematch

1.0.1

rematch2

2.1.2

remotes

2.4.2

reprex

2.0.2

reshape2

1.4.4

rlang

1.0.6

rmarkdown

2.20

RODBC

1.3-20

roxygen2

7.2.3

rpart

4.1.19

rprojroot

2.0.3

Rserve

1.8-12

RSQLite

2.2.20

rstudioapi

0.14

rversions

2.1.2

rvest

1.0.3

sass

0.4.5

scales

1.2.1

selectr

0.4-2

sessioninfo

1.2.2

shape

1.4.6

shiny

1.7.4

sourcetools

0.1.7-1

sparklyr

1.7.9

SparkR

3.4.0

spatial

7.3-15

splines

4.2.2

sqldf

0.4-11

SQUAREM

2021.1

stats

4.2.2

stats4

4.2.2

stringi

1.7.12

stringr

1.5.0

survival

3.5-3

sys

3.4.1

systemfonts

1.0.4

tcltk

4.2.2

testthat

3.1.6

textshaping

0.3.6

tibble

3.1.8

tidyr

1.3.0

tidyselect

1.2.0

tidyverse

1.3.2

timechange

0.2.0

timeDate

4022.108

tinytex

0.44

tools

4.2.2

tzdb

0.3.0

urlchecker

1.0.1

usethis

2.1.6

utf8

1.2.3

utils

4.2.2

uuid

1.1-0

vctrs

0.5.2

viridisLite

0.4.1

vroom

1.6.1

waldo

0.4.0

whisker

0.4.1

withr

2.5.0

xfun

0.37

xml2

1.3.3

xopen

1.0.0

xtable

1.8-4

yaml

2.3.7

zip

2.2.2

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID

Artifact ID

Version

antlr

antlr

2.7.7

com.amazonaws

amazon-kinesis-client

1.12.0

com.amazonaws

aws-java-sdk-autoscaling

1.12.390

com.amazonaws

aws-java-sdk-cloudformation

1.12.390

com.amazonaws

aws-java-sdk-cloudfront

1.12.390

com.amazonaws

aws-java-sdk-cloudhsm

1.12.390

com.amazonaws

aws-java-sdk-cloudsearch

1.12.390

com.amazonaws

aws-java-sdk-cloudtrail

1.12.390

com.amazonaws

aws-java-sdk-cloudwatch

1.12.390

com.amazonaws

aws-java-sdk-cloudwatchmetrics

1.12.390

com.amazonaws

aws-java-sdk-codedeploy

1.12.390

com.amazonaws

aws-java-sdk-cognitoidentity

1.12.390

com.amazonaws

aws-java-sdk-cognitosync

1.12.390

com.amazonaws

aws-java-sdk-config

1.12.390

com.amazonaws

aws-java-sdk-core

1.12.390

com.amazonaws

aws-java-sdk-datapipeline

1.12.390

com.amazonaws

aws-java-sdk-directconnect

1.12.390

com.amazonaws

aws-java-sdk-directory

1.12.390

com.amazonaws

aws-java-sdk-dynamodb

1.12.390

com.amazonaws

aws-java-sdk-ec2

1.12.390

com.amazonaws

aws-java-sdk-ecs

1.12.390

com.amazonaws

aws-java-sdk-efs

1.12.390

com.amazonaws

aws-java-sdk-elasticache

1.12.390

com.amazonaws

aws-java-sdk-elasticbeanstalk

1.12.390

com.amazonaws

aws-java-sdk-elasticloadbalancing

1.12.390

com.amazonaws

aws-java-sdk-elastictranscoder

1.12.390

com.amazonaws

aws-java-sdk-emr

1.12.390

com.amazonaws

aws-java-sdk-glacier

1.12.390

com.amazonaws

aws-java-sdk-glue

1.12.390

com.amazonaws

aws-java-sdk-iam

1.12.390

com.amazonaws

aws-java-sdk-importexport

1.12.390

com.amazonaws

aws-java-sdk-kinesis

1.12.390

com.amazonaws

aws-java-sdk-kms

1.12.390

com.amazonaws

aws-java-sdk-lambda

1.12.390

com.amazonaws

aws-java-sdk-logs

1.12.390

com.amazonaws

aws-java-sdk-machinelearning

1.12.390

com.amazonaws

aws-java-sdk-opsworks

1.12.390

com.amazonaws

aws-java-sdk-rds

1.12.390

com.amazonaws

aws-java-sdk-redshift

1.12.390

com.amazonaws

aws-java-sdk-route53

1.12.390

com.amazonaws

aws-java-sdk-s3

1.12.390

com.amazonaws

aws-java-sdk-ses

1.12.390

com.amazonaws

aws-java-sdk-simpledb

1.12.390

com.amazonaws

aws-java-sdk-simpleworkflow

1.12.390

com.amazonaws

aws-java-sdk-sns

1.12.390

com.amazonaws

aws-java-sdk-sqs

1.12.390

com.amazonaws

aws-java-sdk-ssm

1.12.390

com.amazonaws

aws-java-sdk-storagegateway

1.12.390

com.amazonaws

aws-java-sdk-sts

1.12.390

com.amazonaws

aws-java-sdk-support

1.12.390

com.amazonaws

aws-java-sdk-swf-libraries

1.11.22

com.amazonaws

aws-java-sdk-workspaces

1.12.390

com.amazonaws

jmespath-java

1.12.390

com.clearspring.analytics

stream

2.9.6

com.databricks

Rserve

1.8-3

com.databricks

jets3t

0.7.1-0

com.databricks.scalapb

compilerplugin_2.12

0.4.15-10

com.databricks.scalapb

scalapb-runtime_2.12

0.4.15-10

com.esotericsoftware

kryo-shaded

4.0.2

com.esotericsoftware

minlog

1.3.0

com.fasterxml

classmate

1.3.4

com.fasterxml.jackson.core

jackson-annotations

2.14.2

com.fasterxml.jackson.core

jackson-core

2.14.2

com.fasterxml.jackson.core

jackson-databind

2.14.2

com.fasterxml.jackson.dataformat

jackson-dataformat-cbor

2.14.2

com.fasterxml.jackson.datatype

jackson-datatype-joda

2.14.2

com.fasterxml.jackson.datatype

jackson-datatype-jsr310

2.13.4

com.fasterxml.jackson.module

jackson-module-paranamer

2.14.2

com.fasterxml.jackson.module

jackson-module-scala_2.12

2.14.2

com.github.ben-manes.caffeine

caffeine

2.9.3

com.github.fommil

jniloader

1.1

com.github.fommil.netlib

native_ref-java

1.1

com.github.fommil.netlib

native_ref-java

1.1-natives

com.github.fommil.netlib

native_system-java

1.1

com.github.fommil.netlib

native_system-java

1.1-natives

com.github.fommil.netlib

netlib-native_ref-linux-x86_64

1.1-natives

com.github.fommil.netlib

netlib-native_system-linux-x86_64

1.1-natives

com.github.luben

zstd-jni

1.5.2-5

com.github.wendykierp

JTransforms

3.1

com.google.code.findbugs

jsr305

3.0.0

com.google.code.gson

gson

2.8.9

com.google.crypto.tink

tink

1.7.0

com.google.errorprone

error_prone_annotations

2.10.0

com.google.flatbuffers

flatbuffers-java

1.12.0

com.google.guava

guava

15.0

com.google.protobuf

protobuf-java

2.6.1

com.h2database

h2

2.1.214

com.helger

profiler

1.1.1

com.jcraft

jsch

0.1.55

com.jolbox

bonecp

0.8.0.RELEASE

com.lihaoyi

sourcecode_2.12

0.1.9

com.microsoft.azure

azure-data-lake-store-sdk

2.3.9

com.microsoft.sqlserver

mssql-jdbc

11.2.2.jre8

com.ning

compress-lzf

1.1.2

com.sun.mail

javax.mail

1.5.2

com.sun.xml.bind

jaxb-core

2.2.11

com.sun.xml.bind

jaxb-impl

2.2.11

com.tdunning

json

1.8

com.thoughtworks.paranamer

paranamer

2.8

com.trueaccord.lenses

lenses_2.12

0.4.12

com.twitter

chill-java

0.10.0

com.twitter

chill_2.12

0.10.0

com.twitter

util-app_2.12

7.1.0

com.twitter

util-core_2.12

7.1.0

com.twitter

util-function_2.12

7.1.0

com.twitter

util-jvm_2.12

7.1.0

com.twitter

util-lint_2.12

7.1.0

com.twitter

util-registry_2.12

7.1.0

com.twitter

util-stats_2.12

7.1.0

com.typesafe

config

1.2.1

com.typesafe.scala-logging

scala-logging_2.12

3.7.2

com.uber

h3

3.7.0

com.univocity

univocity-parsers

2.9.1

com.zaxxer

HikariCP

4.0.3

commons-cli

commons-cli

1.5.0

commons-codec

commons-codec

1.15

commons-collections

commons-collections

3.2.2

commons-dbcp

commons-dbcp

1.4

commons-fileupload

commons-fileupload

1.5

commons-httpclient

commons-httpclient

3.1

commons-io

commons-io

2.11.0

commons-lang

commons-lang

2.6

commons-logging

commons-logging

1.1.3

commons-pool

commons-pool

1.5.4

dev.ludovic.netlib

arpack

3.0.3

dev.ludovic.netlib

blas

3.0.3

dev.ludovic.netlib

lapack

3.0.3

info.ganglia.gmetric4j

gmetric4j

1.0.10

io.airlift

aircompressor

0.21

io.delta

delta-sharing-spark_2.12

0.6.4

io.dropwizard.metrics

metrics-core

4.2.10

io.dropwizard.metrics

metrics-graphite

4.2.10

io.dropwizard.metrics

metrics-healthchecks

4.2.10

io.dropwizard.metrics

metrics-jetty9

4.2.10

io.dropwizard.metrics

metrics-jmx

4.2.10

io.dropwizard.metrics

metrics-json

4.2.10

io.dropwizard.metrics

metrics-jvm

4.2.10

io.dropwizard.metrics

metrics-servlets

4.2.10

io.netty

netty-all

4.1.87.Final

io.netty

netty-buffer

4.1.87.Final

io.netty

netty-codec

4.1.87.Final

io.netty

netty-codec-http

4.1.87.Final

io.netty

netty-codec-http2

4.1.87.Final

io.netty

netty-codec-socks

4.1.87.Final

io.netty

netty-common

4.1.87.Final

io.netty

netty-handler

4.1.87.Final

io.netty

netty-handler-proxy

4.1.87.Final

io.netty

netty-resolver

4.1.87.Final

io.netty

netty-transport

4.1.87.Final

io.netty

netty-transport-classes-epoll

4.1.87.Final

io.netty

netty-transport-classes-kqueue

4.1.87.Final

io.netty

netty-transport-native-epoll

4.1.87.Final

io.netty

netty-transport-native-epoll

4.1.87.Final-linux-aarch_64

io.netty

netty-transport-native-epoll

4.1.87.Final-linux-x86_64

io.netty

netty-transport-native-kqueue

4.1.87.Final-osx-aarch_64

io.netty

netty-transport-native-kqueue

4.1.87.Final-osx-x86_64

io.netty

netty-transport-native-unix-common

4.1.87.Final

io.prometheus

simpleclient

0.7.0

io.prometheus

simpleclient_common

0.7.0

io.prometheus

simpleclient_dropwizard

0.7.0

io.prometheus

simpleclient_pushgateway

0.7.0

io.prometheus

simpleclient_servlet

0.7.0

io.prometheus.jmx

collector

0.12.0

jakarta.annotation

jakarta.annotation-api

1.3.5

jakarta.servlet

jakarta.servlet-api

4.0.3

jakarta.validation

jakarta.validation-api

2.0.2

jakarta.ws.rs

jakarta.ws.rs-api

2.1.6

javax.activation

activation

1.1.1

javax.el

javax.el-api

2.2.4

javax.jdo

jdo-api

3.0.1

javax.transaction

jta

1.1

javax.transaction

transaction-api

1.1

javax.xml.bind

jaxb-api

2.2.11

javolution

javolution

5.5.1

jline

jline

2.14.6

joda-time

joda-time

2.12.1

ml.combust.mleap

mleap-databricks-runtime_2.12

v0.20.0-db2

net.java.dev.jna

jna

5.8.0

net.razorvine

pickle

1.3

net.sf.jpam

jpam

1.1

net.sf.opencsv

opencsv

2.3

net.sf.supercsv

super-csv

2.2.0

net.snowflake

snowflake-ingest-sdk

0.9.6

net.snowflake

snowflake-jdbc

3.13.22

net.sourceforge.f2j

arpack_combined_all

0.1

org.acplt.remotetea

remotetea-oncrpc

1.1.2

org.antlr

ST4

4.0.4

org.antlr

antlr-runtime

3.5.2

org.antlr

antlr4-runtime

4.9.3

org.antlr

stringtemplate

3.2.1

org.apache.ant

ant

1.9.16

org.apache.ant

ant-jsch

1.9.16

org.apache.ant

ant-launcher

1.9.16

org.apache.arrow

arrow-format

11.0.0

org.apache.arrow

arrow-memory-core

11.0.0

org.apache.arrow

arrow-memory-netty

11.0.0

org.apache.arrow

arrow-vector

11.0.0

org.apache.avro

avro

1.11.1

org.apache.avro

avro-ipc

1.11.1

org.apache.avro

avro-mapred

1.11.1

org.apache.commons

commons-collections4

4.4

org.apache.commons

commons-compress

1.21

org.apache.commons

commons-crypto

1.1.0

org.apache.commons

commons-lang3

3.12.0

org.apache.commons

commons-math3

3.6.1

org.apache.commons

commons-text

1.10.0

org.apache.curator

curator-client

2.13.0

org.apache.curator

curator-framework

2.13.0

org.apache.curator

curator-recipes

2.13.0

org.apache.datasketches

datasketches-java

3.1.0

org.apache.datasketches

datasketches-memory

2.0.0

org.apache.derby

derby

10.14.2.0

org.apache.hadoop

hadoop-client-runtime

3.3.4

org.apache.hive

hive-beeline

2.3.9

org.apache.hive

hive-cli

2.3.9

org.apache.hive

hive-jdbc

2.3.9

org.apache.hive

hive-llap-client

2.3.9

org.apache.hive

hive-llap-common

2.3.9

org.apache.hive

hive-serde

2.3.9

org.apache.hive

hive-shims

2.3.9

org.apache.hive

hive-storage-api

2.8.1

org.apache.hive.shims

hive-shims-0.23

2.3.9

org.apache.hive.shims

hive-shims-common

2.3.9

org.apache.hive.shims

hive-shims-scheduler

2.3.9

org.apache.httpcomponents

httpclient

4.5.14

org.apache.httpcomponents

httpcore

4.4.16

org.apache.ivy

ivy

2.5.1

org.apache.logging.log4j

log4j-1.2-api

2.19.0

org.apache.logging.log4j

log4j-api

2.19.0

org.apache.logging.log4j

log4j-core

2.19.0

org.apache.logging.log4j

log4j-slf4j2-impl

2.19.0

org.apache.mesos

mesos

1.11.0-shaded-protobuf

org.apache.orc

orc-core

1.8.3-shaded-protobuf

org.apache.orc

orc-mapreduce

1.8.3-shaded-protobuf

org.apache.orc

orc-shims

1.8.3

org.apache.thrift

libfb303

0.9.3

org.apache.thrift

libthrift

0.12.0

org.apache.xbean

xbean-asm9-shaded

4.22

org.apache.yetus

audience-annotations

0.13.0

org.apache.zookeeper

zookeeper

3.6.3

org.apache.zookeeper

zookeeper-jute

3.6.3

org.checkerframework

checker-qual

3.19.0

org.codehaus.jackson

jackson-core-asl

1.9.13

org.codehaus.jackson

jackson-mapper-asl

1.9.13

org.codehaus.janino

commons-compiler

3.0.16

org.codehaus.janino

janino

3.0.16

org.datanucleus

datanucleus-api-jdo

4.2.4

org.datanucleus

datanucleus-core

4.1.17

org.datanucleus

datanucleus-rdbms

4.1.19

org.datanucleus

javax.jdo

3.2.0-m3

org.eclipse.jetty

jetty-client

9.4.50.v20221201

org.eclipse.jetty

jetty-continuation

9.4.50.v20221201

org.eclipse.jetty

jetty-http

9.4.50.v20221201

org.eclipse.jetty

jetty-io

9.4.50.v20221201

org.eclipse.jetty

jetty-jndi

9.4.50.v20221201

org.eclipse.jetty

jetty-plus

9.4.50.v20221201

org.eclipse.jetty

jetty-proxy

9.4.50.v20221201

org.eclipse.jetty

jetty-security

9.4.50.v20221201

org.eclipse.jetty

jetty-server

9.4.50.v20221201

org.eclipse.jetty

jetty-servlet

9.4.50.v20221201

org.eclipse.jetty

jetty-servlets

9.4.50.v20221201

org.eclipse.jetty

jetty-util

9.4.50.v20221201

org.eclipse.jetty

jetty-util-ajax

9.4.50.v20221201

org.eclipse.jetty

jetty-webapp

9.4.50.v20221201

org.eclipse.jetty

jetty-xml

9.4.50.v20221201

org.eclipse.jetty.websocket

websocket-api

9.4.50.v20221201

org.eclipse.jetty.websocket

websocket-client

9.4.50.v20221201

org.eclipse.jetty.websocket

websocket-common

9.4.50.v20221201

org.eclipse.jetty.websocket

websocket-server

9.4.50.v20221201

org.eclipse.jetty.websocket

websocket-servlet

9.4.50.v20221201

org.fusesource.leveldbjni

leveldbjni-all

1.8

org.glassfish.hk2

hk2-api

2.6.1

org.glassfish.hk2

hk2-locator

2.6.1

org.glassfish.hk2

hk2-utils

2.6.1

org.glassfish.hk2

osgi-resource-locator

1.0.3

org.glassfish.hk2.external

aopalliance-repackaged

2.6.1

org.glassfish.hk2.external

jakarta.inject

2.6.1

org.glassfish.jersey.containers

jersey-container-servlet

2.36

org.glassfish.jersey.containers

jersey-container-servlet-core

2.36

org.glassfish.jersey.core

jersey-client

2.36

org.glassfish.jersey.core

jersey-common

2.36

org.glassfish.jersey.core

jersey-server

2.36

org.glassfish.jersey.inject

jersey-hk2

2.36

org.hibernate.validator

hibernate-validator

6.1.7.Final

org.javassist

javassist

3.25.0-GA

org.jboss.logging

jboss-logging

3.3.2.Final

org.jdbi

jdbi

2.63.1

org.jetbrains

annotations

17.0.0

org.joda

joda-convert

1.7

org.jodd

jodd-core

3.5.2

org.json4s

json4s-ast_2.12

3.7.0-M11

org.json4s

json4s-core_2.12

3.7.0-M11

org.json4s

json4s-jackson_2.12

3.7.0-M11

org.json4s

json4s-scalap_2.12

3.7.0-M11

org.lz4

lz4-java

1.8.0

org.mariadb.jdbc

mariadb-java-client

2.7.4

org.mlflow

mlflow-spark

2.2.0

org.objenesis

objenesis

2.5.1

org.postgresql

postgresql

42.3.8

org.roaringbitmap

RoaringBitmap

0.9.39

org.roaringbitmap

shims

0.9.39

org.rocksdb

rocksdbjni

7.8.3

org.rosuda.REngine

REngine

2.1.0

org.scala-lang

scala-compiler_2.12

2.12.15

org.scala-lang

scala-library_2.12

2.12.15

org.scala-lang

scala-reflect_2.12

2.12.15

org.scala-lang.modules

scala-collection-compat_2.12

2.4.3

org.scala-lang.modules

scala-parser-combinators_2.12

1.1.2

org.scala-lang.modules

scala-xml_2.12

1.2.0

org.scala-sbt

test-interface

1.0

org.scalacheck

scalacheck_2.12

1.14.2

org.scalactic

scalactic_2.12

3.2.15

org.scalanlp

breeze-macros_2.12

2.1.0

org.scalanlp

breeze_2.12

2.1.0

org.scalatest

scalatest-compatible

3.2.15

org.scalatest

scalatest-core_2.12

3.2.15

org.scalatest

scalatest-diagrams_2.12

3.2.15

org.scalatest

scalatest-featurespec_2.12

3.2.15

org.scalatest

scalatest-flatspec_2.12

3.2.15

org.scalatest

scalatest-freespec_2.12

3.2.15

org.scalatest

scalatest-funspec_2.12

3.2.15

org.scalatest

scalatest-funsuite_2.12

3.2.15

org.scalatest

scalatest-matchers-core_2.12

3.2.15

org.scalatest

scalatest-mustmatchers_2.12

3.2.15

org.scalatest

scalatest-propspec_2.12

3.2.15

org.scalatest

scalatest-refspec_2.12

3.2.15

org.scalatest

scalatest-shouldmatchers_2.12

3.2.15

org.scalatest

scalatest-wordspec_2.12

3.2.15

org.scalatest

scalatest_2.12

3.2.15

org.slf4j

jcl-over-slf4j

2.0.6

org.slf4j

jul-to-slf4j

2.0.6

org.slf4j

slf4j-api

2.0.6

org.threeten

threeten-extra

1.7.1

org.tukaani

xz

1.9

org.typelevel

algebra_2.12

2.0.1

org.typelevel

cats-kernel_2.12

2.1.1

org.typelevel

spire-macros_2.12

0.17.0

org.typelevel

spire-platform_2.12

0.17.0

org.typelevel

spire-util_2.12

0.17.0

org.typelevel

spire_2.12

0.17.0

org.wildfly.openssl

wildfly-openssl

1.1.3.Final

org.xerial

sqlite-jdbc

3.42.0.0

org.xerial.snappy

snappy-java

1.1.8.4

org.yaml

snakeyaml

1.33

oro

oro

2.0.8

pl.edu.icm

JLargeArrays

1.5

software.amazon.cryptools

AmazonCorrettoCryptoProvider

1.6.1-linux-x86_64

software.amazon.ion

ion-java

1.0.2

stax

stax-api

1.0.1