Databricks Runtime 11.0 (unsupported)

The following release notes provide information about Databricks Runtime 11.0, powered by Apache Spark 3.3.0. Databricks released these images in June 2022.

Note

These release notes may include references to features that are not available on Google Cloud as of this release.

New features and improvements

New version of Apache Spark

Databricks Runtime 11.0 and Databricks Runtime 11.0 Photon include Apache Spark 3.3.0. For details, see Apache Spark.

Python notebooks now use the IPython kernel

In Databricks Runtime 11.0 and above, Python notebooks use the IPython kernel to execute Python code. See IPython kernel.

Synapse connector now writes Parquet data in non-legacy mode

The Azure Synapse connector now writes Parquet data in non-legacy mode. It preserves the INT96 timestamp format when using PolyBase and COPY commands for both batch and streaming workloads.

HTTPS scheme now enforced when ABFS client uses SAS token

When the Azure Blob File System (ABFS) client uses a Shared Access Signature (SAS) token, the HTTPS scheme is now enforced.

SQL: DESC is now an alias for DESCRIBE

You can now use DESC as an alias for DESCRIBE when you describe external locations or storage credentials. For example:

-- Describe an external location.
DESC EXTERNAL LOCATION location_name;

-- Describe a storage credential.
DESC STORAGE CREDENTIAL credential_name;

SQL: New current_version function outputs version details

The new current_version function outputs the current Databricks Runtime version if available, the current Databricks SQL version if available, and other related version details. Use this new function to query version-related information. See current_version function.

Dropping a missing Delta table constraint now raises an error

If you now try to drop a Delta table constraint by name, and that constraint does not exist, you will get an error. To get the previous behavior, which does not throw an error if the constraint does not exist, you must now use the IF EXISTS statement. See ALTER TABLE.

SQL: New EXCEPT clause in SELECT statement excludes columns from selection

SELECT statements now support the EXCEPT clause to exclude columns from selection. For example, SELECT * EXCEPT (x) FROM table returns all of table’s columns except x. Nested columns are also allowed. For example, SELECT * EXCEPT (x.a) FROM table returns all of table’s columns but omits field a from struct x.

Support for dropping columns in Delta tables (Public Preview)

You can use ALTER TABLE <table-name> DROP COLUMN [IF EXISTS] <column-name> or ALTER TABLE <table-name> DROP COLUMNS [IF EXISTS] (<column-name>, *) to drop a column or a list of columns, respectively, from a Delta table as a metadata-only operation. The columns are effectively “soft-deleted,” as they are still in the underlying Parquet files but are no longer visible to the Delta table.

You can use REORG TABLE <table-name> APPLY (PURGE) to trigger a file rewrite on the files that contain any soft-deleted data such as dropped columns.

You can use VACUUM to remove the deleted files from physical storage, including the old files that contain dropped columns and have been rewritten by REORG TABLE.

COPY INTO improvements

You can now create empty placeholder Delta tables so that the schema is later inferred during a COPY INTO command:

CREATE TABLE IF NOT EXISTS my_table
[COMMENT <table-description>]
[TBLPROPERTIES (<table-properties>)];

COPY INTO my_table
FROM '/path/to/files'
FILEFORMAT = <format>
FORMAT_OPTIONS ('mergeSchema' = 'true')
COPY_OPTIONS ('mergeSchema' = 'true');

The preceding SQL statement is idempotent and can be scheduled to run to ingest data exactly once into a Delta table.

Note

The empty Delta table is not usable outside of COPY INTO. You cannot use INSERT INTO and MERGE INTO to write data into schema-less Delta tables. After data is inserted into the table with COPY INTO, the table is queryable.

If the data you are ingesting cannot be read due to some corruption issue, you can skip corrupted files by setting ignoreCorruptFiles to true in the FORMAT_OPTIONS:

COPY INTO my_table
FROM '/path/to/files'
FILEFORMAT = <format>
FORMAT_OPTIONS ('ignoreCorruptFiles' = 'true')

The COPY INTO command returns the number of files that were skipped due to corruption in the num_skipped_corrupt_files column. This metric also appears in the operationMetrics column under numSkippedCorruptFiles after you run DESCRIBE HISTORY on the Delta table.

Corrupt files are not tracked by COPY INTO, so they can be reloaded in a subsequent run if the corruption is fixed. You can see which files are corrupt by running COPY INTO in VALIDATE mode.

Behavior changes

SQL: lpad and rpad functions now support byte sequences

The lpad and rpad functions have been updated to add support for byte sequences in addition to strings.

String format in format_string and printf no longer allows %0$

Specifying the format %0$ in the format_string and printf functions now throws an error by default. This change is to preserve expected behavior with previous versions of the Databricks Runtime and common third-party databases. The first argument should always reference %1$ when you use an argument index to indicate the position of the argument in the argument list.

Null values in CSV files are now written as unquoted empty strings by default

Null values in CSV files were previously written as quoted empty strings. With this release, null values in CSV files are now written as unquoted empty strings by default. To change back to the previous behavior, set the nullValue option to "" for write operations.

The table property external is now reserved

The property external is now a reserved table property by default. Exceptions are now thrown when you use the external property with the CREATE TABLE ... TBLPROPERTIES and ALTER TABLE ... SET TBLPROPERTIES clauses.

Log4j is upgraded from Log4j 1 to Log4j 2

Log4j 1 is being upgraded to Log4j 2. Legacy Log4j 1 dependencies are being removed.

If you depend on the Log4j 1 classes that were previously included inside the Databricks Runtime, those classes no longer exist. You should upgrade your dependencies to Log4j 2.

If you have custom plugins or configuration files that depend on Log4j 2, they may no longer work with the version of Log4j 2 in this release. For assistance, contact your Databricks account team.

Libraries installed from Maven are now resolved in your compute plane by default

Maven libraries now resolve in your compute plane by default when you install libraries on a cluster. You cluster must have access to Maven Central. Alternatively, you can revert to the previous behavior by setting the Spark configuration property:

spark.databricks.libraries.enableMavenResolution false

Bug fixes

  • Binary compatibility for LeafNode, UnaryNode, and BinaryNode between Apache Spark and the Databricks Runtime has been fixed and the classes are now compatible with Apache Spark 3.3.0 and above. If you encounter the following or a similar message when using a third-party package with the Databricks Runtime, rebuild the package with Apache Spark 3.3.0 or above: Found interface org.apache.spark.sql.catalyst.plans.logical.UnaryNode, but class was expected.

Library upgrades

  • Upgraded Python libraries:

    • platformdirs from 2.5.1 to 2.5.2

    • protobuf from 3.20.0 to 3.20.1

  • Upgraded R libraries:

    • blob from 1.2.2 to 1.2.3

    • broom from 0.7.12 to 0.8.0

    • caret from 6.0-91 to 6.0-92

    • cli from 3.2.0 to 3.3.0

    • dplyr from 1.0.8 to 1.0.9

    • future from 1.24.0 to 1.25.0

    • future.apply from 1.8.1 to 1.9.0

    • gert from 1.5.0 to 1.6.0

    • ggplot2 from 3.3.5 to 3.3.6

    • glmnet from 4.1-3 to 4.1-4

    • haven from 2.4.3 to 2.5.0

    • httr from 1.4.2 to 1.4.3

    • knitr from 1.38 to 1.39

    • magrittr from 2.0.2 to 2.0.3

    • parallelly from 1.30.0 to 1.31.1

    • ps from 1.6.0 to 1.7.0

    • RColorBrewer from 1.1-2 to 1.1-3

    • RcppEigen from 0.3.3.9.1 to 0.3.3.9.2

    • readxl from 1.3.1 to 1.4.0

    • rmarkdown from 2.13 to 2.14

    • rprojroot from 2.0.2 to 2.0.3

    • RSQLite from 2.2.11 to 2.2.13

    • scales from 1.1.1 to 1.2.0

    • testthat from 3.1.2 to 3.1.4

    • tibble from 3.1.6 to 3.1.7

    • tinytex from 0.37 to 0.38

    • tzdb from 0.2.0 to 0.3.0

    • uuid from 1.0-4 to 1.1-0

    • vctrs from 0.3.8 to 0.4.1

  • Upgraded Java libraries:

    • com.fasterxml.jackson.core.jackson-annotations from 2.13.0 to 2.13.3

    • com.fasterxml.jackson.core.jackson-core from 2.13.0 to 2.13.3

    • com.fasterxml.jackson.core.jackson-databind from 2.13.0 to 2.13.3

    • com.fasterxml.jackson.dataformat.jackson-dataformat-cbor from 2.13.0 to 2.13.3

    • com.fasterxml.jackson.datatype.jackson-datatype-joda from 2.13.0 to 2.13.3

    • com.fasterxml.jackson.module.jackson-module-paranamer from 2.13.0 to 2.13.3

    • com.fasterxml.jackson.module.jackson-module-scala_2.12 from 2.13.0 to 2.13.3

    • com.google.crypto.tink.tink from 1.6.0 to 1.6.1

    • com.ning.compress-lzf from 1.0.3 to 1.1

    • dev.ludovic.netlib.arpack from 2.2.0 to 2.2.1

    • dev.ludovic.netlib.blas from 2.2.0 to 2.2.1

    • dev.ludovic.netlib.lapack from 2.2.0 to 2.2.1

    • io.netty.netty-all from 4.1.73.Final to 4.1.74.Final

    • io.netty.netty-buffer from 4.1.73.Final to 4.1.74.Final

    • io.netty.netty-codec from 4.1.73.Final to 4.1.74.Final

    • io.netty.netty-common from 4.1.73.Final to 4.1.74.Final

    • io.netty.netty-handler from 4.1.73.Final to 4.1.74.Final

    • io.netty.netty-resolver from 4.1.73.Final to 4.1.74.Final

    • io.netty.netty-tcnative-classes from 2.0.46.Final to 2.0.48.Final

    • io.netty.netty-transport from 4.1.73.Final to 4.1.74.Final

    • io.netty.netty-transport-classes-epoll from 4.1.73.Final to 4.1.74.Final

    • io.netty.netty-transport-classes-kqueue from 4.1.73.Final to 4.1.74.Final

    • io.netty.netty-transport-native-epoll-linux-aarch_64 from 4.1.73.Final to 4.1.74.Final

    • io.netty.netty-transport-native-epoll-linux-x86_64 from 4.1.73.Final to 4.1.74.Final

    • io.netty.netty-transport-native-kqueue-osx-aarch_64 from 4.1.73.Final to 4.1.74.Final

    • io.netty.netty-transport-native-kqueue-osx-x86_64 from 4.1.73.Final to 4.1.74.Final

    • io.netty.netty-transport-native-unix-common from 4.1.73.Final to 4.1.74.Final

    • joda-time.joda-time from 2.10.12 to 2.10.13

    • org.apache.commons.commons-math3 from 3.4.1 to 3.6.1

    • org.apache.httpcomponents.httpcore from 4.4.12 to 4.4.14

    • org.apache.orc.orc-core from 1.7.3 to 1.7.4

    • org.apache.orc.orc-mapreduce from 1.7.3 to 1.7.4

    • org.apache.orc.orc-shims from 1.7.3 to 1.7.4

    • org.eclipse.jetty.jetty-client from 9.4.43.v20210629 to 9.4.46.v20220331

    • org.eclipse.jetty.jetty-continuation from 9.4.43.v20210629 to 9.4.46.v20220331

    • org.eclipse.jetty.jetty-http from 9.4.43.v20210629 to 9.4.46.v20220331

    • org.eclipse.jetty.jetty-io from 9.4.43.v20210629 to 9.4.46.v20220331

    • org.eclipse.jetty.jetty-jndi from 9.4.43.v20210629 to 9.4.46.v20220331

    • org.eclipse.jetty.jetty-plus from 9.4.43.v20210629 to 9.4.46.v20220331

    • org.eclipse.jetty.jetty-proxy from 9.4.43.v20210629 to 9.4.46.v20220331

    • org.eclipse.jetty.jetty-security from 9.4.43.v20210629 to 9.4.46.v20220331

    • org.eclipse.jetty.jetty-server from 9.4.43.v20210629 to 9.4.46.v20220331

    • org.eclipse.jetty.jetty-servlet from 9.4.43.v20210629 to 9.4.46.v20220331

    • org.eclipse.jetty.jetty-servlets from 9.4.43.v20210629 to 9.4.46.v20220331

    • org.eclipse.jetty.jetty-util from 9.4.43.v20210629 to 9.4.46.v20220331

    • org.eclipse.jetty.jetty-util-ajax from 9.4.43.v20210629 to 9.4.46.v20220331

    • org.eclipse.jetty.jetty-webapp from 9.4.43.v20210629 to 9.4.46.v20220331

    • org.eclipse.jetty.jetty-xml from 9.4.43.v20210629 to 9.4.46.v20220331

    • org.eclipse.jetty.websocket.websocket-api from 9.4.43.v20210629 to 9.4.46.v20220331

    • org.eclipse.jetty.websocket.websocket-client from 9.4.43.v20210629 to 9.4.46.v20220331

    • org.eclipse.jetty.websocket.websocket-common from 9.4.43.v20210629 to 9.4.46.v20220331

    • org.eclipse.jetty.websocket.websocket-server from 9.4.43.v20210629 to 9.4.46.v20220331

    • org.eclipse.jetty.websocket.websocket-servlet from 9.4.43.v20210629 to 9.4.46.v20220331

    • org.mariadb.jdbc.mariadb-java-client from 2.2.5 to 2.7.4

    • org.postgresql.postgresql from 42.2.19 to 42.3.3

    • org.roaringbitmap.RoaringBitmap from 0.9.23 to 0.9.25

    • org.roaringbitmap.shims from 0.9.23 to 0.9.25

    • org.rocksdb.rocksdbjni from 6.20.3 to 6.24.2

    • org.slf4j.jcl-over-slf4j from 1.7.32 to 1.7.36

    • org.slf4j.jul-to-slf4j from 1.7.32 to 1.7.36

    • org.slf4j.slf4j-api from 1.7.30 to 1.7.36

Apache Spark

Databricks Runtime 11.0 includes Apache Spark 3.3.0.

Spark SQL and Core

ANSI mode

  • New explicit cast syntax rules in ANSI mode (SPARK-33354)

  • Elt() should return null if index is null under ANSI mode (SPARK-38304)

  • Optionally return null result if element not exists in array/map (SPARK-37750)

  • Allow casting between numeric type and timestamp type (SPARK-37714)

  • Disable ANSI reserved keywords by default (SPARK-37724)

  • Use store assignment rules for resolving function invocation (SPARK-37438)

  • Add a config to allow casting between Datetime and Numeric (SPARK-37179)

  • Add a config to optionally enforce ANSI reserved keywords (SPARK-37133)

  • Disallow binary operations between Interval and String literal (SPARK-36508)

Feature enhancements

Performance enhancements

  • Whole-stage code generation

    • Add code-gen for sort aggregate without grouping keys (SPARK-37564)

    • Add code-gen for full outer sort merge join (SPARK-35352)

    • Add code-gen for full outer shuffled hash join (SPARK-32567)

    • Add code-gen for existence sort merge join (SPARK-37316)

  • Push down (filters)

    • Push down filters through RebalancePartitions (SPARK-37828)

    • Push down boolean column filter (SPARK-36644)

    • Push down limit 1 for right side of left semi/anti join if join condition is empty (SPARK-37917)

    • Translate more standard aggregate functions for pushdown (SPARK-37527)

    • Support propagate empty relation through aggregate/union (SPARK-35442)

    • Row-level Runtime Filtering (SPARK-32268)

    • Support Left Semi join in row level runtime filters (SPARK-38565)

    • Support predicate pushdown and column pruning for de-duped CTEs (SPARK-37670)

  • Vectorization

    • Implement a ConstantColumnVector and improve performance of the hidden file metadata (SPARK-37896)

    • Enable vectorized read for VectorizedPlainValuesReader.readBooleans (SPARK-35867)

  • Combine/remove/replace nodes

    • Combine unions if there is a project between them (SPARK-37915)

    • Combine to one cast if we can safely up-cast two casts (SPARK-37922)

    • Remove the Sort if it is the child of RepartitionByExpression (SPARK-36703)

    • Removes outer join if it only has DISTINCT on streamed side with alias (SPARK-37292)

    • Replace hash with sort aggregate if child is already sorted (SPARK-37455)

    • Only collapse projects if we don’t duplicate expensive expressions (SPARK-36718)

    • Remove redundant aliases after RewritePredicateSubquery (SPARK-36280)

    • Merge non-correlated scalar subqueries (SPARK-34079)

  • Partitioning

    • Do not add dynamic partition pruning if there exists static partition pruning (SPARK-38148)

    • Improve RebalancePartitions in rules of Optimizer (SPARK-37904)

    • Add small partition factor for rebalance partitions (SPARK-37357)

  • Join

    • Fine tune logic to demote Broadcast hash join in DynamicJoinSelection (SPARK-37753)

    • Ignore duplicated join keys when building relation for SEMI/ANTI shuffled hash join (SPARK-36794)

    • Support optimize skewed join even if introduce extra shuffle (SPARK-33832)

  • AQE

    • Support eliminate limits in AQE Optimizer (SPARK-36424)

    • Optimize one row plan in normal and AQE Optimizer (SPARK-38162)

  • Aggregate.groupOnly support foldable expressions (SPARK-38489)

  • ByteArrayMethods arrayEquals should fast skip the check of aligning with unaligned platform (SPARK-37796)

  • Add tree pattern pruning to CTESubstitution rule (SPARK-37379)

  • Add more Not operator simplifications (SPARK-36665)

  • Support BooleanType in UnwrapCastInBinaryComparison (SPARK-36607)

  • Coalesce drop all expressions after the first non nullable expression (SPARK-36359)

  • Add a logical plan visitor to propagate the distinct attributes (SPARK-36194)

Built-in connector enhancements

  • General

    • Lenient serialization of datetime from datasource (SPARK-38437)

    • Treat table location as absolute when the first letter of its path is slash in create/alter table (SPARK-38236)

    • Remove leading zeros from empty static number type partition (SPARK-35561)

    • Support ignoreCorruptFiles and ignoreMissingFiles in Data Source options (SPARK-38767)

    • Add command SHOW CATALOGS (SPARK-35973)

  • Parquet

    • Enable matching schema column names by field ids (SPARK-38094)

    • Remove check field name when reading/writing data in parquet (SPARK-27442)

    • Support vectorized read boolean values use RLE encoding with Parquet DataPage V2 (SPARK-37864)

    • Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path (SPARK-36879)

    • Rebase timestamps in the session time zone saved in Parquet/Avro metadata (SPARK-37705)

    • Push down group by partition column for aggregate (SPARK-36646)

    • Aggregate (Min/Max/Count) push down for Parquet (SPARK-36645)

    • Parquet: enable matching schema columns by field id (SPARK-38094)

    • Reduce default page size by LONG_ARRAY_OFFSET if G1GC and ON_HEAP are used (SPARK-37593)

    • Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support (SPARK-37974)

    • Support complex types for Parquet vectorized reader (SPARK-34863)

  • ORC

    • Remove check field name when reading/writing existing data in Orc (SPARK-37965)

    • Aggregate push down for ORC (SPARK-34960)

    • Support reading and writing ANSI intervals from/to ORC datasources (SPARK-36931)

    • Support number-only column names in ORC data sources (SPARK-36663)

  • JSON

    • Respect allowNonNumericNumbers when parsing quoted NaN and Infinity values in JSON reader (SPARK-38060)

    • Use CAST for datetime in CSV/JSON by default (SPARK-36536)

    • Align error message for unsupported key types in MapType in Json reader (SPARK-35320)

  • CSV

    • Fix referring to the corrupt record column from CSV (SPARK-38534)

    • null values should be saved as nothing instead of quoted empty Strings “” by default (SPARK-37575)

  • JDBC

    • Add the IMMEDIATE statement to the DB2 dialect truncate implementation (SPARK-30062)

  • Hive

    • Support writing Hive bucketed table (Hive file formats with Hive hash) (SPARK-32712)

    • Use expressions to filter Hive partitions at client side (SPARK-35437)

    • Support Dynamic Partition pruning for HiveTableScanExec (SPARK-36876)

    • InsertIntoHiveDir should use data source if it’s convertible (SPARK-38215)

    • Support writing Hive bucketed table (Parquet/ORC format with Hive hash) (SPARK-32709)

Node decommission

  • FallbackStorage shouldn’t attempt to resolve arbitrary “remote” hostname (SPARK-38062)

  • ExecutorMonitor.onExecutorRemoved should handle ExecutorDecommission as finished (SPARK-38023)

Other notable changes

  • Add fine grained locking to BlockInfoManager (SPARK-37356)

  • Support mapping Spark gpu/fpga resource types to custom YARN resource type (SPARK-37208)

  • Report accurate shuffle block size if its skewed (SPARK-36967)

  • Supporting Netty Logging at the network layer (SPARK-36719)

Structured Streaming

Major features

  • Introduce Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches (SPARK-36533)

Other notable changes

  • Use StatefulOpClusteredDistribution for stateful operators with respecting backward compatibility (SPARK-38204)

  • Fix flatMapGroupsWithState timeout in batch with data for key (SPARK-38320)

  • Fix correctness issue on stream-stream outer join with RocksDB state store provider (SPARK-38684)

  • Support Trigger.AvailableNow on Kafka data source (SPARK-36649)

  • Optimize write path on RocksDB state store provider (SPARK-37224)

  • Introduce a new data source for providing a consistent set of rows per microbatch (SPARK-37062)

  • Use HashClusteredDistribution for stateful operators with respecting backward compatibility (SPARK-38204)

PySpark

Pandas API on Spark

Major improvements

  • ‘distributed-sequence’ index optimization with being default (SPARK-36559, SPARK-36338)

    • Support to specify index type and name in pandas API on Spark (SPARK-36709)

    • Show default index type in SQL plans for pandas API on Spark (SPARK-38654)

Major features

  • Implement SparkSQL native ps.merge_asof (SPARK-36813)

  • Support TimedeltaIndex in pandas API on Spark (SPARK-37525)

  • Support Python’s timedelta (SPARK-37275, SPARK-37510)

  • Implement functions in CategoricalAccessor/CategoricalIndex (SPARK-36185)

  • Uses Python’s standard string formatter for SQL API in pandas API on Spark (SPARK-37436)

  • Support basic operations of timedelta Series/Index (SPARK-37510)

  • Support ps.MultiIndex.dtypes (SPARK-36930)

  • Implement Index.map (SPARK-36469)

  • Implement Series.__xor__ and Series.__rxor__ (SPARK-36653)

  • Implement unary operator invert of integral ps.Series/Index (SPARK-36003)

  • Implement DataFrame.cov (SPARK-36396)

  • Support str and timestamp for (Series|DataFrame).describe() (SPARK-37657)

  • Support lambda column parameter of DataFrame.rename(SPARK-38763)

Other notable changes

Breaking changes

  • Drop references to Python 3.6 support in docs and python/docs (SPARK-36977)

  • Remove namedtuple hack by replacing built-in pickle to cloudpickle (SPARK-32079)

  • Bump minimum pandas version to 1.0.5 (SPARK-37465)

  • Major improvements

    • Provide a profiler for Python/Pandas UDFs (SPARK-37443)

    • Uses Python’s standard string formatter for SQL API in PySpark (SPARK-37516)

    • Expose SQL state and error class in PySpark exceptions (SPARK-36953)

    • Try to capture faulthanlder when a Python worker crashes (SPARK-36062)

Major features

  • Implement DataFrame.mapInArrow in Python (SPARK-37228)

  • Uses Python’s standard string formatter for SQL API in PySpark (SPARK-37516)

  • Add df.withMetadata pyspark API (SPARK-36642)

  • Support Python’s timedelta (SPARK-37275)

  • Expose tableExists in pyspark.sql.catalog (SPARK-36176)

  • Expose databaseExists in pyspark.sql.catalog (SPARK-36207)

  • Exposing functionExists in pyspark sql catalog (SPARK-36258)

  • Add Dataframe.observation to PySpark (SPARK-36263)

  • Add max_by/min_by API to PySpark (SPARK-36972)

  • Support to infer nested dict as a struct when creating a DataFrame (SPARK-35929)

  • Add bit/octet_length APIs to Scala, Python and R (SPARK-36751)

  • Support ILIKE API on Python (SPARK-36882)

  • Add isEmpty method for the Python DataFrame API (SPARK-37207)

  • Add multiple columns adding support (SPARK-35173)

  • Add SparkContext.addArchive in PySpark (SPARK-38278)

  • Make sql type reprs eval-able (SPARK-18621)

  • Inline type hints for fpm.py in python/pyspark/mllib (SPARK-37396)

  • Implement dropna parameter of SeriesGroupBy.value_counts (SPARK-38837)

MLLIB

Major features

  • Add distanceMeasure param to trainKMeansModel (SPARK-37118)

  • Expose LogisticRegression.setInitialModel, like KMeans et al do (SPARK-36481)

  • Support CrossValidatorModel get standard deviation of metrics for each paramMap (SPARK-36425)

Major improvements

  • Optimize some treeAggregates in MLlib by delaying allocations (SPARK-35848)

  • Rewrite _shared_params_code_gen.py to inline type hints for ml/param/shared.py (SPARK-37419)

Other notable changes

SparkR

UI

  • Speculation metrics summary at stage level (SPARK-36038)

  • Unified shuffle read block time to shuffle read fetch wait time in StagePage (SPARK-37469)

  • Add modified configs for SQL execution in UI (SPARK-34735)

  • Make ThriftServer recognize spark.sql.redaction.string.regex (SPARK-36400)

  • Attach and start handler after application started in UI (SPARK-36237)

  • Add commit duration to SQL tab’s graph node (SPARK-34399)

  • Support RocksDB backend in Spark History Server (SPARK-37680)

  • Show options for Pandas API on Spark in UI (SPARK-38656)

  • Rename ‘SQL’ to ‘SQL / DataFrame’ in SQL UI page (SPARK-38657)

Build

Maintenance updates

See Databricks Runtime 11.0 (unsupported).

System environment

  • Operating System: Ubuntu 20.04.4 LTS

  • Java: Zulu 8.56.0.21-CA-linux64

  • Scala: 2.12.14

  • Python: 3.9.5

  • R: 4.1.3

  • Delta Lake: 1.2.1

Installed Python libraries

Library

Version

Library

Version

Library

Version

Antergos Linux

2015.10 (ISO-Rolling)

argon2-cffi

20.1.0

async-generator

1.10

attrs

21.2.0

backcall

0.2.0

backports.entry-points-selectable

1.1.1

bleach

4.0.0

boto3

1.21.18

botocore

1.24.18

certifi

2021.10.8

cffi

1.14.6

chardet

4.0.0

charset-normalizer

2.0.4

cycler

0.10.0

Cython

0.29.24

dbus-python

1.2.16

debugpy

1.4.1

decorator

5.1.0

defusedxml

0.7.1

distlib

0.3.4

distro-info

0.23ubuntu1

entrypoints

0.3

facets-overview

1.0.0

filelock

3.6.0

idna

3.2

ipykernel

6.12.1

ipython

7.32.0

ipython-genutils

0.2.0

ipywidgets

7.7.0

jedi

0.18.0

Jinja2

2.11.3

jmespath

0.10.0

joblib

1.0.1

jsonschema

3.2.0

jupyter-client

6.1.12

jupyter-core

4.8.1

jupyterlab-pygments

0.1.2

jupyterlab-widgets

1.0.0

kiwisolver

1.3.1

MarkupSafe

2.0.1

matplotlib

3.4.3

matplotlib-inline

0.1.2

mistune

0.8.4

nbclient

0.5.3

nbconvert

6.1.0

nbformat

5.1.3

nest-asyncio

1.5.1

notebook

6.4.5

numpy

1.20.3

packaging

21.0

pandas

1.3.4

pandocfilters

1.4.3

parso

0.8.2

patsy

0.5.2

pexpect

4.8.0

pickleshare

0.7.5

Pillow

8.4.0

pip

21.2.4

platformdirs

2.5.2

plotly

5.6.0

prometheus-client

0.11.0

prompt-toolkit

3.0.20

protobuf

3.20.1

psutil

5.8.0

psycopg2

2.9.3

ptyprocess

0.7.0

pyarrow

7.0.0

pycparser

2.20

Pygments

2.10.0

PyGObject

3.36.0

pyodbc

4.0.31

pyparsing

3.0.4

pyrsistent

0.18.0

python-apt

2.0.0+ubuntu0.20.4.7

python-dateutil

2.8.2

pytz

2021.3

pyzmq

22.2.1

requests

2.26.0

requests-unixsocket

0.2.0

s3transfer

0.5.2

scikit-learn

0.24.2

scipy

1.7.1

seaborn

0.11.2

Send2Trash

1.8.0

setuptools

58.0.4

six

1.16.0

ssh-import-id

5.10

statsmodels

0.12.2

tenacity

8.0.1

terminado

0.9.4

testpath

0.5.0

threadpoolctl

2.2.0

tornado

6.1

traitlets

5.1.0

unattended-upgrades

0.1

urllib3

1.26.7

virtualenv

20.8.0

wcwidth

0.2.5

webencodings

0.5.1

wheel

0.37.0

widgetsnbextension

3.6.0

Installed R libraries

R libraries are installed from the Microsoft CRAN snapshot on 2022-05-06.

Library

Version

Library

Version

Library

Version

askpass

1.1

assertthat

0.2.1

backports

1.4.1

base

4.1.3

base64enc

0.1-3

bit

4.0.4

bit64

4.0.5

blob

1.2.3

boot

1.3-28

brew

1.0-7

brio

1.1.3

broom

0.8.0

bslib

0.3.1

cachem

1.0.6

callr

3.7.0

caret

6.0-92

cellranger

1.1.0

chron

2.3-56

class

7.3-20

cli

3.3.0

clipr

0.8.0

cluster

2.1.3

codetools

0.2-18

colorspace

2.0-3

commonmark

1.8.0

compiler

4.1.3

config

0.3.1

cpp11

0.4.2

crayon

1.5.1

credentials

1.3.2

curl

4.3.2

data.table

1.14.2

datasets

4.1.3

DBI

1.1.2

dbplyr

2.1.1

desc

1.4.1

devtools

2.4.3

diffobj

0.3.5

digest

0.6.29

dplyr

1.0.9

dtplyr

1.2.1

e1071

1.7-9

ellipsis

0.3.2

evaluate

0.15

fansi

1.0.3

farver

2.1.0

fastmap

1.1.0

fontawesome

0.2.2

forcats

0.5.1

foreach

1.5.2

foreign

0.8-82

forge

0.2.0

fs

1.5.2

future

1.25.0

future.apply

1.9.0

gargle

1.2.0

generics

0.1.2

gert

1.6.0

ggplot2

3.3.6

gh

1.3.0

gitcreds

0.1.1

glmnet

4.1-4

globals

0.14.0

glue

1.6.2

googledrive

2.0.0

googlesheets4

1.0.0

gower

1.0.0

graphics

4.1.3

grDevices

4.1.3

grid

4.1.3

gridExtra

2.3

gsubfn

0.7

gtable

0.3.0

hardhat

0.2.0

haven

2.5.0

highr

0.9

hms

1.1.1

htmltools

0.5.2

htmlwidgets

1.5.4

httpuv

1.6.5

httr

1.4.3

ids

1.0.1

ini

0.3.1

ipred

0.9-12

isoband

0.2.5

iterators

1.0.14

jquerylib

0.1.4

jsonlite

1.8.0

KernSmooth

2.23-20

knitr

1.39

labeling

0.4.2

later

1.3.0

lattice

0.20-45

lava

1.6.10

lifecycle

1.0.1

listenv

0.8.0

lubridate

1.8.0

magrittr

2.0.3

markdown

1.1

MASS

7.3-56

Matrix

1.4-1

memoise

2.0.1

methods

4.1.3

mgcv

1.8-40

mime

0.12

ModelMetrics

1.2.2.2

modelr

0.1.8

munsell

0.5.0

nlme

3.1-157

nnet

7.3-17

numDeriv

2016.8-1.1

openssl

2.0.0

parallel

4.1.3

parallelly

1.31.1

pillar

1.7.0

pkgbuild

1.3.1

pkgconfig

2.0.3

pkgload

1.2.4

plogr

0.2.0

plyr

1.8.7

praise

1.0.0

prettyunits

1.1.1

pROC

1.18.0

processx

3.5.3

prodlim

2019.11.13

progress

1.2.2

progressr

0.10.0

promises

1.2.0.1

proto

1.0.0

proxy

0.4-26

ps

1.7.0

purrr

0.3.4

r2d3

0.2.6

R6

2.5.1

randomForest

4.7-1

rappdirs

0.3.3

rcmdcheck

1.4.0

RColorBrewer

1.1-3

Rcpp

1.0.8.3

RcppEigen

0.3.3.9.2

readr

2.1.2

readxl

1.4.0

recipes

0.2.0

rematch

1.0.1

rematch2

2.1.2

remotes

2.4.2

reprex

2.0.1

reshape2

1.4.4

rlang

1.0.2

rmarkdown

2.14

RODBC

1.3-19

roxygen2

7.1.2

rpart

4.1.16

rprojroot

2.0.3

Rserve

1.8-10

RSQLite

2.2.13

rstudioapi

0.13

rversions

2.1.1

rvest

1.0.2

sass

0.4.1

scales

1.2.0

selectr

0.4-2

sessioninfo

1.2.2

shape

1.4.6

shiny

1.7.1

sourcetools

0.1.7

sparklyr

1.7.5

SparkR

3.3.0

spatial

7.3-11

splines

4.1.3

sqldf

0.4-11

SQUAREM

2021.1

stats

4.1.3

stats4

4.1.3

stringi

1.7.6

stringr

1.4.0

survival

3.2-13

sys

3.4

tcltk

4.1.3

testthat

3.1.4

tibble

3.1.7

tidyr

1.2.0

tidyselect

1.1.2

tidyverse

1.3.1

timeDate

3043.102

tinytex

0.38

tools

4.1.3

tzdb

0.3.0

usethis

2.1.5

utf8

1.2.2

utils

4.1.3

uuid

1.1-0

vctrs

0.4.1

viridisLite

0.4.0

vroom

1.5.7

waldo

0.4.0

whisker

0.4

withr

2.5.0

xfun

0.30

xml2

1.3.3

xopen

1.0.0

xtable

1.8-4

yaml

2.3.5

zip

2.2.0

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID

Artifact ID

Version

antlr

antlr

2.7.7

com.amazonaws

amazon-kinesis-client

1.12.0

com.amazonaws

aws-java-sdk-autoscaling

1.12.189

com.amazonaws

aws-java-sdk-cloudformation

1.12.189

com.amazonaws

aws-java-sdk-cloudfront

1.12.189

com.amazonaws

aws-java-sdk-cloudhsm

1.12.189

com.amazonaws

aws-java-sdk-cloudsearch

1.12.189

com.amazonaws

aws-java-sdk-cloudtrail

1.12.189

com.amazonaws

aws-java-sdk-cloudwatch

1.12.189

com.amazonaws

aws-java-sdk-cloudwatchmetrics

1.12.189

com.amazonaws

aws-java-sdk-codedeploy

1.12.189

com.amazonaws

aws-java-sdk-cognitoidentity

1.12.189

com.amazonaws

aws-java-sdk-cognitosync

1.12.189

com.amazonaws

aws-java-sdk-config

1.12.189

com.amazonaws

aws-java-sdk-core

1.12.189

com.amazonaws

aws-java-sdk-datapipeline

1.12.189

com.amazonaws

aws-java-sdk-directconnect

1.12.189

com.amazonaws

aws-java-sdk-directory

1.12.189

com.amazonaws

aws-java-sdk-dynamodb

1.12.189

com.amazonaws

aws-java-sdk-ec2

1.12.189

com.amazonaws

aws-java-sdk-ecs

1.12.189

com.amazonaws

aws-java-sdk-efs

1.12.189

com.amazonaws

aws-java-sdk-elasticache

1.12.189

com.amazonaws

aws-java-sdk-elasticbeanstalk

1.12.189

com.amazonaws

aws-java-sdk-elasticloadbalancing

1.12.189

com.amazonaws

aws-java-sdk-elastictranscoder

1.12.189

com.amazonaws

aws-java-sdk-emr

1.12.189

com.amazonaws

aws-java-sdk-glacier

1.12.189

com.amazonaws

aws-java-sdk-glue

1.12.189

com.amazonaws

aws-java-sdk-iam

1.12.189

com.amazonaws

aws-java-sdk-importexport

1.12.189

com.amazonaws

aws-java-sdk-kinesis

1.12.189

com.amazonaws

aws-java-sdk-kms

1.12.189

com.amazonaws

aws-java-sdk-lambda

1.12.189

com.amazonaws

aws-java-sdk-logs

1.12.189

com.amazonaws

aws-java-sdk-machinelearning

1.12.189

com.amazonaws

aws-java-sdk-opsworks

1.12.189

com.amazonaws

aws-java-sdk-rds

1.12.189

com.amazonaws

aws-java-sdk-redshift

1.12.189

com.amazonaws

aws-java-sdk-route53

1.12.189

com.amazonaws

aws-java-sdk-s3

1.12.189

com.amazonaws

aws-java-sdk-ses

1.12.189

com.amazonaws

aws-java-sdk-simpledb

1.12.189

com.amazonaws

aws-java-sdk-simpleworkflow

1.12.189

com.amazonaws

aws-java-sdk-sns

1.12.189

com.amazonaws

aws-java-sdk-sqs

1.12.189

com.amazonaws

aws-java-sdk-ssm

1.12.189

com.amazonaws

aws-java-sdk-storagegateway

1.12.189

com.amazonaws

aws-java-sdk-sts

1.12.189

com.amazonaws

aws-java-sdk-support

1.12.189

com.amazonaws

aws-java-sdk-swf-libraries

1.11.22

com.amazonaws

aws-java-sdk-workspaces

1.12.189

com.amazonaws

jmespath-java

1.12.189

com.chuusai

shapeless_2.12

2.3.3

com.clearspring.analytics

stream

2.9.6

com.databricks

Rserve

1.8-3

com.databricks

jets3t

0.7.1-0

com.databricks.scalapb

compilerplugin_2.12

0.4.15-10

com.databricks.scalapb

scalapb-runtime_2.12

0.4.15-10

com.esotericsoftware

kryo-shaded

4.0.2

com.esotericsoftware

minlog

1.3.0

com.fasterxml

classmate

1.3.4

com.fasterxml.jackson.core

jackson-annotations

2.13.3

com.fasterxml.jackson.core

jackson-core

2.13.3

com.fasterxml.jackson.core

jackson-databind

2.13.3

com.fasterxml.jackson.dataformat

jackson-dataformat-cbor

2.13.3

com.fasterxml.jackson.datatype

jackson-datatype-joda

2.13.3

com.fasterxml.jackson.module

jackson-module-paranamer

2.13.3

com.fasterxml.jackson.module

jackson-module-scala_2.12

2.13.3

com.github.ben-manes.caffeine

caffeine

2.3.4

com.github.fommil

jniloader

1.1

com.github.fommil.netlib

core

1.1.2

com.github.fommil.netlib

native_ref-java

1.1

com.github.fommil.netlib

native_ref-java-natives

1.1

com.github.fommil.netlib

native_system-java

1.1

com.github.fommil.netlib

native_system-java-natives

1.1

com.github.fommil.netlib

netlib-native_ref-linux-x86_64-natives

1.1

com.github.fommil.netlib

netlib-native_system-linux-x86_64-natives

1.1

com.github.luben

zstd-jni

1.5.2-1

com.github.wendykierp

JTransforms

3.1

com.google.code.findbugs

jsr305

3.0.0

com.google.code.gson

gson

2.8.6

com.google.crypto.tink

tink

1.6.1

com.google.flatbuffers

flatbuffers-java

1.12.0

com.google.guava

guava

15.0

com.google.protobuf

protobuf-java

2.6.1

com.h2database

h2

2.0.204

com.helger

profiler

1.1.1

com.jcraft

jsch

0.1.50

com.jolbox

bonecp

0.8.0.RELEASE

com.lihaoyi

sourcecode_2.12

0.1.9

com.microsoft.azure

azure-data-lake-store-sdk

2.3.9

com.ning

compress-lzf

1.1

com.sun.mail

javax.mail

1.5.2

com.tdunning

json

1.8

com.thoughtworks.paranamer

paranamer

2.8

com.trueaccord.lenses

lenses_2.12

0.4.12

com.twitter

chill-java

0.10.0

com.twitter

chill_2.12

0.10.0

com.twitter

util-app_2.12

7.1.0

com.twitter

util-core_2.12

7.1.0

com.twitter

util-function_2.12

7.1.0

com.twitter

util-jvm_2.12

7.1.0

com.twitter

util-lint_2.12

7.1.0

com.twitter

util-registry_2.12

7.1.0

com.twitter

util-stats_2.12

7.1.0

com.typesafe

config

1.2.1

com.typesafe.scala-logging

scala-logging_2.12

3.7.2

com.univocity

univocity-parsers

2.9.1

com.zaxxer

HikariCP

4.0.3

commons-cli

commons-cli

1.5.0

commons-codec

commons-codec

1.15

commons-collections

commons-collections

3.2.2

commons-dbcp

commons-dbcp

1.4

commons-fileupload

commons-fileupload

1.3.3

commons-httpclient

commons-httpclient

3.1

commons-io

commons-io

2.11.0

commons-lang

commons-lang

2.6

commons-logging

commons-logging

1.1.3

commons-pool

commons-pool

1.5.4

dev.ludovic.netlib

arpack

2.2.1

dev.ludovic.netlib

blas

2.2.1

dev.ludovic.netlib

lapack

2.2.1

hive-2.3__hadoop-3.2

jets3t-0.7

liball_deps_2.12

info.ganglia.gmetric4j

gmetric4j

1.0.10

io.airlift

aircompressor

0.21

io.delta

delta-sharing-spark_2.12

0.4.0

io.dropwizard.metrics

metrics-core

4.1.1

io.dropwizard.metrics

metrics-graphite

4.1.1

io.dropwizard.metrics

metrics-healthchecks

4.1.1

io.dropwizard.metrics

metrics-jetty9

4.1.1

io.dropwizard.metrics

metrics-jmx

4.1.1

io.dropwizard.metrics

metrics-json

4.1.1

io.dropwizard.metrics

metrics-jvm

4.1.1

io.dropwizard.metrics

metrics-servlets

4.1.1

io.netty

netty-all

4.1.74.Final

io.netty

netty-buffer

4.1.74.Final

io.netty

netty-codec

4.1.74.Final

io.netty

netty-common

4.1.74.Final

io.netty

netty-handler

4.1.74.Final

io.netty

netty-resolver

4.1.74.Final

io.netty

netty-tcnative-classes

2.0.48.Final

io.netty

netty-transport

4.1.74.Final

io.netty

netty-transport-classes-epoll

4.1.74.Final

io.netty

netty-transport-classes-kqueue

4.1.74.Final

io.netty

netty-transport-native-epoll-linux-aarch_64

4.1.74.Final

io.netty

netty-transport-native-epoll-linux-x86_64

4.1.74.Final

io.netty

netty-transport-native-kqueue-osx-aarch_64

4.1.74.Final

io.netty

netty-transport-native-kqueue-osx-x86_64

4.1.74.Final

io.netty

netty-transport-native-unix-common

4.1.74.Final

io.prometheus

simpleclient

0.7.0

io.prometheus

simpleclient_common

0.7.0

io.prometheus

simpleclient_dropwizard

0.7.0

io.prometheus

simpleclient_pushgateway

0.7.0

io.prometheus

simpleclient_servlet

0.7.0

io.prometheus.jmx

collector

0.12.0

jakarta.annotation

jakarta.annotation-api

1.3.5

jakarta.servlet

jakarta.servlet-api

4.0.3

jakarta.validation

jakarta.validation-api

2.0.2

jakarta.ws.rs

jakarta.ws.rs-api

2.1.6

javax.activation

activation

1.1.1

javax.annotation

javax.annotation-api

1.3.2

javax.el

javax.el-api

2.2.4

javax.jdo

jdo-api

3.0.1

javax.transaction

jta

1.1

javax.transaction

transaction-api

1.1

javax.xml.bind

jaxb-api

2.2.11

javolution

javolution

5.5.1

jline

jline

2.14.6

joda-time

joda-time

2.10.13

maven-trees

hive-2.3__hadoop-3.2

liball_deps_2.12

net.java.dev.jna

jna

5.8.0

net.razorvine

pickle

1.2

net.sf.jpam

jpam

1.1

net.sf.opencsv

opencsv

2.3

net.sf.supercsv

super-csv

2.2.0

net.snowflake

snowflake-ingest-sdk

0.9.6

net.snowflake

snowflake-jdbc

3.13.14

net.snowflake

spark-snowflake_2.12

2.10.0-spark_3.1

net.sourceforge.f2j

arpack_combined_all

0.1

org.acplt.remotetea

remotetea-oncrpc

1.1.2

org.antlr

ST4

4.0.4

org.antlr

antlr-runtime

3.5.2

org.antlr

antlr4-runtime

4.8

org.antlr

stringtemplate

3.2.1

org.apache.ant

ant

1.9.2

org.apache.ant

ant-jsch

1.9.2

org.apache.ant

ant-launcher

1.9.2

org.apache.arrow

arrow-format

7.0.0

org.apache.arrow

arrow-memory-core

7.0.0

org.apache.arrow

arrow-memory-netty

7.0.0

org.apache.arrow

arrow-vector

7.0.0

org.apache.avro

avro

1.11.0

org.apache.avro

avro-ipc

1.11.0

org.apache.avro

avro-mapred

1.11.0

org.apache.commons

commons-collections4

4.4

org.apache.commons

commons-compress

1.21

org.apache.commons

commons-crypto

1.1.0

org.apache.commons

commons-lang3

3.12.0

org.apache.commons

commons-math3

3.6.1

org.apache.commons

commons-text

1.9

org.apache.curator

curator-client

2.13.0

org.apache.curator

curator-framework

2.13.0

org.apache.curator

curator-recipes

2.13.0

org.apache.derby

derby

10.14.2.0

org.apache.hadoop

hadoop-client-api

3.3.2-databricks

org.apache.hadoop

hadoop-client-runtime

3.3.2

org.apache.hive

hive-beeline

2.3.9

org.apache.hive

hive-cli

2.3.9

org.apache.hive

hive-jdbc

2.3.9

org.apache.hive

hive-llap-client

2.3.9

org.apache.hive

hive-llap-common

2.3.9

org.apache.hive

hive-serde

2.3.9

org.apache.hive

hive-shims

2.3.9

org.apache.hive

hive-storage-api

2.7.2

org.apache.hive.shims

hive-shims-0.23

2.3.9

org.apache.hive.shims

hive-shims-common

2.3.9

org.apache.hive.shims

hive-shims-scheduler

2.3.9

org.apache.httpcomponents

httpclient

4.5.13

org.apache.httpcomponents

httpcore

4.4.14

org.apache.ivy

ivy

2.5.0

org.apache.logging.log4j

log4j-1.2-api

2.17.2

org.apache.logging.log4j

log4j-api

2.17.2

org.apache.logging.log4j

log4j-core

2.17.2

org.apache.logging.log4j

log4j-slf4j-impl

2.17.2

org.apache.mesos

mesos-shaded-protobuf

1.4.0

org.apache.orc

orc-core

1.7.4

org.apache.orc

orc-mapreduce

1.7.4

org.apache.orc

orc-shims

1.7.4

org.apache.parquet

parquet-column

1.12.0-databricks-0004

org.apache.parquet

parquet-common

1.12.0-databricks-0004

org.apache.parquet

parquet-encoding

1.12.0-databricks-0004

org.apache.parquet

parquet-format-structures

1.12.0-databricks-0004

org.apache.parquet

parquet-hadoop

1.12.0-databricks-0004

org.apache.parquet

parquet-jackson

1.12.0-databricks-0004

org.apache.thrift

libfb303

0.9.3

org.apache.thrift

libthrift

0.12.0

org.apache.xbean

xbean-asm9-shaded

4.20

org.apache.yetus

audience-annotations

0.5.0

org.apache.zookeeper

zookeeper

3.6.2

org.apache.zookeeper

zookeeper-jute

3.6.2

org.checkerframework

checker-qual

3.5.0

org.codehaus.jackson

jackson-core-asl

1.9.13

org.codehaus.jackson

jackson-mapper-asl

1.9.13

org.codehaus.janino

commons-compiler

3.0.16

org.codehaus.janino

janino

3.0.16

org.datanucleus

datanucleus-api-jdo

4.2.4

org.datanucleus

datanucleus-core

4.1.17

org.datanucleus

datanucleus-rdbms

4.1.19

org.datanucleus

javax.jdo

3.2.0-m3

org.eclipse.jetty

jetty-client

9.4.46.v20220331

org.eclipse.jetty

jetty-continuation

9.4.46.v20220331

org.eclipse.jetty

jetty-http

9.4.46.v20220331

org.eclipse.jetty

jetty-io

9.4.46.v20220331

org.eclipse.jetty

jetty-jndi

9.4.46.v20220331

org.eclipse.jetty

jetty-plus

9.4.46.v20220331

org.eclipse.jetty

jetty-proxy

9.4.46.v20220331

org.eclipse.jetty

jetty-security

9.4.46.v20220331

org.eclipse.jetty

jetty-server

9.4.46.v20220331

org.eclipse.jetty

jetty-servlet

9.4.46.v20220331

org.eclipse.jetty

jetty-servlets

9.4.46.v20220331

org.eclipse.jetty

jetty-util

9.4.46.v20220331

org.eclipse.jetty

jetty-util-ajax

9.4.46.v20220331

org.eclipse.jetty

jetty-webapp

9.4.46.v20220331

org.eclipse.jetty

jetty-xml

9.4.46.v20220331

org.eclipse.jetty.websocket

websocket-api

9.4.46.v20220331

org.eclipse.jetty.websocket

websocket-client

9.4.46.v20220331

org.eclipse.jetty.websocket

websocket-common

9.4.46.v20220331

org.eclipse.jetty.websocket

websocket-server

9.4.46.v20220331

org.eclipse.jetty.websocket

websocket-servlet

9.4.46.v20220331

org.fusesource.leveldbjni

leveldbjni-all

1.8

org.glassfish.hk2

hk2-api

2.6.1

org.glassfish.hk2

hk2-locator

2.6.1

org.glassfish.hk2

hk2-utils

2.6.1

org.glassfish.hk2

osgi-resource-locator

1.0.3

org.glassfish.hk2.external

aopalliance-repackaged

2.6.1

org.glassfish.hk2.external

jakarta.inject

2.6.1

org.glassfish.jersey.containers

jersey-container-servlet

2.34

org.glassfish.jersey.containers

jersey-container-servlet-core

2.34

org.glassfish.jersey.core

jersey-client

2.34

org.glassfish.jersey.core

jersey-common

2.34

org.glassfish.jersey.core

jersey-server

2.34

org.glassfish.jersey.inject

jersey-hk2

2.34

org.hibernate.validator

hibernate-validator

6.1.0.Final

org.javassist

javassist

3.25.0-GA

org.jboss.logging

jboss-logging

3.3.2.Final

org.jdbi

jdbi

2.63.1

org.jetbrains

annotations

17.0.0

org.joda

joda-convert

1.7

org.jodd

jodd-core

3.5.2

org.json4s

json4s-ast_2.12

3.7.0-M11

org.json4s

json4s-core_2.12

3.7.0-M11

org.json4s

json4s-jackson_2.12

3.7.0-M11

org.json4s

json4s-scalap_2.12

3.7.0-M11

org.lz4

lz4-java

1.8.0

org.mariadb.jdbc

mariadb-java-client

2.7.4

org.objenesis

objenesis

2.5.1

org.postgresql

postgresql

42.3.3

org.roaringbitmap

RoaringBitmap

0.9.25

org.roaringbitmap

shims

0.9.25

org.rocksdb

rocksdbjni

6.24.2

org.rosuda.REngine

REngine

2.1.0

org.scala-lang

scala-compiler_2.12

2.12.14

org.scala-lang

scala-library_2.12

2.12.14

org.scala-lang

scala-reflect_2.12

2.12.14

org.scala-lang.modules

scala-collection-compat_2.12

2.4.3

org.scala-lang.modules

scala-parser-combinators_2.12

1.1.2

org.scala-lang.modules

scala-xml_2.12

1.2.0

org.scala-sbt

test-interface

1.0

org.scalacheck

scalacheck_2.12

1.14.2

org.scalactic

scalactic_2.12

3.0.8

org.scalanlp

breeze-macros_2.12

1.2

org.scalanlp

breeze_2.12

1.2

org.scalatest

scalatest_2.12

3.0.8

org.slf4j

jcl-over-slf4j

1.7.36

org.slf4j

jul-to-slf4j

1.7.36

org.slf4j

slf4j-api

1.7.36

org.spark-project.spark

unused

1.0.0

org.threeten

threeten-extra

1.5.0

org.tukaani

xz

1.8

org.typelevel

algebra_2.12

2.0.1

org.typelevel

cats-kernel_2.12

2.1.1

org.typelevel

macro-compat_2.12

1.1.1

org.typelevel

spire-macros_2.12

0.17.0

org.typelevel

spire-platform_2.12

0.17.0

org.typelevel

spire-util_2.12

0.17.0

org.typelevel

spire_2.12

0.17.0

org.wildfly.openssl

wildfly-openssl

1.0.7.Final

org.xerial

sqlite-jdbc

3.8.11.2

org.xerial.snappy

snappy-java

1.1.8.4

org.yaml

snakeyaml

1.24

oro

oro

2.0.8

pl.edu.icm

JLargeArrays

1.5

software.amazon.ion

ion-java

1.0.2

stax

stax-api

1.0.1