Databricks Runtime 9.0 (EoS)

Note

Support for this Databricks Runtime version has ended. For the end-of-support date, see End-of-support history. For all supported Databricks Runtime versions, see Databricks Runtime release notes versions and compatibility.

The following release notes provide information about Databricks Runtime 9.0 and Databricks Runtime 9.0 Photon, powered by Apache Spark 3.1.2. Databricks released this version in August 2021. Photon is in Public Preview.

Note

These release notes may include references to features that are not available on Google Cloud as of this release.

Correction

A previous version of these release notes incorrectly stated that Apache Parquet dependencies were upgraded from 1.10 to 1.12. In fact, Parquet dependencies remain at version 1.10. The incorrect release note has been removed.

New features and improvements

New API for summary statistics of datasets (Public Preview)

The new dbutils.data.summarize command in Databricks Utilities allows you to launch a Spark job that automatically computes summary statistics on the columns of a Spark DataFrame and then displays the results interactively. This function is available in Scala and Python. See Data utility (dbutils.data).

Easier external data source configuration for the Azure Synapse connector

The new externalDataSource option in the Query data in Azure Synapse Analytics connector allows you to use a pre-provisioned external data source to read from an Azure Synapse database. The externalDataSource option removes the need for the CONTROL permission previously required.

When setting externalDataSource, the external data source and the tempDir option used to configure temporary storage must reference the same container in the storage account.

Optionally limit the session to a specified duration for the Amazon Redshift connector

The new fs.s3a.assumed.role.session.duration option in the Query Amazon Redshift using Databricks connector allows you to optionally set a duration for the session when Redshift is accessing the temporary S3 bucket with an assumed role.

Auto Loader

Optimized file listing

Auto Loader optimizations provide performance improvements and cost savings when listing nested directories in cloud storage, including AWS S3, Azure Data Lake Storage Gen2 (ADLS Gen2), and Google Cloud Storage (GCS).

For example, if you had files being uploaded as /some/path/YYYY/MM/DD/HH/fileName, to find all the files in these directories, Auto Loader used to do a parallel listing of all subdirectories, causing 365 (per day) * 24 (per hour) = 8760 LIST API directory calls to the underlying storage for each year directory. By receiving a flattened response from these storage systems, Auto Loader reduces the number of API calls to the number of files in the storage system divided by the number of results returned by each API call (1000 for S3, 5000 for ADLS Gen2, and 1024 for GCS), greatly reducing your cloud costs.

Optimized image data storage

Auto Loader can now auto-detect image data that is being ingested and optimize its storage in Delta tables to improve read and write performance. See Ingest image or binary data to Delta Lake for ML.

Image thumbnails for binary files (Public Preview)

Images in binaryFile format loaded or saved as Delta tables using Auto Loader have annotations attached so that the image thumbnails appear when you display the table in a Databricks notebook. For more information, see Images.

DirectoryRename events enable atomic processing of multiple files

Auto Loader streams created in Databricks Runtime 9.0 and above on Azure Data Lake Storage Gen2 set up file event notifications to include directory renames and listen to RenameDirectory events. You can use directory renames to make multiple files appear atomically to Auto Loader.

SQL

Exclude columns in SELECT * (Public Preview)

SELECT * now supports an EXCEPT keyword, which allows you to exclude specified top level columns from the expansion. For example, SELECT * EXCEPT (b) FROM tbl from a table with schema (a, b, c) expands to (a, c).

SQL scalar functions (Public Preview)

CREATE FUNCTION now supports SQL scalar functions. You can create scalar functions that take a set of arguments and return a single scalar type value. The SQL function body can be any expression. For example:

CREATE FUNCTION square(x DOUBLE) RETURNS DOUBLE RETURN x * x;
SELECT square(2);

For details, see CREATE FUNCTION (SQL and Python).

Reference preceding aliases and columns in FROM subqueries (Public Preview)

Subqueries in the FROM clause of a query can now be preceded by the LATERAL keyword which allows them to reference aliases and columns in the preceding FROM items. For example:

SELECT * FROM t1, LATERAL (SELECT * FROM t2 WHERE t1.c1 = t2.c1)

The LATERAL keyword supports INNER, CROSS, and LEFT (OUTER) JOIN.

See Parameters and Parameters.

R support

Notebook-scoped R libraries (Public Preview)

Notebook-scoped libraries allow you to install libraries and create an environment scoped to a notebook session. These libraries do not affect other notebooks running on the same cluster. The libraries are available both on the driver and worker nodes, so you can reference them in user-defined functions. See Notebook-scoped R libraries.

Warning messages in R notebooks

The default value of the warn option is now set to 1 inside R notebooks. As a result, all warnings are now exposed as part of the command result. To learn more about the warn option, see Options Settings.

Avoid redo by specifying initial state for Structured Streaming stateful processing

You can now specify a user defined initial state for structured streaming stateful processing using [flat]MapGroupsWithState operator.

See Specify initial state for mapGroupsWithState.

Install libraries from CRAN

You can now install cluster-scoped libraries from CRAN. For details, see CRAN package.

A low-shuffle implementation of the Delta MERGE INTO command is now available (Public Preview)

The Delta MERGE INTO command has a new implementation available which reduces shuffling of unmodified rows. This improves performance of the command and helps to preserve existing clustering on the table, such as Z-ordering. To enable low shuffle merge, set spark.databricks.delta.merge.enableLowShuffle to true. See Low shuffle merge on Databricks.

Bug fixes

  • You can no longer overwrite a view by creating a view with the same name.

Library upgrades

  • Upgraded Python libraries:

    • certifi from 2021.5.30 to 2020.12.5

    • chardet from 3.0.4 to 4.0.0

    • Cython from 0.29.21 to 0.29.23

    • decorator from 4.4.2 to 5.0.6

    • ipython from 7.19.0 to 7.22.0

    • joblib from 0.17.0 to 1.0.1

    • jupyter-client from 6.1.7 to 6.1.12

    • jupyter-core from 4.6.3 to 4.7.1

    • kiwisolver from 1.3.0 to 1.3.1

    • matplotlib from 3.2.2 to 3.4.2

    • pandas from 1.1.5 to 1.2.4

    • pip from 20.2.4 to 21.0.1

    • prompt-toolkit from 3.0.8 to 3.0.17

    • protobuf from 3.17.3 to 3.17.2

    • ptyprocess from 0.6.0 to 0.7.0

    • pyarrow from 1.0.1 to 4.0.0

    • Pygments from 2.7.2 to 2.8.1

    • pyzmq from 19.0.2 to 20.0.0

    • requests from 2.24.0 to 2.25.1

    • s3transfer from 0.3.6 to 0.3.7

    • scikit-learn from 0.23.2 to 0.24.1

    • scipy from 1.5.2 to 1.6.2

    • seaborn from 0.10.0 to 0.11.1

    • setuptools from 50.3.1 to 52.0.0

    • statsmodels from 0.12.0 to 0.12.2

    • tornado from 6.0.4 to 6.1

    • virtualenv from 20.2.1 to 20.4.1

    • wheel from 0.35.1 to 0.36.2

  • Upgraded R libraries:

    • Matrix from 1.3-3 to 1.3-4

Apache Spark

Databricks Runtime 9.0 includes Apache Spark 3.1.2. This release includes all Spark fixes and improvements included in Databricks Runtime 8.4 (EoS), as well as the following additional bug fixes and improvements made to Spark:

  • [SPARK-35886] [SQL][3.1] PromotePrecision should not overwrite genCodePromotePrecision should not overwrite genCode

  • [SPARK-35879] [CORE][SHUFFLE] Fix performance regression caused by collectFetchRequests

  • [SPARK-35817] [SQL][3.1] Restore performance of queries against wide Avro tables

  • [SPARK-35841] [SQL] Casting string to decimal type doesn’t work if the…

  • [SPARK-35783] [SQL] Set the list of read columns in the task configuration to reduce reading of ORC data

  • [SPARK-35576] [SQL][3.1] Redact the sensitive info in the result of Set command

  • [SPARK-35449] [SQL][3.1] Only extract common expressions from CaseWhen values if elseValue is set

  • [SPARK-35288] [SQL] StaticInvoke should find the method without exact argument classes match

  • [SPARK-34794] [SQL] Fix lambda variable name issues in nested DataFrame functions

  • [SPARK-35278] [SQL] Invoke should find the method with correct number of parameters

  • [SPARK-35226] [SQL] Support refreshKrb5Config option in JDBC datasources

  • [SPARK-35244] [SQL] Invoke should throw the original exception

  • [SPARK-35213] [SQL] Keep the correct ordering of nested structs in chained withField operations

  • [SPARK-35087] [UI] Some columns in table Aggregated Metrics by Executor of stage-detail page shows incorrectly.

  • [SPARK-35168] [SQL] mapred.reduce.tasks should be shuffle.partitions not adaptive.coalescePartitions.initialPartitionNum

  • [SPARK-35127] [UI] When we switch between different stage-detail pages, the entry item in the newly-opened page may be blank

  • [SPARK-35142] [PYTHON][ML] Fix incorrect return type for rawPredictionUDF in OneVsRestModel

  • [SPARK-35096] [SQL] SchemaPruning should adhere spark.sql.caseSensitive config

  • [SPARK-34639] [SQL][3.1] RelationalGroupedDataset.alias should not create UnresolvedAlias

  • [SPARK-35080] [SQL] Only allow a subset of correlated equality predicates when a subquery is aggregated

  • [SPARK-35117] [UI] Change progress bar back to highlight ratio of tasks in progress

  • [SPARK-35136] Remove initial null value of LiveStage.info

  • [SPARK-34834] [NETWORK] Fix a potential Netty memory leak in TransportResponseHandler

  • [SPARK-35045] [SQL] Add an internal option to control input buffer in univocity

  • [SPARK-35014] Fix the PhysicalAggregation pattern to not rewrite foldable expressions

  • [SPARK-35019] [PYTHON][SQL] Fix type hints mismatches in pyspark.sql.*

  • [SPARK-34926] [SQL][3.1] PartitioningUtils.getPathFragment() should respect partition value is null

  • [SPARK-34630] [PYTHON] Add typehint for pyspark.version

  • [SPARK-34963] [SQL] Fix nested column pruning for extracting case-insensitive struct field from array of struct

  • [SPARK-34988] [CORE][3.1] Upgrade Jetty for CVE-2021-28165

  • [SPARK-34922] [SQL][3.1] Use a relative cost comparison function in the CBO

  • [SPARK-34970] [SQL][SECURITY][3.1] Redact map-type options in the output of explain()

  • [SPARK-34923] [SQL] Metadata output should be empty for more plans

  • [SPARK-34949] [CORE] Prevent BlockManager reregister when Executor is shutting down

  • [SPARK-34939] [CORE] Throw fetch failure exception when unable to deserialize broadcasted map statuses

  • [SPARK-34909] [SQL] Fix conversion of negative to unsigned in conv()

  • [SPARK-34845] [CORE] ProcfsMetricsGetter shouldn’t return partial procfs metrics

  • [SPARK-34814] [SQL] LikeSimplification should handle NULL

  • [SPARK-34876] [SQL] Fill defaultResult of non-nullable aggregates

  • [SPARK-34829] [SQL] Fix higher order function results

  • [SPARK-34840] [SHUFFLE] Fixes cases of corruption in merged shuffle …

  • [SPARK-34833] [SQL] Apply right-padding correctly for correlated subqueries

  • [SPARK-34630] [PYTHON][SQL] Added typehint for pyspark.sql.Column.contains

  • [SPARK-34763] [SQL] col(), $”name” and df(“name”) should handle quoted column names properly

  • [SPARK-33482] [SPARK-34756] [SQL] Fix FileScan equality check

  • [SPARK-34790] [CORE] Disable fetching shuffle blocks in batch when io encryption is enabled

  • [SPARK-34803] [PYSPARK] Pass the raised ImportError if pandas or pyarrow fail to import

  • [SPARK-34225] [CORE] Don’t encode further when a URI form string is passed to addFile or addJar

  • [SPARK-34811] [CORE] Redact fs.s3a.access.key like secret and token

  • [SPARK-34796] [SQL][3.1] Initialize counter variable for LIMIT code-gen in doProduce()

  • [SPARK-34128] [SQL] Suppress undesirable TTransportException warnings involved in THRIFT-4805

  • [SPARK-34776] [SQL] Nested column pruning should not prune Window produced attributes

  • [SPARK-34087] [3.1][SQL] Fix memory leak of ExecutionListenerBus

  • [SPARK-34772] [SQL] RebaseDateTime loadRebaseRecords should use Spark classloader instead of context

  • [SPARK-34719] [SQL][3.1] Correctly resolve the view query with duplicated column names

  • [SPARK-34766] [SQL][3.1] Do not capture maven config for views

  • [SPARK-34731] [CORE] Avoid ConcurrentModificationException when redacting properties in EventLoggingListener

  • [SPARK-34737] [SQL][3.1] Cast input float to double in TIMESTAMP_SECONDS

  • [SPARK-34749] [SQL][3.1] Simplify ResolveCreateNamedStruct

  • [SPARK-34768] [SQL] Respect the default input buffer size in Univocity

  • [SPARK-34770] [SQL] InMemoryCatalog.tableExists should not fail if database doesn’t exist

  • [SPARK-34504] [SQL] Avoid unnecessary resolving of SQL temp views for DDL commands

  • [SPARK-34727] [SQL] Fix discrepancy in casting float to timestamp

  • [SPARK-34723] [SQL] Correct parameter type for subexpression elimination under whole-stage

  • [SPARK-34724] [SQL] Fix Interpreted evaluation by using getMethod instead of getDeclaredMethod

  • [SPARK-34713] [SQL] Fix group by CreateStruct with ExtractValue

  • [SPARK-34697] [SQL] Allow DESCRIBE FUNCTION and SHOW FUNCTIONS explain about || (string concatenation operator)

  • [SPARK-34682] [SQL] Use PrivateMethodTester instead of reflection

  • [SPARK-34682] [SQL] Fix regression in canonicalization error check in CustomShuffleReaderExec

  • [SPARK-34681] [SQL] Fix bug for full outer shuffled hash join when building left side with non-equal condition

  • [SPARK-34545] [SQL] Fix issues with valueCompare feature of pyrolite

  • [SPARK-34607] [SQL][3.1] Add Utils.isMemberClass to fix a malformed class name error on jdk8u

  • [SPARK-34596] [SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode

  • [SPARK-34613] [SQL] Fix view does not capture disable hint config

  • [SPARK-32924] [WEBUI] Make duration column in master UI sorted in the correct order

  • [SPARK-34482] [SS] Correct the active SparkSession for StreamExecution.logicalPlan

  • [SPARK-34567] [SQL] CreateTableAsSelect should update metrics too

  • [SPARK-34599] [SQL] Fix the issue that INSERT INTO OVERWRITE doesn’t support partition columns containing dot for DSv2

  • [SPARK-34577] [SQL] Fix drop/add columns to a dataset of DESCRIBE NAMESPACE

  • [SPARK-34584] [SQL] Static partition should also follow StoreAssignmentPolicy when insert into v2 tables

  • [SPARK-34555] [SQL] Resolve metadata output from DataFrame

  • [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

  • [SPARK-34547] [SQL] Only use metadata columns for resolution as last resort

  • [SPARK-34417] [SQL] org.apache.spark.sql.DataFrameNaFunctions.fillMap fails for column name having a dot

  • [SPARK-34561] [SQL] Fix drop/add columns from/to a dataset of v2 DESCRIBE TABLE

  • [SPARK-34556] [SQL] Checking duplicate static partition columns should respect case sensitive conf

  • [SPARK-34392] [SQL] Support ZoneOffset +h:mm in DateTimeUtils. getZoneId

  • [SPARK-34550] [SQL] Skip InSet null value during push filter to Hive metastore

  • [SPARK-34543] [SQL] Respect the spark.sql.caseSensitive config while resolving partition spec in v1 SET LOCATION

  • [SPARK-34436] [SQL] DPP support LIKE ANY/ALL expression

  • [SPARK-34531] [CORE] Remove Experimental API tag in PrometheusServlet

  • [SPARK-34497] [SQL] Fix built-in JDBC connection providers to restore JVM security context changes

  • [SPARK-34515] [SQL] Fix NPE if InSet contains null value during getPartitionsByFilter

  • [SPARK-34490] [SQL] Analysis should fail if the view refers a dropped table

  • [SPARK-34473] [SQL] Avoid NPE in DataFrameReader.schema(StructType)

  • [SPARK-34384] [CORE] Add missing docs for ResourceProfile APIs

  • [SPARK-34373] [SQL] HiveThriftServer2 startWithContext may hang with a race issue

  • [SPARK-20977] [CORE] Use a non-final field for the state of CollectionAccumulator

  • [SPARK-34421] [SQL] Resolve temporary functions and views in views with CTEs

  • [SPARK-34431] [CORE] Only load hive-site.xml once

  • [SPARK-34405] [CORE] Fix mean value of timersLabels in the PrometheusServlet class

  • [SPARK-33438] [SQL] Eagerly init objects with defined SQL Confs for command set -v

  • [SPARK-34158] Incorrect url of the only developer Matei in pom.xml

  • [SPARK-34346] [CORE][SQL][3.1] io.file.buffer.size set by spark.buffer.size will override by loading hive-site.xml accidentally may cause perf regression

  • [SPARK-34359] [SQL][3.1] Add a legacy config to restore the output schema of SHOW DATABASES

  • [SPARK-34331] [SQL] Speed up DS v2 metadata col resolution

  • [SPARK-34318] [SQL][3.1] Dataset.colRegex should work with column names and qualifiers which contain newlines

  • [SPARK-34326] [CORE][SQL] Fix UTs added in SPARK-31793 depending on the length of temp path

  • [SPARK-34319] [SQL] Resolve duplicate attributes for FlatMapCoGroupsInPandas/MapInPandas

  • [SPARK-34310] [CORE][SQL] Replaces map and flatten with flatMap

  • [SPARK-34083] [SQL][3.1] Using TPCDS original definitions for char/varchar colums

  • [SPARK-34233] [SQL][3.1] FIX NPE for char padding in binary comparison

  • [SPARK-34270] [SS] Combine StateStoreMetrics should not override StateStoreCustomMetric

  • [SPARK-34144] [SQL] Exception thrown when trying to write LocalDate and Instant values to a JDBC relation

  • [SPARK-34273] [CORE] Do not reregister BlockManager when SparkContext is stopped

  • [SPARK-34262] [SQL][3.1] Refresh cached data of v1 table in ALTER TABLE .. SET LOCATION

  • [SPARK-34275] [CORE][SQL][MLLIB] Replaces filter and size with count

  • [SPARK-34260] [SQL] Fix UnresolvedException when creating temp view twice

  • [SPARK-33867] [SQL] Instant and LocalDate values aren’t handled when generating SQL queries

  • [SPARK-34193] [CORE] TorrentBroadcast block manager decommissioning race fix

  • [SPARK-34221] [WEBUI] Ensure if a stage fails in the UI page, the corresponding error message can be displayed correctly

  • [SPARK-34236] [SQL] Fix v2 Overwrite w/ null static partition raise Cannot translate expression to source filter: null

  • [SPARK-34212] [SQL] Fix incorrect decimal reading from Parquet files

  • [SPARK-34244] [SQL] Remove the Scala function version of regexp_extract_all

  • [SPARK-34235] [SS] Make spark.sql.hive as a private package

  • [SPARK-34232] [CORE] Redact SparkListenerEnvironmentUpdate event in log

  • [SPARK-34229] [SQL] Avro should read decimal values with the file schema

  • [SPARK-34223] [SQL] FIX NPE for static partition with null in InsertIntoHadoopFsRelationCommand

  • [SPARK-34192] [SQL] Move char padding to write side and remove length check on read side too

  • [SPARK-34203] [SQL] Convert null partition values to __HIVE_DEFAULT_PARTITION__ in v1 In-Memory catalog

  • [SPARK-33726] [SQL] Fix for Duplicate field names during Aggregation

  • [SPARK-34133] [AVRO] Respect case sensitivity when performing Catalyst-to-Avro field matching

  • [SPARK-34187] [SS] Use available offset range obtained during polling when checking offset validation

  • [SPARK-34052] [SQL][3.1] store SQL text for a temp view created using “CACHE TABLE .. AS SELECT …”

  • [SPARK-34213] [SQL] Refresh cached data of v1 table in LOAD DATA

  • [SPARK-34191] [PYTHON][SQL] Add typing for udf overload

  • [SPARK-34200] [SQL] Ambiguous column reference should consider attribute availability

  • [SPARK-33813] [SQL][3.1] Fix the issue that JDBC source can’t treat MS SQL Server’s spatial types

  • [SPARK-34178] [SQL] Copy tags for the new node created by MultiInstanceRelation.newInstance

  • [SPARK-34005] [CORE][3.1] Update peak memory metrics for each Executor on task end

  • [SPARK-34115] [CORE] Check SPARK_TESTING as lazy val to avoid slowdown

  • [SPARK-34153] [SQL][3.1][3.0] Remove unused getRawTable() from HiveExternalCatalog.alterPartitions()

  • [SPARK-34130] [SQL] Impove preformace for char varchar padding and length check with StaticInvoke

  • [SPARK-34027] [SQL][3.1] Refresh cache in ALTER TABLE .. RECOVER PARTITIONS

  • [SPARK-34151] [SQL] Replaces java.io.File.toURL with java.io.File.toURI.toURL

  • [SPARK-34140] [SQL][3.1] Move QueryCompilationErrors.scala to org/apache/spark/sql/errors

  • [SPARK-34080] [ML][PYTHON] Add UnivariateFeatureSelector

  • [SPARK-33790] [CORE][3.1] Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader

  • [SPARK-34118] [CORE][SQL][3.1] Replaces filter and check for emptiness with exists or forall

  • [SPARK-34114] [SQL] should not trim right for read-side char length check and padding

  • [SPARK-34086] [SQL][3.1] RaiseError generates too much code and may fails codegen in length check for char varchar

  • [SPARK-34075] [SQL][CORE] Hidden directories are being listed for partition inference

  • [SPARK-34076] [SQL] SQLContext.dropTempTable fails if cache is non-empty

  • [SPARK-34084] [SQL][3.1] Fix auto updating of table stats in ALTER TABLE .. ADD PARTITION

  • [SPARK-34090] [SS] Cache HadoopDelegationTokenManager.isServiceEnabled result used in KafkaTokenUtil.needTokenUpdate

  • [SPARK-34069] [CORE] Kill barrier tasks should respect SPARK_JOB_INTERRUPT_ON_CANCEL

  • [SPARK-34091] [SQL] Shuffle batch fetch should be able to disable after it’s been enabled

  • [SPARK-34059] [SQL][CORE][3.1] Use for/foreach rather than map to make sure execute it eagerly

  • [SPARK-34002] [SQL] Fix the usage of encoder in ScalaUDF

  • [SPARK-34060] [SQL][3.1] Fix Hive table caching while updating stats by ALTER TABLE .. DROP PARTITION

  • [SPARK-31952] [SQL] Fix incorrect memory spill metric when doing Aggregate

  • [SPARK-33591] [SQL][3.1] Recognize null in partition spec values

  • [SPARK-34055] [SQL][3.1] Refresh cache in ALTER TABLE .. ADD PARTITION

  • [SPARK-34039] [SQL][3.1] ReplaceTable should invalidate cache

  • [SPARK-34003] [SQL] Fix Rule conflicts between PaddingAndLengthCheckForCharVarchar and ResolveAggregateFunctions

  • [SPARK-33938] [SQL][3.1] Optimize Like Any/All by LikeSimplification

  • [SPARK-34021] [R] Fix hyper links in SparkR documentation for CRAN submission

  • [SPARK-34011] [SQL][3.1][3.0] Refresh cache in ALTER TABLE .. RENAME TO PARTITION

  • [SPARK-33948] [SQL] Fix CodeGen error of MapObjects.doGenCode method in Scala 2.13

  • [SPARK-33635] [SS] Adjust the order of check in KafkaTokenUtil.needTokenUpdate to remedy perf regression

  • [SPARK-33029] [CORE][WEBUI] Fix the UI executor page incorrectly marking the driver as excluded

  • [SPARK-34015] [R] Fixing input timing in gapply

  • [SPARK-34012] [SQL] Keep behavior consistent when conf spark.sql.legacy.parser.havingWithoutGroupByAsWhere is true with migration guide

  • [SPARK-33844] [SQL][3.1] InsertIntoHiveDir command should check col name too

  • [SPARK-33935] [SQL] Fix CBO cost function

  • [SPARK-33100] [SQL] Ignore a semicolon inside a bracketed comment in spark-sql

  • [SPARK-34000] [CORE] Fix stageAttemptToNumSpeculativeTasks java.util.NoSuchElementException

  • [SPARK-33992] [SQL] override transformUpWithNewOutput to add allowInvokingTransformsInAnalyzer

  • [SPARK-33894] [SQL] Change visibility of private case classes in mllib to avoid runtime compilation errors with Scala 2.13

  • [SPARK-33950] [SQL][3.1][3.0] Refresh cache in v1 ALTER TABLE .. DROP PARTITION

  • [SPARK-33980] [SS] Invalidate char/varchar in spark.readStream.schema

  • [SPARK-33945] [SQL][3.1] Handles a random seed consisting of an expr tree

  • [SPARK-33398] Fix loading tree models prior to Spark 3.0

  • [SPARK-33963] [SQL] Canonicalize HiveTableRelation w/o table stats

  • [SPARK-33906] [WEBUI] Fix the bug of UI Executor page stuck due to undefined peakMemoryMetrics

  • [SPARK-33944] [SQL] Incorrect logging for warehouse keys in SharedState options

  • [SPARK-33936] [SQL][3.1] Add the version when connector’s interfaces were added

  • [SPARK-33916] [CORE] Fix fallback storage offset and improve compression codec test coverage

  • [SPARK-33899] [SQL][3.1] Fix assert failure in v1 SHOW TABLES/VIEWS on spark_catalog

  • [SPARK-33901] [SQL] Fix Char and Varchar display error after DDLs

  • [SPARK-33897] [SQL] Can’t set option ‘cross’ in join method

  • [SPARK-33907] [SQL][3.1] Only prune columns of JsonToStructs if parsing options is empty

  • [SPARK-33621] [SPARK-33784] [SQL][3.1] Add a way to inject data source rewrite rules

  • [SPARK-33900] [WEBUI] Show shuffle read size / records correctly when only remotebytesread is available

  • [SPARK-33892] [SQL] Display char/varchar in DESC and SHOW CREATE TABLE

  • [SPARK-33895] [SQL] Char and Varchar fail in MetaOperation of ThriftServer

  • [SPARK-33659] [SS] Document the current behavior for DataStreamWriter.toTable API

  • [SPARK-33893] [CORE] Exclude fallback block manager from executorList

  • [SPARK-33277] [PYSPARK][SQL] Use ContextAwareIterator to stop consuming after the task ends

  • [SPARK-33889] [SQL][3.1] Fix NPE from SHOW PARTITIONS on V2 tables

  • [SPARK-33879] [SQL] Char Varchar values fails w/ match error as partition columns

  • [SPARK-33877] [SQL] SQL reference documents for INSERT w/ a column list

  • [SPARK-33876] [SQL] Add length-check for reading char/varchar from tables w/ a external location

  • [SPARK-33846] [SQL] Include Comments for a nested schema in StructType.toDDL

  • [SPARK-33860] [SQL] Make CatalystTypeConverters.convertToCatalyst match special Array value

  • [SPARK-33834] [SQL] Verify ALTER TABLE CHANGE COLUMN with Char and Varchar

  • [SPARK-33853] [SQL] EXPLAIN CODEGEN and BenchmarkQueryTest don’t show subquery code

  • [SPARK-33836] [SS][PYTHON] Expose DataStreamReader.table and DataStreamWriter.toTable

  • [SPARK-33829] [SQL][3.1] Renaming v2 tables should recreate the cache

  • [SPARK-33756] [SQL] Make BytesToBytesMap’s MapIterator idempotent

  • [SPARK-33850] [SQL] EXPLAIN FORMATTED doesn’t show the plan for subqueries if AQE is enabled

  • [SPARK-33841] [CORE][3.1] Fix issue with jobs disappearing intermittently from the SHS under high load

  • [SPARK-33593] [SQL] Vector reader got incorrect data with binary partition value

  • [SPARK-26341] [WEBUI] Expose executor memory metrics at the stage level, in the Stages tab

  • [SPARK-33831] [UI] Update to jetty 9.4.34

  • [SPARK-33822] [SQL] Use the CastSupport.cast method in HashJoin

  • [SPARK-33774] [UI][CORE] Back to Master” returns 500 error in Standalone cluster

  • [SPARK-26199] [SPARK-31517] [R] Fix strategy for handling … names in mutate

  • [SPARK-33819] [CORE][3.1] SingleFileEventLogFileReader/RollingEventLogFilesFileReader should be package private

  • [SPARK-33697] [SQL] RemoveRedundantProjects should require column ordering by default

  • [SPARK-33752] [SQL][3.1] Avoid the getSimpleMessage of AnalysisException adds semicolon repeatedly

  • [SPARK-33788] [SQL][3.1][3.0][2.4] Throw NoSuchPartitionsException from HiveExternalCatalog.dropPartitions()

  • [SPARK-33803] [SQL] Sort table properties by key in DESCRIBE TABLE command

  • [SPARK-33786] [SQL] The storage level for a cache should be respected when a table name is altered

  • [SPARK-33273] [SQL] Fix a race condition in subquery execution

  • [SPARK-33653] [SQL][3.1] DSv2: REFRESH TABLE should recache the table itself

  • [SPARK-33777] [SQL] Sort output of V2 SHOW PARTITIONS

  • [SPARK-33733] [SQL] PullOutNondeterministic should check and collect deterministic field

  • [SPARK-33764] [SS] Make state store maintenance interval as SQL config

  • [SPARK-33729] [SQL] When refreshing cache, Spark should not use cached plan when recaching data

  • [SPARK-33742] [SQL][3.1] Throw PartitionsAlreadyExistException from HiveExternalCatalog.createPartitions()

  • [SPARK-33706] [SQL] Require fully specified partition identifier in partitionExists()

  • [SPARK-33740] [SQL] hadoop configs in hive-site.xml can overrides pre-existing hadoop ones

  • [SPARK-33692] [SQL] View should use captured catalog and namespace to lookup function

  • [SPARK-33669] Wrong error message from YARN application state monitor when sc.stop in yarn client mode

  • [SPARK-32110] [SQL] normalize special floating numbers in HyperLogLog++

  • [SPARK-33677] [SQL] Skip LikeSimplification rule if pattern contains any escapeChar

  • [SPARK-33693] [SQL] deprecate spark.sql.hive.convertCTAS

  • [SPARK-33641] [SQL] Invalidate new char/varchar types in public APIs that produce incorrect results

  • [SPARK-32680] [SQL] Don’t Preprocess V2 CTAS with Unresolved Query

  • [SPARK-33676] [SQL] Require exact matching of partition spec to the schema in V2 ALTER TABLE .. ADD/DROP PARTITION

  • [SPARK-33670] [SQL] Verify the partition provider is Hive in v1 SHOW TABLE EXTENDED

  • [SPARK-33663] [SQL] Uncaching should not be called on non-existing temp views

  • [SPARK-33667] [SQL] Respect the spark.sql.caseSensitive config while resolving partition spec in v1 SHOW PARTITIONS

  • [SPARK-33652] [SQL] DSv2: DeleteFrom should refresh cache

System environment

  • Operating System: Ubuntu 20.04.2 LTS

  • Java: Zulu 8.54.0.21-CA-linux64

  • Scala: 2.12.10

  • Python: 3.8.10

  • R: 4.1.0 (2021-05-18)

  • Delta Lake 1.0.0

Installed Python libraries

Library

Version

Library

Version

Library

Version

Antergos Linux

2015.10 (ISO-Rolling)

appdirs

1.4.4

backcall

0.2.0

boto3

1.16.7

botocore

1.19.7

certifi

2020.12.5

chardet

4.0.0

cycler

0.10.0

Cython

0.29.23

dbus-python

1.2.16

decorator

5.0.6

distlib

0.3.2

distro-info

0.23ubuntu1

facets-overview

1.0.0

filelock

3.0.12

idna

2.10

ipykernel

5.3.4

ipython

7.22.0

ipython-genutils

0.2.0

jedi

0.17.2

jmespath

0.10.0

joblib

1.0.1

jupyter-client

6.1.12

jupyter-core

4.7.1

kiwisolver

1.3.1

koalas

1.8.1

matplotlib

3.4.2

numpy

1.19.2

pandas

1.2.4

parso

0.7.0

patsy

0.5.1

pexpect

4.8.0

pickleshare

0.7.5

Pillow

8.2.0

pip

21.0.1

plotly

4.14.3

prompt-toolkit

3.0.17

protobuf

3.17.2

psycopg2

2.8.5

ptyprocess

0.7.0

pyarrow

4.0.0

Pygments

2.8.1

PyGObject

3.36.0

pyparsing

2.4.7

python-apt

2.0.0+ubuntu0.20.4.5

python-dateutil

2.8.1

pytz

2020.5

pyzmq

20.0.0

requests

2.25.1

requests-unixsocket

0.2.0

retrying

1.3.3

s3transfer

0.3.7

scikit-learn

0.24.1

scipy

1.6.2

seaborn

0.11.1

setuptools

52.0.0

six

1.15.0

ssh-import-id

5.10

statsmodels

0.12.2

threadpoolctl

2.1.0

tornado

6.1

traitlets

5.0.5

unattended-upgrades

0.1

urllib3

1.25.11

virtualenv

20.4.1

wcwidth

0.2.5

wheel

0.36.2

Installed R libraries

R libraries are installed from the Microsoft CRAN snapshot on 2021-07-28.

Library

Version

Library

Version

Library

Version

askpass

1.1

assertthat

0.2.1

backports

1.2.1

base

4.1.0

base64enc

0.1-3

BH

1.72.0-3

bit

4.0.4

bit64

4.0.5

blob

1.2.1

boot

1.3-28

brew

1.0-6

brio

1.1.0

broom

0.7.2

callr

3.5.1

caret

6.0-86

cellranger

1.1.0

chron

2.3-56

class

7.3-19

cli

2.2.0

clipr

0.7.1

cluster

2.1.2

codetools

0.2-18

colorspace

2.0-0

commonmark

1.7

compiler

4.1.0

config

0.3

covr

3.5.1

cpp11

0.2.4

crayon

1.3.4

credentials

1.3.0

crosstalk

1.1.0.1

curl

4.3

data.table

1.13.4

datasets

4.1.0

DBI

1.1.0

dbplyr

2.0.0

desc

1.2.0

devtools

2.3.2

diffobj

0.3.2

digest

0.6.27

dplyr

1.0.2

DT

0.16

ellipsis

0.3.1

evaluate

0.14

fansi

0.4.1

farver

2.0.3

fastmap

1.0.1

forcats

0.5.0

foreach

1.5.1

foreign

0.8-81

forge

0.2.0

fs

1.5.0

future

1.21.0

generics

0.1.0

gert

1.0.2

ggplot2

3.3.2

gh

1.2.0

gitcreds

0.1.1

glmnet

4.0-2

globals

0.14.0

glue

1.4.2

gower

0.2.2

graphics

4.1.0

grDevices

4.1.0

grid

4.1.0

gridExtra

2.3

gsubfn

0.7

gtable

0.3.0

haven

2.3.1

highr

0.8

hms

0.5.3

htmltools

0.5.0

htmlwidgets

1.5.3

httpuv

1.5.4

httr

1.4.2

hwriter

1.3.2

hwriterPlus

1.0-3

ini

0.3.1

ipred

0.9-9

isoband

0.2.3

iterators

1.0.13

jsonlite

1.7.2

KernSmooth

2.23-20

knitr

1.30

labeling

0.4.2

later

1.1.0.1

lattice

0.20-44

lava

1.6.8.1

lazyeval

0.2.2

lifecycle

0.2.0

listenv

0.8.0

lubridate

1.7.9.2

magrittr

2.0.1

markdown

1.1

MASS

7.3-54

Matrix

1.3-4

memoise

1.1.0

methods

4.1.0

mgcv

1.8-36

mime

0.9

ModelMetrics

1.2.2.2

modelr

0.1.8

munsell

0.5.0

nlme

3.1-152

nnet

7.3-16

numDeriv

2016.8-1.1

openssl

1.4.3

parallel

4.1.0

parallelly

1.22.0

pillar

1.4.7

pkgbuild

1.1.0

pkgconfig

2.0.3

pkgload

1.1.0

plogr

0.2.0

plyr

1.8.6

praise

1.0.0

prettyunits

1.1.1

pROC

1.16.2

processx

3.4.5

prodlim

2019.11.13

progress

1.2.2

promises

1.1.1

proto

1.0.0

ps

1.5.0

purrr

0.3.4

r2d3

0.2.3

R6

2.5.0

randomForest

4.6-14

rappdirs

0.3.1

rcmdcheck

1.3.3

RColorBrewer

1.1-2

Rcpp

1.0.5

readr

1.4.0

readxl

1.3.1

recipes

0.1.15

rematch

1.0.1

rematch2

2.1.2

remotes

2.2.0

reprex

0.3.0

reshape2

1.4.4

rex

1.2.0

rlang

0.4.9

rmarkdown

2.6

RODBC

1.3-17

roxygen2

7.1.1

rpart

4.1-15

rprojroot

2.0.2

Rserve

1.8-8

RSQLite

2.2.1

rstudioapi

0.13

rversions

2.0.2

rvest

0.3.6

scales

1.1.1

selectr

0.4-2

sessioninfo

1.1.1

shape

1.4.5

shiny

1.5.0

sourcetools

0.1.7

sparklyr

1.5.2

SparkR

3.1.1

spatial

7.3-11

splines

4.1.0

sqldf

0.4-11

SQUAREM

2020.5

stats

4.1.0

stats4

4.1.0

stringi

1.5.3

stringr

1.4.0

survival

3.2-11

sys

3.4

tcltk

4.1.0

TeachingDemos

2.10

testthat

3.0.0

tibble

3.0.4

tidyr

1.1.2

tidyselect

1.1.0

tidyverse

1.3.0

timeDate

3043.102

tinytex

0.28

tools

4.1.0

usethis

2.0.0

utf8

1.1.4

utils

4.1.0

uuid

0.1-4

vctrs

0.3.5

viridisLite

0.3.0

waldo

0.2.3

whisker

0.4

withr

2.3.0

xfun

0.19

xml2

1.3.2

xopen

1.0.0

xtable

1.8-4

yaml

2.2.1

zip

2.1.1

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID

Artifact ID

Version

antlr

antlr

2.7.7

com.amazonaws

amazon-kinesis-client

1.12.0

com.amazonaws

aws-java-sdk-autoscaling

1.11.655

com.amazonaws

aws-java-sdk-cloudformation

1.11.655

com.amazonaws

aws-java-sdk-cloudfront

1.11.655

com.amazonaws

aws-java-sdk-cloudhsm

1.11.655

com.amazonaws

aws-java-sdk-cloudsearch

1.11.655

com.amazonaws

aws-java-sdk-cloudtrail

1.11.655

com.amazonaws

aws-java-sdk-cloudwatch

1.11.655

com.amazonaws

aws-java-sdk-cloudwatchmetrics

1.11.655

com.amazonaws

aws-java-sdk-codedeploy

1.11.655

com.amazonaws

aws-java-sdk-cognitoidentity

1.11.655

com.amazonaws

aws-java-sdk-cognitosync

1.11.655

com.amazonaws

aws-java-sdk-config

1.11.655

com.amazonaws

aws-java-sdk-core

1.11.655

com.amazonaws

aws-java-sdk-datapipeline

1.11.655

com.amazonaws

aws-java-sdk-directconnect

1.11.655

com.amazonaws

aws-java-sdk-directory

1.11.655

com.amazonaws

aws-java-sdk-dynamodb

1.11.655

com.amazonaws

aws-java-sdk-ec2

1.11.655

com.amazonaws

aws-java-sdk-ecs

1.11.655

com.amazonaws

aws-java-sdk-efs

1.11.655

com.amazonaws

aws-java-sdk-elasticache

1.11.655

com.amazonaws

aws-java-sdk-elasticbeanstalk

1.11.655

com.amazonaws

aws-java-sdk-elasticloadbalancing

1.11.655

com.amazonaws

aws-java-sdk-elastictranscoder

1.11.655

com.amazonaws

aws-java-sdk-emr

1.11.655

com.amazonaws

aws-java-sdk-glacier

1.11.655

com.amazonaws

aws-java-sdk-glue

1.11.655

com.amazonaws

aws-java-sdk-iam

1.11.655

com.amazonaws

aws-java-sdk-importexport

1.11.655

com.amazonaws

aws-java-sdk-kinesis

1.11.655

com.amazonaws

aws-java-sdk-kms

1.11.655

com.amazonaws

aws-java-sdk-lambda

1.11.655

com.amazonaws

aws-java-sdk-logs

1.11.655

com.amazonaws

aws-java-sdk-machinelearning

1.11.655

com.amazonaws

aws-java-sdk-marketplacecommerceanalytics

1.11.655

com.amazonaws

aws-java-sdk-marketplacemeteringservice

1.11.655

com.amazonaws

aws-java-sdk-opsworks

1.11.655

com.amazonaws

aws-java-sdk-rds

1.11.655

com.amazonaws

aws-java-sdk-redshift

1.11.655

com.amazonaws

aws-java-sdk-route53

1.11.655

com.amazonaws

aws-java-sdk-s3

1.11.655

com.amazonaws

aws-java-sdk-ses

1.11.655

com.amazonaws

aws-java-sdk-simpledb

1.11.655

com.amazonaws

aws-java-sdk-simpleworkflow

1.11.655

com.amazonaws

aws-java-sdk-sns

1.11.655

com.amazonaws

aws-java-sdk-sqs

1.11.655

com.amazonaws

aws-java-sdk-ssm

1.11.655

com.amazonaws

aws-java-sdk-storagegateway

1.11.655

com.amazonaws

aws-java-sdk-sts

1.11.655

com.amazonaws

aws-java-sdk-support

1.11.655

com.amazonaws

aws-java-sdk-swf-libraries

1.11.22

com.amazonaws

aws-java-sdk-workspaces

1.11.655

com.amazonaws

jmespath-java

1.11.655

com.chuusai

shapeless_2.12

2.3.3

com.clearspring.analytics

stream

2.9.6

com.databricks

Rserve

1.8-3

com.databricks

jets3t

0.7.1-0

com.databricks.scalapb

compilerplugin_2.12

0.4.15-10

com.databricks.scalapb

scalapb-runtime_2.12

0.4.15-10

com.esotericsoftware

kryo-shaded

4.0.2

com.esotericsoftware

minlog

1.3.0

com.fasterxml

classmate

1.3.4

com.fasterxml.jackson.core

jackson-annotations

2.10.0

com.fasterxml.jackson.core

jackson-core

2.10.0

com.fasterxml.jackson.core

jackson-databind

2.10.0

com.fasterxml.jackson.dataformat

jackson-dataformat-cbor

2.10.0

com.fasterxml.jackson.datatype

jackson-datatype-joda

2.10.0

com.fasterxml.jackson.module

jackson-module-paranamer

2.10.0

com.fasterxml.jackson.module

jackson-module-scala_2.12

2.10.0

com.github.ben-manes.caffeine

caffeine

2.3.4

com.github.fommil

jniloader

1.1

com.github.fommil.netlib

core

1.1.2

com.github.fommil.netlib

native_ref-java

1.1

com.github.fommil.netlib

native_ref-java-natives

1.1

com.github.fommil.netlib

native_system-java

1.1

com.github.fommil.netlib

native_system-java-natives

1.1

com.github.fommil.netlib

netlib-native_ref-linux-x86_64-natives

1.1

com.github.fommil.netlib

netlib-native_system-linux-x86_64-natives

1.1

com.github.joshelser

dropwizard-metrics-hadoop-metrics2-reporter

0.1.2

com.github.luben

zstd-jni

1.4.8-1

com.github.wendykierp

JTransforms

3.1

com.google.code.findbugs

jsr305

3.0.0

com.google.code.gson

gson

2.2.4

com.google.flatbuffers

flatbuffers-java

1.9.0

com.google.guava

guava

15.0

com.google.protobuf

protobuf-java

2.6.1

com.h2database

h2

1.4.195

com.helger

profiler

1.1.1

com.jcraft

jsch

0.1.50

com.jolbox

bonecp

0.8.0.RELEASE

com.lihaoyi

sourcecode_2.12

0.1.9

com.microsoft.azure

azure-data-lake-store-sdk

2.3.9

com.microsoft.sqlserver

mssql-jdbc

9.2.1.jre8

com.ning

compress-lzf

1.0.3

com.sun.mail

javax.mail

1.5.2

com.tdunning

json

1.8

com.thoughtworks.paranamer

paranamer

2.8

com.trueaccord.lenses

lenses_2.12

0.4.12

com.twitter

chill-java

0.9.5

com.twitter

chill_2.12

0.9.5

com.twitter

util-app_2.12

7.1.0

com.twitter

util-core_2.12

7.1.0

com.twitter

util-function_2.12

7.1.0

com.twitter

util-jvm_2.12

7.1.0

com.twitter

util-lint_2.12

7.1.0

com.twitter

util-registry_2.12

7.1.0

com.twitter

util-stats_2.12

7.1.0

com.typesafe

config

1.2.1

com.typesafe.scala-logging

scala-logging_2.12

3.7.2

com.univocity

univocity-parsers

2.9.1

com.zaxxer

HikariCP

3.1.0

commons-beanutils

commons-beanutils

1.9.4

commons-cli

commons-cli

1.2

commons-codec

commons-codec

1.10

commons-collections

commons-collections

3.2.2

commons-configuration

commons-configuration

1.6

commons-dbcp

commons-dbcp

1.4

commons-digester

commons-digester

1.8

commons-fileupload

commons-fileupload

1.3.3

commons-httpclient

commons-httpclient

3.1

commons-io

commons-io

2.4

commons-lang

commons-lang

2.6

commons-logging

commons-logging

1.1.3

commons-net

commons-net

3.1

commons-pool

commons-pool

1.5.4

hive-2.3__hadoop-2.7

jets3t-0.7

liball_deps_2.12

hive-2.3__hadoop-2.7

zookeeper-3.4

liball_deps_2.12

info.ganglia.gmetric4j

gmetric4j

1.0.10

io.airlift

aircompressor

0.10

io.delta

delta-sharing-spark_2.12

0.1.0

io.dropwizard.metrics

metrics-core

4.1.1

io.dropwizard.metrics

metrics-graphite

4.1.1

io.dropwizard.metrics

metrics-healthchecks

4.1.1

io.dropwizard.metrics

metrics-jetty9

4.1.1

io.dropwizard.metrics

metrics-jmx

4.1.1

io.dropwizard.metrics

metrics-json

4.1.1

io.dropwizard.metrics

metrics-jvm

4.1.1

io.dropwizard.metrics

metrics-servlets

4.1.1

io.netty

netty-all

4.1.51.Final

io.prometheus

simpleclient

0.7.0

io.prometheus

simpleclient_common

0.7.0

io.prometheus

simpleclient_dropwizard

0.7.0

io.prometheus

simpleclient_pushgateway

0.7.0

io.prometheus

simpleclient_servlet

0.7.0

io.prometheus.jmx

collector

0.12.0

jakarta.annotation

jakarta.annotation-api

1.3.5

jakarta.validation

jakarta.validation-api

2.0.2

jakarta.ws.rs

jakarta.ws.rs-api

2.1.6

javax.activation

activation

1.1.1

javax.el

javax.el-api

2.2.4

javax.jdo

jdo-api

3.0.1

javax.servlet

javax.servlet-api

3.1.0

javax.servlet.jsp

jsp-api

2.1

javax.transaction

jta

1.1

javax.transaction

transaction-api

1.1

javax.xml.bind

jaxb-api

2.2.2

javax.xml.stream

stax-api

1.0-2

javolution

javolution

5.5.1

jline

jline

2.14.6

joda-time

joda-time

2.10.5

log4j

apache-log4j-extras

1.2.17

log4j

log4j

1.2.17

maven-trees

hive-2.3__hadoop-2.7

liball_deps_2.12

net.java.dev.jna

jna

5.8.0

net.razorvine

pyrolite

4.30

net.sf.jpam

jpam

1.1

net.sf.opencsv

opencsv

2.3

net.sf.supercsv

super-csv

2.2.0

net.snowflake

snowflake-ingest-sdk

0.9.6

net.snowflake

snowflake-jdbc

3.13.3

net.snowflake

spark-snowflake_2.12

2.9.0-spark_3.1

net.sourceforge.f2j

arpack_combined_all

0.1

org.acplt.remotetea

remotetea-oncrpc

1.1.2

org.antlr

ST4

4.0.4

org.antlr

antlr-runtime

3.5.2

org.antlr

antlr4-runtime

4.8-1

org.antlr

stringtemplate

3.2.1

org.apache.ant

ant

1.9.2

org.apache.ant

ant-jsch

1.9.2

org.apache.ant

ant-launcher

1.9.2

org.apache.arrow

arrow-format

2.0.0

org.apache.arrow

arrow-memory-core

2.0.0

org.apache.arrow

arrow-memory-netty

2.0.0

org.apache.arrow

arrow-vector

2.0.0

org.apache.avro

avro

1.8.2

org.apache.avro

avro-ipc

1.8.2

org.apache.avro

avro-mapred-hadoop2

1.8.2

org.apache.commons

commons-compress

1.20

org.apache.commons

commons-crypto

1.1.0

org.apache.commons

commons-lang3

3.10

org.apache.commons

commons-math3

3.4.1

org.apache.commons

commons-text

1.6

org.apache.curator

curator-client

2.7.1

org.apache.curator

curator-framework

2.7.1

org.apache.curator

curator-recipes

2.7.1

org.apache.derby

derby

10.12.1.1

org.apache.directory.api

api-asn1-api

1.0.0-M20

org.apache.directory.api

api-util

1.0.0-M20

org.apache.directory.server

apacheds-i18n

2.0.0-M15

org.apache.directory.server

apacheds-kerberos-codec

2.0.0-M15

org.apache.hadoop

hadoop-annotations

2.7.4

org.apache.hadoop

hadoop-auth

2.7.4

org.apache.hadoop

hadoop-client

2.7.4

org.apache.hadoop

hadoop-common

2.7.4

org.apache.hadoop

hadoop-hdfs

2.7.4

org.apache.hadoop

hadoop-mapreduce-client-app

2.7.4

org.apache.hadoop

hadoop-mapreduce-client-common

2.7.4

org.apache.hadoop

hadoop-mapreduce-client-core

2.7.4

org.apache.hadoop

hadoop-mapreduce-client-jobclient

2.7.4

org.apache.hadoop

hadoop-mapreduce-client-shuffle

2.7.4

org.apache.hadoop

hadoop-yarn-api

2.7.4

org.apache.hadoop

hadoop-yarn-client

2.7.4

org.apache.hadoop

hadoop-yarn-common

2.7.4

org.apache.hadoop

hadoop-yarn-server-common

2.7.4

org.apache.hive

hive-beeline

2.3.7

org.apache.hive

hive-cli

2.3.7

org.apache.hive

hive-jdbc

2.3.7

org.apache.hive

hive-llap-client

2.3.7

org.apache.hive

hive-llap-common

2.3.7

org.apache.hive

hive-serde

2.3.7

org.apache.hive

hive-shims

2.3.7

org.apache.hive

hive-storage-api

2.7.2

org.apache.hive.shims

hive-shims-0.23

2.3.7

org.apache.hive.shims

hive-shims-common

2.3.7

org.apache.hive.shims

hive-shims-scheduler

2.3.7

org.apache.htrace

htrace-core

3.1.0-incubating

org.apache.httpcomponents

httpclient

4.5.6

org.apache.httpcomponents

httpcore

4.4.12

org.apache.ivy

ivy

2.4.0

org.apache.mesos

mesos-shaded-protobuf

1.4.0

org.apache.orc

orc-core

1.5.12

org.apache.orc

orc-mapreduce

1.5.12

org.apache.orc

orc-shims

1.5.12

org.apache.parquet

parquet-column

1.10.1-databricks9

org.apache.parquet

parquet-common

1.10.1-databricks9

org.apache.parquet

parquet-encoding

1.10.1-databricks9

org.apache.parquet

parquet-format

2.4.0

org.apache.parquet

parquet-hadoop

1.10.1-databricks9

org.apache.parquet

parquet-jackson

1.10.1-databricks9

org.apache.thrift

libfb303

0.9.3

org.apache.thrift

libthrift

0.12.0

org.apache.xbean

xbean-asm7-shaded

4.15

org.apache.yetus

audience-annotations

0.5.0

org.apache.zookeeper

zookeeper

3.4.14

org.codehaus.jackson

jackson-core-asl

1.9.13

org.codehaus.jackson

jackson-jaxrs

1.9.13

org.codehaus.jackson

jackson-mapper-asl

1.9.13

org.codehaus.jackson

jackson-xc

1.9.13

org.codehaus.janino

commons-compiler

3.0.16

org.codehaus.janino

janino

3.0.16

org.datanucleus

datanucleus-api-jdo

4.2.4

org.datanucleus

datanucleus-core

4.1.17

org.datanucleus

datanucleus-rdbms

4.1.19

org.datanucleus

javax.jdo

3.2.0-m3

org.eclipse.jetty

jetty-client

9.4.36.v20210114

org.eclipse.jetty

jetty-continuation

9.4.36.v20210114

org.eclipse.jetty

jetty-http

9.4.36.v20210114

org.eclipse.jetty

jetty-io

9.4.36.v20210114

org.eclipse.jetty

jetty-jndi

9.4.36.v20210114

org.eclipse.jetty

jetty-plus

9.4.36.v20210114

org.eclipse.jetty

jetty-proxy

9.4.36.v20210114

org.eclipse.jetty

jetty-security

9.4.36.v20210114

org.eclipse.jetty

jetty-server

9.4.36.v20210114

org.eclipse.jetty

jetty-servlet

9.4.36.v20210114

org.eclipse.jetty

jetty-servlets

9.4.36.v20210114

org.eclipse.jetty

jetty-util

9.4.36.v20210114

org.eclipse.jetty

jetty-util-ajax

9.4.36.v20210114

org.eclipse.jetty

jetty-webapp

9.4.36.v20210114

org.eclipse.jetty

jetty-xml

9.4.36.v20210114

org.fusesource.leveldbjni

leveldbjni-all

1.8

org.glassfish.hk2

hk2-api

2.6.1

org.glassfish.hk2

hk2-locator

2.6.1

org.glassfish.hk2

hk2-utils

2.6.1

org.glassfish.hk2

osgi-resource-locator

1.0.3

org.glassfish.hk2.external

aopalliance-repackaged

2.6.1

org.glassfish.hk2.external

jakarta.inject

2.6.1

org.glassfish.jersey.containers

jersey-container-servlet

2.30

org.glassfish.jersey.containers

jersey-container-servlet-core

2.30

org.glassfish.jersey.core

jersey-client

2.30

org.glassfish.jersey.core

jersey-common

2.30

org.glassfish.jersey.core

jersey-server

2.30

org.glassfish.jersey.inject

jersey-hk2

2.30

org.glassfish.jersey.media

jersey-media-jaxb

2.30

org.hibernate.validator

hibernate-validator

6.1.0.Final

org.javassist

javassist

3.25.0-GA

org.jboss.logging

jboss-logging

3.3.2.Final

org.jdbi

jdbi

2.63.1

org.joda

joda-convert

1.7

org.jodd

jodd-core

3.5.2

org.json4s

json4s-ast_2.12

3.7.0-M5

org.json4s

json4s-core_2.12

3.7.0-M5

org.json4s

json4s-jackson_2.12

3.7.0-M5

org.json4s

json4s-scalap_2.12

3.7.0-M5

org.lz4

lz4-java

1.7.1

org.mariadb.jdbc

mariadb-java-client

2.2.5

org.objenesis

objenesis

2.5.1

org.postgresql

postgresql

42.1.4

org.roaringbitmap

RoaringBitmap

0.9.14

org.roaringbitmap

shims

0.9.14

org.rocksdb

rocksdbjni

6.20.3

org.rosuda.REngine

REngine

2.1.0

org.scala-lang

scala-compiler_2.12

2.12.10

org.scala-lang

scala-library_2.12

2.12.10

org.scala-lang

scala-reflect_2.12

2.12.10

org.scala-lang.modules

scala-collection-compat_2.12

2.1.1

org.scala-lang.modules

scala-parser-combinators_2.12

1.1.2

org.scala-lang.modules

scala-xml_2.12

1.2.0

org.scala-sbt

test-interface

1.0

org.scalacheck

scalacheck_2.12

1.14.2

org.scalactic

scalactic_2.12

3.0.8

org.scalanlp

breeze-macros_2.12

1.0

org.scalanlp

breeze_2.12

1.0

org.scalatest

scalatest_2.12

3.0.8

org.slf4j

jcl-over-slf4j

1.7.30

org.slf4j

jul-to-slf4j

1.7.30

org.slf4j

slf4j-api

1.7.30

org.slf4j

slf4j-log4j12

1.7.30

org.spark-project.spark

unused

1.0.0

org.springframework

spring-core

4.1.4.RELEASE

org.springframework

spring-test

4.1.4.RELEASE

org.threeten

threeten-extra

1.5.0

org.tukaani

xz

1.5

org.typelevel

algebra_2.12

2.0.0-M2

org.typelevel

cats-kernel_2.12

2.0.0-M4

org.typelevel

machinist_2.12

0.6.8

org.typelevel

macro-compat_2.12

1.1.1

org.typelevel

spire-macros_2.12

0.17.0-M1

org.typelevel

spire-platform_2.12

0.17.0-M1

org.typelevel

spire-util_2.12

0.17.0-M1

org.typelevel

spire_2.12

0.17.0-M1

org.wildfly.openssl

wildfly-openssl

1.0.7.Final

org.xerial

sqlite-jdbc

3.8.11.2

org.xerial.snappy

snappy-java

1.1.8.2

org.yaml

snakeyaml

1.24

oro

oro

2.0.8

pl.edu.icm

JLargeArrays

1.5

software.amazon.ion

ion-java

1.0.2

stax

stax-api

1.0.1

xmlenc

xmlenc

0.52