Databricks Runtime 13.3 LTS
The following release notes provide information about Databricks Runtime 13.3 LTS, powered by Apache Spark 3.4.1.
Databricks released this version in August 2023.
Note
LTS means this version is under long-term support. See Databricks Runtime LTS version lifecycle.
Note
These release notes may include references to features that are not available on Google Cloud as of this release.
Tip
For information on migrating between Databricks Runtime versions, see the Databricks Runtime migration guide.
To see release notes for Databricks Runtime versions that have reached end-of-support (EoS), see Unsupported Databricks Runtime release notes. The EoS Databricks Runtime versions have been retired and might not be updated.
New features and improvements
Volumes support for JARs is in Public Preview
You can now use Unity Catalog volumes to store JARs on compute with assigned or shared access modes running Databricks Runtime 13.3 and above. See Install libraries from a volume.
Safe casts enabled by default for Delta Lake operations
Delta UPDATE
and MERGE
operations now result in an error when values cannot be safely cast to the type in the target table schema. All implicit casts in Delta now follow spark.sql.storeAssignmentPolicy
instead of spark.sql.ansi.enabled
. See ANSI compliance in Databricks Runtime.
Introducing tags with Unity Catalog
With Databricks Runtime 13.3 LTS and later, you can apply tags to your securable objects. Tags help you organize securable objects and simplify search and discovery of your data assets. See Apply tags to Unity Catalog securable objects.
Databricks Runtime returns an error if a file is modified between query planning and invocation
Databricks Runtime queries now return an error if a file is updated between query planning and invocation. Before this change, Databricks Runtime would read a file between these stages, which occasionally lead to unpredictable results.
Block schema overwrite when using dynamic partition overwrites
You cannot set overwriteSchema
to true
in combination with dynamic partition overwrites in Delta Lake. This prevents possible table corruption due to schema mismatch. See Dynamic partition overwrites.
Integrate copyFile API into dbutils.fs.cp
The dbutils copy command, dbutils.fs.cp
is now optimized for faster copying. With this improvement, copy operations can be up to 100x faster, depending on the file size. The feature is available across all file systems accessible in Databricks, including for Unity Catalog Volumes and Databricks Filesystem mounts.
Databricks ODBC/JDBC driver support
Databricks supports ODBC and JDBC drivers released in the past 2 years. Download the latest driver at Download ODBC Driver and Download JDBC Driver.
Support for unnamed (positional) parameter markers
The SparkSession.sql()
API now supports parameterized queries using the ?
syntax in addition to parameterized queries using the :parm
syntax. See parameter markers.
Enable time series column labeling
You can now label time series columns when creating primary key constraints.
New bitmap SQL functions
Databricks now provided a set of functions which can be used to map integral numeric values onto a bitmap and aggregate them. This can be used to pre-compute number of distinct values.
Improved encryption functions
The aes_encrypt, aes_decrypt, and try_aes_decrypt functions have been improved with support for initialization vectors (IV) and authenticated additional data (AAD).
Unity Catalog support for REFRESH FOREIGN
You can now update foreign catalogs, schemas, and tables in Unity Catalog using REFRESH FOREIGN CATALOG
, REFRESH FOREIGN SCHEMA
, and REFRESH FOREIGN TABLE
. See REFRESH FOREIGN (CATALOG, SCHEMA, and TABLE).
INSERT BY NAME
When inserting rows into a table you can now let Databricks map columns and fields in the source query automatically to the table’s columns by their name. See INSERT INTO.
Breaking changes
Changes to implicit casting when storing rows in a table with MERGE INTO and UPDATE
Databricks now follows the configuration spark.sql.storeAssignmentPolicy
for implicit casting when storing rows in a table. The default value ANSI
throws an error when storing values that overflow. Previously, values would be stored as NULL
by default.
As in the following example, you can address overflows by rewriting the table to use a broader type for the column causing the error.
-- Enable column mapping
ALTER TABLE MyTable SET TBLPROPERTIES (
'delta.minReaderVersion' = '2',
'delta.minWriterVersion' = '5',
'delta.columnMapping.mode' = 'name'
)
-- Rename the old column and create a new column with the new type using the old name
ALTER TABLE MyTable RENAME ID to ID_old
ALTER TABLE MyTable ADD COLUMN ID BIGINT
-- Rewrite the entire table
UPDATE MyTable SET ID = ID_old
-- Drop the old column
ALTER TABLE MyTable DROP COLUMN ID_old
Parquet Schema Inference Changes
When inferring schemas from Parquet files not written by Spark, int64
timestamp columns annotated with isAdjustedToUTC=false
will now default to TIMESTAMP_NTZ
type. Previously, these were inferred as TIMESTAMP
type. This update enhances the accuracy of mapping Parquet types to Spark SQL types.
As a result, reading data from external Parquet files into a Delta table may lead to an error if the timestampNtz
feature is not enabled. The error will display as follows:
Fatal exception of the update:
com.databricks.sql.transaction.tahoe.DeltaTableFeatureException
Your table schema requires manual enablement of the following table feature(s): timestampNtz.
To maintain the previous behavior and prevent this error, you can set the Spark configuration parameter spark.sql.parquet.inferTimestampNTZ.enabled
to false
.
Library upgrades
Upgraded Python libraries:
debugpy from 1.5.1 to 1.6.0
distlib from 0.3.6 to 0.3.7
fastjsonschema from 2.16.3 to 2.18.0
filelock from 3.12.0 to 3.12.2
Upgraded R libraries:
SparkR from 3.4.0 to 3.4.1
Upgraded Java libraries:
org.apache.orc.orc-core from 1.8.3-shaded-protobuf to 1.8.4-shaded-protobuf
org.apache.orc.orc-mapreduce from 1.8.3-shaded-protobuf to 1.8.4-shaded-protobuf
org.apache.orc.orc-shims from 1.8.3 to 1.8.4
org.eclipse.jetty.jetty-client from 9.4.50.v20221201 to 9.4.51.v20230217
org.eclipse.jetty.jetty-continuation from 9.4.50.v20221201 to 9.4.51.v20230217
org.eclipse.jetty.jetty-http from 9.4.50.v20221201 to 9.4.51.v20230217
org.eclipse.jetty.jetty-io from 9.4.50.v20221201 to 9.4.51.v20230217
org.eclipse.jetty.jetty-jndi from 9.4.50.v20221201 to 9.4.51.v20230217
org.eclipse.jetty.jetty-plus from 9.4.50.v20221201 to 9.4.51.v20230217
org.eclipse.jetty.jetty-proxy from 9.4.50.v20221201 to 9.4.51.v20230217
org.eclipse.jetty.jetty-security from 9.4.50.v20221201 to 9.4.51.v20230217
org.eclipse.jetty.jetty-server from 9.4.50.v20221201 to 9.4.51.v20230217
org.eclipse.jetty.jetty-servlet from 9.4.50.v20221201 to 9.4.51.v20230217
org.eclipse.jetty.jetty-servlets from 9.4.50.v20221201 to 9.4.51.v20230217
org.eclipse.jetty.jetty-util from 9.4.50.v20221201 to 9.4.51.v20230217
org.eclipse.jetty.jetty-util-ajax from 9.4.50.v20221201 to 9.4.51.v20230217
org.eclipse.jetty.jetty-webapp from 9.4.50.v20221201 to 9.4.51.v20230217
org.eclipse.jetty.jetty-xml from 9.4.50.v20221201 to 9.4.51.v20230217
org.eclipse.jetty.websocket.websocket-api from 9.4.50.v20221201 to 9.4.51.v20230217
org.eclipse.jetty.websocket.websocket-client from 9.4.50.v20221201 to 9.4.51.v20230217
org.eclipse.jetty.websocket.websocket-common from 9.4.50.v20221201 to 9.4.51.v20230217
org.eclipse.jetty.websocket.websocket-server from 9.4.50.v20221201 to 9.4.51.v20230217
org.eclipse.jetty.websocket.websocket-servlet from 9.4.50.v20221201 to 9.4.51.v20230217
org.xerial.snappy.snappy-java from 1.1.8.4 to 1.1.10.1
Apache Spark
Databricks Runtime 13.3 includes Apache Spark 3.4.1. This release includes all Spark fixes and improvements included in Databricks Runtime 13.2 (EoS), as well as the following additional bug fixes and improvements made to Spark:
You can now set cluster environment variable
SNOWFLAKE_SPARK_CONNECTOR_VERSION=2.12
to use Spark-snowflake connector v2.12.0.[SPARK-43380] [DBRRM-383] Revert “[SC-133100][SQL] Fix Avro data type conver…
[SPARK-44504] [Backport][13.2][13.x][13.3] Unload provider thereby forcing DB instance close and releasing resources on maintenance task error
[SPARK-44059] [SC-137023] Add analyzer support of named arguments for built-in functions
[SPARK-44485] [SC-137653][SQL] Optimize TreeNode.generateTreeString
[SPARK-42944] [SC-137164][SS][PYTHON] Streaming ForeachBatch in Python
[SPARK-44446] [SC-136994][PYTHON] Add checks for expected list type special cases
[SPARK-44439] [SC-136973][CONNECT][SS]Fixed listListeners to only send ids back to client
[SPARK-44216] [SC-136941] [PYTHON] Make assertSchemaEqual API public
[SPARK-44201] [SC-136778][CONNECT][SS]Add support for Streaming Listener in Scala for Spark Connect
[SPARK-43915] [SC-134766][SQL] Assign names to the error class LEGACYERROR_TEMP_[2438-2445]
[SPARK-43798] Revert “[SC-133990][SQL][PYTHON] Support Python user-defined table functions”
[SPARK-44398] [SC-136720][CONNECT] Scala foreachBatch API
[SPARK-44476] [SC-137169][CORE][CONNECT] Fix population of artifacts for a JobArtifactState with no associated artifacts
[SPARK-44269] [SC-135817][SQL] Assign names to the error class LEGACYERROR_TEMP_[2310-2314]
[SPARK-44395] [SC-136744][SQL] Update TVF arguments to require parentheses around identifier after TABLE keyword
[SPARK-43915] [SC-134766][SQL] Assign names to the error class LEGACYERROR_TEMP_[2438-2445]
[SPARK-43995] [SC-136794][SPARK-43996][CONNECT] Add support for UDFRegistration to the Connect Scala Client
[SPARK-44109] [SC-134755][CORE] Remove duplicate preferred locations of each RDD partition
[SPARK-44169] [SC-135497][SQL] Assign names to the error class LEGACYERROR_TEMP_[2300-2304]
[SPARK-41487] [SC-135636][SQL] Assign name to LEGACYERROR_TEMP_1020
[SPARK-44398] [SC-136720][CONNECT] Scala foreachBatch API
[SPARK-44153] [SC-134977][CORE][UI] Support
Heap Histogram
column inExecutors
tab[SPARK-44044] [SC-135695][SS] Improve Error message for Window functions with streaming
[SPARK-43914] [SC-135414][SQL] Assign names to the error class LEGACYERROR_TEMP_[2433-2437]
[SPARK-44217] [SC-136805][PYTHON] Allow custom precision for fp approx equality
[SPARK-44370] [SC-136575][CONNECT] Migrate Buf remote generation alpha to remote plugins
[SPARK-44410] [SC-136930][PYTHON][CONNECT] Set active session in create, not just getOrCreate
[SPARK-44348] [SAS-1910][SC-136644][13.X][CORE][CONNECT][PYTHON] Reenable test_artifact with relevant changes
[SPARK-44145] [SC-136698][SQL] Callback when ready for execution
[SPARK-44264] [SC-136870][PYTHON][ML] FunctionPickler Class
[SPARK-43986] [SC-135623][SQL] Create error classes for HyperLogLog function call failures
[SPARK-44139] [SC-134967][SQL] Discard completely pushed down filters in group-based MERGE operations
[SPARK-44338] [SC-136718][SQL] Fix view schema mismatch error message
[SPARK-44208] [SC-135505][CORE][SQL] Assign clear error class names for some logic that directly uses exceptions
[SPARK-44364] [SC-136773] [PYTHON] Add support for List[Row] data type for expected
[SPARK-44180] [SC-136792][SQL] DistributionAndOrderingUtils should apply ResolveTimeZone
[SPARK-43389] [SC-136772][SQL] Added a null check for lineSep option
[SPARK-44388] [SC-136695][CONNECT] Fix protobuf cast issue when UDF instance is updated
[SPARK-44352] [SC-136373][CONNECT] Put back sameType and friends in DataType.
[SPARK-43975] [SC-134265][SQL] DataSource V2: Handle UPDATE commands for group-based sources
[SPARK-44360] [SC-136461][SQL] Support schema pruning in delta-based MERGE operations
[SPARK-44138] [SC-134907][SQL] Prohibit non-deterministic expressions, subqueries and aggregates in MERGE conditions
[SPARK-43321] [SC-136678][Connect][Followup] Better names for APIs used in Scala Client joinWith
[SPARK-44263] [SC-136447][CONNECT] Custom Interceptors Support
[SPARK-39851] [SC-123383][SQL] Improve join stats estimation if one side can keep uniqueness
[SPARK-44390] [SC-136677][CORE][SQL] Rename
SparkSerDerseUtils
toSparkSerDeUtils
[SPARK-44004] [SC-134751][SQL] Assign name & improve error message for frequent LEGACY errors.
[SPARK-44322] [SC-136204][CONNECT] Make parser use SqlApiConf instead of SQLConf.
[SPARK-44271] [SC-136347][SQL] Move default values functions from StructType to ResolveDefaultColumns
[SPARK-44332] [SC-136413][CORE][WEBUI] Fix the sorting error of Executor ID Column on Executors UI Page
[SPARK-44282] [SC-135948][CONNECT] Prepare DataType parsing for use in Spark Connect Scala Client
[SPARK-44349] [SC-136581][R] Add math functions to SparkR
[SPARK-44397] [SC-136647][PYTHON] Expose assertDataFrameEqual in pyspark.testing.utils
[SPARK-42981] [SC-136620][CONNECT] Add direct arrow serialization
[SPARK-44373] [SC-136577][SQL] Wrap withActive for Dataset API w/ parse logic to make parser related configuration work
[SPARK-44340] [SC-136574][SQL] Define the computing logic through PartitionEvaluator API and use it in WindowGroupLimitExec
[SPARK-43665] [SC-136573][CONNECT][PS] Enable PandasSQLStringFormatter.vformat to work with Spark Connect
[SPARK-44325] [SC-136572][SQL] Use PartitionEvaluator API in SortMergeJoinExec
[SPARK-44315] [SC-136171][SQL][CONNECT] Move DefinedByConstructorParams to sql/api
[SPARK-44275] [SC-136184][CONNECT] Add configurable retry mechanism to Scala Spark Connect
[SPARK-44326] [SC-136208][SQL][CONNECT] Move utils that are used from Scala client to the common modules
[SPARK-43321] [SC-136156][Connect] Dataset#Joinwith
[SPARK-44343] [SC-136372][CONNECT] Prepare ScalaReflection to the move to SQL/API
[SPARK-44313] [SC-136081][SQL] Fix generated column expression validation when there is a char/varchar column in the schema
[SPARK-43963] [SC-134145][SQL] DataSource V2: Handle MERGE commands for group-based sources
[SPARK-44363] [SC-136433] [PYTHON] Display percent of unequal rows in DataFrame comparison
[SPARK-44251] [SC-136432][SQL] Set nullable correctly on coalesced join key in full outer USING join
[SPARK-43948] [SC-133366][SQL] Assign names to the error class LEGACYERROR_TEMP_[0050|0057|0058|0059]
[SPARK-44329] [SC-136326][CONNECT][PYTHON] Add hll_sketch_agg, hll_union_agg, to_varchar, try_aes_decrypt to Scala and Python
[SPARK-44351] [SC-136353][SQL] Make some syntactic simplification
[SPARK-44281] [SC-135963][SQL] Move QueryCompilation error that used by DataType to sql/api as DataTypeErrors
[SPARK-44283] [SC-136109][CONNECT] Move Origin to SQL/API
[SPARK-43926] [SC-135590][CONNECT][PYTHON] Add array_agg, array_size, cardinality, count_min_sketch,mask,named_struct,json_* to Scala and Python
[SPARK-44327] [SC-136187][SQL][CONNECT] Add functions
any
andlen
to Scala[SPARK-44290] [SC-136300][CONNECT] Session-based files and archives in Spark Connect
[SPARK-44061] [SC-136299][PYTHON] Add assertDataFrameEqual util function
[SPARK-44331] [SC-136324][CONNECT][PYTHON] Add bitmap functions to Scala and Python
[SPARK-44342] [SC-136334][SQL] Replace SQLContext with SparkSession for GenTPCDSData
[SPARK-42583] [SC-124190][SQL] Remove the outer join if they are all distinct aggregate functions
[SPARK-44333] [SC-136206][CONNECT][SQL] Move EnhancedLogicalPlan out of ParserUtils
[SPARK-43824] [SC-132655][SPARK-43825] [SQL] Assign names to the error class LEGACYERROR_TEMP_128[1-2]
[SPARK-43939] [SC-134602][CONNECT][PYTHON] Add try_* functions to Scala and Python
[SPARK-44193] [SC-135886][CONNECT] Implement GRPC exceptions interception for conversion
[SPARK-44284] [SC-136107][CONNECT] Create simple conf system for sql/api
[SPARK-43913] [SC-133368][SQL] Assign names to the error class LEGACYERROR_TEMP_[2426-2432]
[SPARK-44291] Revert “[SC-135924][SPARK-43416][CONNECT] Fix incorrect schema for range query”
[SPARK-44312] [SC-136185][CONNECT][PYTHON] Allow to set a user agent with an environment variable
[SPARK-44154] [SC-136060] Implement bitmap functions
[SPARK-43885] [SC-133996][SQL] DataSource V2: Handle MERGE commands for delta-based sources
[SPARK-43924] [SC-135161][CONNECT][PYTHON] Add misc functions to Scala and Python
[SPARK-43969] [SC-134556][SQL] Refactor & Assign names to the error class LEGACYERROR_TEMP_1170
[SPARK-44185] [SC-135730][SQL] Fix inconsistent path qualifying between catalog and data operations
[SPARK-44073] [SC-134562][SQL][PYTHON][CONNECT] Add date time functions to Scala, Python and Connect - part 2
[SPARK-43942] [SC-134527][CONNECT][PYTHON] Add string functions to Scala and Python - part 1
[SPARK-44266] [SC-135813][SQL] Move Util.truncatedString to sql/api
[SPARK-44195] [SC-135722][R] Add JobTag APIs to SparkR SparkContext
[SPARK-44200] [SC-135736][SQL] Support TABLE argument parser rule for TableValuedFunction
[SPARK-44254] [SC-135709][SQL] Move QueryExecutionErrors that used by DataType to sql/api as DataTypeErrors
[SPARK-44291] [SC-135924][SPARK-43416][CONNECT] Fix incorrect schema for range query
[SPARK-43922] [SC-135612][SQL] Add named parameter support in parser for function calls
[SPARK-44300] [SC-135882][CONNECT] Fix artifact cleanup to limit deletion scope to session specific artifacts
[SPARK-44293] [SC-135849][CONNECT] Fix invalid URI for custom JARs in Spark Connect
[SPARK-44215] [SC-135850][SHUFFLE] If num chunks are 0, then server should throw a RuntimeException
[SPARK-44245] [SC-135851][PYTHON] pyspark.sql.dataframe sample() doctests should be illustrative-only
[SPARK-44274] [SC-135814][CONNECT] Move out util functions used by ArtifactManager to common/utils
[SPARK-44194] [SC-135815][PYTHON][CORE] Add JobTag APIs to PySpark SparkContext
[SPARK-42828] [SC-135733][PYTHON][SQL] More explicit Python type annotations for GroupedData
[SPARK-44079] [SC-135486][SQL] Fix
ArrayIndexOutOfBoundsException
when parse array as struct using PERMISSIVE mode with corrupt record[SPARK-44255] [SC-135809][SQL] Relocate StorageLevel to common/utils
[SPARK-42169] [SC-135795] [SQL] Implement code generation for to_csv function (StructsToCsv)
[SPARK-44249] [SC-135719][SQL][PYTHON] Refactor PythonUDTFRunner to send its return type separately
[SPARK-43353] [SC-132734][PYTHON] Migrate remaining session errors into error class
[SPARK-44211] [SC-135718][PYTHON][CONNECT] Implement SparkSession.is_stopped
[SPARK-42784] [SC-135691] should still create subDir when the number of subDir in merge dir is less than conf
[SPARK-41599] [SC-135616] Memory leak in FileSystem.CACHE when submitting apps to secure cluster using InProcessLauncher
[SPARK-44241] [SC-135613][Core] Mistakenly set io.connectionTimeout/connectionCreationTimeout to zero or negative will cause incessant executor cons/destructions
[SPARK-44133] [SC-134795][13.X][PYTHON] Upgrade MyPy from 0.920 to 0.982
[SPARK-42941] [SC-134707][SS][CONNECT][1/2] StreamingQueryListener - Event Serde in JSON format
[SPARK-44188] [SC-135173][CORE] Remove useless
resetAllPartitions
method inActiveJob
[SPARK-43757] [SC-135418][CONNECT] Change client compatibility from allow list to deny list
[SPARK-43474] [SC-135521] [SS] [CONNECT] Add a spark connect access to runtime Dataframes by ID.
[SPARK-44183] [SC-135209][PYTHON] Increate
PyArrow
minimum version to 4.0.0[SPARK-44248] [SC-135554][SS][SQL][Kafka] Add preferred location in kafka source v2
[SPARK-43929] [SC-134510][SQL][PYTHON][CONNECT] Add date time functions to Scala, Python and Connect API - part 1
[SPARK-44161] [SC-135326][Connect] Handle Row input for UDFs
[SPARK-44227] [SC-135520][SQL] Extract SchemaUtils from StructField
[SPARK-44064] [SC-135116][CORE][SQL] Add a new
apply
function toNonFateSharingCache
[SPARK-44171] [SC-135269][SQL] Assign names to the error class LEGACYERROR_TEMP_[2279-2282] & delete some unused error classes
[SPARK-44056] [SC-134758][SQL] Include UDF name in UDF execution failure error message when available
[SPARK-44205] [SC-135422][SQL] Extract Catalyst Code from DecimalType
[SPARK-44220] [SC-135484][SQL] Move StringConcat to sql/api
[SPARK-44146] [SC-135230][CONNECT] Isolate Spark Connect Session jars and classfiles
[SPARK-44206] [SC-135411][SQL] DataSet.selectExpr scope Session.active
[SPARK-40850] [SC-135417][SQL] Fix test case interpreted queries may execute Codegen
[SPARK-44204] [SC-135253][SQL][HIVE] Add missing recordHiveCall for getPartitionNames
[SPARK-44237] [SC-135485][CORE] Simplify DirectByteBuffer constructor lookup logic
[SPARK-44189] [SC-135170][CONNECT][PYTHON] Support positional parameters by
sql()
[SPARK-44140] [SC-134893][SQL][PYTHON] Support positional parameters in Python
sql()
[SPARK-44164] [SC-135158][SQL] Extract toAttribute method from StructField to Util class
[SPARK-44178] [SC-135113][CONNECT] Support positional parameters in
sql()
[SPARK-43208] [SC-128955][SQL][HIVE] IsolatedClassLoader should close barrier class InputStream after reading
[SPARK-43063] [SC-128382][SQL]
df.show
handle null should print NULL instead of null[SPARK-42539] [SC-124464][SQL][HIVE] Eliminate separate classloader when using ‘builtin’ Hive version for metadata client
[SPARK-43992] [SC-133645][SQL][PYTHON][CONNECT] Add optional pattern for Catalog.listFunctions
[SPARK-44066] [SC-134878][SQL] Support positional parameters in Scala/Java
sql()
[SPARK-44078] [SC-134869][CONNECT][CORE] Add support for classloader/resource isolation
[SPARK-43470] [SC-135187][CORE] Add OS, Java, Python version information to application log
[SPARK-43136] [SC-135238][CONNECT][Followup] Adding tests for KeyAs
[SPARK-39740] [SC-135093][UI]: Upgrade vis timeline to 7.7.2 to fix CVE-2020-28487
[SPARK-42298] [SC-133952][SQL] Assign name to LEGACYERROR_TEMP_2132
[SPARK-43961] [SC-133459][SQL][PYTHON][CONNECT] Add optional pattern for Catalog.listTables
[SPARK-44026] [SC-135181] Allow providing initial value for SQLMetrics
[SPARK-44202] [SC-135244][CORE] Add JobTag APIs to JavaSparkContext
[SPARK-44030] [SC-135237][SQL] Implement DataTypeExpression to offer Unapply for expression
[SPARK-43876] Revert “[SC-134603][SQL] Enable fast hashmap for distinct queries”
[SPARK-40082] [SC-126437] Schedule mergeFinalize when push merge shuffleMapStage retry but no running tasks
[SPARK-43888] [SC-132893][CORE] Relocate Logging to common/utils
[SPARK-43474] [SC-134400] [SS] [CONNECT] Add SessionHolder to SparkConnectPlanner
[SPARK-43944] [SC-134490][CONNECT][PYTHON] Add string functions to Scala and Python - part 2
[SPARK-44134] [SC-134954][CORE] Fix setting resources (GPU/FPGA) to 0 when they are set in spark-defaults.conf
[SPARK-44142] [SC-134947][PYTHON] Replace type with tpe in utility to convert python types to spark types
[SPARK-43952] [SC-134695][CORE][CONNECT][SQL] Add SparkContext APIs for query cancellation by tag
[SPARK-44000] [SC-133634][SQL] Add hint to disable broadcasting and replicating one side of join
[SPARK-43937] [SC-134362][CONNECT][PYTHON] Add ifnull,isnotnull,equal_null,nullif,nvl,nvl2 to Scala and Python
[SPARK-43624] [13.x][SC-134557][PS][CONNECT] Add
EWM
to SparkConnectPlanner.[SPARK-44040] [SC-134366][SQL] Fix compute stats when AggregateExec node above QueryStageExec
[SPARK-43485] [SC-131264][SQL] Fix the error message for the
unit
argument of the datetime add/diff functions[SPARK-43794] [SC-132561][SQL] Assign a name to the error class LEGACYERROR_TEMP_1335
[SPARK-43511] [SC-134749][CONNECT][SS]Implemented MapGroupsWithState and FlatMapGroupsWithState APIs for Spark Connect
[SPARK-43529] [SQL] Support CREATE/REPLACE TABLE OPTIONS expressions + code cleanup
[SPARK-44106] [SC-134750][PYTHON][CONNECT] Add
__repr__
forGroupedData
[SPARK-42299] [SC-133190] Assign name to LEGACYERROR_TEMP_2206
[SPARK-43290] [SC-134388][SQL] Adds support for aes_encrypt IVs and AAD
[SPARK-43932] [SC-134138][SQL][PYTHON][CONNECT] Add
current
like functions to Scala and Python[SPARK-43934] [SC-134108][SQL][PYTHON][CONNECT] Add regexp_* functions to Scala and Python
[SPARK-44107] [SC-134746][CONNECT][PYTHON] Hide unsupported Column methods from auto-completion
[SPARK-42941] [SC-134707][SS][CONNECT][1/2] StreamingQueryListener - Event Serde in JSON format
[SPARK-43773] [SC-132659][CONNECT][PYTHON] Implement ‘levenshtein(str1, str2[, threshold])’ functions in python client
[SPARK-44125] [SC-134745][R] Support Java 21 in SparkR
[SPARK-43919] [SC-133374][SQL] Extract JSON functionality out of Row
[SPARK-43769] [SC-132520][CONNECT] Implement ‘levenshtein(str1, str2[, threshold])’ functions
[SPARK-44012] [SC-134662][SS] KafkaDataConsumer to print some read status
[SPARK-43876] [SC-134603][SQL] Enable fast hashmap for distinct queries
[SPARK-44024] [SC-134497][SQL] Change to use
map
whenunzip
only used to extract a single element[SPARK-43928] [SC-134467][SQL][PYTHON][CONNECT] Add bit operations to Scala, Python and Connect API
[SPARK-44092] [SC-134506][CORE] Add
Utils.isJavaVersionAtLeast21
and makecore
module pass with Java 21[SPARK-44018] [SC-134512][SQL] Improve the hashCode and toString for some DS V2 Expression
[SPARK-44105] [SC-134582][SQL]
LastNonNull
should be lazily resolved[SPARK-44081] [SC-134434] Simplify PartitionedFileUtil API a little
[SPARK-43791] [SC-132531][SQL] Assign a name to the error class LEGACYERROR_TEMP_1336
[SPARK-44071] [SC-134435] Define and use Unresolved[Leaf|Unary]Node traits.
[SPARK-43852] [SC-132890][SPARK-43853][SPARK-43854][SPARK-43855][SPARK-43856] Assign names to the error class LEGACYERROR_TEMP_2418-2425
[SPARK-43742] [SC-132684][SQL] Refactor default column value resolution
[SPARK-43493] [SC-132063][SQL] Add a max distance argument to the levenshtein() function
[SPARK-44075] [SC-134382][CONNECT] Make
transformStatCorr
lazy[SPARK-43925] [SC-134381][SQL][PYTHON][CONNECT] Add some, bool_or,bool_and,every to Scala, Python and Connect
[SPARK-43931] [SC-134158][SQL][PYTHON][CONNECT] Add make_* functions to Scala and Python
[SPARK-43762] [SC-132602][SPARK-43763][SPARK-43764][SPARK-43765][SPARK-43766][SQL] Assign names to the error class LEGACYERROR_TEMP_24[06-10]
[SPARK-43962] [SC-133367][SQL] Improve error messages:
CANNOT_DECODE_URL
,CANNOT_MERGE_INCOMPATIBLE_DATA_TYPE
,CANNOT_PARSE_DECIMAL
,CANNOT_READ_FILE_FOOTER
,CANNOT_RECOGNIZE_HIVE_TYPE
.[SPARK-43938] [SC-133887][CONNECT][PYTHON] Add to_* functions to Scala and Python
[SPARK-44055] [SC-134321][CORE] Remove redundant
override
functions fromCheckpointRDD
[SPARK-43802] [SC-132597][SQL] Fix codegen for unhex and unbase64 with failOnError=true
[SPARK-43798] [SC-133990][SQL][PYTHON] Support Python user-defined table functions
[SPARK-43941] [SC-134286][SQL][PYTHON][CONNECT] Add any_value, approx_percentile, count_if, first_value, histogram_numeric, last_value, reduce to Scala, Python and Connect API
[SPARK-43380] [SC-133100][SQL] Fix Avro data type conversion issues to avoid producing incorrect results
[SPARK-43803] [SC-133729] [SS] [CONNECT] Improve awaitTermination() to handle client disconnects
[SPARK-43179] [SC-129070][SHUFFLE] Allowing apps to control whether their metadata gets saved in the db by the External Shuffle Service
Databricks ODBC/JDBC driver support
Databricks supports ODBC/JDBC drivers released in the past 2 years. Please download the recently released drivers and upgrade (download ODBC, download JDBC).
System environment
Operating System: Ubuntu 22.04.2 LTS
Java: Zulu 8.70.0.23-CA-linux64
Scala: 2.12.15
Python: 3.10.12
R: 4.2.2
Delta Lake: 2.4.0
Installed Python libraries
Library |
Version |
Library |
Version |
Library |
Version |
---|---|---|---|---|---|
appdirs |
1.4.4 |
argon2-cffi |
21.3.0 |
argon2-cffi-bindings |
21.2.0 |
asttokens |
2.2.1 |
attrs |
21.4.0 |
backcall |
0.2.0 |
beautifulsoup4 |
4.11.1 |
black |
22.6.0 |
bleach |
4.1.0 |
blinker |
1.4 |
boto3 |
1.24.28 |
botocore |
1.27.28 |
certifi |
2022.9.14 |
cffi |
1.15.1 |
chardet |
4.0.0 |
charset-normalizer |
2.0.4 |
click |
8.0.4 |
cryptography |
37.0.1 |
cycler |
0.11.0 |
Cython |
0.29.32 |
databricks-sdk |
0.1.6 |
dbus-python |
1.2.18 |
debugpy |
1.6.0 |
decorator |
5.1.1 |
defusedxml |
0.7.1 |
distlib |
0.3.7 |
docstring-to-markdown |
0.12 |
entrypoints |
0.4 |
executing |
1.2.0 |
facets-overview |
1.0.3 |
fastjsonschema |
2.18.0 |
filelock |
3.12.2 |
fonttools |
4.25.0 |
googleapis-common-protos |
1.56.4 |
grpcio |
1.48.1 |
grpcio-status |
1.48.1 |
httplib2 |
0.20.2 |
idna |
3.3 |
importlib-metadata |
4.6.4 |
ipykernel |
6.17.1 |
ipython |
8.10.0 |
ipython-genutils |
0.2.0 |
ipywidgets |
7.7.2 |
jedi |
0.18.1 |
jeepney |
0.7.1 |
Jinja2 |
2.11.3 |
jmespath |
0.10.0 |
joblib |
1.2.0 |
jsonschema |
4.16.0 |
jupyter-client |
7.3.4 |
jupyter_core |
4.11.2 |
jupyterlab-pygments |
0.1.2 |
jupyterlab-widgets |
1.0.0 |
keyring |
23.5.0 |
kiwisolver |
1.4.2 |
launchpadlib |
1.10.16 |
lazr.restfulclient |
0.14.4 |
lazr.uri |
1.0.6 |
MarkupSafe |
2.0.1 |
matplotlib |
3.5.2 |
matplotlib-inline |
0.1.6 |
mccabe |
0.7.0 |
mistune |
0.8.4 |
more-itertools |
8.10.0 |
mypy-extensions |
0.4.3 |
nbclient |
0.5.13 |
nbconvert |
6.4.4 |
nbformat |
5.5.0 |
nest-asyncio |
1.5.5 |
nodeenv |
1.8.0 |
notebook |
6.4.12 |
numpy |
1.21.5 |
oauthlib |
3.2.0 |
packaging |
21.3 |
pandas |
1.4.4 |
pandocfilters |
1.5.0 |
parso |
0.8.3 |
pathspec |
0.9.0 |
patsy |
0.5.2 |
pexpect |
4.8.0 |
pickleshare |
0.7.5 |
Pillow |
9.2.0 |
pip |
22.2.2 |
platformdirs |
2.5.2 |
plotly |
5.9.0 |
pluggy |
1.0.0 |
prometheus-client |
0.14.1 |
prompt-toolkit |
3.0.36 |
protobuf |
3.19.4 |
psutil |
5.9.0 |
psycopg2 |
2.9.3 |
ptyprocess |
0.7.0 |
pure-eval |
0.2.2 |
pyarrow |
8.0.0 |
pycparser |
2.21 |
pydantic |
1.10.6 |
pyflakes |
3.0.1 |
Pygments |
2.11.2 |
PyGObject |
3.42.1 |
PyJWT |
2.3.0 |
pyodbc |
4.0.32 |
pyparsing |
3.0.9 |
pyright |
1.1.294 |
pyrsistent |
0.18.0 |
python-dateutil |
2.8.2 |
python-lsp-jsonrpc |
1.0.0 |
python-lsp-server |
1.7.1 |
pytoolconfig |
1.2.2 |
pytz |
2022.1 |
pyzmq |
23.2.0 |
requests |
2.28.1 |
rope |
1.7.0 |
s3transfer |
0.6.0 |
scikit-learn |
1.1.1 |
scipy |
1.9.1 |
seaborn |
0.11.2 |
SecretStorage |
3.3.1 |
Send2Trash |
1.8.0 |
setuptools |
63.4.1 |
six |
1.16.0 |
soupsieve |
2.3.1 |
ssh-import-id |
5.11 |
stack-data |
0.6.2 |
statsmodels |
0.13.2 |
tenacity |
8.1.0 |
terminado |
0.13.1 |
testpath |
0.6.0 |
threadpoolctl |
2.2.0 |
tokenize-rt |
4.2.1 |
tomli |
2.0.1 |
tornado |
6.1 |
traitlets |
5.1.1 |
typing_extensions |
4.3.0 |
ujson |
5.4.0 |
unattended-upgrades |
0.1 |
urllib3 |
1.26.11 |
virtualenv |
20.16.3 |
wadllib |
1.3.6 |
wcwidth |
0.2.5 |
webencodings |
0.5.1 |
whatthepatch |
1.0.2 |
wheel |
0.37.1 |
widgetsnbextension |
3.6.1 |
yapf |
0.31.0 |
zipp |
1.0.0 |
Installed R libraries
R libraries are installed from the Microsoft CRAN snapshot on 2023-02-10.
Library |
Version |
Library |
Version |
Library |
Version |
---|---|---|---|---|---|
arrow |
10.0.1 |
askpass |
1.1 |
assertthat |
0.2.1 |
backports |
1.4.1 |
base |
4.2.2 |
base64enc |
0.1-3 |
bit |
4.0.5 |
bit64 |
4.0.5 |
blob |
1.2.3 |
boot |
1.3-28 |
brew |
1.0-8 |
brio |
1.1.3 |
broom |
1.0.3 |
bslib |
0.4.2 |
cachem |
1.0.6 |
callr |
3.7.3 |
caret |
6.0-93 |
cellranger |
1.1.0 |
chron |
2.3-59 |
class |
7.3-21 |
cli |
3.6.0 |
clipr |
0.8.0 |
clock |
0.6.1 |
cluster |
2.1.4 |
codetools |
0.2-19 |
colorspace |
2.1-0 |
commonmark |
1.8.1 |
compiler |
4.2.2 |
config |
0.3.1 |
cpp11 |
0.4.3 |
crayon |
1.5.2 |
credentials |
1.3.2 |
curl |
5.0.0 |
data.table |
1.14.6 |
datasets |
4.2.2 |
DBI |
1.1.3 |
dbplyr |
2.3.0 |
desc |
1.4.2 |
devtools |
2.4.5 |
diffobj |
0.3.5 |
digest |
0.6.31 |
downlit |
0.4.2 |
dplyr |
1.1.0 |
dtplyr |
1.2.2 |
e1071 |
1.7-13 |
ellipsis |
0.3.2 |
evaluate |
0.20 |
fansi |
1.0.4 |
farver |
2.1.1 |
fastmap |
1.1.0 |
fontawesome |
0.5.0 |
forcats |
1.0.0 |
foreach |
1.5.2 |
foreign |
0.8-82 |
forge |
0.2.0 |
fs |
1.6.1 |
future |
1.31.0 |
future.apply |
1.10.0 |
gargle |
1.3.0 |
generics |
0.1.3 |
gert |
1.9.2 |
ggplot2 |
3.4.0 |
gh |
1.3.1 |
gitcreds |
0.1.2 |
glmnet |
4.1-6 |
globals |
0.16.2 |
glue |
1.6.2 |
googledrive |
2.0.0 |
googlesheets4 |
1.0.1 |
gower |
1.0.1 |
graphics |
4.2.2 |
grDevices |
4.2.2 |
grid |
4.2.2 |
gridExtra |
2.3 |
gsubfn |
0.7 |
gtable |
0.3.1 |
hardhat |
1.2.0 |
haven |
2.5.1 |
highr |
0.10 |
hms |
1.1.2 |
htmltools |
0.5.4 |
htmlwidgets |
1.6.1 |
httpuv |
1.6.8 |
httr |
1.4.4 |
ids |
1.0.1 |
ini |
0.3.1 |
ipred |
0.9-13 |
isoband |
0.2.7 |
iterators |
1.0.14 |
jquerylib |
0.1.4 |
jsonlite |
1.8.4 |
KernSmooth |
2.23-20 |
knitr |
1.42 |
labeling |
0.4.2 |
later |
1.3.0 |
lattice |
0.20-45 |
lava |
1.7.1 |
lifecycle |
1.0.3 |
listenv |
0.9.0 |
lubridate |
1.9.1 |
magrittr |
2.0.3 |
markdown |
1.5 |
MASS |
7.3-58.2 |
Matrix |
1.5-1 |
memoise |
2.0.1 |
methods |
4.2.2 |
mgcv |
1.8-41 |
mime |
0.12 |
miniUI |
0.1.1.1 |
ModelMetrics |
1.2.2.2 |
modelr |
0.1.10 |
munsell |
0.5.0 |
nlme |
3.1-162 |
nnet |
7.3-18 |
numDeriv |
2016.8-1.1 |
openssl |
2.0.5 |
parallel |
4.2.2 |
parallelly |
1.34.0 |
pillar |
1.8.1 |
pkgbuild |
1.4.0 |
pkgconfig |
2.0.3 |
pkgdown |
2.0.7 |
pkgload |
1.3.2 |
plogr |
0.2.0 |
plyr |
1.8.8 |
praise |
1.0.0 |
prettyunits |
1.1.1 |
pROC |
1.18.0 |
processx |
3.8.0 |
prodlim |
2019.11.13 |
profvis |
0.3.7 |
progress |
1.2.2 |
progressr |
0.13.0 |
promises |
1.2.0.1 |
proto |
1.0.0 |
proxy |
0.4-27 |
ps |
1.7.2 |
purrr |
1.0.1 |
r2d3 |
0.2.6 |
R6 |
2.5.1 |
ragg |
1.2.5 |
randomForest |
4.7-1.1 |
rappdirs |
0.3.3 |
rcmdcheck |
1.4.0 |
RColorBrewer |
1.1-3 |
Rcpp |
1.0.10 |
RcppEigen |
0.3.3.9.3 |
readr |
2.1.3 |
readxl |
1.4.2 |
recipes |
1.0.4 |
rematch |
1.0.1 |
rematch2 |
2.1.2 |
remotes |
2.4.2 |
reprex |
2.0.2 |
reshape2 |
1.4.4 |
rlang |
1.0.6 |
rmarkdown |
2.20 |
RODBC |
1.3-20 |
roxygen2 |
7.2.3 |
rpart |
4.1.19 |
rprojroot |
2.0.3 |
Rserve |
1.8-12 |
RSQLite |
2.2.20 |
rstudioapi |
0.14 |
rversions |
2.1.2 |
rvest |
1.0.3 |
sass |
0.4.5 |
scales |
1.2.1 |
selectr |
0.4-2 |
sessioninfo |
1.2.2 |
shape |
1.4.6 |
shiny |
1.7.4 |
sourcetools |
0.1.7-1 |
sparklyr |
1.7.9 |
SparkR |
3.4.1 |
spatial |
7.3-15 |
splines |
4.2.2 |
sqldf |
0.4-11 |
SQUAREM |
2021.1 |
stats |
4.2.2 |
stats4 |
4.2.2 |
stringi |
1.7.12 |
stringr |
1.5.0 |
survival |
3.5-3 |
sys |
3.4.1 |
systemfonts |
1.0.4 |
tcltk |
4.2.2 |
testthat |
3.1.6 |
textshaping |
0.3.6 |
tibble |
3.1.8 |
tidyr |
1.3.0 |
tidyselect |
1.2.0 |
tidyverse |
1.3.2 |
timechange |
0.2.0 |
timeDate |
4022.108 |
tinytex |
0.44 |
tools |
4.2.2 |
tzdb |
0.3.0 |
urlchecker |
1.0.1 |
usethis |
2.1.6 |
utf8 |
1.2.3 |
utils |
4.2.2 |
uuid |
1.1-0 |
vctrs |
0.5.2 |
viridisLite |
0.4.1 |
vroom |
1.6.1 |
waldo |
0.4.0 |
whisker |
0.4.1 |
withr |
2.5.0 |
xfun |
0.37 |
xml2 |
1.3.3 |
xopen |
1.0.0 |
xtable |
1.8-4 |
yaml |
2.3.7 |
zip |
2.2.2 |
Installed Java and Scala libraries (Scala 2.12 cluster version)
Group ID |
Artifact ID |
Version |
---|---|---|
antlr |
antlr |
2.7.7 |
com.amazonaws |
amazon-kinesis-client |
1.12.0 |
com.amazonaws |
aws-java-sdk-autoscaling |
1.12.390 |
com.amazonaws |
aws-java-sdk-cloudformation |
1.12.390 |
com.amazonaws |
aws-java-sdk-cloudfront |
1.12.390 |
com.amazonaws |
aws-java-sdk-cloudhsm |
1.12.390 |
com.amazonaws |
aws-java-sdk-cloudsearch |
1.12.390 |
com.amazonaws |
aws-java-sdk-cloudtrail |
1.12.390 |
com.amazonaws |
aws-java-sdk-cloudwatch |
1.12.390 |
com.amazonaws |
aws-java-sdk-cloudwatchmetrics |
1.12.390 |
com.amazonaws |
aws-java-sdk-codedeploy |
1.12.390 |
com.amazonaws |
aws-java-sdk-cognitoidentity |
1.12.390 |
com.amazonaws |
aws-java-sdk-cognitosync |
1.12.390 |
com.amazonaws |
aws-java-sdk-config |
1.12.390 |
com.amazonaws |
aws-java-sdk-core |
1.12.390 |
com.amazonaws |
aws-java-sdk-datapipeline |
1.12.390 |
com.amazonaws |
aws-java-sdk-directconnect |
1.12.390 |
com.amazonaws |
aws-java-sdk-directory |
1.12.390 |
com.amazonaws |
aws-java-sdk-dynamodb |
1.12.390 |
com.amazonaws |
aws-java-sdk-ec2 |
1.12.390 |
com.amazonaws |
aws-java-sdk-ecs |
1.12.390 |
com.amazonaws |
aws-java-sdk-efs |
1.12.390 |
com.amazonaws |
aws-java-sdk-elasticache |
1.12.390 |
com.amazonaws |
aws-java-sdk-elasticbeanstalk |
1.12.390 |
com.amazonaws |
aws-java-sdk-elasticloadbalancing |
1.12.390 |
com.amazonaws |
aws-java-sdk-elastictranscoder |
1.12.390 |
com.amazonaws |
aws-java-sdk-emr |
1.12.390 |
com.amazonaws |
aws-java-sdk-glacier |
1.12.390 |
com.amazonaws |
aws-java-sdk-glue |
1.12.390 |
com.amazonaws |
aws-java-sdk-iam |
1.12.390 |
com.amazonaws |
aws-java-sdk-importexport |
1.12.390 |
com.amazonaws |
aws-java-sdk-kinesis |
1.12.390 |
com.amazonaws |
aws-java-sdk-kms |
1.12.390 |
com.amazonaws |
aws-java-sdk-lambda |
1.12.390 |
com.amazonaws |
aws-java-sdk-logs |
1.12.390 |
com.amazonaws |
aws-java-sdk-machinelearning |
1.12.390 |
com.amazonaws |
aws-java-sdk-opsworks |
1.12.390 |
com.amazonaws |
aws-java-sdk-rds |
1.12.390 |
com.amazonaws |
aws-java-sdk-redshift |
1.12.390 |
com.amazonaws |
aws-java-sdk-route53 |
1.12.390 |
com.amazonaws |
aws-java-sdk-s3 |
1.12.390 |
com.amazonaws |
aws-java-sdk-ses |
1.12.390 |
com.amazonaws |
aws-java-sdk-simpledb |
1.12.390 |
com.amazonaws |
aws-java-sdk-simpleworkflow |
1.12.390 |
com.amazonaws |
aws-java-sdk-sns |
1.12.390 |
com.amazonaws |
aws-java-sdk-sqs |
1.12.390 |
com.amazonaws |
aws-java-sdk-ssm |
1.12.390 |
com.amazonaws |
aws-java-sdk-storagegateway |
1.12.390 |
com.amazonaws |
aws-java-sdk-sts |
1.12.390 |
com.amazonaws |
aws-java-sdk-support |
1.12.390 |
com.amazonaws |
aws-java-sdk-swf-libraries |
1.11.22 |
com.amazonaws |
aws-java-sdk-workspaces |
1.12.390 |
com.amazonaws |
jmespath-java |
1.12.390 |
com.clearspring.analytics |
stream |
2.9.6 |
com.databricks |
Rserve |
1.8-3 |
com.databricks |
databricks-sdk-java |
0.2.0 |
com.databricks |
jets3t |
0.7.1-0 |
com.databricks.scalapb |
compilerplugin_2.12 |
0.4.15-10 |
com.databricks.scalapb |
scalapb-runtime_2.12 |
0.4.15-10 |
com.esotericsoftware |
kryo-shaded |
4.0.2 |
com.esotericsoftware |
minlog |
1.3.0 |
com.fasterxml |
classmate |
1.3.4 |
com.fasterxml.jackson.core |
jackson-annotations |
2.14.2 |
com.fasterxml.jackson.core |
jackson-core |
2.14.2 |
com.fasterxml.jackson.core |
jackson-databind |
2.14.2 |
com.fasterxml.jackson.dataformat |
jackson-dataformat-cbor |
2.14.2 |
com.fasterxml.jackson.datatype |
jackson-datatype-joda |
2.14.2 |
com.fasterxml.jackson.datatype |
jackson-datatype-jsr310 |
2.13.4 |
com.fasterxml.jackson.module |
jackson-module-paranamer |
2.14.2 |
com.fasterxml.jackson.module |
jackson-module-scala_2.12 |
2.14.2 |
com.github.ben-manes.caffeine |
caffeine |
2.9.3 |
com.github.fommil |
jniloader |
1.1 |
com.github.fommil.netlib |
native_ref-java |
1.1 |
com.github.fommil.netlib |
native_ref-java |
1.1-natives |
com.github.fommil.netlib |
native_system-java |
1.1 |
com.github.fommil.netlib |
native_system-java |
1.1-natives |
com.github.fommil.netlib |
netlib-native_ref-linux-x86_64 |
1.1-natives |
com.github.fommil.netlib |
netlib-native_system-linux-x86_64 |
1.1-natives |
com.github.luben |
zstd-jni |
1.5.2-5 |
com.github.wendykierp |
JTransforms |
3.1 |
com.google.code.findbugs |
jsr305 |
3.0.0 |
com.google.code.gson |
gson |
2.8.9 |
com.google.crypto.tink |
tink |
1.7.0 |
com.google.errorprone |
error_prone_annotations |
2.10.0 |
com.google.flatbuffers |
flatbuffers-java |
1.12.0 |
com.google.guava |
guava |
15.0 |
com.google.protobuf |
protobuf-java |
2.6.1 |
com.h2database |
h2 |
2.1.214 |
com.helger |
profiler |
1.1.1 |
com.jcraft |
jsch |
0.1.55 |
com.jolbox |
bonecp |
0.8.0.RELEASE |
com.lihaoyi |
sourcecode_2.12 |
0.1.9 |
com.microsoft.azure |
azure-data-lake-store-sdk |
2.3.9 |
com.microsoft.sqlserver |
mssql-jdbc |
11.2.2.jre8 |
com.ning |
compress-lzf |
1.1.2 |
com.sun.mail |
javax.mail |
1.5.2 |
com.sun.xml.bind |
jaxb-core |
2.2.11 |
com.sun.xml.bind |
jaxb-impl |
2.2.11 |
com.tdunning |
json |
1.8 |
com.thoughtworks.paranamer |
paranamer |
2.8 |
com.trueaccord.lenses |
lenses_2.12 |
0.4.12 |
com.twitter |
chill-java |
0.10.0 |
com.twitter |
chill_2.12 |
0.10.0 |
com.twitter |
util-app_2.12 |
7.1.0 |
com.twitter |
util-core_2.12 |
7.1.0 |
com.twitter |
util-function_2.12 |
7.1.0 |
com.twitter |
util-jvm_2.12 |
7.1.0 |
com.twitter |
util-lint_2.12 |
7.1.0 |
com.twitter |
util-registry_2.12 |
7.1.0 |
com.twitter |
util-stats_2.12 |
7.1.0 |
com.typesafe |
config |
1.2.1 |
com.typesafe.scala-logging |
scala-logging_2.12 |
3.7.2 |
com.uber |
h3 |
3.7.0 |
com.univocity |
univocity-parsers |
2.9.1 |
com.zaxxer |
HikariCP |
4.0.3 |
commons-cli |
commons-cli |
1.5.0 |
commons-codec |
commons-codec |
1.15 |
commons-collections |
commons-collections |
3.2.2 |
commons-dbcp |
commons-dbcp |
1.4 |
commons-fileupload |
commons-fileupload |
1.5 |
commons-httpclient |
commons-httpclient |
3.1 |
commons-io |
commons-io |
2.11.0 |
commons-lang |
commons-lang |
2.6 |
commons-logging |
commons-logging |
1.1.3 |
commons-pool |
commons-pool |
1.5.4 |
dev.ludovic.netlib |
arpack |
3.0.3 |
dev.ludovic.netlib |
blas |
3.0.3 |
dev.ludovic.netlib |
lapack |
3.0.3 |
info.ganglia.gmetric4j |
gmetric4j |
1.0.10 |
io.airlift |
aircompressor |
0.21 |
io.delta |
delta-sharing-spark_2.12 |
0.7.1 |
io.dropwizard.metrics |
metrics-core |
4.2.10 |
io.dropwizard.metrics |
metrics-graphite |
4.2.10 |
io.dropwizard.metrics |
metrics-healthchecks |
4.2.10 |
io.dropwizard.metrics |
metrics-jetty9 |
4.2.10 |
io.dropwizard.metrics |
metrics-jmx |
4.2.10 |
io.dropwizard.metrics |
metrics-json |
4.2.10 |
io.dropwizard.metrics |
metrics-jvm |
4.2.10 |
io.dropwizard.metrics |
metrics-servlets |
4.2.10 |
io.netty |
netty-all |
4.1.87.Final |
io.netty |
netty-buffer |
4.1.87.Final |
io.netty |
netty-codec |
4.1.87.Final |
io.netty |
netty-codec-http |
4.1.87.Final |
io.netty |
netty-codec-http2 |
4.1.87.Final |
io.netty |
netty-codec-socks |
4.1.87.Final |
io.netty |
netty-common |
4.1.87.Final |
io.netty |
netty-handler |
4.1.87.Final |
io.netty |
netty-handler-proxy |
4.1.87.Final |
io.netty |
netty-resolver |
4.1.87.Final |
io.netty |
netty-transport |
4.1.87.Final |
io.netty |
netty-transport-classes-epoll |
4.1.87.Final |
io.netty |
netty-transport-classes-kqueue |
4.1.87.Final |
io.netty |
netty-transport-native-epoll |
4.1.87.Final |
io.netty |
netty-transport-native-epoll |
4.1.87.Final-linux-aarch_64 |
io.netty |
netty-transport-native-epoll |
4.1.87.Final-linux-x86_64 |
io.netty |
netty-transport-native-kqueue |
4.1.87.Final-osx-aarch_64 |
io.netty |
netty-transport-native-kqueue |
4.1.87.Final-osx-x86_64 |
io.netty |
netty-transport-native-unix-common |
4.1.87.Final |
io.prometheus |
simpleclient |
0.7.0 |
io.prometheus |
simpleclient_common |
0.7.0 |
io.prometheus |
simpleclient_dropwizard |
0.7.0 |
io.prometheus |
simpleclient_pushgateway |
0.7.0 |
io.prometheus |
simpleclient_servlet |
0.7.0 |
io.prometheus.jmx |
collector |
0.12.0 |
jakarta.annotation |
jakarta.annotation-api |
1.3.5 |
jakarta.servlet |
jakarta.servlet-api |
4.0.3 |
jakarta.validation |
jakarta.validation-api |
2.0.2 |
jakarta.ws.rs |
jakarta.ws.rs-api |
2.1.6 |
javax.activation |
activation |
1.1.1 |
javax.el |
javax.el-api |
2.2.4 |
javax.jdo |
jdo-api |
3.0.1 |
javax.transaction |
jta |
1.1 |
javax.transaction |
transaction-api |
1.1 |
javax.xml.bind |
jaxb-api |
2.2.11 |
javolution |
javolution |
5.5.1 |
jline |
jline |
2.14.6 |
joda-time |
joda-time |
2.12.1 |
ml.combust.mleap |
mleap-databricks-runtime_2.12 |
v0.20.0-db2 |
net.java.dev.jna |
jna |
5.8.0 |
net.razorvine |
pickle |
1.3 |
net.sf.jpam |
jpam |
1.1 |
net.sf.opencsv |
opencsv |
2.3 |
net.sf.supercsv |
super-csv |
2.2.0 |
net.snowflake |
snowflake-ingest-sdk |
0.9.6 |
net.snowflake |
snowflake-jdbc |
3.13.33 |
net.sourceforge.f2j |
arpack_combined_all |
0.1 |
org.acplt.remotetea |
remotetea-oncrpc |
1.1.2 |
org.antlr |
ST4 |
4.0.4 |
org.antlr |
antlr-runtime |
3.5.2 |
org.antlr |
antlr4-runtime |
4.9.3 |
org.antlr |
stringtemplate |
3.2.1 |
org.apache.ant |
ant |
1.9.16 |
org.apache.ant |
ant-jsch |
1.9.16 |
org.apache.ant |
ant-launcher |
1.9.16 |
org.apache.arrow |
arrow-format |
11.0.0 |
org.apache.arrow |
arrow-memory-core |
11.0.0 |
org.apache.arrow |
arrow-memory-netty |
11.0.0 |
org.apache.arrow |
arrow-vector |
11.0.0 |
org.apache.avro |
avro |
1.11.1 |
org.apache.avro |
avro-ipc |
1.11.1 |
org.apache.avro |
avro-mapred |
1.11.1 |
org.apache.commons |
commons-collections4 |
4.4 |
org.apache.commons |
commons-compress |
1.21 |
org.apache.commons |
commons-crypto |
1.1.0 |
org.apache.commons |
commons-lang3 |
3.12.0 |
org.apache.commons |
commons-math3 |
3.6.1 |
org.apache.commons |
commons-text |
1.10.0 |
org.apache.curator |
curator-client |
2.13.0 |
org.apache.curator |
curator-framework |
2.13.0 |
org.apache.curator |
curator-recipes |
2.13.0 |
org.apache.datasketches |
datasketches-java |
3.1.0 |
org.apache.datasketches |
datasketches-memory |
2.0.0 |
org.apache.derby |
derby |
10.14.2.0 |
org.apache.hadoop |
hadoop-client-runtime |
3.3.4 |
org.apache.hive |
hive-beeline |
2.3.9 |
org.apache.hive |
hive-cli |
2.3.9 |
org.apache.hive |
hive-jdbc |
2.3.9 |
org.apache.hive |
hive-llap-client |
2.3.9 |
org.apache.hive |
hive-llap-common |
2.3.9 |
org.apache.hive |
hive-serde |
2.3.9 |
org.apache.hive |
hive-shims |
2.3.9 |
org.apache.hive |
hive-storage-api |
2.8.1 |
org.apache.hive.shims |
hive-shims-0.23 |
2.3.9 |
org.apache.hive.shims |
hive-shims-common |
2.3.9 |
org.apache.hive.shims |
hive-shims-scheduler |
2.3.9 |
org.apache.httpcomponents |
httpclient |
4.5.14 |
org.apache.httpcomponents |
httpcore |
4.4.16 |
org.apache.ivy |
ivy |
2.5.1 |
org.apache.logging.log4j |
log4j-1.2-api |
2.19.0 |
org.apache.logging.log4j |
log4j-api |
2.19.0 |
org.apache.logging.log4j |
log4j-core |
2.19.0 |
org.apache.logging.log4j |
log4j-slf4j2-impl |
2.19.0 |
org.apache.mesos |
mesos |
1.11.0-shaded-protobuf |
org.apache.orc |
orc-core |
1.8.4-shaded-protobuf |
org.apache.orc |
orc-mapreduce |
1.8.4-shaded-protobuf |
org.apache.orc |
orc-shims |
1.8.4 |
org.apache.thrift |
libfb303 |
0.9.3 |
org.apache.thrift |
libthrift |
0.12.0 |
org.apache.xbean |
xbean-asm9-shaded |
4.22 |
org.apache.yetus |
audience-annotations |
0.13.0 |
org.apache.zookeeper |
zookeeper |
3.6.3 |
org.apache.zookeeper |
zookeeper-jute |
3.6.3 |
org.checkerframework |
checker-qual |
3.19.0 |
org.codehaus.jackson |
jackson-core-asl |
1.9.13 |
org.codehaus.jackson |
jackson-mapper-asl |
1.9.13 |
org.codehaus.janino |
commons-compiler |
3.0.16 |
org.codehaus.janino |
janino |
3.0.16 |
org.datanucleus |
datanucleus-api-jdo |
4.2.4 |
org.datanucleus |
datanucleus-core |
4.1.17 |
org.datanucleus |
datanucleus-rdbms |
4.1.19 |
org.datanucleus |
javax.jdo |
3.2.0-m3 |
org.eclipse.jetty |
jetty-client |
9.4.51.v20230217 |
org.eclipse.jetty |
jetty-continuation |
9.4.51.v20230217 |
org.eclipse.jetty |
jetty-http |
9.4.51.v20230217 |
org.eclipse.jetty |
jetty-io |
9.4.51.v20230217 |
org.eclipse.jetty |
jetty-jndi |
9.4.51.v20230217 |
org.eclipse.jetty |
jetty-plus |
9.4.51.v20230217 |
org.eclipse.jetty |
jetty-proxy |
9.4.51.v20230217 |
org.eclipse.jetty |
jetty-security |
9.4.51.v20230217 |
org.eclipse.jetty |
jetty-server |
9.4.51.v20230217 |
org.eclipse.jetty |
jetty-servlet |
9.4.51.v20230217 |
org.eclipse.jetty |
jetty-servlets |
9.4.51.v20230217 |
org.eclipse.jetty |
jetty-util |
9.4.51.v20230217 |
org.eclipse.jetty |
jetty-util-ajax |
9.4.51.v20230217 |
org.eclipse.jetty |
jetty-webapp |
9.4.51.v20230217 |
org.eclipse.jetty |
jetty-xml |
9.4.51.v20230217 |
org.eclipse.jetty.websocket |
websocket-api |
9.4.51.v20230217 |
org.eclipse.jetty.websocket |
websocket-client |
9.4.51.v20230217 |
org.eclipse.jetty.websocket |
websocket-common |
9.4.51.v20230217 |
org.eclipse.jetty.websocket |
websocket-server |
9.4.51.v20230217 |
org.eclipse.jetty.websocket |
websocket-servlet |
9.4.51.v20230217 |
org.fusesource.leveldbjni |
leveldbjni-all |
1.8 |
org.glassfish.hk2 |
hk2-api |
2.6.1 |
org.glassfish.hk2 |
hk2-locator |
2.6.1 |
org.glassfish.hk2 |
hk2-utils |
2.6.1 |
org.glassfish.hk2 |
osgi-resource-locator |
1.0.3 |
org.glassfish.hk2.external |
aopalliance-repackaged |
2.6.1 |
org.glassfish.hk2.external |
jakarta.inject |
2.6.1 |
org.glassfish.jersey.containers |
jersey-container-servlet |
2.36 |
org.glassfish.jersey.containers |
jersey-container-servlet-core |
2.36 |
org.glassfish.jersey.core |
jersey-client |
2.36 |
org.glassfish.jersey.core |
jersey-common |
2.36 |
org.glassfish.jersey.core |
jersey-server |
2.36 |
org.glassfish.jersey.inject |
jersey-hk2 |
2.36 |
org.hibernate.validator |
hibernate-validator |
6.1.7.Final |
org.ini4j |
ini4j |
0.5.4 |
org.javassist |
javassist |
3.25.0-GA |
org.jboss.logging |
jboss-logging |
3.3.2.Final |
org.jdbi |
jdbi |
2.63.1 |
org.jetbrains |
annotations |
17.0.0 |
org.joda |
joda-convert |
1.7 |
org.jodd |
jodd-core |
3.5.2 |
org.json4s |
json4s-ast_2.12 |
3.7.0-M11 |
org.json4s |
json4s-core_2.12 |
3.7.0-M11 |
org.json4s |
json4s-jackson_2.12 |
3.7.0-M11 |
org.json4s |
json4s-scalap_2.12 |
3.7.0-M11 |
org.lz4 |
lz4-java |
1.8.0 |
org.mariadb.jdbc |
mariadb-java-client |
2.7.4 |
org.mlflow |
mlflow-spark |
2.2.0 |
org.objenesis |
objenesis |
2.5.1 |
org.postgresql |
postgresql |
42.3.8 |
org.roaringbitmap |
RoaringBitmap |
0.9.39 |
org.roaringbitmap |
shims |
0.9.39 |
org.rocksdb |
rocksdbjni |
7.8.3 |
org.rosuda.REngine |
REngine |
2.1.0 |
org.scala-lang |
scala-compiler_2.12 |
2.12.15 |
org.scala-lang |
scala-library_2.12 |
2.12.15 |
org.scala-lang |
scala-reflect_2.12 |
2.12.15 |
org.scala-lang.modules |
scala-collection-compat_2.12 |
2.4.3 |
org.scala-lang.modules |
scala-parser-combinators_2.12 |
1.1.2 |
org.scala-lang.modules |
scala-xml_2.12 |
1.2.0 |
org.scala-sbt |
test-interface |
1.0 |
org.scalacheck |
scalacheck_2.12 |
1.14.2 |
org.scalactic |
scalactic_2.12 |
3.2.15 |
org.scalanlp |
breeze-macros_2.12 |
2.1.0 |
org.scalanlp |
breeze_2.12 |
2.1.0 |
org.scalatest |
scalatest-compatible |
3.2.15 |
org.scalatest |
scalatest-core_2.12 |
3.2.15 |
org.scalatest |
scalatest-diagrams_2.12 |
3.2.15 |
org.scalatest |
scalatest-featurespec_2.12 |
3.2.15 |
org.scalatest |
scalatest-flatspec_2.12 |
3.2.15 |
org.scalatest |
scalatest-freespec_2.12 |
3.2.15 |
org.scalatest |
scalatest-funspec_2.12 |
3.2.15 |
org.scalatest |
scalatest-funsuite_2.12 |
3.2.15 |
org.scalatest |
scalatest-matchers-core_2.12 |
3.2.15 |
org.scalatest |
scalatest-mustmatchers_2.12 |
3.2.15 |
org.scalatest |
scalatest-propspec_2.12 |
3.2.15 |
org.scalatest |
scalatest-refspec_2.12 |
3.2.15 |
org.scalatest |
scalatest-shouldmatchers_2.12 |
3.2.15 |
org.scalatest |
scalatest-wordspec_2.12 |
3.2.15 |
org.scalatest |
scalatest_2.12 |
3.2.15 |
org.slf4j |
jcl-over-slf4j |
2.0.6 |
org.slf4j |
jul-to-slf4j |
2.0.6 |
org.slf4j |
slf4j-api |
2.0.6 |
org.threeten |
threeten-extra |
1.7.1 |
org.tukaani |
xz |
1.9 |
org.typelevel |
algebra_2.12 |
2.0.1 |
org.typelevel |
cats-kernel_2.12 |
2.1.1 |
org.typelevel |
spire-macros_2.12 |
0.17.0 |
org.typelevel |
spire-platform_2.12 |
0.17.0 |
org.typelevel |
spire-util_2.12 |
0.17.0 |
org.typelevel |
spire_2.12 |
0.17.0 |
org.wildfly.openssl |
wildfly-openssl |
1.1.3.Final |
org.xerial |
sqlite-jdbc |
3.42.0.0 |
org.xerial.snappy |
snappy-java |
1.1.10.1 |
org.yaml |
snakeyaml |
1.33 |
oro |
oro |
2.0.8 |
pl.edu.icm |
JLargeArrays |
1.5 |
software.amazon.cryptools |
AmazonCorrettoCryptoProvider |
1.6.1-linux-x86_64 |
software.amazon.ion |
ion-java |
1.0.2 |
stax |
stax-api |
1.0.1 |