Google
Cloud Platform
Amazon Web Services
Microsoft AzureDatabricks Runtime 16.0
The following release notes provide information about Databricks Runtime 16.0, powered by Apache Spark 3.5.0.
Databricks released this version in November 2024.
Note
These release notes may include references to features that are not available on Google Cloud as of this release.
Tip
To see release notes for Databricks Runtime versions that have reached end-of-support (EoS), see End-of-support Databricks Runtime release notes. The EoS Databricks Runtime versions have been retired and might not be updated.
Behavioral changes
Breaking change: Removal of support for changing
byte,short,intandlongtypes to wider typesCorrect parsing of regex patterns with negation in nested character grouping
Cluster library installation method can no longer be overridden
Default timeout of two hours for cluster-scoped library installs
The
addArtifact()functionality is now consistent across compute types
Breaking change: JDK 17 is now the default
In Databricks Runtime 16.0 and above, the default JDK version is switched from JDK 8 to JDK 17. This change is made because of the planned deprecation and end-of-support for JDK 8. This affects the following:
Java code running on Databricks compute must be compatible with Java 17.
Scala code running in notebooks or Databricks compute must be compatible with Java 17.
Java and Scala libraries installed on compute must be compatible with Java 17.
Versions of the Apache Hive metastore client below 2.x. Setting the Spark configuration
spark.sql.hive.metastore.versionto a version lower than 2.x will cause compatibility issues with Java 17 and connection failures to the Hive metastore. Databricks recommends upgrading Hive to a version above 2.0.0.
If you need to revert to Java 8, add the following to the Spark environment variables when you configure your Databricks compute:
JNAME=zulu8-ca-amd64
If you’re using ARM instances, use the following:
JNAME=zulu8-ca-arm64
To learn more about specifying JDK versions with Databricks compute, see Create a cluster that uses JDK 17.
For help with migrating your code from Java 8, see the following guides:
Breaking change: Hosted RStudio is end-of-life
With this release, Databricks-hosted RStudio Server is end-of-life and unavailable on any Databricks workspace running Databricks Runtime 16.0 and above. To learn more and see a list of alternatives to RStudio, see Hosted RStudio Server deprecation.
Breaking change: Removal of support for changing byte, short, int and long types to wider types
In Databricks Runtime 15.4.3 and above, the following data type changes can no longer be applied to tables with the type widening feature enabled:
byte,short,intandlongtodecimal.byte,short, andinttodouble.
This change is made to ensure consistent behavior across Delta and Iceberg tables. To learn more about type widening, see Type widening.
Correct parsing of regex patterns with negation in nested character grouping
This release includes a change to support the correct parsing of regex patterns with negation in nested character grouping. For example, [^[abc]] will be parsed as “any character that is NOT one of ‘abc’”.
Additionally, Photon behavior was inconsistent with Spark for nested character classes. Regex patterns containing nested character classes will no longer use Photon, and instead will use Spark. A nested character class is any pattern containing square brackets within square brackets, such as [[a-c][1-3]].
Improve duplicate match detection in Delta Lake MERGE
In Databricks Runtime 15.4 LTS and below, MERGE operations fail if more than one row in the source table matches the same row in the target table based on the MERGE condition specified in the ON clause. In Databricks Runtime 16.0 and above, MERGE also considers conditions specified in the WHEN MATCHED clause. See Upsert into a Delta Lake table using merge.
Cluster library installation method can no longer be overridden
The Spark configurations spark.databricks.libraries.enableSparkPyPI, spark.databricks.libraries.enableMavenResolution, and spark.databricks.libraries.enableCRANResolutionnow now default to true and cannot be overridden.
Default timeout of two hours for cluster-scoped library installs
In Databricks Runtime 16.0 and above, cluster-scoped library installation has a default timeout of two hours. Library installations that take longer than this timeout will fail, and the installation is terminated. When configuring a cluster, you can change the timeout period using the Spark configuration spark.databricks.driver.clusterLibraryInstallationTimeoutSec.
Installing libraries from DBFS and setting the spark conf spark.databricks.driver.dbfsLibraryInstallationAllowed are disabled
In Databricks Runtime 16.0 and above, installing libraries from DBFS is fully disabled. This change is made to improve the security of libraries in a Databricks workspace. Additionally, in Databricks Runtime 16.0 and above, you can no longer use the Spark configuration spark.databricks.driver.dbfsLibraryInstallationAllowed.
The addArtifact() functionality is now consistent across compute types
With this release, the archive is automatically unpacked when you use addArtifact(archive = True) to add a dependency to shared or serverless Databricks compute. This change makes the `addArtifact(archive = True) behavior on these compute types consistent with single user compute, which already supports automatically unpacking archives.
New features and improvements
More reliable reloading of modified Python modules with improvements to autoreload
In Databricks Runtime 16.0 and above, updates to the autoreload extension improve the safety and reliability of reloading modified Python modules imported from workspace files. With these changes, autoreload, when possible, reloads only the portion of a module that has changed instead of the entire module. Additionally, Databricks now automatically suggests using the autoreload extension if the module has changed since its last import. See Autoreload for Python modules.
Avro support for recursive schema
You can now use the recursiveFieldMaxDepth option with the from_avro function and the avro data source. This option sets the maximum depth for schema recursion on the Avro data source. See Read and write streaming Avro data.
Expanded support for Confluent Schema Registry for Avro
Databricks now supports Avro schema reference with the Confluent Schema Registry. See Authenticate to an external Confluent Schema Registry.
Force reclustering on tables with liquid clustering
In Databricks Runtime 16.0 and above, you can use the OPTIMIZE FULL syntax to force the reclustering of all records in a table with liquid clustering enabled. See Force reclustering for all records.
The Delta APIs for Python and Scala now support identity columns
You can now use the Delta APIs for Python and Scala to create tables with identity columns. See Use identity columns in Delta Lake.
Create liquid clustered tables during streaming writes
You can now use clusterBy to enable liquid clustering when creating new tables with Structured Streaming writes. See Enable liquid clustering.
Support for the OPTIMIZE FULL clause
Databricks Runtime 16.0 supports the OPTIMIZE FULL clause. This clause optimizes all records in a table that uses liquid clustering, including data that might have previously been clustered.
Support for WITH options specification in INSERT and table-reference
Databricks Runtime 16.0 supports an options specification for table references and table names of an INSERT statement which can be used to control the behavior of data sources.
New SQL functions
The following SQL functions are added in Databricks Runtime 16.0:
-
This function is an error-tolerant version of url_decode. This function returns
NULLif the input is not a valid URL-encoded string. -
If the input expression to the
zeroifnull()function isNULL, then the function returns 0. Otherwise, the value of the input expression is returned. -
Returns
NULLif the input is 0 or its input if it is not 0. If the input expression to thenullifzero()function is 0, then the function returnsNULL. If the input expression is not 0, the value of the input expression is returned
Enable automatic schema evolution when merging data into a Delta table
This release adds support for the withSchemaEvolution() member of the DeltaMergeBuilder class. Use withSchemaEvolution() to enable automatic schema evolution during MERGE operations. For example, mergeBuilder.whenMatched(...).withSchemaEvolution().execute()}}.
Other changes
SparkR is now deprecated
In Databricks Runtime 16.0 and above, SparkR in Databricks is deprecated in preparation for its deprecation in the upcoming Spark 4 release. See the Apache Spark Deprecate SparkR thread.
Databricks recommends using sparklyr instead.
Bug fixes
Auto Loader now rescues Avro record types with empty schemas
When loading an Avro file into a Delta table using Auto Loader, record types in the file that have an empty schema are now added to the rescued data column. Because you cannot ingest empty complex data types into a Delta table, this resolves an issue with loading some Avro files. To learn more about rescued data, see What is the rescued data column?.
Fix for error writing timestamps with timezones containing a second offset.
This release fixes a bug affecting some timestamps with timezones containing a second offset. This bug causes the seconds to be omitted when writing to JSON, XML, or CSV, leading to incorrect timestamp values.
To return to the previous behavior, use the following option when writing to one of the affected formats: .option("yyyy-MM-dd'T'HH:mm:ss[.SSS][XXX]").
Library upgrades
Upgraded Python libraries:
azure-core from 1.30.2 to 1.31.0
azure-storage-blob from 12.19.1 to 12.23.0
azure-storage-file-datalake from 12.14.0 to 12.17.0
black from 23.3.0 to 24.4.2
blinker from 1.4 to 1.7.0
boto3 from 1.34.39 to 1.34.69
botocore from 1.34.39 to 1.34.69
certifi from 2023.7.22 to 2024.6.2
cffi from 1.15.1 to 1.16.0
click from 8.0.4 to 8.1.7
comm from 0.1.2 to 0.2.1
contourpy from 1.0.5 to 1.2.0
cryptography from 41.0.3 to 42.0.5
Cython from 0.29.32 to 3.0.11
databricks-sdk from 0.20.0 to 0.30.0
dbus-python from 1.2.18 to 1.3.2
filelock from 3.13.4 to 3.15.4
fonttools from 4.25.0 to 4.51.0
GitPython from 3.1.43 to 3.1.37
google-api-core from 2.18.0 to 2.20.0
google-auth from 2.31.0 to 2.35.0
google-cloud-storage from 2.17.0 to 2.18.2
google-crc32c from 1.5.0 to 1.6.0
google-resumable-media from 2.7.1 to 2.7.2
googleapis-common-protos from 1.63.2 to 1.65.0
httplib2 from 0.20.2 to 0.20.4
idna from 3.4 to 3.7
ipykernel from 6.25.1 to 6.28.0
ipython from 8.15.0 to 8.25.0
jedi from 0.18.1 to 0.19.1
jmespath from 0.10.0 to 1.0.1
joblib from 1.2.0 to 1.4.2
jupyter_client from 7.4.9 to 8.6.0
jupyter_core from 5.3.0 to 5.7.2
launchpadlib from 1.10.16 to 1.11.0
lazr.restfulclient from 0.14.4 to 0.14.6
matplotlib from 3.7.2 to 3.8.4
mlflow-skinny from 2.11.4 to 2.15.1
more-itertools from 8.10.0 to 10.3.0
mypy-extensions from 0.4.3 to 1.0.0
nest-asyncio from 1.5.6 to 1.6.0
numpy from 1.23.5 to 1.26.4
oauthlib from 3.2.0 to 3.2.2
packaging from 23.2 to 24.1
patsy from 0.5.3 to 0.5.6
pip from 23.2.1 to 24.2
plotly from 5.9.0 to 5.22.0
prompt-toolkit from 3.0.36 to 3.0.43
pyarrow from 14.0.1 to 15.0.2
pydantic from 1.10.6 to 2.8.2
PyGObject from 3.42.1 to 3.48.2
PyJWT from 2.3.0 to 2.7.0
pyodbc from 4.0.38 to 5.0.1
python-dateutil from 2.8.2 to 2.9.0.post0
python-lsp-jsonrpc from 1.1.1 to 1.1.2
pytz from 2022.7 to 2024.1
PyYAML from 6.0 to 6.0.1
pyzmq from 23.2.0 to 25.1.2
requests from 2.31.0 to 2.32.2
scikit-learn from 1.3.0 to 1.4.2
scipy from 1.11.1 to 1.13.1
seaborn from 0.12.2 to 0.13.2
setuptools from 68.0.0 to 74.0.0
smmap from 5.0.1 to 5.0.0
sqlparse from 0.5.0 to 0.5.1
statsmodels from 0.14.0 to 0.14.2
tornado from 6.3.2 to 6.4.1
traitlets from 5.7.1 to 5.14.3
typing_extensions from 4.10.0 to 4.11.0
ujson from 5.4.0 to 5.10.0
virtualenv from 20.24.2 to 20.26.2
wheel from 0.38.4 to 0.43.0
zipp from 3.11.0 to 3.17.0
Upgraded R libraries:
arrow from 14.0.0.2 to 16.1.0
backports from 1.4.1 to 1.5.0
base from 4.3.2 to 4.4.0
bitops from 1.0-7 to 1.0-8
boot from 1.3-28 to 1.3-30
brio from 1.1.4 to 1.1.5
broom from 1.0.5 to 1.0.6
bslib from 0.6.1 to 0.8.0
cachem from 1.0.8 to 1.1.0
callr from 3.7.3 to 3.7.6
cli from 3.6.2 to 3.6.3
clock from 0.7.0 to 0.7.1
cluster from 2.1.4 to 2.1.6
codetools from 0.2-19 to 0.2-20
colorspace from 2.1-0 to 2.1-1
compiler from 4.3.2 to 4.4.0
crayon from 1.5.2 to 1.5.3
curl from 5.2.0 to 5.2.1
data.table from 1.15.0 to 1.15.4
datasets from 4.3.2 to 4.4.0
DBI from 1.2.1 to 1.2.3
dbplyr from 2.4.0 to 2.5.0
digest from 0.6.34 to 0.6.36
downlit from 0.4.3 to 0.4.4
evaluate from 0.23 to 0.24.0
farver from 2.1.1 to 2.1.2
fastmap from 1.1.1 to 1.2.0
foreign from 0.8-85 to 0.8-86
fs from 1.6.3 to 1.6.4
future from 1.33.1 to 1.34.0
future.apply from 1.11.1 to 1.11.2
gert from 2.0.1 to 2.1.0
ggplot2 from 3.4.4 to 3.5.1
gh from 1.4.0 to 1.4.1
globals from 0.16.2 to 0.16.3
graphics from 4.3.2 to 4.4.0
grDevices from 4.3.2 to 4.4.0
grid from 4.3.2 to 4.4.0
gt from 0.10.1 to 0.11.0
gtable from 0.3.4 to 0.3.5
hardhat from 1.3.1 to 1.4.0
highr from 0.10 to 0.11
htmltools from 0.5.7 to 0.5.8.1
httpuv from 1.6.14 to 1.6.15
httr2 from 1.0.0 to 1.0.2
ipred from 0.9-14 to 0.9-15
KernSmooth from 2.23-21 to 2.23-22
knitr from 1.45 to 1.48
lattice from 0.21-8 to 0.22-5
lava from 1.7.3 to 1.8.0
markdown from 1.12 to 1.13
MASS from 7.3-60 to 7.3-60.0.1
Matrix from 1.5-4.1 to 1.6-5
methods from 4.3.2 to 4.4.0
mgcv from 1.8-42 to 1.9-1
mlflow from 2.10.0 to 2.14.1
munsell from 0.5.0 to 0.5.1
nlme from 3.1-163 to 3.1-165
openssl from 2.1.1 to 2.2.0
parallel from 4.3.2 to 4.4.0
parallelly from 1.36.0 to 1.38.0
pkgbuild from 1.4.3 to 1.4.4
pkgdown from 2.0.7 to 2.1.0
pkgload from 1.3.4 to 1.4.0
processx from 3.8.3 to 3.8.4
prodlim from 2023.08.28 to 2024.06.25
promises from 1.2.1 to 1.3.0
ps from 1.7.6 to 1.7.7
ragg from 1.2.7 to 1.3.2
Rcpp from 1.0.12 to 1.0.13
RcppEigen from 0.3.3.9.4 to 0.3.4.0.0
reactR from 0.5.0 to 0.6.0
recipes from 1.0.9 to 1.1.0
remotes from 2.4.2.1 to 2.5.0
reprex from 2.1.0 to 2.1.1
rlang from 1.1.3 to 1.1.4
rmarkdown from 2.25 to 2.27
roxygen2 from 7.3.1 to 7.3.2
rpart from 4.1.21 to 4.1.23
RSQLite from 2.3.5 to 2.3.7
rstudioapi from 0.15.0 to 0.16.0
rvest from 1.0.3 to 1.0.4
sass from 0.4.8 to 0.4.9
shape from 1.4.6 to 1.4.6.1
shiny from 1.8.0 to 1.9.1
sparklyr from 1.8.4 to 1.8.6
spatial from 7.3-15 to 7.3-17
splines from 4.3.2 to 4.4.0
stats from 4.3.2 to 4.4.0
stats4 from 4.3.2 to 4.4.0
stringi from 1.8.3 to 1.8.4
survival from 3.5-5 to 3.6-4
swagger from 3.33.1 to 5.17.14.1
systemfonts from 1.0.5 to 1.1.0
tcltk from 4.3.2 to 4.4.0
testthat from 3.2.1 to 3.2.1.1
textshaping from 0.3.7 to 0.4.0
tidyselect from 1.2.0 to 1.2.1
tinytex from 0.49 to 0.52
tools from 4.3.2 to 4.4.0
usethis from 2.2.2 to 3.0.0
utils from 4.3.2 to 4.4.0
uuid from 1.2-0 to 1.2-1
V8 from 4.4.1 to 4.4.2
withr from 3.0.0 to 3.0.1
xfun from 0.41 to 0.46
xopen from 1.0.0 to 1.0.1
yaml from 2.3.8 to 2.3.10
Upgraded Java libraries:
com.amazonaws.aws-java-sdk-autoscaling from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-cloudformation from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-cloudfront from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-cloudhsm from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-cloudsearch from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-cloudtrail from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-cloudwatch from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-cloudwatchmetrics from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-codedeploy from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-cognitoidentity from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-cognitosync from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-config from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-core from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-datapipeline from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-directconnect from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-directory from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-dynamodb from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-ec2 from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-ecs from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-efs from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-elasticache from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-elasticbeanstalk from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-elasticloadbalancing from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-elastictranscoder from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-emr from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-glacier from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-glue from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-iam from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-importexport from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-kinesis from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-kms from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-lambda from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-logs from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-machinelearning from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-opsworks from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-rds from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-redshift from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-route53 from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-s3 from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-ses from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-simpledb from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-simpleworkflow from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-sns from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-sqs from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-ssm from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-storagegateway from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-sts from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-support from 1.12.610 to 1.12.638
com.amazonaws.aws-java-sdk-workspaces from 1.12.610 to 1.12.638
com.amazonaws.jmespath-java from 1.12.610 to 1.12.638
com.google.protobuf.protobuf-java from 2.6.1 to 3.25.1
io.airlift.aircompressor from 0.25 to 0.27
io.delta.delta-sharing-client_2.12 from 1.1.3 to 1.2.0
io.netty.netty-all from 4.1.96.Final to 4.1.108.Final
io.netty.netty-buffer from 4.1.96.Final to 4.1.108.Final
io.netty.netty-codec from 4.1.96.Final to 4.1.108.Final
io.netty.netty-codec-http from 4.1.96.Final to 4.1.108.Final
io.netty.netty-codec-http2 from 4.1.96.Final to 4.1.108.Final
io.netty.netty-codec-socks from 4.1.96.Final to 4.1.108.Final
io.netty.netty-common from 4.1.96.Final to 4.1.108.Final
io.netty.netty-handler from 4.1.96.Final to 4.1.108.Final
io.netty.netty-handler-proxy from 4.1.96.Final to 4.1.108.Final
io.netty.netty-resolver from 4.1.96.Final to 4.1.108.Final
io.netty.netty-transport from 4.1.96.Final to 4.1.108.Final
io.netty.netty-transport-classes-epoll from 4.1.96.Final to 4.1.108.Final
io.netty.netty-transport-classes-kqueue from 4.1.96.Final to 4.1.108.Final
io.netty.netty-transport-native-epoll from 4.1.96.Final-linux-x86_64 to 4.1.108.Final-linux-x86_64
io.netty.netty-transport-native-kqueue from 4.1.96.Final-osx-x86_64 to 4.1.108.Final-osx-x86_64
io.netty.netty-transport-native-unix-common from 4.1.96.Final to 4.1.108.Final
org.apache.ivy.ivy from 2.5.1 to 2.5.2
org.apache.zookeeper.zookeeper from 3.6.3 to 3.9.2
org.apache.zookeeper.zookeeper-jute from 3.6.3 to 3.9.2
org.rocksdb.rocksdbjni from 8.11.4 to 9.2.1
org.scalactic.scalactic_2.12 from 3.2.15 to 3.2.16
org.scalatest.scalatest-compatible from 3.2.15 to 3.2.16
org.scalatest.scalatest-core_2.12 from 3.2.15 to 3.2.16
org.scalatest.scalatest-diagrams_2.12 from 3.2.15 to 3.2.16
org.scalatest.scalatest-featurespec_2.12 from 3.2.15 to 3.2.16
org.scalatest.scalatest-flatspec_2.12 from 3.2.15 to 3.2.16
org.scalatest.scalatest-freespec_2.12 from 3.2.15 to 3.2.16
org.scalatest.scalatest-funspec_2.12 from 3.2.15 to 3.2.16
org.scalatest.scalatest-funsuite_2.12 from 3.2.15 to 3.2.16
org.scalatest.scalatest-matchers-core_2.12 from 3.2.15 to 3.2.16
org.scalatest.scalatest-mustmatchers_2.12 from 3.2.15 to 3.2.16
org.scalatest.scalatest-propspec_2.12 from 3.2.15 to 3.2.16
org.scalatest.scalatest-refspec_2.12 from 3.2.15 to 3.2.16
org.scalatest.scalatest-shouldmatchers_2.12 from 3.2.15 to 3.2.16
org.scalatest.scalatest-wordspec_2.12 from 3.2.15 to 3.2.16
org.scalatest.scalatest_2.12 from 3.2.15 to 3.2.16
Apache Spark
Databricks Runtime 16.0 includes Apache Spark 3.5.0. This release includes all Spark fixes and improvements included in Databricks Runtime 15.4 LTS, as well as the following additional bug fixes and improvements made to Spark:
[SPARK-49093] [DBRRM-1371] Revert “[SC-172958][SQL] GROUP BY with MapType nes…
[SPARK-49898] [DBRRM-1282][SC-178410] Fix documentation and default for event log task metrics accumulator logging flag from SPARK-42204
[SPARK-49743] [ES-1260022][BEHAVE-157][SC-177475][SQL] OptimizeCsvJsonExpr should not change schema fields when pruning GetArrayStructFields
[SPARK-49816] [SC-177896][SQL] Should only update out-going-ref-count for referenced outer CTE relation
[SPARK-48939] [SC-177022][SC-172766][AVRO] Support reading Avro with recursive schema reference
[SPARK-49688] [SC-177468][ES-1242349][CONNECT] Fix a data race between interrupt and execute plan
[SPARK-49771] [SC-177466][PYTHON] Improve Pandas Scalar Iter UDF error when output rows exceed input rows
[SPARK-48866] [SC-170772][SQL] Fix hints of valid charset in the error message of INVALID_PARAMETER_VALUE.CHARSET
[SPARK-48195] [FIXFORWARD][SC-177267][CORE] Save and reuse RDD/Broadcast created by SparkPlan
[SPARK-49585] [CONNECT] Replace executions map in SessionHolder with operationID set
[SPARK-49211] [SC-174257][SQL] V2 Catalog can also support built-in data sources
[SPARK-49684] Minimise the lifetime of the session restoration lock
[SPARK-48059] [SPARK-48145][SPARK-48134][SPARK-48182][SPARK-48209][SPARK-48291] Structured log framework on the java side
[SPARK-48857] [SC-170661][SQL] Restrict charsets in CSVOptions
[SPARK-49152] [SC-173690][SQL] V2SessionCatalog should use V2Command
[SPARK-42846] [SC-176588][SQL] Remove error condition LEGACYERROR_TEMP_2011
[SPARK-48195] [SC-177267][CORE] Save and reuse RDD/Broadcast created by SparkPlan
[SPARK-49630] [SC-177379][SS] Add flatten option to process collection types with state data source reader
[SPARK-49699] [SC-177154][SS] Disable PruneFilters for streaming workloads
[SPARK-48781] [SC-175282][SQL] Add Catalog APIs for loading stored procedures
[SPARK-49667] [SC-177068][SQL] Disallow CS_AI collators with expressions that use StringSearch
[SPARK-49737] [SC-177207][SQL] Disable bucketing on collated columns in complex types
[SPARK-48712] [SC-169794][SQL] Perf Improvement for encode with empty values or UTF-8 charset
[SPARK-49038] [SC-173933][SQL] SQLMetric should report the raw value in the accumulator update event
[SPARK-48541] [SC-169066][CORE] Add a new exit code for executors killed by TaskReaper
[SPARK-48774] [SC-170138][SQL] Use SparkSession in SQLImplicits
[SPARK-49719] [SC-177139][SQL] Make
UUIDandSHUFFLEaccept integerseed[SPARK-49713] [SC-177135][PYTHON][CONNECT] Make function
count_min_sketchaccept number arguments[SPARK-47601] [SC-162499][GRAPHX] Graphx: Migrate logs with variables to structured logging framework
[SPARK-49738] [SC-177219][SQL] Endswith bug fix
[SPARK-48623] [SC-170822][CORE] Structured logging migrations [Part 3]
[SPARK-49677] [SC-177148][SS] Ensure that changelog files are written on commit and forceSnapshot flag is also reset
[SPARK-49684] [SC-177040][CONNECT] Remove global locks from session and execution managers
[SPARK-48302] [SC-168814][PYTHON] Preserve nulls in map columns in PyArrow Tables
[SPARK-48601] [SC-169025][SQL] Give a more user friendly error message when setting a null value for JDBC Option
[SPARK-48635] [SC-169263][SQL] Assign classes to join type errors and as-of join error
[SPARK-49673] [SC-177032][CONNECT] Increase CONNECT_GRPC_ARROW_MAX_BATCH_SIZE to 0.7 * CONNECT_GRPC_MAX_MESSAGE_SIZE
[SPARK-49693] [SC-177071][PYTHON][CONNECT] Refine the string representation of
timedelta[SPARK-49687] [SC-176901][SQL] Delay sorting in
validateAndMaybeEvolveStateSchema[SPARK-49718] [SC-177112][PS] Switch
Scatterplot to sampled data[SPARK-48472] [SC-169044][SQL] Enable reflect expressions with collated strings
[SPARK-48484] [SC-167484][SQL] Fix: V2Write use the same TaskAttemptId for different task attempts
[SPARK-48341] [SC-166560][CONNECT] Allow plugins to use QueryTest in their tests
[SPARK-42252] [SC-168723][CORE] Add
spark.shuffle.localDisk.file.output.bufferand deprecatespark.shuffle.unsafe.file.output.buffer[SPARK-48314] [SC-166565][SS] Don’t double cache files for FileStreamSource using Trigger.AvailableNow
[SPARK-49567] [SC-176241][PYTHON] Use
classicinstead ofvanillafrom PySpark code base[SPARK-48374] [SC-167596][PYTHON] Support additional PyArrow Table column types
[SPARK-48300] [SC-166481][SQL] Codegen Support for
from_xml[SPARK-49412] [SC-177059][PS] Compute all box plot metrics in single job
[SPARK-49692] [SC-177031][PYTHON][CONNECT] Refine the string representation of literal date and datetime
[SPARK-49392] [ES-1130351][SC-176705][SQL] Catch errors when failing to write to external data source
[SPARK-48306] [SC-166241][SQL] Improve UDT in error message
[SPARK-44924] [SC-166379][SS] Add config for FileStreamSource cached files
[SPARK-48176] [SC-165644][SQL] Adjust name of FIELD_ALREADY_EXISTS error condition
[SPARK-49691] [SC-176988][PYTHON][CONNECT] Function
substringshould accept column names[SPARK-49502] [SC-176077][CORE] Avoid NPE in SparkEnv.get.shuffleManager.unregisterShuffle
[SPARK-49244] [SC-176703][SQL] Further exception improvements for parser/interpreter
[SPARK-48355] [SC-176684][SQL] Support for CASE statement
[SPARK-49355] [SC-175121][SQL]
levenshteinshould check whether thecollationvalues of all parameter types are the same[SPARK-49640] [SC-176953][PS] Apply reservoir sampling in
SampledPlotBase[SPARK-49678] [SC-176857][CORE] Support
spark.test.masterinSparkSubmitArguments[SPARK-49680] [SC-176856][PYTHON] Limit
Sphinxbuild parallelism to 4 by default[SPARK-49396] Revert “[SC-176030][SQL] Modify nullability check for CaseWhen expression”
[SPARK-48419] [SC-167443][SQL] Foldable propagation replace foldable column shoul…
[SPARK-49556] [SC-176757][SQL] Add SQL pipe syntax for the SELECT operator
[SPARK-49438] [SC-175237][SQL] Fix the pretty name of the
FromAvro&ToAvroexpression[SPARK-49659] [SC-1229924][SQL] Add a nice user-facing error for scalar subqueries inside VALUES clause
[SPARK-49646] [SC-176778][SQL] fix subquery decorrelation for union/set operations when parentOuterReferences has references not covered in collectedChildOuterReferences
[SPARK-49354] [SC-175034][SQL]
split_partshould check whether thecollationvalues of all parameter types are the same[SPARK-49478] [SC-175914][CONNECT] Handle null metrics in ConnectProgressExecutionListener
[SPARK-48358] [SC-176374][SQL] Support for REPEAT statement
[SPARK-49183] [SC-173680][SQL] V2SessionCatalog.createTable should respect PROP_IS_MANAGED_LOCATION
[SPARK-49611] [SC-176791][SQL] Introduce TVF
collations()& remove theSHOW COLLATIONScommand[SPARK-49261] [SC-176589][SQL] Don’t replace literals in aggregate expressions with group-by expressions
[SPARK-49099] [SC-173229][SQL] CatalogManager.setCurrentNamespace should respect custom session catalog
[SPARK-49594] [SC-176569][SS] Adding check on whether columnFamilies were added or removed to write StateSchemaV3 file
[SPARK-49578] [SC-176385][SQL] Remove the ANSI config suggestion in CAST_INVALID_INPUT and CAST_OVERFLOW
[SPARK-48882] [SC-174256][SS] Assign names to streaming output mode related error classes
[SPARK-49155] [SC-176506][SQL][SS] Use more appropriate parameter type to construct
GenericArrayData[SPARK-49519] [SC-176388][SQL] Merge options of table and relation when constructing FileScanBuilder
[SPARK-49591] [SC-176587][SQL] Add Logical Type column to variant readme
[SPARK-49596] [SC-176423][SQL] Improve performance of
FormatString[SPARK-49525] [SC-176044][SS][CONNECT] Minor log improvement to Server Side Streaming Query ListenerBus Listener
[SPARK-49583] [SC-176272][SQL] Define the error sub-condition
SECONDS_FRACTIONfor invalid seconds fraction pattern[SPARK-49536] [SC-176242] Handle error in python streaming data source record prefetching
[SPARK-49443] [SC-176273][SQL][PYTHON] Implement to_variant_object expression and make schema_of_variant expressions print OBJECT for for Variant Objects
[SPARK-49544] [SASP-3990][SC-176557][CONNECT] Replace coarse-locking in SparkConnectExecutionManager with ConcurrentMap
[SPARK-49548] [SASP-3990][SC-176556][CONNECT] Replace coarse-locking in SparkConnectSessionManager with ConcurrentMap
[SPARK-49551] [SC-176218][SS] Improve RocksDB log for replayChangelog
[SPARK-49595] [SC-176396][CONNECT][SQL] Fix
DataFrame.unpivot/meltin Spark Connect Scala Client[SPARK-49006] [SC-176162] Implement purging for OperatorStateMetadataV2 and StateSchemaV3 files
[SPARK-49600] [SC-176426][PYTHON] Remove
Python 3.6 and older-related logic fromtry_simplify_traceback[SPARK-49303] [SC-176013][SS] Implement TTL for ValueState in transformWithStateInPandas API
[SPARK-49191] [SC-176243][SS] Add support for reading transformWithState map state variables with state data source reader
[SPARK-49593] [SC-176371][SS] Throw RocksDB exception to the caller on DB close if an error is seen
[SPARK-49334] [SC-174803][SQL]
str_to_mapshould check whether thecollationvalues of all parameter types are the same[SPARK-42204] [SC-176126][CORE] Add option to disable redundant logging of TaskMetrics internal accumulators in event logs
[SPARK-49575] [SC-176256][SS] Add logging for lock release only if acquiredThreadInfo is not null
[SPARK-49539] [SC-176250][SS] Update internal col families start identifier to a different one
[SPARK-49205] [SC-173853][SQL] KeyGroupedPartitioning should inherit HashPartitioningLike
[SPARK-49396] [SC-176030][SQL] Modify nullability check for CaseWhen expression
[SPARK-49476] [SC-175700][SQL] Fix nullability of base64 function
[SPARK-47262] [SC-174829][SQL] Assign names to error conditions for parquet conversions
[SPARK-47247] [SC-158349][SQL] Use smaller target size when coalescing partitions with exploding joins
[SPARK-49501] [SC-176166][SQL] Fix double-escaping of table location
[SPARK-49083] [SC-173214][CONNECT] Allow from_xml and from_json to natively work with json schemas
[SPARK-49043] [SC-174673][SQL] Fix interpreted codepath group by on map containing collated strings
[SPARK-48986] [SC-172265][CONNECT][SQL] Add ColumnNode Intermediate Representation
[SPARK-49326] [SC-176016][SS] Classify Error class for Foreach sink user function error
[SPARK-48348] [SC-175950][SPARK-48376][SQL] Introduce
LEAVEandITERATEstatements[SPARK-49523] [SC-175949][CONNECT] Increase maximum wait time for connect server to come up for testing
[SPARK-49000] [BEHAVE-105][ES-1194747][SQL] Fix “select count(distinct 1) from t” where t is empty table by expanding RewriteDistinctAggregates - DBR version 16.x
[SPARK-49311] [SC-175038][SQL] Make it possible for large ‘interval second’ values to be cast to decimal
[SPARK-49200] [SC-173699][SQL] Fix null type non-codegen ordering exception
[SPARK-49467] [SC-176051][SS] Add support for state data source reader and list state
[SPARK-47307] [SC-170891][SQL] Add a config to optionally chunk base64 strings
[SPARK-49391] [SC-176032][PS] Box plot select outliers by distance from fences
[SPARK-49445] [SC-175845][UI] Support show tooltip in the progress bar of UI
[SPARK-49451] [SC-175702] Allow duplicate keys in parse_json.
[SPARK-49275] [SC-175701][SQL] Fix return type nullness of the xpath expression
[SPARK-49021] [SC-175578][SS] Add support for reading transformWithState value state variables with state data source reader
[SPARK-49474] [BEHAVE-143][SC-169253][SC-175933][SS] Classify Error class for FlatMapGroupsWithState user function error
[SPARK-49408] [SC-175932][SQL] Use IndexedSeq in ProjectingInternalRow
[SPARK-49509] [SC-175853][CORE] Use
Platform.allocateDirectBufferinstead ofByteBuffer.allocateDirect[SPARK-49382] [SC-175013][PS] Make frame box plot properly render the fliers/outliers
[SPARK-49002] [SC-172846][SQL] Consistently handle invalid locations in WAREHOUSE/SCHEMA/TABLE/PARTITION/DIRECTORY
[SPARK-49480] [SC-175699][CORE] Fix NullPointerException from
SparkThrowableHelper.isInternalError[SPARK-49477] [SC-175828][PYTHON] Improve pandas udf invalid return type error message
[SPARK-48693] [SC-169492][SQL] Simplify and unify toString of Invoke and StaticInvoke
[SPARK-49441] [SC-175716][ML]
StringIndexersort arrays in executors[SPARK-49347] [SC-175004][R] Deprecate SparkR
[SPARK-49357] [SC-175227][CONNECT][PYTHON] Vertically truncate deeply nested protobuf message
[SPARK-41982] [SC-120604][SQL] Partitions of type string should not be treated as numeric types
[SPARK-48776] [SC-170452][BEHAVE-72] Fix timestamp formatting for json, xml and csv
[SPARK-49223] [SC-174800][ML] Simplify the StringIndexer.countByValue with builtin functions
[SPARK-49016] Revert “[SC-174663][SQL] Restore the behavior thatqueries from raw CSV files are disallowed when only include corrupt record column and assign name to
_LEGACY_ERROR_TEMP_1285”[SPARK-49041] [SC-172392][PYTHON][CONNECT] Raise proper error for
dropDuplicateswhen wrongsubsetis given[SPARK-49050] [SC-175235] Enabling deleteIfExists operator in TWS with Virtual Column Families
[SPARK-49216] [SC-173919][CORE]Fix to not log message context with explicitly LogEntry constructed when Structured Logging conf is off
[SPARK-49252] [SC-175596][CORE] Make`TaskSetExcludeList` and
HeathTrackerindependent[SPARK-49352] [SC-174971][SQL] Avoid redundant array transform for identical expression
[SPARK-42307] [SC-173863][SQL] Assign name for error LEGACYERROR_TEMP_2232
[SPARK-49197] [SC-173732][CORE] Redact
Spark Commandoutput inlaunchermodule[SPARK-48913] [SC-173934][SQL] Implement IndentingXMLStreamWriter
[SPARK-49306] [SC-175363][PYTHON][SQL] Create SQL function aliases for ‘zeroifnull’ and ‘nullifzero’
[SPARK-48344] [SQL] SQL Scripting execution (including Spark Connect)
[SPARK-49402] [SC-175122][PYTHON] Fix Binder integration in PySpark documentation
[SPARK-49017] [SC-174664][SQL] Insert statement fails when multiple parameters are being used
[SPARK-49318] [SC-174733][SQL] Preempt low priority error on LCA until end of check analysis to improve error experience
[SPARK-49016] [SC-174663][SQL] Restore the behavior that queries from raw CSV files are disallowed when only include corrupt record column and assign name to
_LEGACY_ERROR_TEMP_1285[SPARK-49387] [SC-175124][PYTHON] Fix type hint for
accuracyinpercentile_approxandapprox_percentile[SPARK-49131] [SC-174666][SS] TransformWithState should properly set implicit grouping keys even with lazy iterators
[SPARK-49301] [SC-174795][SS] Chunk arrow data passed to Python worker
[SPARK-49039] [SC-174651][UI] Reset checkbox when executor metrics are loaded in the Stages tab
[SPARK-48428] [SC-169806][SQL]: Fix IllegalStateException in NestedColumnAliasing
[SPARK-49353] [SC-174830][SQL] Update docs related to
UTF-32encoding/decoding[SPARK-48613] [SC-170966][SQL] SPJ: Support auto-shuffle one side + less join keys than partition keys
[SPARK-47473] [SC-160450][BEHAVE-127][SQL] Fix correctness issue of converting postgres INFINITY timestamps
[SPARK-49142] [SC-173658][CONNECT][PYTHON] Follow up to revert proto to string performance cost
[SPARK-49300] [SC-175008][CORE] Fix Hadoop delegation token leak when tokenRenewalInterval is not set.
[SPARK-49367] [SC-175012][PS] Parallelize the KDE computation for multiple columns (plotly backend)
[SPARK-49365] [SC-175011][PS] Simplify the bucket aggregation in hist plot
[SPARK-49372] [SC-175003][SS] Ensure that latestSnapshot is set to none on close to avoid subsequent use
[SPARK-49341] [SC-174785] Remove
connector/dockerin favor ofApache Spark Operator[SPARK-49344] [SC-174894][PS] Support
json_normalizefor Pandas API on Spark[SPARK-49306] [SC-174794][SQL] Create new SQL functions ‘zeroifnull’ and ‘nullifzero’
[SPARK-48796] [SC-174668][SS] Load Column Family Id from RocksDBCheckpointMetadata for VCF when restarting
[SPARK-49342] [SC-174899][SQL] Make TO_AVRO SQL function ‘jsonFormatSchema’ argument optional
[SPARK-48628] [SC-174695][CORE] Add task peak on/off heap memory metrics
[SPARK-47407] [SC-159379][BEHAVE-126][SQL] Support java.sql.Types.NULL map to NullType
[SPARK-48628] [SC-173407][CORE] Add task peak on/off heap memory metrics
[SPARK-49166] [SC-173987][SQL] Support OFFSET in correlated subquery
[SPARK-49269] [SC-174676][SQL] Eagerly evaluate VALUES() list in AstBuilder
[SPARK-49281] [SC-174782][SQL] Optimze parquet binary getBytes with getBytesUnsafe to avoid copy cost
[SPARK-49113] [SC-174734] Dont assert on translation bugs - silently swallow the exception
[SPARK-49098] [SC-173253][SQL] Add write options for INSERT
[SPARK-48638] [SC-174694][FOLLOW][CONNECT] Fix documentation for ExecutionInfo
[SPARK-49250] [ES-1222826][SQL] Improve error message for nested UnresolvedWindowExpression in CheckAnalysis
[SPARK-48755] [SC-174258][SS][PYTHON] transformWithState pyspark base implementation and ValueState support
[SPARK-48966] [SC-174329][SQL] Improve error message with invalid unresolved column reference in UDTF call
[SPARK-46590] [SC-154115][SQL] Fix coalesce failed with unexpected partition indeces
[SPARK-49235] [SC-174159][SQL] Refactor ResolveInlineTables rule so it doesn’t traverse the whole tree
[SPARK-49060] [SC-173107][CONNECT] Clean up Mima rules for SQL-Connect binary compatibility checks
[SPARK-48762] [SC-172525][SQL] Introduce clusterBy DataFrameWriter API for Python
[SPARK-49207] [SC-173852][SQL] Fix one-to-many case mapping in SplitPart and StringSplitSQL
[SPARK-49204] [SC-173850][SQL] Fix surrogate pair handling in StringInstr and StringLocate
[SPARK-36680] [SC-170640][SQL] Supports Dynamic Table Options for Spark SQL
[SPARK-49204] [SC-173849][SQL] Fix surrogate pair handling in SubstringIndex
[SPARK-49204] [SC-173848][SQL] Fix surrogate pair handling in StringTrim
[SPARK-48967] [SC-173993]Fix forward SparkConfigOwnershipSuite test for OPTIMIZE_INSERT_INTO_VALUES_PARSER
[SPARK-49204] [SC-173851][SQL] Fix surrogate pair handling in StringReplace
[SPARK-48967] [SC-173993][SQL][16.x] Improve performance and memory footprint of “INSERT INTO … VALUES” Statements
[SPARK-49099] Revert “[SC-173229][SQL] CatalogManager.setCurrent…
[SPARK-48347] [SC-173812][SQL] Support for WHILE statement
[SPARK-49128] [SC-173344][CORE] Support custom History Server UI title
[SPARK-49146] [SC-173825][SS] Move assertion errors related to watermark missing in append mode streaming queries to error framework
[SPARK-45787] [SC-172197][SQL] Support Catalog.listColumns for clustering columns
[SPARK-49099] [SC-173229][SQL] CatalogManager.setCurrentNamespace should respect custom session catalog
[SPARK-49138] [SC-173483][SQL] Fix CollationTypeCasts of several expressions
[SPARK-49163] [SC-173666][SQL] Attempt to create table based on broken parquet partition data results should return user-facing error
[SPARK-49201] [SC-173793][PS][PYTHON][CONNECT] Reimplement
histplot with Spark SQL[SPARK-49188] [SC-173682][SQL] Internal error on concat_ws called on array of arrays of string
[SPARK-49137] [SC-173677][SQL] When the Boolean condition in the
if statementis invalid, an exception should be thrown[SPARK-49193] [SC-173672][SQL] Improve the performance of RowSetUtils.toColumnBasedSet
[SPARK-49078] [SC-173078][SQL] Support show columns syntax in v2 table
[SPARK-49141] [SC-173388][SQL] Mark variant as hive incompatible data type
[SPARK-49059] [Cherry-Pick][15.x][SC-172528][CONNECT] Move
SessionHolder.forTesting(...)to the test package[SPARK-49111] [SC-173661][SQL] Move withProjectAndFilter to the companion object of DataSourceV2Strategy
[SPARK-49185] [SC-173688][PS][PYTHON][CONNECT] Reimplement
kdeplot with Spark SQL[SPARK-49178] [SC-173673][SQL] Optimize performance of
Row#getSeqto match the performance when using Spark 3.5 with Scala 2.12[SPARK-49093] [SC-172958][SQL] GROUP BY with MapType nested inside complex type
[SPARK-49142] [SC-173469][CONNECT][PYTHON] Lower Spark Connect client log level to debug
[SPARK-48761] [SC-172048][SQL] Introduce clusterBy DataFrameWriter API for Scala
[SPARK-48346] [SC-173083][SQL] Support for IF ELSE statements in SQL scripts
[SPARK-48338] [SC-173112][SQL] Improve exceptions thrown from parser/interpreter
[SPARK-48658] [SC-169474][SQL] Encode/Decode functions report coding errors instead of mojibake for unmappable characters
[SPARK-49071] [SC-172954][SQL] Remove ArraySortLike trait
[SPARK-49107] Revert “Revert “[SC-173103][SQL]
ROUTINE_ALREADY_EXISTSsupports RoutineType””[SPARK-49070] [SC-172907][SS][SQL] TransformWithStateExec.initialState is rewritten incorrectly to produce invalid query plan
[SPARK-49114] [SC-173217] Sub-categorize cannot load state store errors
[SPARK-49107] Revert “[SC-173103][SQL]
ROUTINE_ALREADY_EXISTSsupports RoutineType”[SPARK-49048] [SC-173223][SS] Add support for reading relevant operator metadata at given batch id
[SPARK-49094] [SC-173049][SQL] Fix ignoreCorruptFiles non-functioning for hive orc impl with mergeSchema off
[SPARK-49108] [SC-173102][EXAMPLE] Add
submit_pi.shREST API example[SPARK-49107] [SC-173103][SQL]
ROUTINE_ALREADY_EXISTSsupports RoutineType[SPARK-48997] [SC-172484][SS] Implement individual unloads for maintenance thread pool thread failures
[SPARK-49063] [SC-173094][SQL] Fix Between with ScalarSubqueries
[SPARK-45891] [SC-172305][SQL][PYTHON][VARIANT] Add support for interval types in the Variant Spec
[SPARK-49074] [BEHAVE-110][SC-172815][SQL] Fix variant with
df.cache()[SPARK-49003] [SC-172613][SQL] Fix interpreted code path hashing to be collation aware
[SPARK-48740] [SC-172430][SQL] Catch missing window specification error early
[SPARK-48999] [SC-172245][SS] Divide PythonStreamingDataSourceSimpleSuite
[SPARK-49031] [SC-172602] Implement validation for the TransformWithStateExec operator using OperatorStateMetadataV2
[SPARK-49053] [SC-172494][PYTHON][ML] Make model save/load helper functions accept spark session
[SPARK-49032] [Backport][15.x][SS] Add schema path in metadata table entry, verify expected version and add operator metadata related test for operator metadata format v2
[SPARK-49034] [SC-172306][CORE] Support server-side
sparkPropertiesreplacement in REST Submission API[SPARK-48931] [SC-171895][SS] Reduce Cloud Store List API cost for state store maintenance task
[SPARK-48849] [SC-172068][SS]Create OperatorStateMetadataV2 for the TransformWithStateExec operator
[SPARK-49013] [SC-172322] Change key in collationsMap for Map and Array types in scala
[SPARK-48414] [SC-171884][PYTHON] Fix breaking change in python’s
fromJson[SPARK-48910] [SC-171001][SQL] Use HashSet/HashMap to avoid linear searches in PreprocessTableCreation
[SPARK-49007] [SC-172204][CORE] Improve
MasterPageto support custom title[SPARK-49009] [SC-172263][SQL][PYTHON] Make Column APIs and functions accept Enums
[SPARK-49033] [SC-172303][CORE] Support server-side
environmentVariablesreplacement in REST Submission API[SPARK-48363] [SC-166470][SQL] Cleanup some redundant codes in
from_xml[SPARK-46743] [SC-170867][SQL][BEHAVE-84] Count bug after ScalarSubqery is folded if it has an empty relation
[SPARK-49040] [SC-172351][SQL] Fix doc
sql-ref-syntax-aux-exec-imm.md[SPARK-48998] [SC-172212][ML] Meta algorithms save/load model with SparkSession
[SPARK-48959] [SC-171708][SQL] Make
NoSuchNamespaceExceptionextendNoSuchDatabaseExceptionto restore the exception handling[SPARK-48996] [SC-172130][SQL][PYTHON] Allow bare literals for and and or of Column
[SPARK-48990] [SC-171936] Follow up for #101759 - test fix
[SPARK-48338] [SC-171912][SQL] Check variable declarations
[SPARK-48990] [SC-171936][SQL] Unified variable related SQL syntax keywords
[SPARK-48988] [SC-171915][ML] Make
DefaultParamsReader/Writerhandle metadata with spark session[SPARK-48974] [SC-171978][SQL][SS][ML][MLLIB] Use
SparkSession.implicitsinstead ofSQLContext.implicits[SPARK-48760] [SC-170870][SQL] Fix CatalogV2Util.applyClusterByChanges
[SPARK-48928] [SC-171956] Log Warning for Calling .unpersist() on Locally Checkpointed RDDs
[SPARK-48760] [SC-170139][SQL] Introduce ALTER TABLE … CLUSTER BY SQL syntax to change clustering columns
[SPARK-48844] Revert “[SC-170669][SQL] USE INVALID_EMPTY_LOCATION instead of UNSUPPORTED_DATASOURCE_FOR_DIRECT_QUERY when path is empty”
[SPARK-48833] [SC-171914][SQL][VARIANT] Support variant in
InMemoryTableScan[SPARK-48975] [SC-171894][PROTOBUF] Remove unnecessary
ScalaReflectionLockdefinition fromprotobuf[SPARK-48970] [SC-171800][PYTHON][ML] Avoid using SparkSession.getActiveSession in spark ML reader/writer
[SPARK-48844] [SC-170669][SQL] USE INVALID_EMPTY_LOCATION instead of UNSUPPORTED_DATASOURCE_FOR_DIRECT_QUERY when path is empty
[SPARK-48714] [SC-170136] Fix failing df.mergeInto tests in PySpark and UC
[SPARK-48957] [SC-171797][SS] Return sub-classified error class on state store load for hdfs and rocksdb provider
[SPARK-48891] [Backport][15x][SC-171677][SS] Refactor StateSchemaCompatibilityChecker to unify all state schema formats
[SPARK-48972] [SC-171795][PYTHON] Unify the literal string handling in functions
[SPARK-48388] [SC-171337][SQL] Fix SET statement behavior for SQL Scripts
[SPARK-48743] [SC-170552][SQL][SS] MergingSessionIterator should better handle when getStruct returns null
[SPARK-48623] [15.x][SC-171322][CORE] Migrate FileAppender logs to structured logging
[SPARK-36680] [DBRRM-1123] Revert “[SC-170640][SQL] Supports Dynamic Table Options for Spark SQL”
[SPARK-48841] [SC-170868][BEHAVE-83][SQL] Include
collationNametosql()ofCollate[SPARK-48941] [SC-171703][PYTHON][ML] Replace RDD read / write API invocation with Dataframe read / write API
[SPARK-48938] [SC-171577][PYTHON] Improve error messages when registering Python UDTFs
[SPARK-48350] [SC-171040][SQL] Introduction of Custom Exceptions for Sql Scripting
[SPARK-48907] [SC-171158][SQL] Fix the value
explicitTypesinCOLLATION_MISMATCH.EXPLICIT[SPARK-48945] [SC-171658][PYTHON] Simplify regex functions with
lit[SPARK-48944] [SC-171576][CONNECT] Unify the JSON-format schema handling in Connect Server
[SPARK-48836] [SC-171569] Integrate SQL schema with state schema/metadata
[SPARK-48946] [SC-171504][SQL] NPE in redact method when session is null
[SPARK-48921] [SC-171412][SQL] ScalaUDF encoders in subquery should be resolved for MergeInto
[SPARK-45155] [SC-171048][CONNECT] Add API Docs for Spark Connect JVM/Scala Client
[SPARK-48900] [SC-171319] Add
reasonfield forcancelJobGroupandcancelJobsWithTag[SPARK-48865] [SC-171154][SQL] Add try_url_decode function
[SPARK-48851] [SC-170767][SQL] Change the value of
SCHEMA_NOT_FOUNDfromnamespacetocatalog.namespace[SPARK-48510] [SC-170893][2/2] Support UDAF
toColumnAPI in Spark Connect[SPARK-45190] [SC-171055][SPARK-48897][PYTHON][CONNECT] Make
from_xmlsupport StructType schema[SPARK-48930] [SC-171304][CORE] Redact
awsAccessKeyIdby includingaccesskeypattern[SPARK-48909] [SC-171080][ML][MLLIB] Uses SparkSession over SparkContext when writing metadata
[SPARK-48883] [SC-171133][ML][R] Replace RDD read / write API invocation with Dataframe read / write API
[SPARK-48924] [SC-171313][PS] Add a pandas-like
make_intervalhelper function[SPARK-48884] [SC-171051][PYTHON] Remove unused helper function
PythonSQLUtils.makeInterval[SPARK-48817] [SC-170636][SQL] Eagerly execute union multi commands together
[SPARK-48896] [SC-171079][ML][MLLIB] Avoid repartition when writing out the metadata
[SPARK-48892] [SC-171127][ML] Avoid per-row param read in
Tokenizer[SPARK-48927] [SC-171227][CORE] Show the number of cached RDDs in
StoragePage[SPARK-48886] [15.x][Backport][SC-171039][SS] Add version info to changelog v2 to allow for easier evolution
[SPARK-48903] [SC-171136][SS] Set the RocksDB last snapshot version correctly on remote load
[SPARK-48742] [SC-170538][SS] Virtual Column Family for RocksDB
[SPARK-48726] [15.x][SC-170753][SS] Create the StateSchemaV3 file format, and write this out for the TransformWithStateExec operator
[SPARK-48794] [SC-170882][CONNECT][15.x] df.mergeInto support for Spark Connect (Scala and Python)
[SPARK-48714] [SC-170136][PYTHON] Implement
DataFrame.mergeIntoin PySpark[SPARK-48772] [SC-170642][SS][SQL] State Data Source Change Feed Reader Mode
[SPARK-48666] [SC-170887][SQL] Do not push down filter if it contains PythonUDFs
[SPARK-48845] [SC-170889][SQL] GenericUDF catch exceptions from children
[SPARK-48880] [SC-170974][CORE] Avoid throw NullPointerException if driver plugin fails to initialize
[SPARK-48888] [Backport][15x][SC-170973][SS] Remove snapshot creation based on changelog ops size
[SPARK-48871] [SC-170876] Fix INVALID_NON_DETERMINISTIC_EXPRESSIONS validation in…
[SPARK-48883] [SC-170894][ML][R] Replace RDD read / write API invocation with Dataframe read / write API
[SPARK-36680] [SC-170640][SQL] Supports Dynamic Table Options for Spark SQL
[SPARK-48804] [SC-170558][SQL] Add classIsLoadable & OutputCommitter.isAssignableFrom check for output committer class configrations
[SPARK-46738] [SC-170791][PYTHON] Reenable a group of doctests
[SPARK-48858] [SC-170756][PYTHON] Remove deprecated
setDaemonmethod call ofThreadinlog_communication.py[SPARK-48639] [SC-169801][CONNECT][PYTHON] Add Origin to RelationCommon
[SPARK-48863] [SC-170770][ES-1133940][SQL] Fix ClassCastException when parsing JSON with “spark.sql.json.enablePartialResults” enabled
[SPARK-48343] [SC-170450][SQL] Introduction of SQL Scripting interpreter
[SPARK-48529] [SC-170755][SQL] Introduction of Labels in SQL Scripting
[SPARK-45292] Revert “[SC-151609][SQL][HIVE] Remove Guava from shared classes from IsolatedClientLoader”
[SPARK-48037] [SC-165330][CORE][3.5] Fix SortShuffleWriter lacks shuffle write related metrics resulting in potentially inaccurate data
[SPARK-48720] [SC-170551][SQL] Align the command
ALTER TABLE ... UNSET TBLPROPERTIES ...in v1 and v2[SPARK-48485] [SC-167825][CONNECT][SS] Support interruptTag and interruptAll in streaming queries
[SPARK-45292] [SC-151609][SQL][HIVE] Remove Guava from shared classes from IsolatedClientLoader
[SPARK-48668] [SC-169815][SQL] Support ALTER NAMESPACE … UNSET PROPERTIES in v2
[SPARK-47914] [SC-165313][SQL] Do not display the splits parameter in Range
[SPARK-48807] [SC-170643][SQL] Binary Support for CSV datasource
[SPARK-48220] [SC-167592][PYTHON][15.X] Allow passing PyArrow Table to createDataFrame()
[SPARK-48545] [SC-169543][SQL] Create to_avro and from_avro SQL functions to match DataFrame equivalents
[SPARK-47577] [SC-168875][SPARK-47579] Correct misleading usage of log key TASK_ID
Databricks ODBC/JDBC driver support
Databricks supports ODBC/JDBC drivers released in the past 2 years. Please download the recently released drivers and upgrade (download ODBC, download JDBC).
System environment
Operating System: Ubuntu 24.04.1 LTS
Java: Zulu17.50+19-CA
Scala: 2.12.18
Python: 3.12.3
R: 4.4.0
Delta Lake: 3.2.1
Installed Python libraries
Library |
Version |
Library |
Version |
Library |
Version |
|---|---|---|---|---|---|
annotated-types |
0.7.0 |
asttokens |
2.0.5 |
astunparse |
1.6.3 |
autocommand |
2.2.2 |
azure-core |
1.31.0 |
azure-storage-blob |
12.23.0 |
azure-storage-file-datalake |
12.17.0 |
backports.tarfile |
1.2.0 |
black |
24.4.2 |
blinker |
1.7.0 |
boto3 |
1.34.69 |
botocore |
1.34.69 |
cachetools |
5.3.3 |
certifi |
2024.6.2 |
cffi |
1.16.0 |
chardet |
4.0.0 |
charset-normalizer |
2.0.4 |
click |
8.1.7 |
cloudpickle |
2.2.1 |
comm |
0.2.1 |
contourpy |
1.2.0 |
cryptography |
42.0.5 |
cycler |
0.11.0 |
Cython |
3.0.11 |
databricks-sdk |
0.30.0 |
dbus-python |
1.3.2 |
debugpy |
1.6.7 |
decorator |
5.1.1 |
Deprecated |
1.2.14 |
distlib |
0.3.8 |
docstring-to-markdown |
0.11 |
entrypoints |
0.4 |
executing |
0.8.3 |
facets-overview |
1.1.1 |
filelock |
3.15.4 |
fonttools |
4.51.0 |
gitdb |
4.0.11 |
GitPython |
3.1.37 |
google-api-core |
2.20.0 |
google-auth |
2.35.0 |
google-cloud-core |
2.4.1 |
google-cloud-storage |
2.18.2 |
google-crc32c |
1.6.0 |
google-resumable-media |
2.7.2 |
googleapis-common-protos |
1.65.0 |
grpcio |
1.60.0 |
grpcio-status |
1.60.0 |
httplib2 |
0.20.4 |
idna |
3.7 |
importlib-metadata |
6.0.0 |
importlib_resources |
6.4.0 |
inflect |
7.3.1 |
ipyflow-core |
0.0.198 |
ipykernel |
6.28.0 |
ipython |
8.25.0 |
ipython-genutils |
0.2.0 |
ipywidgets |
7.7.2 |
isodate |
0.6.1 |
jaraco.context |
5.3.0 |
jaraco.functools |
4.0.1 |
jaraco.text |
3.12.1 |
jedi |
0.19.1 |
jmespath |
1.0.1 |
joblib |
1.4.2 |
jupyter_client |
8.6.0 |
jupyter_core |
5.7.2 |
kiwisolver |
1.4.4 |
launchpadlib |
1.11.0 |
lazr.restfulclient |
0.14.6 |
lazr.uri |
1.0.6 |
matplotlib |
3.8.4 |
matplotlib-inline |
0.1.6 |
mccabe |
0.7.0 |
mlflow-skinny |
2.15.1 |
more-itertools |
10.3.0 |
mypy |
1.10.0 |
mypy-extensions |
1.0.0 |
nest-asyncio |
1.6.0 |
nodeenv |
1.9.1 |
numpy |
1.26.4 |
oauthlib |
3.2.2 |
opentelemetry-api |
1.27.0 |
opentelemetry-sdk |
1.27.0 |
opentelemetry-semantic-conventions |
0.48b0 |
packaging |
24.1 |
pandas |
1.5.3 |
parso |
0.8.3 |
pathspec |
0.10.3 |
patsy |
0.5.6 |
pexpect |
4.8.0 |
pillow |
10.3.0 |
pip |
24.2 |
platformdirs |
3.10.0 |
plotly |
5.22.0 |
pluggy |
1.0.0 |
prompt-toolkit |
3.0.43 |
proto-plus |
1.24.0 |
protobuf |
4.24.1 |
psutil |
5.9.0 |
psycopg2 |
2.9.3 |
ptyprocess |
0.7.0 |
pure-eval |
0.2.2 |
pyarrow |
15.0.2 |
pyasn1 |
0.4.8 |
pyasn1-modules |
0.2.8 |
pyccolo |
0.0.52 |
pycparser |
2.21 |
pydantic |
2.8.2 |
pydantic_core |
2.20.1 |
pyflakes |
3.2.0 |
Pygments |
2.15.1 |
PyGObject |
3.48.2 |
PyJWT |
2.7.0 |
pyodbc |
5.0.1 |
pyparsing |
3.0.9 |
pyright |
1.1.294 |
python-dateutil |
2.9.0.post0 |
python-lsp-jsonrpc |
1.1.2 |
python-lsp-server |
1.10.0 |
pytoolconfig |
1.2.6 |
pytz |
2024.1 |
PyYAML |
6.0.1 |
pyzmq |
25.1.2 |
requests |
2.32.2 |
rope |
1.12.0 |
rsa |
4.9 |
s3transfer |
0.10.2 |
scikit-learn |
1.4.2 |
scipy |
1.13.1 |
seaborn |
0.13.2 |
setuptools |
74.0.0 |
six |
1.16.0 |
smmap |
5.0.0 |
sqlparse |
0.5.1 |
ssh-import-id |
5.11 |
stack-data |
0.2.0 |
statsmodels |
0.14.2 |
tenacity |
8.2.2 |
threadpoolctl |
2.2.0 |
tokenize-rt |
4.2.1 |
tomli |
2.0.1 |
tornado |
6.4.1 |
traitlets |
5.14.3 |
typeguard |
4.3.0 |
types-protobuf |
3.20.3 |
types-psutil |
5.9.0 |
types-pytz |
2023.3.1.1 |
types-PyYAML |
6.0.0 |
types-requests |
2.31.0.0 |
types-setuptools |
68.0.0.0 |
types-six |
1.16.0 |
types-urllib3 |
1.26.25.14 |
typing_extensions |
4.11.0 |
ujson |
5.10.0 |
unattended-upgrades |
0.1 |
urllib3 |
1.26.16 |
virtualenv |
20.26.2 |
wadllib |
1.3.6 |
wcwidth |
0.2.5 |
whatthepatch |
1.0.2 |
wheel |
0.43.0 |
wrapt |
1.14.1 |
yapf |
0.33.0 |
zipp |
3.17.0 |
Installed R libraries
R libraries are installed from the Posit Package Manager CRAN snapshot.
Library |
Version |
Library |
Version |
Library |
Version |
|---|---|---|---|---|---|
arrow |
16.1.0 |
askpass |
1.2.0 |
assertthat |
0.2.1 |
backports |
1.5.0 |
base |
4.4.0 |
base64enc |
0.1-3 |
bigD |
0.2.0 |
bit |
4.0.5 |
bit64 |
4.0.5 |
bitops |
1.0-8 |
blob |
1.2.4 |
boot |
1.3-30 |
brew |
1.0-10 |
brio |
1.1.5 |
broom |
1.0.6 |
bslib |
0.8.0 |
cachem |
1.1.0 |
callr |
3.7.6 |
caret |
6.0-94 |
cellranger |
1.1.0 |
chron |
2.3-61 |
class |
7.3-22 |
cli |
3.6.3 |
clipr |
0.8.0 |
clock |
0.7.1 |
cluster |
2.1.6 |
codetools |
0.2-20 |
colorspace |
2.1-1 |
commonmark |
1.9.1 |
compiler |
4.4.0 |
config |
0.3.2 |
conflicted |
1.2.0 |
cpp11 |
0.4.7 |
crayon |
1.5.3 |
credentials |
2.0.1 |
curl |
5.2.1 |
data.table |
1.15.4 |
datasets |
4.4.0 |
DBI |
1.2.3 |
dbplyr |
2.5.0 |
desc |
1.4.3 |
devtools |
2.4.5 |
diagram |
1.6.5 |
diffobj |
0.3.5 |
digest |
0.6.36 |
downlit |
0.4.4 |
dplyr |
1.1.4 |
dtplyr |
1.3.1 |
e1071 |
1.7-14 |
ellipsis |
0.3.2 |
evaluate |
0.24.0 |
fansi |
1.0.6 |
farver |
2.1.2 |
fastmap |
1.2.0 |
fontawesome |
0.5.2 |
forcats |
1.0.0 |
foreach |
1.5.2 |
foreign |
0.8-86 |
forge |
0.2.0 |
fs |
1.6.4 |
future |
1.34.0 |
future.apply |
1.11.2 |
gargle |
1.5.2 |
generics |
0.1.3 |
gert |
2.1.0 |
ggplot2 |
3.5.1 |
gh |
1.4.1 |
git2r |
0.33.0 |
gitcreds |
0.1.2 |
glmnet |
4.1-8 |
globals |
0.16.3 |
glue |
1.7.0 |
googledrive |
2.1.1 |
googlesheets4 |
1.1.1 |
gower |
1.0.1 |
graphics |
4.4.0 |
grDevices |
4.4.0 |
grid |
4.4.0 |
gridExtra |
2.3 |
gsubfn |
0.7 |
gt |
0.11.0 |
gtable |
0.3.5 |
hardhat |
1.4.0 |
haven |
2.5.4 |
highr |
0.11 |
hms |
1.1.3 |
htmltools |
0.5.8.1 |
htmlwidgets |
1.6.4 |
httpuv |
1.6.15 |
httr |
1.4.7 |
httr2 |
1.0.2 |
ids |
1.0.1 |
ini |
0.3.1 |
ipred |
0.9-15 |
isoband |
0.2.7 |
iterators |
1.0.14 |
jquerylib |
0.1.4 |
jsonlite |
1.8.8 |
juicyjuice |
0.1.0 |
KernSmooth |
2.23-22 |
knitr |
1.48 |
labeling |
0.4.3 |
later |
1.3.2 |
lattice |
0.22-5 |
lava |
1.8.0 |
lifecycle |
1.0.4 |
listenv |
0.9.1 |
lubridate |
1.9.3 |
magrittr |
2.0.3 |
markdown |
1.13 |
MASS |
7.3-60.0.1 |
Matrix |
1.6-5 |
memoise |
2.0.1 |
methods |
4.4.0 |
mgcv |
1.9-1 |
mime |
0.12 |
miniUI |
0.1.1.1 |
mlflow |
2.14.1 |
ModelMetrics |
1.2.2.2 |
modelr |
0.1.11 |
munsell |
0.5.1 |
nlme |
3.1-165 |
nnet |
7.3-19 |
numDeriv |
2016.8-1.1 |
openssl |
2.2.0 |
parallel |
4.4.0 |
parallelly |
1.38.0 |
pillar |
1.9.0 |
pkgbuild |
1.4.4 |
pkgconfig |
2.0.3 |
pkgdown |
2.1.0 |
pkgload |
1.4.0 |
plogr |
0.2.0 |
plyr |
1.8.9 |
praise |
1.0.0 |
prettyunits |
1.2.0 |
pROC |
1.18.5 |
processx |
3.8.4 |
prodlim |
2024.06.25 |
profvis |
0.3.8 |
progress |
1.2.3 |
progressr |
0.14.0 |
promises |
1.3.0 |
proto |
1.0.0 |
proxy |
0.4-27 |
ps |
1.7.7 |
purrr |
1.0.2 |
R6 |
2.5.1 |
ragg |
1.3.2 |
randomForest |
4.7-1.1 |
rappdirs |
0.3.3 |
rcmdcheck |
1.4.0 |
RColorBrewer |
1.1-3 |
Rcpp |
1.0.13 |
RcppEigen |
0.3.4.0.0 |
reactable |
0.4.4 |
reactR |
0.6.0 |
readr |
2.1.5 |
readxl |
1.4.3 |
recipes |
1.1.0 |
rematch |
2.0.0 |
rematch2 |
2.1.2 |
remotes |
2.5.0 |
reprex |
2.1.1 |
reshape2 |
1.4.4 |
rlang |
1.1.4 |
rmarkdown |
2.27 |
RODBC |
1.3-23 |
roxygen2 |
7.3.2 |
rpart |
4.1.23 |
rprojroot |
2.0.4 |
Rserve |
1.8-13 |
RSQLite |
2.3.7 |
rstudioapi |
0.16.0 |
rversions |
2.1.2 |
rvest |
1.0.4 |
sass |
0.4.9 |
scales |
1.3.0 |
selectr |
0.4-2 |
sessioninfo |
1.2.2 |
shape |
1.4.6.1 |
shiny |
1.9.1 |
sourcetools |
0.1.7-1 |
sparklyr |
1.8.6 |
spatial |
7.3-17 |
splines |
4.4.0 |
sqldf |
0.4-11 |
SQUAREM |
2021.1 |
stats |
4.4.0 |
stats4 |
4.4.0 |
stringi |
1.8.4 |
stringr |
1.5.1 |
survival |
3.6-4 |
swagger |
5.17.14.1 |
sys |
3.4.2 |
systemfonts |
1.1.0 |
tcltk |
4.4.0 |
testthat |
3.2.1.1 |
textshaping |
0.4.0 |
tibble |
3.2.1 |
tidyr |
1.3.1 |
tidyselect |
1.2.1 |
tidyverse |
2.0.0 |
timechange |
0.3.0 |
timeDate |
4032.109 |
tinytex |
0.52 |
tools |
4.4.0 |
tzdb |
0.4.0 |
urlchecker |
1.0.1 |
usethis |
3.0.0 |
utf8 |
1.2.4 |
utils |
4.4.0 |
uuid |
1.2-1 |
V8 |
4.4.2 |
vctrs |
0.6.5 |
viridisLite |
0.4.2 |
vroom |
1.6.5 |
waldo |
0.5.2 |
whisker |
0.4.1 |
withr |
3.0.1 |
xfun |
0.46 |
xml2 |
1.3.6 |
xopen |
1.0.1 |
xtable |
1.8-4 |
yaml |
2.3.10 |
zeallot |
0.1.0 |
zip |
2.3.1 |
Installed Java and Scala libraries (Scala 2.12 cluster version)
Group ID |
Artifact ID |
Version |
|---|---|---|
antlr |
antlr |
2.7.7 |
com.amazonaws |
amazon-kinesis-client |
1.12.0 |
com.amazonaws |
aws-java-sdk-autoscaling |
1.12.638 |
com.amazonaws |
aws-java-sdk-cloudformation |
1.12.638 |
com.amazonaws |
aws-java-sdk-cloudfront |
1.12.638 |
com.amazonaws |
aws-java-sdk-cloudhsm |
1.12.638 |
com.amazonaws |
aws-java-sdk-cloudsearch |
1.12.638 |
com.amazonaws |
aws-java-sdk-cloudtrail |
1.12.638 |
com.amazonaws |
aws-java-sdk-cloudwatch |
1.12.638 |
com.amazonaws |
aws-java-sdk-cloudwatchmetrics |
1.12.638 |
com.amazonaws |
aws-java-sdk-codedeploy |
1.12.638 |
com.amazonaws |
aws-java-sdk-cognitoidentity |
1.12.638 |
com.amazonaws |
aws-java-sdk-cognitosync |
1.12.638 |
com.amazonaws |
aws-java-sdk-config |
1.12.638 |
com.amazonaws |
aws-java-sdk-core |
1.12.638 |
com.amazonaws |
aws-java-sdk-datapipeline |
1.12.638 |
com.amazonaws |
aws-java-sdk-directconnect |
1.12.638 |
com.amazonaws |
aws-java-sdk-directory |
1.12.638 |
com.amazonaws |
aws-java-sdk-dynamodb |
1.12.638 |
com.amazonaws |
aws-java-sdk-ec2 |
1.12.638 |
com.amazonaws |
aws-java-sdk-ecs |
1.12.638 |
com.amazonaws |
aws-java-sdk-efs |
1.12.638 |
com.amazonaws |
aws-java-sdk-elasticache |
1.12.638 |
com.amazonaws |
aws-java-sdk-elasticbeanstalk |
1.12.638 |
com.amazonaws |
aws-java-sdk-elasticloadbalancing |
1.12.638 |
com.amazonaws |
aws-java-sdk-elastictranscoder |
1.12.638 |
com.amazonaws |
aws-java-sdk-emr |
1.12.638 |
com.amazonaws |
aws-java-sdk-glacier |
1.12.638 |
com.amazonaws |
aws-java-sdk-glue |
1.12.638 |
com.amazonaws |
aws-java-sdk-iam |
1.12.638 |
com.amazonaws |
aws-java-sdk-importexport |
1.12.638 |
com.amazonaws |
aws-java-sdk-kinesis |
1.12.638 |
com.amazonaws |
aws-java-sdk-kms |
1.12.638 |
com.amazonaws |
aws-java-sdk-lambda |
1.12.638 |
com.amazonaws |
aws-java-sdk-logs |
1.12.638 |
com.amazonaws |
aws-java-sdk-machinelearning |
1.12.638 |
com.amazonaws |
aws-java-sdk-opsworks |
1.12.638 |
com.amazonaws |
aws-java-sdk-rds |
1.12.638 |
com.amazonaws |
aws-java-sdk-redshift |
1.12.638 |
com.amazonaws |
aws-java-sdk-route53 |
1.12.638 |
com.amazonaws |
aws-java-sdk-s3 |
1.12.638 |
com.amazonaws |
aws-java-sdk-ses |
1.12.638 |
com.amazonaws |
aws-java-sdk-simpledb |
1.12.638 |
com.amazonaws |
aws-java-sdk-simpleworkflow |
1.12.638 |
com.amazonaws |
aws-java-sdk-sns |
1.12.638 |
com.amazonaws |
aws-java-sdk-sqs |
1.12.638 |
com.amazonaws |
aws-java-sdk-ssm |
1.12.638 |
com.amazonaws |
aws-java-sdk-storagegateway |
1.12.638 |
com.amazonaws |
aws-java-sdk-sts |
1.12.638 |
com.amazonaws |
aws-java-sdk-support |
1.12.638 |
com.amazonaws |
aws-java-sdk-swf-libraries |
1.11.22 |
com.amazonaws |
aws-java-sdk-workspaces |
1.12.638 |
com.amazonaws |
jmespath-java |
1.12.638 |
com.clearspring.analytics |
stream |
2.9.6 |
com.databricks |
Rserve |
1.8-3 |
com.databricks |
databricks-sdk-java |
0.27.0 |
com.databricks |
jets3t |
0.7.1-0 |
com.databricks.scalapb |
scalapb-runtime_2.12 |
0.4.15-10 |
com.esotericsoftware |
kryo-shaded |
4.0.2 |
com.esotericsoftware |
minlog |
1.3.0 |
com.fasterxml |
classmate |
1.3.4 |
com.fasterxml.jackson.core |
jackson-annotations |
2.15.2 |
com.fasterxml.jackson.core |
jackson-core |
2.15.2 |
com.fasterxml.jackson.core |
jackson-databind |
2.15.2 |
com.fasterxml.jackson.dataformat |
jackson-dataformat-cbor |
2.15.2 |
com.fasterxml.jackson.dataformat |
jackson-dataformat-yaml |
2.15.2 |
com.fasterxml.jackson.datatype |
jackson-datatype-joda |
2.15.2 |
com.fasterxml.jackson.datatype |
jackson-datatype-jsr310 |
2.16.0 |
com.fasterxml.jackson.module |
jackson-module-paranamer |
2.15.2 |
com.fasterxml.jackson.module |
jackson-module-scala_2.12 |
2.15.2 |
com.github.ben-manes.caffeine |
caffeine |
2.9.3 |
com.github.fommil |
jniloader |
1.1 |
com.github.fommil.netlib |
native_ref-java |
1.1 |
com.github.fommil.netlib |
native_ref-java |
1.1-natives |
com.github.fommil.netlib |
native_system-java |
1.1 |
com.github.fommil.netlib |
native_system-java |
1.1-natives |
com.github.fommil.netlib |
netlib-native_ref-linux-x86_64 |
1.1-natives |
com.github.fommil.netlib |
netlib-native_system-linux-x86_64 |
1.1-natives |
com.github.luben |
zstd-jni |
1.5.5-4 |
com.github.wendykierp |
JTransforms |
3.1 |
com.google.code.findbugs |
jsr305 |
3.0.0 |
com.google.code.gson |
gson |
2.10.1 |
com.google.crypto.tink |
tink |
1.9.0 |
com.google.errorprone |
error_prone_annotations |
2.10.0 |
com.google.flatbuffers |
flatbuffers-java |
23.5.26 |
com.google.guava |
guava |
15.0 |
com.google.protobuf |
protobuf-java |
3.25.1 |
com.helger |
profiler |
1.1.1 |
com.ibm.icu |
icu4j |
75.1 |
com.jcraft |
jsch |
0.1.55 |
com.jolbox |
bonecp |
0.8.0.RELEASE |
com.lihaoyi |
sourcecode_2.12 |
0.1.9 |
com.microsoft.azure |
azure-data-lake-store-sdk |
2.3.9 |
com.microsoft.sqlserver |
mssql-jdbc |
11.2.2.jre8 |
com.ning |
compress-lzf |
1.1.2 |
com.sun.mail |
javax.mail |
1.5.2 |
com.sun.xml.bind |
jaxb-core |
2.2.11 |
com.sun.xml.bind |
jaxb-impl |
2.2.11 |
com.tdunning |
json |
1.8 |
com.thoughtworks.paranamer |
paranamer |
2.8 |
com.trueaccord.lenses |
lenses_2.12 |
0.4.12 |
com.twitter |
chill-java |
0.10.0 |
com.twitter |
chill_2.12 |
0.10.0 |
com.twitter |
util-app_2.12 |
7.1.0 |
com.twitter |
util-core_2.12 |
7.1.0 |
com.twitter |
util-function_2.12 |
7.1.0 |
com.twitter |
util-jvm_2.12 |
7.1.0 |
com.twitter |
util-lint_2.12 |
7.1.0 |
com.twitter |
util-registry_2.12 |
7.1.0 |
com.twitter |
util-stats_2.12 |
7.1.0 |
com.typesafe |
config |
1.4.3 |
com.typesafe.scala-logging |
scala-logging_2.12 |
3.7.2 |
com.uber |
h3 |
3.7.3 |
com.univocity |
univocity-parsers |
2.9.1 |
com.zaxxer |
HikariCP |
4.0.3 |
commons-cli |
commons-cli |
1.5.0 |
commons-codec |
commons-codec |
1.16.0 |
commons-collections |
commons-collections |
3.2.2 |
commons-dbcp |
commons-dbcp |
1.4 |
commons-fileupload |
commons-fileupload |
1.5 |
commons-httpclient |
commons-httpclient |
3.1 |
commons-io |
commons-io |
2.13.0 |
commons-lang |
commons-lang |
2.6 |
commons-logging |
commons-logging |
1.1.3 |
commons-pool |
commons-pool |
1.5.4 |
dev.ludovic.netlib |
arpack |
3.0.3 |
dev.ludovic.netlib |
blas |
3.0.3 |
dev.ludovic.netlib |
lapack |
3.0.3 |
info.ganglia.gmetric4j |
gmetric4j |
1.0.10 |
io.airlift |
aircompressor |
0.27 |
io.delta |
delta-sharing-client_2.12 |
1.2.0 |
io.dropwizard.metrics |
metrics-annotation |
4.2.19 |
io.dropwizard.metrics |
metrics-core |
4.2.19 |
io.dropwizard.metrics |
metrics-graphite |
4.2.19 |
io.dropwizard.metrics |
metrics-healthchecks |
4.2.19 |
io.dropwizard.metrics |
metrics-jetty9 |
4.2.19 |
io.dropwizard.metrics |
metrics-jmx |
4.2.19 |
io.dropwizard.metrics |
metrics-json |
4.2.19 |
io.dropwizard.metrics |
metrics-jvm |
4.2.19 |
io.dropwizard.metrics |
metrics-servlets |
4.2.19 |
io.netty |
netty-all |
4.1.108.Final |
io.netty |
netty-buffer |
4.1.108.Final |
io.netty |
netty-codec |
4.1.108.Final |
io.netty |
netty-codec-http |
4.1.108.Final |
io.netty |
netty-codec-http2 |
4.1.108.Final |
io.netty |
netty-codec-socks |
4.1.108.Final |
io.netty |
netty-common |
4.1.108.Final |
io.netty |
netty-handler |
4.1.108.Final |
io.netty |
netty-handler-proxy |
4.1.108.Final |
io.netty |
netty-resolver |
4.1.108.Final |
io.netty |
netty-tcnative-boringssl-static |
2.0.61.Final |
io.netty |
netty-tcnative-boringssl-static |
2.0.61.Final-linux-aarch_64 |
io.netty |
netty-tcnative-boringssl-static |
2.0.61.Final-linux-x86_64 |
io.netty |
netty-tcnative-boringssl-static |
2.0.61.Final-osx-aarch_64 |
io.netty |
netty-tcnative-boringssl-static |
2.0.61.Final-osx-x86_64 |
io.netty |
netty-tcnative-boringssl-static |
2.0.61.Final-windows-x86_64 |
io.netty |
netty-tcnative-classes |
2.0.61.Final |
io.netty |
netty-transport |
4.1.108.Final |
io.netty |
netty-transport-classes-epoll |
4.1.108.Final |
io.netty |
netty-transport-classes-kqueue |
4.1.108.Final |
io.netty |
netty-transport-native-epoll |
4.1.108.Final |
io.netty |
netty-transport-native-epoll |
4.1.108.Final-linux-aarch_64 |
io.netty |
netty-transport-native-epoll |
4.1.108.Final-linux-riscv64 |
io.netty |
netty-transport-native-epoll |
4.1.108.Final-linux-x86_64 |
io.netty |
netty-transport-native-kqueue |
4.1.108.Final-osx-aarch_64 |
io.netty |
netty-transport-native-kqueue |
4.1.108.Final-osx-x86_64 |
io.netty |
netty-transport-native-unix-common |
4.1.108.Final |
io.prometheus |
simpleclient |
0.7.0 |
io.prometheus |
simpleclient_common |
0.7.0 |
io.prometheus |
simpleclient_dropwizard |
0.7.0 |
io.prometheus |
simpleclient_pushgateway |
0.7.0 |
io.prometheus |
simpleclient_servlet |
0.7.0 |
io.prometheus.jmx |
collector |
0.12.0 |
jakarta.annotation |
jakarta.annotation-api |
1.3.5 |
jakarta.servlet |
jakarta.servlet-api |
4.0.3 |
jakarta.validation |
jakarta.validation-api |
2.0.2 |
jakarta.ws.rs |
jakarta.ws.rs-api |
2.1.6 |
javax.activation |
activation |
1.1.1 |
javax.el |
javax.el-api |
2.2.4 |
javax.jdo |
jdo-api |
3.0.1 |
javax.transaction |
jta |
1.1 |
javax.transaction |
transaction-api |
1.1 |
javax.xml.bind |
jaxb-api |
2.2.11 |
javolution |
javolution |
5.5.1 |
jline |
jline |
2.14.6 |
joda-time |
joda-time |
2.12.1 |
net.java.dev.jna |
jna |
5.8.0 |
net.razorvine |
pickle |
1.3 |
net.sf.jpam |
jpam |
1.1 |
net.sf.opencsv |
opencsv |
2.3 |
net.sf.supercsv |
super-csv |
2.2.0 |
net.snowflake |
snowflake-ingest-sdk |
0.9.6 |
net.sourceforge.f2j |
arpack_combined_all |
0.1 |
org.acplt.remotetea |
remotetea-oncrpc |
1.1.2 |
org.antlr |
ST4 |
4.0.4 |
org.antlr |
antlr-runtime |
3.5.2 |
org.antlr |
antlr4-runtime |
4.9.3 |
org.antlr |
stringtemplate |
3.2.1 |
org.apache.ant |
ant |
1.10.11 |
org.apache.ant |
ant-jsch |
1.10.11 |
org.apache.ant |
ant-launcher |
1.10.11 |
org.apache.arrow |
arrow-format |
15.0.0 |
org.apache.arrow |
arrow-memory-core |
15.0.0 |
org.apache.arrow |
arrow-memory-netty |
15.0.0 |
org.apache.arrow |
arrow-vector |
15.0.0 |
org.apache.avro |
avro |
1.11.3 |
org.apache.avro |
avro-ipc |
1.11.3 |
org.apache.avro |
avro-mapred |
1.11.3 |
org.apache.commons |
commons-collections4 |
4.4 |
org.apache.commons |
commons-compress |
1.23.0 |
org.apache.commons |
commons-crypto |
1.1.0 |
org.apache.commons |
commons-lang3 |
3.12.0 |
org.apache.commons |
commons-math3 |
3.6.1 |
org.apache.commons |
commons-text |
1.10.0 |
org.apache.curator |
curator-client |
2.13.0 |
org.apache.curator |
curator-framework |
2.13.0 |
org.apache.curator |
curator-recipes |
2.13.0 |
org.apache.datasketches |
datasketches-java |
3.1.0 |
org.apache.datasketches |
datasketches-memory |
2.0.0 |
org.apache.derby |
derby |
10.14.2.0 |
org.apache.hadoop |
hadoop-client-runtime |
3.3.6 |
org.apache.hive |
hive-beeline |
2.3.9 |
org.apache.hive |
hive-cli |
2.3.9 |
org.apache.hive |
hive-jdbc |
2.3.9 |
org.apache.hive |
hive-llap-client |
2.3.9 |
org.apache.hive |
hive-llap-common |
2.3.9 |
org.apache.hive |
hive-serde |
2.3.9 |
org.apache.hive |
hive-shims |
2.3.9 |
org.apache.hive |
hive-storage-api |
2.8.1 |
org.apache.hive.shims |
hive-shims-0.23 |
2.3.9 |
org.apache.hive.shims |
hive-shims-common |
2.3.9 |
org.apache.hive.shims |
hive-shims-scheduler |
2.3.9 |
org.apache.httpcomponents |
httpclient |
4.5.14 |
org.apache.httpcomponents |
httpcore |
4.4.16 |
org.apache.ivy |
ivy |
2.5.2 |
org.apache.logging.log4j |
log4j-1.2-api |
2.22.1 |
org.apache.logging.log4j |
log4j-api |
2.22.1 |
org.apache.logging.log4j |
log4j-core |
2.22.1 |
org.apache.logging.log4j |
log4j-layout-template-json |
2.22.1 |
org.apache.logging.log4j |
log4j-slf4j2-impl |
2.22.1 |
org.apache.orc |
orc-core |
1.9.2-shaded-protobuf |
org.apache.orc |
orc-mapreduce |
1.9.2-shaded-protobuf |
org.apache.orc |
orc-shims |
1.9.2 |
org.apache.thrift |
libfb303 |
0.9.3 |
org.apache.thrift |
libthrift |
0.12.0 |
org.apache.ws.xmlschema |
xmlschema-core |
2.3.0 |
org.apache.xbean |
xbean-asm9-shaded |
4.23 |
org.apache.yetus |
audience-annotations |
0.13.0 |
org.apache.zookeeper |
zookeeper |
3.9.2 |
org.apache.zookeeper |
zookeeper-jute |
3.9.2 |
org.checkerframework |
checker-qual |
3.31.0 |
org.codehaus.jackson |
jackson-core-asl |
1.9.13 |
org.codehaus.jackson |
jackson-mapper-asl |
1.9.13 |
org.codehaus.janino |
commons-compiler |
3.0.16 |
org.codehaus.janino |
janino |
3.0.16 |
org.datanucleus |
datanucleus-api-jdo |
4.2.4 |
org.datanucleus |
datanucleus-core |
4.1.17 |
org.datanucleus |
datanucleus-rdbms |
4.1.19 |
org.datanucleus |
javax.jdo |
3.2.0-m3 |
org.eclipse.collections |
eclipse-collections |
11.1.0 |
org.eclipse.collections |
eclipse-collections-api |
11.1.0 |
org.eclipse.jetty |
jetty-client |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-continuation |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-http |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-io |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-jndi |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-plus |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-proxy |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-security |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-server |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-servlet |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-servlets |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-util |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-util-ajax |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-webapp |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-xml |
9.4.52.v20230823 |
org.eclipse.jetty.websocket |
websocket-api |
9.4.52.v20230823 |
org.eclipse.jetty.websocket |
websocket-client |
9.4.52.v20230823 |
org.eclipse.jetty.websocket |
websocket-common |
9.4.52.v20230823 |
org.eclipse.jetty.websocket |
websocket-server |
9.4.52.v20230823 |
org.eclipse.jetty.websocket |
websocket-servlet |
9.4.52.v20230823 |
org.fusesource.leveldbjni |
leveldbjni-all |
1.8 |
org.glassfish.hk2 |
hk2-api |
2.6.1 |
org.glassfish.hk2 |
hk2-locator |
2.6.1 |
org.glassfish.hk2 |
hk2-utils |
2.6.1 |
org.glassfish.hk2 |
osgi-resource-locator |
1.0.3 |
org.glassfish.hk2.external |
aopalliance-repackaged |
2.6.1 |
org.glassfish.hk2.external |
jakarta.inject |
2.6.1 |
org.glassfish.jersey.containers |
jersey-container-servlet |
2.40 |
org.glassfish.jersey.containers |
jersey-container-servlet-core |
2.40 |
org.glassfish.jersey.core |
jersey-client |
2.40 |
org.glassfish.jersey.core |
jersey-common |
2.40 |
org.glassfish.jersey.core |
jersey-server |
2.40 |
org.glassfish.jersey.inject |
jersey-hk2 |
2.40 |
org.hibernate.validator |
hibernate-validator |
6.1.7.Final |
org.ini4j |
ini4j |
0.5.4 |
org.javassist |
javassist |
3.29.2-GA |
org.jboss.logging |
jboss-logging |
3.3.2.Final |
org.jdbi |
jdbi |
2.63.1 |
org.jetbrains |
annotations |
17.0.0 |
org.joda |
joda-convert |
1.7 |
org.jodd |
jodd-core |
3.5.2 |
org.json4s |
json4s-ast_2.12 |
3.7.0-M11 |
org.json4s |
json4s-core_2.12 |
3.7.0-M11 |
org.json4s |
json4s-jackson_2.12 |
3.7.0-M11 |
org.json4s |
json4s-scalap_2.12 |
3.7.0-M11 |
org.lz4 |
lz4-java |
1.8.0 |
org.mlflow |
mlflow-spark_2.12 |
2.9.1 |
org.objenesis |
objenesis |
2.5.1 |
org.postgresql |
postgresql |
42.6.1 |
org.roaringbitmap |
RoaringBitmap |
0.9.45-databricks |
org.roaringbitmap |
shims |
0.9.45-databricks |
org.rocksdb |
rocksdbjni |
9.2.1 |
org.rosuda.REngine |
REngine |
2.1.0 |
org.scala-lang |
scala-compiler_2.12 |
2.12.15 |
org.scala-lang |
scala-library_2.12 |
2.12.15 |
org.scala-lang |
scala-reflect_2.12 |
2.12.15 |
org.scala-lang.modules |
scala-collection-compat_2.12 |
2.11.0 |
org.scala-lang.modules |
scala-java8-compat_2.12 |
0.9.1 |
org.scala-lang.modules |
scala-parser-combinators_2.12 |
1.1.2 |
org.scala-lang.modules |
scala-xml_2.12 |
1.2.0 |
org.scala-sbt |
test-interface |
1.0 |
org.scalacheck |
scalacheck_2.12 |
1.14.2 |
org.scalactic |
scalactic_2.12 |
3.2.16 |
org.scalanlp |
breeze-macros_2.12 |
2.1.0 |
org.scalanlp |
breeze_2.12 |
2.1.0 |
org.scalatest |
scalatest-compatible |
3.2.16 |
org.scalatest |
scalatest-core_2.12 |
3.2.16 |
org.scalatest |
scalatest-diagrams_2.12 |
3.2.16 |
org.scalatest |
scalatest-featurespec_2.12 |
3.2.16 |
org.scalatest |
scalatest-flatspec_2.12 |
3.2.16 |
org.scalatest |
scalatest-freespec_2.12 |
3.2.16 |
org.scalatest |
scalatest-funspec_2.12 |
3.2.16 |
org.scalatest |
scalatest-funsuite_2.12 |
3.2.16 |
org.scalatest |
scalatest-matchers-core_2.12 |
3.2.16 |
org.scalatest |
scalatest-mustmatchers_2.12 |
3.2.16 |
org.scalatest |
scalatest-propspec_2.12 |
3.2.16 |
org.scalatest |
scalatest-refspec_2.12 |
3.2.16 |
org.scalatest |
scalatest-shouldmatchers_2.12 |
3.2.16 |
org.scalatest |
scalatest-wordspec_2.12 |
3.2.16 |
org.scalatest |
scalatest_2.12 |
3.2.16 |
org.slf4j |
jcl-over-slf4j |
2.0.7 |
org.slf4j |
jul-to-slf4j |
2.0.7 |
org.slf4j |
slf4j-api |
2.0.7 |
org.slf4j |
slf4j-simple |
1.7.25 |
org.threeten |
threeten-extra |
1.7.1 |
org.tukaani |
xz |
1.9 |
org.typelevel |
algebra_2.12 |
2.0.1 |
org.typelevel |
cats-kernel_2.12 |
2.1.1 |
org.typelevel |
spire-macros_2.12 |
0.17.0 |
org.typelevel |
spire-platform_2.12 |
0.17.0 |
org.typelevel |
spire-util_2.12 |
0.17.0 |
org.typelevel |
spire_2.12 |
0.17.0 |
org.wildfly.openssl |
wildfly-openssl |
1.1.3.Final |
org.xerial |
sqlite-jdbc |
3.42.0.0 |
org.xerial.snappy |
snappy-java |
1.1.10.3 |
org.yaml |
snakeyaml |
2.0 |
oro |
oro |
2.0.8 |
pl.edu.icm |
JLargeArrays |
1.5 |
software.amazon.cryptools |
AmazonCorrettoCryptoProvider |
1.6.2-linux-x86_64 |
stax |
stax-api |
1.0.1 |