Databricks Runtime 13.0
The following release notes provide information about Databricks Runtime 13.0, powered by Apache Spark 3.4.0.
Databricks released these images in April 2023.
Note
These release notes may include references to features that are not available on Google Cloud as of this release.
New features and improvements
Photon returns an error if a file is modified between query planning and execution
Use the Databricks connector to connect to another Databricks workspace
Additional Kafka options for Structured Streaming with shared cluster on Unity Catalog
New metadata column fields denoting file block start and length
Cluster-scoped Python libraries are installed using a non-root user
Ganglia replaced with cluster metrics
For Databricks Runtime versions 13 and above, Ganglia metrics are replaced with Databricks cluster metrics. For Databricks Runtime versions 12 and below, you can continue to use Ganglia metrics. If you have a workflow that depends on Ganglia metrics that cannot be satisfied with cluster metrics, contact your Databricks representative. See _.
Spark 3.4.0 released
Apache Spark 3.4.0 is now generally available. See Spark Release 3.4.0.
Photon returns an error if a file is modified between query planning and execution
Photon queries now return an error if a file is updated between query planning and execution. Before this change, Photon would read a file even if its modification time changes between query planning and execution, which could lead to unpredictable results.
New features for Predictive I/O
Photon now supports sliding frames, for both the ROWS and RANGE modes, using {ROWS | RANGE} BETWEEN offset_start { PRECEDING | FOLLOWING } AND offset_stop { PRECEDING | FOLLOWING }.
Extended support for Predictive I/O features
Photon extended support for CSV, AVRO, DataFrame caching data sources when your schema exclusively contains scalar columns. Support is also extended for nested types, such as maps, structs, and arrays.
Use the Databricks connector to connect to another Databricks workspace
The Databricks connector allows you to connect to compute resources configured in another workspace and return results to your current Databricks workspace. The Databricks JDBC driver is now included in Databricks Runtime. See Use the Databricks connector to connect to another Databricks workspace.
Skip Delta table modifications with Structured Streaming
You can now use skipChangeCommits to ignore data changing commits to Delta table for UPDATE, MERGE, and DELETE operations. This replaces ignoreChanges, which is now deprecated. See Ignore updates and deletes.
CREATE TABLE LIKE feature for Delta tables
`CREATE TABLE LIKE` is now supported when using Delta tables.
New metadata column fields denoting file block start and length
You can now get the following metadata information for input files with the _metadata column: file_block_start and file_block_length. For more information, see File metadata column.
New H3 geospatial functions
The new h3_coverash3 and h3_coverash3string expressions are available for geospatial processing when Photon is enabled. See H3 geospatial functions.
New SQL built-in functions
The following functions have been added:
array_insert(array, index, elem) Returns an expanded
arraywhereelemis inserted at theindexposition.luhn_check(numStr) Returns
trueifnumStrpasses the Luhn algorithm check.
%sql in Python notebooks
All non-empty %sql results in Python notebooks are now accessible using _sqldf or IPython’s output caching system. This simplifies usage of the DataFrame result in Python. This is a behavior change from earlier DBR versions, as previously, _sqldf was not always reassigned.
_sqldf is not reassigned on a %sql cell result if the cell was executed using parallel SQL cell execution.
%pip does not automatically restart Python
After running %pip commands inside Python notebooks that modify the virtualEnv, Databricks no longer automatically restarts the Python process. To restart the Python process, call dbutils.library.restartPython(). With this change, %pip commands on Databricks behave similarly to Jupyter Notebooks and are consistent with !pip and %sh pip commands. Databricks recommends %pip instead of !pip or %sh pip.
Cluster-scoped Python libraries are installed using a non-root user
Libraries are now installed using a non-root user who is responsible for library installations. If you want to configure pip installation behavior in init scripts, use the [global level file https://pip.pypa.io/en/stable/topics/configuration/#configuration-files).
Bug fixes
In
%shcommands, the following fixes were made:Output containing non UTF-8 characters is now supported.
Subprocesses are now interrupted when the command is cancelled.
Library upgrades
Upgraded Python libraries:
asttokensfrom 2.0.5 to 2.2.1blackfrom 22.3.0 to 22.6.0boto3from 1.21.32 to 1.24.28botocorefrom 1.24.32 to 1.27.28certififrom 2021.10.8 to 2022.9.14cffifrom 1.15.0 to 1.15.1cryptographyfrom 3.4.8 to 37.0.1Cythonfrom 0.29.28 to 0.29.32dbus-pythonfrom 1.2.16 to 1.2.18docstring-to-markdownfrom 0.11 to 0.12executingfrom 0.8.3 to 1.2.0facets-overviewfrom 1.0.0 to 1.0.2fastjsonschemafrom 2.16.2 to 2.16.3filelockfrom 3.9.0 to 3.10.7ipykernelfrom 6.15.3 to 6.17.1ipythonfrom 8.5.0 to 8.10.0joblibfrom 1.1.1 to 1.2.0jsonschemafrom 4.4.0 to 4.16.0jupyter-clientfrom 6.1.12 to 7.3.4kiwisolverfrom 1.3.2 to 1.4.2matplotlibfrom 3.5.1 to 3.5.2matplotlib-inlinefrom 0.1.2 to 0.1.6nbformatfrom 5.3.0 to 5.5.0notebookfrom 6.4.8 to 6.4.12pandasfrom 1.4.2 to 1.4.4Pillowfrom 9.0.1 to 9.2.0pipfrom 21.2.4 to 22.2.2platformdirsfrom 2.6.2 to 2.5.2plotlyfrom 5.6.0 to 5.9.0prometheus-clientfrom 0.13.1 to 0.14.1prompt-toolkitfrom 3.0.20 to 3.0.36psutilfrom 5.8.0 to 5.9.0pyflakesfrom 2.5.0 to 3.0.1PyGObjectfrom 3.36.0 to 3.42.1pyparsingfrom 3.0.4 to 3.0.9pyrightfrom 1.1.283 to 1.1.294python-lsp-serverfrom 1.6.0 to 1.7.1pytzfrom 2021.3 to 2022.1pyzmqfrom 22.3.0 to 23.2.0requestsfrom 2.27.1 to 2.28.1ropefrom 0.22.0 to 1.7.0s3transferfrom 0.5.0 to 0.6.0scikit-learnfrom 1.0.2 to 1.1.1scipyfrom 1.7.3 to 1.9.1setuptoolsfrom 61.2.0 to 63.4.1ssh-import-idfrom 5.10 to 5.11stack-datafrom 0.2.0 to 0.6.2testpathfrom 0.5.0 to 0.6.0tomlifrom 1.2.2 to 2.0.1ujsonfrom 5.1.0 to 5.4.0urllib3from 1.26.9 to 1.26.11virtualenvfrom 20.8.0 to 20.16.3whatthepatchfrom 1.0.4 to 1.0.2wheelfrom 0.37.0 to 0.37.1
Upgraded R libraries:
arrowfrom 10.0.0 to 10.0.1bitfrom 4.0.4 to 4.0.5broomfrom 1.0.1 to 1.0.3bslibfrom 0.4.1 to 0.4.2chronfrom 2.3-58 to 2.3-59clifrom 3.4.1 to 3.6.0colorspacefrom 2.0-3 to 2.1-0curlfrom 4.3.3 to 5.0.0data.tablefrom 1.14.4 to 1.14.6dbplyrfrom 2.2.1 to 2.3.0digestfrom 0.6.30 to 0.6.31dplyrfrom 1.0.10 to 1.1.0e1071from 1.7-12 to 1.7-13evaluatefrom 0.18 to 0.20fansifrom 1.0.3 to 1.0.4fontawesomefrom 0.4.0 to 0.5.0forcatsfrom 0.5.2 to 1.0.0fsfrom 1.5.2 to 1.6.1futurefrom 1.29.0 to 1.31.0garglefrom 1.2.1 to 1.3.0gertfrom 1.9.1 to 1.9.2glmnetfrom 4.1-4 to 4.1-6globalsfrom 0.16.1 to 0.16.2gowerfrom 1.0.0 to 1.0.1highrfrom 0.9 to 0.10htmltoolsfrom 0.5.3 to 0.5.4htmlwidgetsfrom 1.5.4 to 1.6.1httpuvfrom 1.6.6 to 1.6.8isobandfrom 0.2.6 to 0.2.7jsonlitefrom 1.8.3 to 1.8.4knitrfrom 1.40 to 1.42lavafrom 1.7.0 to 1.7.1listenvfrom 0.8.0 to 0.9.0lubridatefrom 1.9.0 to 1.9.1markdownfrom 1.3 to 1.5modelrfrom 0.1.9 to 0.1.10opensslfrom 2.0.4 to 2.0.5parallellyfrom 1.32.1 to 1.34.0pkgbuildfrom 1.3.1 to 1.4.0pkgdownfrom 2.0.6 to 2.0.7pkgloadfrom 1.3.1 to 1.3.2plyrfrom 1.8.7 to 1.8.8progressrfrom 0.11.0 to 0.13.0purrrfrom 0.3.5 to 1.0.1raggfrom 1.2.4 to 1.2.5Rcppfrom 1.0.9 to 1.0.10readxlfrom 1.4.1 to 1.4.2recipesfrom 1.0.3 to 1.0.4rmarkdownfrom 2.18 to 2.20RODBCfrom 1.3-19 to 1.3-20roxygen2from 7.2.1 to 7.2.3RSQLitefrom 2.2.18 to 2.2.20sassfrom 0.4.2 to 0.4.5shinyfrom 1.7.3 to 1.7.4sourcetoolsfrom 0.1.7 to 0.1.7-1sparklyrfrom 1.7.8 to 1.7.9SparkRfrom 3.3.2 to 3.4.0spatialfrom 7.3-11 to 7.3-15stringifrom 1.7.8 to 1.7.12stringrfrom 1.4.1 to 1.5.0survivalfrom 3.4-0 to 3.5-3testthatfrom 3.1.5 to 3.1.6tidyrfrom 1.2.1 to 1.3.0timechangefrom 0.1.1 to 0.2.0timeDatefrom 4021.106 to 4022.108tinytexfrom 0.42 to 0.44utf8from 1.2.2 to 1.2.3vctrsfrom 0.5.0 to 0.5.2vroomfrom 1.6.0 to 1.6.1whiskerfrom 0.4 to 0.4.1xfunfrom 0.34 to 0.37yamlfrom 2.3.6 to 2.3.7
Upgraded Java libraries:
com.amazonaws.aws-java-sdk-autoscaling from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-cloudformation from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-cloudfront from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-cloudhsm from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-cloudsearch from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-cloudtrail from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-cloudwatch from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-cloudwatchmetrics from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-codedeploy from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-cognitoidentity from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-cognitosync from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-config from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-core from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-datapipeline from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-directconnect from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-directory from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-dynamodb from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-ec2 from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-ecs from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-efs from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-elasticache from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-elasticbeanstalk from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-elasticloadbalancing from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-elastictranscoder from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-emr from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-glacier from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-glue from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-iam from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-importexport from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-kinesis from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-kms from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-lambda from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-logs from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-machinelearning from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-opsworks from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-rds from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-redshift from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-route53 from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-s3 from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-ses from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-simpledb from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-simpleworkflow from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-sns from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-sqs from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-ssm from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-storagegateway from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-sts from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-support from 1.12.189 to 1.12.390
com.amazonaws.aws-java-sdk-workspaces from 1.12.189 to 1.12.390
com.amazonaws.jmespath-java from 1.12.189 to 1.12.390
com.fasterxml.jackson.core.jackson-annotations from 2.13.4 to 2.14.2
com.fasterxml.jackson.core.jackson-core from 2.13.4 to 2.14.2
com.fasterxml.jackson.core.jackson-databind from 2.13.4.2 to 2.14.2
com.fasterxml.jackson.dataformat.jackson-dataformat-cbor from 2.13.4 to 2.14.2
com.fasterxml.jackson.datatype.jackson-datatype-joda from 2.13.4 to 2.14.2
com.fasterxml.jackson.module.jackson-module-paranamer from 2.13.4 to 2.14.2
com.fasterxml.jackson.module.jackson-module-scala_2.12 from 2.13.4 to 2.14.2
com.github.luben.zstd-jni from 1.5.2-1 to 1.5.2-5
com.google.code.gson.gson from 2.8.6 to 2.8.9
com.google.crypto.tink.tink from 1.6.1 to 1.7.0
com.h2database.h2 from 2.0.204 to 2.1.214
com.jcraft.jsch from 0.1.50 to 0.1.55
com.ning.compress-lzf from 1.1 to 1.1.2
commons-fileupload.commons-fileupload from 1.3.3 to 1.5
dev.ludovic.netlib.arpack from 2.2.1 to 3.0.3
dev.ludovic.netlib.blas from 2.2.1 to 3.0.3
dev.ludovic.netlib.lapack from 2.2.1 to 3.0.3
io.dropwizard.metrics.metrics-core from 4.1.1 to 4.2.10
io.dropwizard.metrics.metrics-graphite from 4.1.1 to 4.2.10
io.dropwizard.metrics.metrics-healthchecks from 4.1.1 to 4.2.10
io.dropwizard.metrics.metrics-jetty9 from 4.1.1 to 4.2.10
io.dropwizard.metrics.metrics-jmx from 4.1.1 to 4.2.10
io.dropwizard.metrics.metrics-json from 4.1.1 to 4.2.10
io.dropwizard.metrics.metrics-jvm from 4.1.1 to 4.2.10
io.dropwizard.metrics.metrics-servlets from 4.1.1 to 4.2.10
io.netty.netty-all from 4.1.74.Final to 4.1.87.Final
io.netty.netty-buffer from 4.1.74.Final to 4.1.87.Final
io.netty.netty-codec from 4.1.74.Final to 4.1.87.Final
io.netty.netty-common from 4.1.74.Final to 4.1.87.Final
io.netty.netty-handler from 4.1.74.Final to 4.1.87.Final
io.netty.netty-resolver from 4.1.74.Final to 4.1.87.Final
io.netty.netty-transport from 4.1.74.Final to 4.1.87.Final
io.netty.netty-transport-classes-epoll from 4.1.74.Final to 4.1.87.Final
io.netty.netty-transport-classes-kqueue from 4.1.74.Final to 4.1.87.Final
io.netty.netty-transport-native-unix-common from 4.1.74.Final to 4.1.87.Final
joda-time.joda-time from 2.10.13 to 2.12.1
net.razorvine.pickle from 1.2 to 1.3
org.antlr.antlr4-runtime from 4.8 to 4.9.3
org.apache.ant.ant from 1.9.2 to 1.9.16
org.apache.ant.ant-jsch from 1.9.2 to 1.9.16
org.apache.ant.ant-launcher from 1.9.2 to 1.9.16
org.apache.arrow.arrow-format from 7.0.0 to 11.0.0
org.apache.arrow.arrow-memory-core from 7.0.0 to 11.0.0
org.apache.arrow.arrow-memory-netty from 7.0.0 to 11.0.0
org.apache.arrow.arrow-vector from 7.0.0 to 11.0.0
org.apache.avro.avro from 1.11.0 to 1.11.1
org.apache.avro.avro-ipc from 1.11.0 to 1.11.1
org.apache.avro.avro-mapred from 1.11.0 to 1.11.1
org.apache.httpcomponents.httpclient from 4.5.13 to 4.5.14
org.apache.httpcomponents.httpcore from 4.4.14 to 4.4.16
org.apache.ivy.ivy from 2.5.0 to 2.5.1
org.apache.logging.log4j.log4j-1.2-api from 2.18.0 to 2.19.0
org.apache.logging.log4j.log4j-api from 2.18.0 to 2.19.0
org.apache.logging.log4j.log4j-core from 2.18.0 to 2.19.0
org.apache.orc.orc-core from 1.7.6 to 1.8.2-shaded-protobuf
org.apache.orc.orc-mapreduce from 1.7.6 to 1.8.2-shaded-protobuf
org.apache.orc.orc-shims from 1.7.6 to 1.8.2
org.apache.xbean.xbean-asm9-shaded from 4.20 to 4.22
org.apache.zookeeper.zookeeper from 3.6.2 to 3.6.3
org.apache.zookeeper.zookeeper-jute from 3.6.2 to 3.6.3
org.eclipse.jetty.jetty-client from 9.4.46.v20220331 to 9.4.50.v20221201
org.eclipse.jetty.jetty-continuation from 9.4.46.v20220331 to 9.4.50.v20221201
org.eclipse.jetty.jetty-http from 9.4.46.v20220331 to 9.4.50.v20221201
org.eclipse.jetty.jetty-io from 9.4.46.v20220331 to 9.4.50.v20221201
org.eclipse.jetty.jetty-jndi from 9.4.46.v20220331 to 9.4.50.v20221201
org.eclipse.jetty.jetty-plus from 9.4.46.v20220331 to 9.4.50.v20221201
org.eclipse.jetty.jetty-proxy from 9.4.46.v20220331 to 9.4.50.v20221201
org.eclipse.jetty.jetty-security from 9.4.46.v20220331 to 9.4.50.v20221201
org.eclipse.jetty.jetty-server from 9.4.46.v20220331 to 9.4.50.v20221201
org.eclipse.jetty.jetty-servlet from 9.4.46.v20220331 to 9.4.50.v20221201
org.eclipse.jetty.jetty-servlets from 9.4.46.v20220331 to 9.4.50.v20221201
org.eclipse.jetty.jetty-util from 9.4.46.v20220331 to 9.4.50.v20221201
org.eclipse.jetty.jetty-util-ajax from 9.4.46.v20220331 to 9.4.50.v20221201
org.eclipse.jetty.jetty-webapp from 9.4.46.v20220331 to 9.4.50.v20221201
org.eclipse.jetty.jetty-xml from 9.4.46.v20220331 to 9.4.50.v20221201
org.eclipse.jetty.websocket.websocket-api from 9.4.46.v20220331 to 9.4.50.v20221201
org.eclipse.jetty.websocket.websocket-client from 9.4.46.v20220331 to 9.4.50.v20221201
org.eclipse.jetty.websocket.websocket-common from 9.4.46.v20220331 to 9.4.50.v20221201
org.eclipse.jetty.websocket.websocket-server from 9.4.46.v20220331 to 9.4.50.v20221201
org.eclipse.jetty.websocket.websocket-servlet from 9.4.46.v20220331 to 9.4.50.v20221201
org.hibernate.validator.hibernate-validator from 6.1.0.Final to 6.1.7.Final
org.mlflow.mlflow-spark from 2.1.1 to 2.2.0
org.postgresql.postgresql from 42.3.3 to 42.3.8
org.roaringbitmap.RoaringBitmap from 0.9.25 to 0.9.39
org.roaringbitmap.shims from 0.9.25 to 0.9.39
org.rocksdb.rocksdbjni from 6.28.2 to 7.8.3
org.scala-lang.scala-compiler_2.12 from 2.12.14 to 2.12.15
org.scala-lang.scala-library_2.12 from 2.12.14 to 2.12.15
org.scala-lang.scala-reflect_2.12 from 2.12.14 to 2.12.15
org.scalanlp.breeze-macros_2.12 from 1.2 to 2.1.0
org.scalanlp.breeze_2.12 from 1.2 to 2.1.0
org.slf4j.jcl-over-slf4j from 1.7.36 to 2.0.6
org.slf4j.jul-to-slf4j from 1.7.36 to 2.0.6
org.slf4j.slf4j-api from 1.7.36 to 2.0.6
org.threeten.threeten-extra from 1.5.0 to 1.7.1
org.wildfly.openssl.wildfly-openssl from 1.0.7.Final to 1.1.3.Final
org.yaml.snakeyaml from 1.24 to 1.33
Apache Spark
Databricks Runtime 13.0 includes Apache Spark 3.4.0.
Spark SQL
Features
Implement support for DEFAULT values for columns in tables (SPARK-38334)
Add Dataset.as(StructType) (SPARK-39625)
Support parameterized SQL (SPARK-41271, SPARK-42702)
Add unpivot / melt (SPARK-38864, SPARK-39876)
Support “lateral column alias references” (SPARK-27561)
Support result offset clause (SPARK-28330, SPARK-39159)
Support Timestamp without time zone data type (SPARK-35662)
Support scalar subquery in time travel (SPARK-39306)
Make Catalog API be compatible with 3-layer-namespace (SPARK-39235)
Support timestamp in seconds for TimeTravel using Dataframe options (SPARK-39633)
Add SparkSession.config(Map) (SPARK-40163)
Support changing session catalog’s default database (SPARK-35242)
Protobuf support for Spark - from_protobuf AND to_protobuf (SPARK-40654)
Add WHEN NOT MATCHED BY SOURCE clause to MERGE INTO (SPARK-40921)
Relax ordering constraint for CREATE TABLE column options (SPARK-40944)
SQL Equivalent for Dataframe overwrite command (SPARK-40956)
Support Generate with no required child output to host outer references (SPARK-41441)
ORDER BY ALL (SPARK-41637)
GROUP BY ALL (SPARK-41635)
Add flatMapSortedGroups and cogroupSorted (SPARK-38591)
Support subqueries with correlated non-equality predicates (SPARK-36114)
Support subqueries with correlation through UNION/INTERSECT/EXCEPT (SPARK-36124)
ANSI Compliance
ANSI SQL mode: always return null on invalid access to map column (SPARK-40066)
Support double quoted identifiers (SPARK-40585)
ANSI SQL mode: Round/Bround should return an error on integer overflow (SPARK-42045)
Support casting of integrals to ANSI intervals (SPARK-40008)
Support cast of decimals to ANSI intervals (SPARK-40014)
Return wider ANSI interval types from the percentile functions (SPARK-40151)
Support cast of ANSI intervals to decimals (SPARK-39470)
Support casting intervals to integrals in ANSI mode (SPARK-39451)
Harden SQLSTATE usage for error classes (SPARK-41994)
Functions
Support table-valued generator functions in the FROM clause (SPARK-41594)
Support ANSI Aggregate Function: REGR_SXY (SPARK-37681)
Support ANSI Aggregate Function: REGR_R2 (SPARK-37641)
Support ANSI Aggregate Function: REGR_SXX (SPARK-37672)
Support ANSI Aggregate Function: REGR_SYY (SPARK-37702)
Support ANSI Aggregate Function: REGR_SLOPE (SPARK-39230)
Support ANSI Aggregate Function: REGR_INTERCEPT (SPARK-37623)
Support ANSI aggregation function PERCENTILE_CONT as window function (SPARK-38219)
Support ANSI Aggregation Function: PERCENTILE_DISC (SPARK-37691)
Support SPLIT_PART function (SPARK-38063)
Support TRY_AVG function (SPARK-38589)
Support TRY_TO_BINARY function (SPARK-38590)
Support the TO_NUMBER and TRY_TO_NUMBER SQL functions according to a new specification (SPARK-38796)
Support ANSI general value specification and function - USER (SPARK-39138)
Support TO_CHAR and TRY_TO_CHAR functions to format Decimal values as strings (SPARK-28516)
Support ANY_VALUE aggregate function (SPARK-39213)
Support EQUAL_NUL function (SPARK-39305)
Support aggregate function MEDIAN (SPARK-39320)
Support REGEXP_COUNT function (SPARK-39618)
Support REGEXP_INSTR function (SPARK-39744)
Support REGEXP_SUBSTR function (SPARK-39695)
Support UNPIVOT function (SPARK-39876)
Support TRY_TO_TIMESTAMP function (SPARK-39795)
Support url encode/decode as built-in function and tidy up url-related functions (SPARK-39741)
Support aggregate function MODE (SPARK-39808)
Support GET function (SPARK-40109)
Add function aliases: LEN, DATEPART, DATEADD, DATE_DIFF, CURDATE (SPARK-40352)
Improve the TO_BINARY function (SPARK-40112)
Support CURRENT_SCHEMA (SPARK-41323)
Support data masking built-in function MASK (SPARK-40687)
Support high-order function: ARRAY_COMPACT (SPARK-41235)
Support ARRAY_APPEND function (SPARK-41232)
Support ARRAY_INSERT function (SPARK-41234)
Support LUHN_CHECK function (SPARK-42191)
Support ARRAY_SORT(column, comparator) (SPARK-39925)
Data Sources
Support Column Stats in DS v2 (SPARK-41378)
Storage Partitioned Join (SPJ) in DS v2 (SPARK-37375)
Row-level operations in DS v2 (SPARK-35801)
Add SupportsReportOrdering mix in interface for DS v2 Scan (SPARK-38647)
Infer DATE type for CSV schema inference (SPARK-39469)
Support driver metrics in DS v2 custom metric API (SPARK-39635)
Distribution and ordering support DS v2 function in writing (SPARK-39607)
StringEndsWith/Contains support push down to Parquet so that we can leverage dictionary filter (SPARK-39002)
Support UDT in Spark Parquet vectorized reader (SPARK-39086)
Extend METADATA column to support row indexes for Parquet files (SPARK-37980)
Support reading parquet FIXED_LEN_BYTE_ARRAY type (SPARK-41096)
Optimize the order of filtering predicates (SPARK-40045)
Support CTE and temp table queries with MSSQL JDBC (SPARK-37259)
Support ignoreCorruptFiles and ignoreMissingFiles in Data Source options (SPARK-38767)
Pull out v1 write to WriteFiles (SPARK-41407)
Add read-side char padding to cover external data files (SPARK-40697)
Query Optimization
Merge non-correlated scalar subqueries (SPARK-34079)
Enable Bloom filter Joins by default (SPARK-38841)
Remove unnecessary distinct in aggregate expression by distinctKeys (SPARK-38832)
Support predicate pushdown and column pruning for de-duped CTEs (SPARK-37670)
Remove outer join if aggregate functions are duplicate agnostic on streamed side (SPARK-38886)
Remove left/right outer join if only left/right side columns are selected and the join keys on the other side are unique (SPARK-39172)
Optimize global Sort to RepartitionByExpression (SPARK-39911)
Optimize TransposeWindow rule (SPARK-38034)
Enhance EliminateSorts to support removing sorts via LocalLimit (SPARK-40050)
Push local limit to both sides if join condition is empty (SPARK-40040)
Add PushProjectionThroughLimit for Optimizer (SPARK-40501)
Support PIVOT/UNPIVOT with join children (SPARK-41195)
Support column pruning with multiple nondeterministic Filters (SPARK-41017)
Improve output partitioning and ordering with AQE cache (SPARK-41048)
Improve multi like performance by creating a balanced expression tree predicate (SPARK-41167)
Remove the Sort if it is the child of RepartitionByExpression (SPARK-36703)
Use available column statistics from completed query stages (SPARK-39991)
Reuse expressions in WindowSpecDefinition (SPARK-41805)
Improve AliasAwareOutputPartitioning and AliasAwareQueryOutputOrdering to take all aliases into account (SPARK-40086, SPARK-42049)
Push down limit through Python UDFs (SPARK-42115)
Code Generation and Query Execution
Make defaultJoin in BroadcastNestedLoopJoinExec running in parallel (SPARK-40487)
Codegen Support for HiveGenericUDF (SPARK-42051)
Introduce shuffle on SinglePartition (SPARK-41986)
Makes DPP support the pruning side has Union (SPARK-39217)
Other Notable Changes
Support Auto Partition Statistics Collection (SPARK-38573)
Format error messages in the Thrift Server (SPARK-40098)
Add an extension API to do plan normalization for caching (SPARK-41183)
Refactor Spark types by introducing physical types (SPARK-41226)
OneOf field support and recursion checks (SPARK-41396)
Centralize the column resolution logic (SPARK-41405)
Improve the plan change validation (SPARK-42081)
Introduce SparkPath for typesafety (SPARK-41970)
Throw Exception for db_name.view_name when creating temp view by Dataset API (SPARK-41090)
Change the default value of argument of Mask function from -1 to NULL (SPARK-42070)
Spark Core
Decommission
Avoid unnecessary task rerun on decommissioned executor lost if shuffle data migrated (SPARK-41469)
Ignore stage fetch failure caused by decommissioned executor (SPARK-40481)
Enable spark.storage.decommission.(rdd|shuffle)Blocks.enabled by default (SPARK-40198)
Add support for YARN decommissioning when ESS is disabled (SPARK-30835)
Scheduler
Make stage scheduling support local-cluster mode (SPARK-41949)
Support stage level task resource profile for standalone cluster when dynamic allocation disabled (SPARK-39853)
Delay onDisconnected to enable Driver receives ExecutorExitCode (SPARK-39957)
Improve the speculation through the stage task metrics (SPARK-32170)
Add stage level resource scheduling support for standalone cluster (SPARK-39062)
Improve LaunchTask process to avoid Stage failures caused by fail-to-send LaunchTask messages (SPARK-39955)
Shuffle
Add Push Based Shuffle client side read metrics (SPARK-36620)
Shuffle server side metrics for Push-based shuffle (SPARK-33573)
Ensure mergedShuffleCleaner have been shutdown before db close (SPARK-40186)
Add RocksDB support for shuffle service state store (SPARK-38888)
Encapsulate LevelDB used to store remote/external shuffle state as DB (SPARK-38909)
Enable spark.dynamicAllocation.shuffleTracking.enabled by default (SPARK-3984)
Enable Push-based shuffle service to store state in NM level DB for work preserving restart (SPARK-33236)
Remove shuffle blocks using the shuffle service for released executors (SPARK-37618)
Other Notable Changes
Support IPv6-only environment (SPARK-39457)
Enable spark.kryo.unsafe by default (SPARK-42137)
Disallow arbitrary custom classpath with proxy user in cluster mode (SPARK-41958)
Avoid BlockManager re-registration if the executor has been lost (SPARK-41360)
Remove the limitation that single task result must fit in 2GB (SPARK-40622)
Remove the support of deprecated spark.akka.* configs (SPARK-40401)
Change default logging to stderr to consistent with the behavior of log4j (SPARK-40406)
Exclude DirectTaskResult metadata when calculating result size (SPARK-40261)
Allow customize initial partitions number in take() behavior (SPARK-40211)
Use interruptible lock instead of synchronized in Executor.updateDependencies() (SPARK-40235)
Task failure should always trigger task failure listeners (SPARK-40106)
Add the ability to selectively disable watching or polling (SPARK-36462)
Do not cache unserialized broadcast relations on the driver (SPARK-39983)
Fix deadlock between TaskMemoryManager and UnsafeExternalSorter.SpillableIterator (SPARK-39283)
Expose the number partitions in a stage to TaskContext (SPARK-38679)
Make memory overhead factor configurable (SPARK-38194)
Avoid using bash -c in ShellBasedGroupsMappingProvider (SPARK-38992)
Structured Streaming
Major Features
Async Progress Tracking (SPARK-39591)
Python Arbitrary Stateful Processing in Structured Streaming (SPARK-40434)
Protobuf Support in Structured Streaming (SPARK-40653)
Fix late record filtering to support chaining of stateful operators (SPARK-40925)
Other Notable Changes
Introducing a streaming checkpoint file manager based on Hadoop’s Abortable interface (SPARK-40039)
Deprecate Trigger.Once and Promote Trigger.AvailableNow (SPARK-39805)
Expose the information of catalog table to the logical plan in streaming query (SPARK-39564)
Support collecting metrics from streaming sinks (SPARK-38564)
Deprecate DStream API (SPARK-42075)
Flip the default value of Kafka offset fetching config (SPARK-40844)
Provide cloned spark session in DataFrame in user function for foreachBatch sink in PySpark (SPARK-41379)
Spark Connect
Python Client
Implement DataFrame API (SPARK-41279)
Implement Column API (SPARK-41282)
Implement Functions API (SPARK-41283)
Implement SparkSession API (SPARK-41281)
Implement I/O API (SPARK-41284)
Implement Catalog API (SPARK-41289)
Support for User-defined Functions in Python (SPARK-41661)
Support for Pandas/Arrow Function API (SPARK-42393)
Support for Runtime SQL configuration (SPARK-42499)
Build, package and infrastructure for Spark Connect (SPARK-41286)
Type annotations for Spark Connect Python Client (SPARK-40451)
Scala Client
Implement basic Scala Client (SPARK-41534, SPARK-42133, SPARK-42043, SPARK-41822)
Implement SparkSession API (SPARK-42639, SPARK-42581, SPARK-42564, SPARK-42544, SPARK-42631)
Implement Dataframe API (SPARK-42440, SPARK-42559, SPARK-42558, SPARK-42556, SPARK-42468, SPARK-42529, SPARK-42561, SPARK-42894, SPARK-41874, SPARK-42691, SPARK-42692, SPARK-42481, SPARK-42541, SPARK-42542, SPARK-42520, SPARK-41823)
Implement Column API (SPARK-42441, SPARK-42560)
Implement Functions API (SPARK-42461, SPARK-42579, SPARK-42527, SPARK-42531, SPARK-42495, SPARK-42557)
Implement I/O API (SPARK-42457, SPARK-42555, SPARK-42690, SPARK-42878, SPARK-42757, SPARK-42482, SPARK-42733, SPARK-42518)
Implement Runtime SQL configuration (SPARK-42586)
Basic User Defined Function support (SPARK-42283, SPARK-42653, SPARK-42543)
Basic Typed API support (SPARK-42580, SPARK-42605)
Test infrastructure for Spark Connect (SPARK-42172, SPARK-42377, SPARK-42599)
Implement REPL Support (SPARK-42656, SPARK-42884)
PySpark
Pandas API on Spark
Major improvement
Python Arbitrary Stateful Processing in Structured Streaming (SPARK-40434)
Implement pandas API missing parameters (SPARK-42883)
Pandas 1.5 support (SPARK-40576)
Major feature
Implement Series.searchsorted (SPARK-40330)
Implement Series.autocorr (SPARK-38774)
Implement DataFrame.mode (SPARK-40138)
Implement DataFrame.boxplot and DataFrame.plot.box (SPARK-38993)
Implement DataFrame.corrwith (SPARK-38907)
Implement DataFrame.resample and Series.resample (SPARK-39081)
Implement DataFrame.interpolate and Series.interpolate (SPARK-38844)
Implement DataFrame.ewm and Series.ewm (SPARK-38785)
Implement GroupBy.prod (SPARK-40334)
Implement GroupBy.nth (SPARK-40333)
Implement GroupBy.quantile (SPARK-40332)
Implement GroupBy.sem (SPARK-40305)
Implement GroupBy.mad (SPARK-39284)
Implement GroupBy.skew (SPARK-39246)
Implement GroupBy.ewm (SPARK-39129)
Support GroupBy positional indexing (SPARK-38947)
Other Notable Changes
Major improvements
Provide a memory profiler for PySpark user-defined functions (SPARK-40281)
Make Catalog API be compatible with 3-layer-namespace (SPARK-39235)
NumPy input support in PySpark (SPARK-39405)
PySpark error improvements (SPARK-41597)
Major features
Support parameterized SQL in PySpark (SPARK-41666)
Implement ‘median’ function (SPARK-40003)
Implement ‘mode’ function (SPARK-40007)
Implement ‘unpivot/melt’ function (SPARK-39877)
Support Varchar in PySpark (SPARK-39760)
Support CharType in PySpark (SPARK-39809)
MLLIB
Implement PyTorch Distributor (SPARK-41589)
Unify the data validation (SPARK-38584)
Reduce the shuffle size of ALS (SPARK-40476, SPARK-40745)
Dedup isotonic regression duplicate features (SPARK-41008)
KMeans blockify input vectors (SPARK-30661)
Add relevance score for nDCG evaluation (SPARK-39446)
SparkR
Add unpivot / melt (SPARK-41267)
Add array_sort(column, comparator) (SPARK-40167)
Support multiple “Column” drop in R (SPARK-40087)
Arrow 9.0.0 support with SparkR (SPARK-40114)
Make Catalog API be compatible with 3-layer-namespace (SPARK-39579, SPARK-39646, SPARK-39645, SPARK-39236, SPARK-39716, SPARK-39719)
Support R 4.2.0 (SPARK-39372)
Live UI and History Server Service
Better Spark UI scalability and Driver stability for large applications(SPARK-41053)
Use RocksDB for spark.history.store.hybridStore.diskBackend by default (SPARK-42277)
Group nested executions under the root execution(SPARK-41752)
Show metrics properties in the environment tab(SPARK-39110)
Fix StagePage input size/records not show when records greater than zero(SPARK-34777)
Improve event logging JsonProtocol performance by using Jackson instead of Json4s (SPARK-39489)
Support spark.history.fs.update.batchSize (SPARK-39225)
Build
Update cloudpickle to v2.2.0 (SPARK-40991)
Deprecate Python 3.7 Support (SPARK-39861)
Support Python 3.11 (SPARK-41454)
Update dev.ludovic.netlib to 3.0.2 (SPARK-40251)
Update breeze to 2.0 (SPARK-39616)
Update slf4j version to 2.0.6 (SPARK-41561)
Update kubernetes-client version to 6.4.1 (SPARK-42362)
Update rocksdbjni to 7.9.2 ( SPARK-42129)
Update Apache Arrow to 11.0.0 (SPARK-42161)
Upgrade Apache Kafka to 3.3.2 (SPARK-42109)
System environment
Operating System: Ubuntu 22.04.2 LTS
Java: Zulu 8.68.0.21-CA-linux64
Scala: 2.12.15
Python: 3.10.6
R: 4.2.2
Delta Lake: 2.3.0
Installed Python libraries
======================== ======================== ======================== ======================== ======================== ========================
Library Version Library Version Library Version
======================== ======================== ======================== ======================== ======================== ========================
appdirs 1.4.4 argon2-cffi 21.3.0 argon2-cffi-bindings 21.2.0
asttokens 2.2.1 attrs 21.4.0 backcall 0.2.0
beautifulsoup4 4.11.1 black 22.6.0 bleach 4.1.0
blinker 1.4 boto3 1.24.28 botocore 1.27.28
certifi 2022.9.14 cffi 1.15.1 chardet 4.0.0
charset-normalizer 2.0.4 click 8.0.4 cryptography 37.0.1
cycler 0.11.0 Cython 0.29.32 dbus-python 1.2.18
debugpy 1.5.1 decorator 5.1.1 defusedxml 0.7.1
distlib 0.3.6 docstring-to-markdown 0.12 entrypoints 0.4
executing 1.2.0 facets-overview 1.0.2 fastjsonschema 2.16.3
filelock 3.10.7 fonttools 4.25.0 googleapis-common-protos 1.56.4
grpcio 1.48.1 grpcio-status 1.48.1 httplib2 0.20.2
idna 3.3 importlib-metadata 4.6.4 ipykernel 6.17.1
ipython 8.10.0 ipython-genutils 0.2.0 ipywidgets 7.7.2
jedi 0.18.1 jeepney 0.7.1 Jinja2 2.11.3
jmespath 0.10.0 joblib 1.2.0 jsonschema 4.16.0
jupyter-client 7.3.4 jupyter_core 4.11.2 jupyterlab-pygments 0.1.2
jupyterlab-widgets 1.0.0 keyring 23.5.0 kiwisolver 1.4.2
launchpadlib 1.10.16 lazr.restfulclient 0.14.4 lazr.uri 1.0.6
MarkupSafe 2.0.1 matplotlib 3.5.2 matplotlib-inline 0.1.6
mccabe 0.7.0 mistune 0.8.4 more-itertools 8.10.0
mypy-extensions 0.4.3 nbclient 0.5.13 nbconvert 6.4.4
nbformat 5.5.0 nest-asyncio 1.5.5 nodeenv 1.7.0
notebook 6.4.12 numpy 1.21.5 oauthlib 3.2.0
packaging 21.3 pandas 1.4.4 pandocfilters 1.5.0
parso 0.8.3 pathspec 0.9.0 patsy 0.5.2
pexpect 4.8.0 pickleshare 0.7.5 Pillow 9.2.0
pip 22.2.2 platformdirs 2.5.2 plotly 5.9.0
pluggy 1.0.0 prometheus-client 0.14.1 prompt-toolkit 3.0.36
protobuf 3.19.4 psutil 5.9.0 psycopg2 2.9.3
ptyprocess 0.7.0 pure-eval 0.2.2 pyarrow 7.0.0
pycparser 2.21 pyflakes 3.0.1 Pygments 2.11.2
PyGObject 3.42.1 PyJWT 2.3.0 pyodbc 4.0.32
pyparsing 3.0.9 pyright 1.1.294 pyrsistent 0.18.0
python-dateutil 2.8.2 python-lsp-jsonrpc 1.0.0 python-lsp-server 1.7.1
pytoolconfig 1.2.2 pytz 2022.1 pyzmq 23.2.0
requests 2.28.1 rope 1.7.0 s3transfer 0.6.0
scikit-learn 1.1.1 scipy 1.9.1 seaborn 0.11.2
SecretStorage 3.3.1 Send2Trash 1.8.0 setuptools 63.4.1
six 1.16.0 soupsieve 2.3.1 ssh-import-id 5.11
stack-data 0.6.2 statsmodels 0.13.2 tenacity 8.0.1
terminado 0.13.1 testpath 0.6.0 threadpoolctl 2.2.0
tokenize-rt 4.2.1 tomli 2.0.1 tornado 6.1
traitlets 5.1.1 ujson 5.4.0 unattended-upgrades 0.1
urllib3 1.26.11 virtualenv 20.16.3 wadllib 1.3.6
wcwidth 0.2.5 webencodings 0.5.1 whatthepatch 1.0.2
wheel 0.37.1 widgetsnbextension 3.6.1 yapf 0.31.0
zipp 1.0.0
======================== ======================== ======================== ======================== ======================== ========================
Installed R libraries
R libraries are installed from the Microsoft CRAN snapshot on 2023-02-10.
============= ============= ============= ============= ============= =============
Library Version Library Version Library Version
============= ============= ============= ============= ============= =============
arrow 10.0.1 askpass 1.1 assertthat 0.2.1
backports 1.4.1 base 4.2.2 base64enc 0.1-3
bit 4.0.5 bit64 4.0.5 blob 1.2.3
boot 1.3-28 brew 1.0-8 brio 1.1.3
broom 1.0.3 bslib 0.4.2 cachem 1.0.6
callr 3.7.3 caret 6.0-93 cellranger 1.1.0
chron 2.3-59 class 7.3-21 cli 3.6.0
clipr 0.8.0 clock 0.6.1 cluster 2.1.4
codetools 0.2-19 colorspace 2.1-0 commonmark 1.8.1
compiler 4.2.2 config 0.3.1 cpp11 0.4.3
crayon 1.5.2 credentials 1.3.2 curl 5.0.0
data.table 1.14.6 datasets 4.2.2 DBI 1.1.3
dbplyr 2.3.0 desc 1.4.2 devtools 2.4.5
diffobj 0.3.5 digest 0.6.31 downlit 0.4.2
dplyr 1.1.0 dtplyr 1.2.2 e1071 1.7-13
ellipsis 0.3.2 evaluate 0.20 fansi 1.0.4
farver 2.1.1 fastmap 1.1.0 fontawesome 0.5.0
forcats 1.0.0 foreach 1.5.2 foreign 0.8-82
forge 0.2.0 fs 1.6.1 future 1.31.0
future.apply 1.10.0 gargle 1.3.0 generics 0.1.3
gert 1.9.2 ggplot2 3.4.0 gh 1.3.1
gitcreds 0.1.2 glmnet 4.1-6 globals 0.16.2
glue 1.6.2 googledrive 2.0.0 googlesheets4 1.0.1
gower 1.0.1 graphics 4.2.2 grDevices 4.2.2
grid 4.2.2 gridExtra 2.3 gsubfn 0.7
gtable 0.3.1 hardhat 1.2.0 haven 2.5.1
highr 0.10 hms 1.1.2 htmltools 0.5.4
htmlwidgets 1.6.1 httpuv 1.6.8 httr 1.4.4
ids 1.0.1 ini 0.3.1 ipred 0.9-13
isoband 0.2.7 iterators 1.0.14 jquerylib 0.1.4
jsonlite 1.8.4 KernSmooth 2.23-20 knitr 1.42
labeling 0.4.2 later 1.3.0 lattice 0.20-45
lava 1.7.1 lifecycle 1.0.3 listenv 0.9.0
lubridate 1.9.1 magrittr 2.0.3 markdown 1.5
MASS 7.3-58.2 Matrix 1.5-1 memoise 2.0.1
methods 4.2.2 mgcv 1.8-41 mime 0.12
miniUI 0.1.1.1 ModelMetrics 1.2.2.2 modelr 0.1.10
munsell 0.5.0 nlme 3.1-162 nnet 7.3-18
numDeriv 2016.8-1.1 openssl 2.0.5 parallel 4.2.2
parallelly 1.34.0 pillar 1.8.1 pkgbuild 1.4.0
pkgconfig 2.0.3 pkgdown 2.0.7 pkgload 1.3.2
plogr 0.2.0 plyr 1.8.8 praise 1.0.0
prettyunits 1.1.1 pROC 1.18.0 processx 3.8.0
prodlim 2019.11.13 profvis 0.3.7 progress 1.2.2
progressr 0.13.0 promises 1.2.0.1 proto 1.0.0
proxy 0.4-27 ps 1.7.2 purrr 1.0.1
r2d3 0.2.6 R6 2.5.1 ragg 1.2.5
randomForest 4.7-1.1 rappdirs 0.3.3 rcmdcheck 1.4.0
RColorBrewer 1.1-3 Rcpp 1.0.10 RcppEigen 0.3.3.9.3
readr 2.1.3 readxl 1.4.2 recipes 1.0.4
rematch 1.0.1 rematch2 2.1.2 remotes 2.4.2
reprex 2.0.2 reshape2 1.4.4 rlang 1.0.6
rmarkdown 2.20 RODBC 1.3-20 roxygen2 7.2.3
rpart 4.1.19 rprojroot 2.0.3 Rserve 1.8-12
RSQLite 2.2.20 rstudioapi 0.14 rversions 2.1.2
rvest 1.0.3 sass 0.4.5 scales 1.2.1
selectr 0.4-2 sessioninfo 1.2.2 shape 1.4.6
shiny 1.7.4 sourcetools 0.1.7-1 sparklyr 1.7.9
SparkR 3.4.0 spatial 7.3-15 splines 4.2.2
sqldf 0.4-11 SQUAREM 2021.1 stats 4.2.2
stats4 4.2.2 stringi 1.7.12 stringr 1.5.0
survival 3.5-3 sys 3.4.1 systemfonts 1.0.4
tcltk 4.2.2 testthat 3.1.6 textshaping 0.3.6
tibble 3.1.8 tidyr 1.3.0 tidyselect 1.2.0
tidyverse 1.3.2 timechange 0.2.0 timeDate 4022.108
tinytex 0.44 tools 4.2.2 tzdb 0.3.0
urlchecker 1.0.1 usethis 2.1.6 utf8 1.2.3
utils 4.2.2 uuid 1.1-0 vctrs 0.5.2
viridisLite 0.4.1 vroom 1.6.1 waldo 0.4.0
whisker 0.4.1 withr 2.5.0 xfun 0.37
xml2 1.3.3 xopen 1.0.0 xtable 1.8-4
yaml 2.3.7 zip 2.2.2
============= ============= ============= ============= ============= =============
Installed Java and Scala libraries (Scala 2.12 cluster version)
Group ID |
Artifact ID |
Version |
|---|---|---|
antlr |
antlr |
2.7.7 |
com.amazonaws |
amazon-kinesis-client |
1.12.0 |
com.amazonaws |
aws-java-sdk-autoscaling |
1.12.390 |
com.amazonaws |
aws-java-sdk-cloudformation |
1.12.390 |
com.amazonaws |
aws-java-sdk-cloudfront |
1.12.390 |
com.amazonaws |
aws-java-sdk-cloudhsm |
1.12.390 |
com.amazonaws |
aws-java-sdk-cloudsearch |
1.12.390 |
com.amazonaws |
aws-java-sdk-cloudtrail |
1.12.390 |
com.amazonaws |
aws-java-sdk-cloudwatch |
1.12.390 |
com.amazonaws |
aws-java-sdk-cloudwatchmetrics |
1.12.390 |
com.amazonaws |
aws-java-sdk-codedeploy |
1.12.390 |
com.amazonaws |
aws-java-sdk-cognitoidentity |
1.12.390 |
com.amazonaws |
aws-java-sdk-cognitosync |
1.12.390 |
com.amazonaws |
aws-java-sdk-config |
1.12.390 |
com.amazonaws |
aws-java-sdk-core |
1.12.390 |
com.amazonaws |
aws-java-sdk-datapipeline |
1.12.390 |
com.amazonaws |
aws-java-sdk-directconnect |
1.12.390 |
com.amazonaws |
aws-java-sdk-directory |
1.12.390 |
com.amazonaws |
aws-java-sdk-dynamodb |
1.12.390 |
com.amazonaws |
aws-java-sdk-ec2 |
1.12.390 |
com.amazonaws |
aws-java-sdk-ecs |
1.12.390 |
com.amazonaws |
aws-java-sdk-efs |
1.12.390 |
com.amazonaws |
aws-java-sdk-elasticache |
1.12.390 |
com.amazonaws |
aws-java-sdk-elasticbeanstalk |
1.12.390 |
com.amazonaws |
aws-java-sdk-elasticloadbalancing |
1.12.390 |
com.amazonaws |
aws-java-sdk-elastictranscoder |
1.12.390 |
com.amazonaws |
aws-java-sdk-emr |
1.12.390 |
com.amazonaws |
aws-java-sdk-glacier |
1.12.390 |
com.amazonaws |
aws-java-sdk-glue |
1.12.390 |
com.amazonaws |
aws-java-sdk-iam |
1.12.390 |
com.amazonaws |
aws-java-sdk-importexport |
1.12.390 |
com.amazonaws |
aws-java-sdk-kinesis |
1.12.390 |
com.amazonaws |
aws-java-sdk-kms |
1.12.390 |
com.amazonaws |
aws-java-sdk-lambda |
1.12.390 |
com.amazonaws |
aws-java-sdk-logs |
1.12.390 |
com.amazonaws |
aws-java-sdk-machinelearning |
1.12.390 |
com.amazonaws |
aws-java-sdk-opsworks |
1.12.390 |
com.amazonaws |
aws-java-sdk-rds |
1.12.390 |
com.amazonaws |
aws-java-sdk-redshift |
1.12.390 |
com.amazonaws |
aws-java-sdk-route53 |
1.12.390 |
com.amazonaws |
aws-java-sdk-s3 |
1.12.390 |
com.amazonaws |
aws-java-sdk-ses |
1.12.390 |
com.amazonaws |
aws-java-sdk-simpledb |
1.12.390 |
com.amazonaws |
aws-java-sdk-simpleworkflow |
1.12.390 |
com.amazonaws |
aws-java-sdk-sns |
1.12.390 |
com.amazonaws |
aws-java-sdk-sqs |
1.12.390 |
com.amazonaws |
aws-java-sdk-ssm |
1.12.390 |
com.amazonaws |
aws-java-sdk-storagegateway |
1.12.390 |
com.amazonaws |
aws-java-sdk-sts |
1.12.390 |
com.amazonaws |
aws-java-sdk-support |
1.12.390 |
com.amazonaws |
aws-java-sdk-swf-libraries |
1.11.22 |
com.amazonaws |
aws-java-sdk-workspaces |
1.12.390 |
com.amazonaws |
jmespath-java |
1.12.390 |
com.clearspring.analytics |
stream |
2.9.6 |
com.databricks |
Rserve |
1.8-3 |
com.databricks |
jets3t |
0.7.1-0 |
com.databricks.scalapb |
compilerplugin_2.12 |
0.4.15-10 |
com.databricks.scalapb |
scalapb-runtime_2.12 |
0.4.15-10 |
com.esotericsoftware |
kryo-shaded |
4.0.2 |
com.esotericsoftware |
minlog |
1.3.0 |
com.fasterxml |
classmate |
1.3.4 |
com.fasterxml.jackson.core |
jackson-annotations |
2.14.2 |
com.fasterxml.jackson.core |
jackson-core |
2.14.2 |
com.fasterxml.jackson.core |
jackson-databind |
2.14.2 |
com.fasterxml.jackson.dataformat |
jackson-dataformat-cbor |
2.14.2 |
com.fasterxml.jackson.datatype |
jackson-datatype-joda |
2.14.2 |
com.fasterxml.jackson.datatype |
jackson-datatype-jsr310 |
2.13.4 |
com.fasterxml.jackson.module |
jackson-module-paranamer |
2.14.2 |
com.fasterxml.jackson.module |
jackson-module-scala_2.12 |
2.14.2 |
com.github.ben-manes.caffeine |
caffeine |
2.3.4 |
com.github.fommil |
jniloader |
1.1 |
com.github.fommil.netlib |
native_ref-java |
1.1 |
com.github.fommil.netlib |
native_ref-java |
1.1-natives |
com.github.fommil.netlib |
native_system-java |
1.1 |
com.github.fommil.netlib |
native_system-java |
1.1-natives |
com.github.fommil.netlib |
netlib-native_ref-linux-x86_64 |
1.1-natives |
com.github.fommil.netlib |
netlib-native_system-linux-x86_64 |
1.1-natives |
com.github.luben |
zstd-jni |
1.5.2-5 |
com.github.wendykierp |
JTransforms |
3.1 |
com.google.code.findbugs |
jsr305 |
3.0.0 |
com.google.code.gson |
gson |
2.8.9 |
com.google.crypto.tink |
tink |
1.7.0 |
com.google.flatbuffers |
flatbuffers-java |
1.12.0 |
com.google.guava |
guava |
15.0 |
com.google.protobuf |
protobuf-java |
2.6.1 |
com.h2database |
h2 |
2.1.214 |
com.helger |
profiler |
1.1.1 |
com.jcraft |
jsch |
0.1.55 |
com.jolbox |
bonecp |
0.8.0.RELEASE |
com.lihaoyi |
sourcecode_2.12 |
0.1.9 |
com.microsoft.azure |
azure-data-lake-store-sdk |
2.3.9 |
com.ning |
compress-lzf |
1.1.2 |
com.sun.mail |
javax.mail |
1.5.2 |
com.tdunning |
json |
1.8 |
com.thoughtworks.paranamer |
paranamer |
2.8 |
com.trueaccord.lenses |
lenses_2.12 |
0.4.12 |
com.twitter |
chill-java |
0.10.0 |
com.twitter |
chill_2.12 |
0.10.0 |
com.twitter |
util-app_2.12 |
7.1.0 |
com.twitter |
util-core_2.12 |
7.1.0 |
com.twitter |
util-function_2.12 |
7.1.0 |
com.twitter |
util-jvm_2.12 |
7.1.0 |
com.twitter |
util-lint_2.12 |
7.1.0 |
com.twitter |
util-registry_2.12 |
7.1.0 |
com.twitter |
util-stats_2.12 |
7.1.0 |
com.typesafe |
config |
1.2.1 |
com.typesafe.scala-logging |
scala-logging_2.12 |
3.7.2 |
com.uber |
h3 |
3.7.0 |
com.univocity |
univocity-parsers |
2.9.1 |
com.zaxxer |
HikariCP |
4.0.3 |
commons-cli |
commons-cli |
1.5.0 |
commons-codec |
commons-codec |
1.15 |
commons-collections |
commons-collections |
3.2.2 |
commons-dbcp |
commons-dbcp |
1.4 |
commons-fileupload |
commons-fileupload |
1.5 |
commons-httpclient |
commons-httpclient |
3.1 |
commons-io |
commons-io |
2.11.0 |
commons-lang |
commons-lang |
2.6 |
commons-logging |
commons-logging |
1.1.3 |
commons-pool |
commons-pool |
1.5.4 |
dev.ludovic.netlib |
arpack |
3.0.3 |
dev.ludovic.netlib |
blas |
3.0.3 |
dev.ludovic.netlib |
lapack |
3.0.3 |
info.ganglia.gmetric4j |
gmetric4j |
1.0.10 |
io.airlift |
aircompressor |
0.21 |
io.delta |
delta-sharing-spark_2.12 |
0.6.3 |
io.dropwizard.metrics |
metrics-core |
4.2.10 |
io.dropwizard.metrics |
metrics-graphite |
4.2.10 |
io.dropwizard.metrics |
metrics-healthchecks |
4.2.10 |
io.dropwizard.metrics |
metrics-jetty9 |
4.2.10 |
io.dropwizard.metrics |
metrics-jmx |
4.2.10 |
io.dropwizard.metrics |
metrics-json |
4.2.10 |
io.dropwizard.metrics |
metrics-jvm |
4.2.10 |
io.dropwizard.metrics |
metrics-servlets |
4.2.10 |
io.netty |
netty-all |
4.1.87.Final |
io.netty |
netty-buffer |
4.1.87.Final |
io.netty |
netty-codec |
4.1.87.Final |
io.netty |
netty-codec-http |
4.1.87.Final |
io.netty |
netty-codec-http2 |
4.1.87.Final |
io.netty |
netty-codec-socks |
4.1.87.Final |
io.netty |
netty-common |
4.1.87.Final |
io.netty |
netty-handler |
4.1.87.Final |
io.netty |
netty-handler-proxy |
4.1.87.Final |
io.netty |
netty-resolver |
4.1.87.Final |
io.netty |
netty-transport |
4.1.87.Final |
io.netty |
netty-transport-classes-epoll |
4.1.87.Final |
io.netty |
netty-transport-classes-kqueue |
4.1.87.Final |
io.netty |
netty-transport-native-epoll |
4.1.87.Final |
io.netty |
netty-transport-native-epoll |
4.1.87.Final-linux-aarch_64 |
io.netty |
netty-transport-native-epoll |
4.1.87.Final-linux-x86_64 |
io.netty |
netty-transport-native-kqueue |
4.1.87.Final-osx-aarch_64 |
io.netty |
netty-transport-native-kqueue |
4.1.87.Final-osx-x86_64 |
io.netty |
netty-transport-native-unix-common |
4.1.87.Final |
io.prometheus |
simpleclient |
0.7.0 |
io.prometheus |
simpleclient_common |
0.7.0 |
io.prometheus |
simpleclient_dropwizard |
0.7.0 |
io.prometheus |
simpleclient_pushgateway |
0.7.0 |
io.prometheus |
simpleclient_servlet |
0.7.0 |
io.prometheus.jmx |
collector |
0.12.0 |
jakarta.annotation |
jakarta.annotation-api |
1.3.5 |
jakarta.servlet |
jakarta.servlet-api |
4.0.3 |
jakarta.validation |
jakarta.validation-api |
2.0.2 |
jakarta.ws.rs |
jakarta.ws.rs-api |
2.1.6 |
javax.activation |
activation |
1.1.1 |
javax.el |
javax.el-api |
2.2.4 |
javax.jdo |
jdo-api |
3.0.1 |
javax.transaction |
jta |
1.1 |
javax.transaction |
transaction-api |
1.1 |
javax.xml.bind |
jaxb-api |
2.2.11 |
javolution |
javolution |
5.5.1 |
jline |
jline |
2.14.6 |
joda-time |
joda-time |
2.12.1 |
net.java.dev.jna |
jna |
5.8.0 |
net.razorvine |
pickle |
1.3 |
net.sf.jpam |
jpam |
1.1 |
net.sf.opencsv |
opencsv |
2.3 |
net.sf.supercsv |
super-csv |
2.2.0 |
net.snowflake |
snowflake-ingest-sdk |
0.9.6 |
net.snowflake |
snowflake-jdbc |
3.13.22 |
net.sourceforge.f2j |
arpack_combined_all |
0.1 |
org.acplt.remotetea |
remotetea-oncrpc |
1.1.2 |
org.antlr |
ST4 |
4.0.4 |
org.antlr |
antlr-runtime |
3.5.2 |
org.antlr |
antlr4-runtime |
4.9.3 |
org.antlr |
stringtemplate |
3.2.1 |
org.apache.ant |
ant |
1.9.16 |
org.apache.ant |
ant-jsch |
1.9.16 |
org.apache.ant |
ant-launcher |
1.9.16 |
org.apache.arrow |
arrow-format |
11.0.0 |
org.apache.arrow |
arrow-memory-core |
11.0.0 |
org.apache.arrow |
arrow-memory-netty |
11.0.0 |
org.apache.arrow |
arrow-vector |
11.0.0 |
org.apache.avro |
avro |
1.11.1 |
org.apache.avro |
avro-ipc |
1.11.1 |
org.apache.avro |
avro-mapred |
1.11.1 |
org.apache.commons |
commons-collections4 |
4.4 |
org.apache.commons |
commons-compress |
1.21 |
org.apache.commons |
commons-crypto |
1.1.0 |
org.apache.commons |
commons-lang3 |
3.12.0 |
org.apache.commons |
commons-math3 |
3.6.1 |
org.apache.commons |
commons-text |
1.10.0 |
org.apache.curator |
curator-client |
2.13.0 |
org.apache.curator |
curator-framework |
2.13.0 |
org.apache.curator |
curator-recipes |
2.13.0 |
org.apache.datasketches |
datasketches-java |
3.1.0 |
org.apache.datasketches |
datasketches-memory |
2.0.0 |
org.apache.derby |
derby |
10.14.2.0 |
org.apache.hadoop |
hadoop-client-runtime |
3.3.4 |
org.apache.hive |
hive-beeline |
2.3.9 |
org.apache.hive |
hive-cli |
2.3.9 |
org.apache.hive |
hive-jdbc |
2.3.9 |
org.apache.hive |
hive-llap-client |
2.3.9 |
org.apache.hive |
hive-llap-common |
2.3.9 |
org.apache.hive |
hive-serde |
2.3.9 |
org.apache.hive |
hive-shims |
2.3.9 |
org.apache.hive |
hive-storage-api |
2.8.1 |
org.apache.hive.shims |
hive-shims-0.23 |
2.3.9 |
org.apache.hive.shims |
hive-shims-common |
2.3.9 |
org.apache.hive.shims |
hive-shims-scheduler |
2.3.9 |
org.apache.httpcomponents |
httpclient |
4.5.14 |
org.apache.httpcomponents |
httpcore |
4.4.16 |
org.apache.ivy |
ivy |
2.5.1 |
org.apache.logging.log4j |
log4j-1.2-api |
2.19.0 |
org.apache.logging.log4j |
log4j-api |
2.19.0 |
org.apache.logging.log4j |
log4j-core |
2.19.0 |
org.apache.logging.log4j |
log4j-slf4j2-impl |
2.19.0 |
org.apache.mesos |
mesos |
1.11.0-shaded-protobuf |
org.apache.orc |
orc-core |
1.8.2-shaded-protobuf |
org.apache.orc |
orc-mapreduce |
1.8.2-shaded-protobuf |
org.apache.orc |
orc-shims |
1.8.2 |
org.apache.thrift |
libfb303 |
0.9.3 |
org.apache.thrift |
libthrift |
0.12.0 |
org.apache.xbean |
xbean-asm9-shaded |
4.22 |
org.apache.yetus |
audience-annotations |
0.13.0 |
org.apache.zookeeper |
zookeeper |
3.6.3 |
org.apache.zookeeper |
zookeeper-jute |
3.6.3 |
org.checkerframework |
checker-qual |
3.5.0 |
org.codehaus.jackson |
jackson-core-asl |
1.9.13 |
org.codehaus.jackson |
jackson-mapper-asl |
1.9.13 |
org.codehaus.janino |
commons-compiler |
3.0.16 |
org.codehaus.janino |
janino |
3.0.16 |
org.datanucleus |
datanucleus-api-jdo |
4.2.4 |
org.datanucleus |
datanucleus-core |
4.1.17 |
org.datanucleus |
datanucleus-rdbms |
4.1.19 |
org.datanucleus |
javax.jdo |
3.2.0-m3 |
org.eclipse.jetty |
jetty-client |
9.4.50.v20221201 |
org.eclipse.jetty |
jetty-continuation |
9.4.50.v20221201 |
org.eclipse.jetty |
jetty-http |
9.4.50.v20221201 |
org.eclipse.jetty |
jetty-io |
9.4.50.v20221201 |
org.eclipse.jetty |
jetty-jndi |
9.4.50.v20221201 |
org.eclipse.jetty |
jetty-plus |
9.4.50.v20221201 |
org.eclipse.jetty |
jetty-proxy |
9.4.50.v20221201 |
org.eclipse.jetty |
jetty-security |
9.4.50.v20221201 |
org.eclipse.jetty |
jetty-server |
9.4.50.v20221201 |
org.eclipse.jetty |
jetty-servlet |
9.4.50.v20221201 |
org.eclipse.jetty |
jetty-servlets |
9.4.50.v20221201 |
org.eclipse.jetty |
jetty-util |
9.4.50.v20221201 |
org.eclipse.jetty |
jetty-util-ajax |
9.4.50.v20221201 |
org.eclipse.jetty |
jetty-webapp |
9.4.50.v20221201 |
org.eclipse.jetty |
jetty-xml |
9.4.50.v20221201 |
org.eclipse.jetty.websocket |
websocket-api |
9.4.50.v20221201 |
org.eclipse.jetty.websocket |
websocket-client |
9.4.50.v20221201 |
org.eclipse.jetty.websocket |
websocket-common |
9.4.50.v20221201 |
org.eclipse.jetty.websocket |
websocket-server |
9.4.50.v20221201 |
org.eclipse.jetty.websocket |
websocket-servlet |
9.4.50.v20221201 |
org.fusesource.leveldbjni |
leveldbjni-all |
1.8 |
org.glassfish.hk2 |
hk2-api |
2.6.1 |
org.glassfish.hk2 |
hk2-locator |
2.6.1 |
org.glassfish.hk2 |
hk2-utils |
2.6.1 |
org.glassfish.hk2 |
osgi-resource-locator |
1.0.3 |
org.glassfish.hk2.external |
aopalliance-repackaged |
2.6.1 |
org.glassfish.hk2.external |
jakarta.inject |
2.6.1 |
org.glassfish.jersey.containers |
jersey-container-servlet |
2.36 |
org.glassfish.jersey.containers |
jersey-container-servlet-core |
2.36 |
org.glassfish.jersey.core |
jersey-client |
2.36 |
org.glassfish.jersey.core |
jersey-common |
2.36 |
org.glassfish.jersey.core |
jersey-server |
2.36 |
org.glassfish.jersey.inject |
jersey-hk2 |
2.36 |
org.hibernate.validator |
hibernate-validator |
6.1.7.Final |
org.javassist |
javassist |
3.25.0-GA |
org.jboss.logging |
jboss-logging |
3.3.2.Final |
org.jdbi |
jdbi |
2.63.1 |
org.jetbrains |
annotations |
17.0.0 |
org.joda |
joda-convert |
1.7 |
org.jodd |
jodd-core |
3.5.2 |
org.json4s |
json4s-ast_2.12 |
3.7.0-M11 |
org.json4s |
json4s-core_2.12 |
3.7.0-M11 |
org.json4s |
json4s-jackson_2.12 |
3.7.0-M11 |
org.json4s |
json4s-scalap_2.12 |
3.7.0-M11 |
org.lz4 |
lz4-java |
1.8.0 |
org.mariadb.jdbc |
mariadb-java-client |
2.7.4 |
org.mlflow |
mlflow-spark |
2.2.0 |
org.objenesis |
objenesis |
2.5.1 |
org.postgresql |
postgresql |
42.3.8 |
org.roaringbitmap |
RoaringBitmap |
0.9.39 |
org.roaringbitmap |
shims |
0.9.39 |
org.rocksdb |
rocksdbjni |
7.8.3 |
org.rosuda.REngine |
REngine |
2.1.0 |
org.scala-lang |
scala-compiler_2.12 |
2.12.15 |
org.scala-lang |
scala-library_2.12 |
2.12.15 |
org.scala-lang |
scala-reflect_2.12 |
2.12.15 |
org.scala-lang.modules |
scala-collection-compat_2.12 |
2.4.3 |
org.scala-lang.modules |
scala-parser-combinators_2.12 |
1.1.2 |
org.scala-lang.modules |
scala-xml_2.12 |
1.2.0 |
org.scala-sbt |
test-interface |
1.0 |
org.scalacheck |
scalacheck_2.12 |
1.14.2 |
org.scalactic |
scalactic_2.12 |
3.0.8 |
org.scalanlp |
breeze-macros_2.12 |
2.1.0 |
org.scalanlp |
breeze_2.12 |
2.1.0 |
org.scalatest |
scalatest_2.12 |
3.0.8 |
org.slf4j |
jcl-over-slf4j |
2.0.6 |
org.slf4j |
jul-to-slf4j |
2.0.6 |
org.slf4j |
slf4j-api |
2.0.6 |
org.threeten |
threeten-extra |
1.7.1 |
org.tukaani |
xz |
1.9 |
org.typelevel |
algebra_2.12 |
2.0.1 |
org.typelevel |
cats-kernel_2.12 |
2.1.1 |
org.typelevel |
spire-macros_2.12 |
0.17.0 |
org.typelevel |
spire-platform_2.12 |
0.17.0 |
org.typelevel |
spire-util_2.12 |
0.17.0 |
org.typelevel |
spire_2.12 |
0.17.0 |
org.wildfly.openssl |
wildfly-openssl |
1.1.3.Final |
org.xerial |
sqlite-jdbc |
3.8.11.2 |
org.xerial.snappy |
snappy-java |
1.1.8.4 |
org.yaml |
snakeyaml |
1.33 |
oro |
oro |
2.0.8 |
pl.edu.icm |
JLargeArrays |
1.5 |
software.amazon.cryptools |
AmazonCorrettoCryptoProvider |
1.6.1-linux-x86_64 |
software.amazon.ion |
ion-java |
1.0.2 |
stax |
stax-api |
1.0.1 |