Limitations with Databricks Connect for Scala

Note

This article covers Databricks Connect for Databricks Runtime 13.3 LTS and above.

This article lists limitations with Databricks Connect for Scala. Databricks Connect enables you to connect popular IDEs, notebook servers, and custom applications to Databricks clusters. See What is Databricks Connect?. For the Python version of this article, see Limitations with Databricks Connect for Python.

Important

Depending on the version of Scala, Java, Databricks Runtime, and Databricks Connect that you are using, there may be version requirements for some features. See Requirements.

Feature availability

Not available on Databricks Connect for Databricks Runtime 13.3 LTS and below:

  • Streaming foreachBatch

  • Creating DataFrames larger than 128 MB

  • Long queries over 3600 seconds

  • Scalar UDFs on clusters that use Single user cluster access mode

Not available:

  • Databricks Utilities: credentials, library, notebook workflow, widgets

  • Spark Context

  • RDDs

  • CREATE TABLE <table-name> AS SELECT (instead, use spark.sql("SELECT ...").write.saveAsTable("table"))

  • Changing the log4j log level through SparkContext

  • Distributed ML training

  • Synchronizing the local development environment with the remote cluster