[LIVY-1041] Add Spark 4 support and make it the default build

[roczei]

What changes were proposed in this pull request?

This PR adds support for building and running Apache Livy against Apache Spark 4.x (pinned to 4.1.2). Spark 4 is now the default build (mvn package with no extra profiles); Spark 3.x remains available via -Pspark3 -Pscala-2.12.

Spark 4 introduced breaking changes that required widespread updates: JDK 17 minimum, Scala 2.13 only, Netty 4.2.x, a rewritten Scala REPL, and json4s 4.0.7.

Build / POM changes

• pom.xml: root properties default to Spark 4.1.2, Scala 2.13.17, Hadoop 3.4.1, and java.version=17 so mvn package works without profile flags. The existing spark3 and scala-2.12 profiles override those defaults for legacy Spark 3 builds; dedicated spark4 / scala-2.13 profiles are no longer needed. Per Gyorgy's suggestion on LIVY-1069, the latest supported Spark/Scala versions are now the default rather than requiring explicit -P flags.
• JDK handling: maven-enforcer follows the active profile — JDK 17 for the default Spark 4 build, JDK 8 minimum when -Pspark3 sets java.version=1.8.
• maven-shade-plugin 3.5.0 -> 3.6.2 (bundles an ASM version that understands Java 22 bytecode, needed to shade Jackson 2.18.2's META-INF/versions/22 multi-release classes).
• repl/pom.xml: added json4s-jackson-core to the shade includes (json4s 4.0.7 split JsonMethods into a separate artifact; missing it caused a NoClassDefFoundError in PythonInterpreter at runtime). Also pulls livy-test-lib in as a test dependency so the repl tests can use the new shared ScalaVersionAware trait.
• New Scala 2.13 module wrapper POMs: core/scala-2.13, repl/scala-2.13, scala-api/scala-2.13.
• README.md: documented default Spark 4 build and Spark 3 profile usage; Spark 4 Python compatibility — supported Python versions for Spark 4.1 are 3.10 – 3.14.

Spark 4 / Scala 2.13 source-level fixes

• repl/scala-2.13/SparkInterpreter.scala: new implementation ported to the Spark 4 SparkILoop (shell.ILoop, PrintWriter, createInterpreter(settings), operations via sparkILoop.intp, ReplCompletion). To make extra JARs supplied via spark.jars.packages visible to import ... we feed those JARs to the Scala compiler through the -classpath argument at Settings construction time, instead of via a post-init IMain.addUrlsToClassPath call. In Scala 2.13 the latter does not reliably register URLs with the compiler's platform.classPath, so import org.codehaus.plexus.util._ fails with object plexus is not a member of package org.codehaus. A new collectUserJarsClasspath() helper walks the context classloader chain to Spark's MutableURLClassLoader, filters out livy-* and the wrong-version scala-reflect (same rules as the old addUrls path), and joins the resulting file paths with File.pathSeparator. If the chain has no MutableURLClassLoader we log a warning and skip the extra -classpath entry rather than passing an empty argument.

• AbstractSparkInterpreter.scala:

  • parseError skips the leading caret line so ename lands on the "error: ..." message on both Scala 2.12 and 2.13 (2.13 prints the caret pointer before the message). Implemented with lines.indexWhere(_.trim != "^") ... lines.patch(idx, Nil, 1).
  • New isEffectivelyEmpty helper strips block/line comments and returns success for comment-only inputs (Scala 2.12 accepted them silently; 2.13 rejects them as compile errors).

• repl/Session.scala: SparkR setJobGroup match now accepts "4" alongside "2" | "3".

• server/interactive/InteractiveSession.scala: datanucleusJars and mergeHiveSiteAndHiveDeps now take an extra scalaVersion parameter, adds case 3 | 4 to the major-version match and replaces the hard-coded assembly/target/scala-2.12/jars with assembly/target/scala-$scalaVersion/jars.

• server/batch/BatchSession.scala: pass --verbose to spark-submit under LivyConf.TEST_MODE so tests can grep the resolved arguments out of the child's log.

• rsc/driver/SparkEntries.java: new reflection-based hiveClassesArePresent helper that probes org.apache.spark.sql.classic.SparkSession$ (Spark 4) first and falls back to org.apache.spark.sql.SparkSession$ (Spark 3).

• utils/SparkProcessBuilder.scala: new verbose(v: Boolean) mutator and an internal _verbose flag that appends --verbose to spark-submit when set (used by BatchSession).

• utils/LivySparkUtils.scala: (4, 0) and (4, 1) -> "2.13" entries in sparkScalaVersionMap; MAX_VERSION bumped from (3, 6) to (4, 2).

• repl/SparkRInterpreter.scala: log a warning on Spark 4 that SparkR was deprecated in SPARK-49347 and may be removed in a future Spark release.

• scala-api/scalaapi/package.scala: inline comment documenting that Duration.isFinite is a parameterless def in Scala 2.13 (took an empty parameter list in 2.12); dropping the parens compiles on both.

• Thriftserver Scala/json4s 4.x updates:

  • session/Get{Columns,Functions,Schemas,Tables}Job.java: scala.collection.JavaConversions.seqAsJavaList (removed in Scala 2.13) replaced with scala.collection.JavaConverters.seqAsJavaListConverter(...).asJava().
  • types/DataTypeUtils.scala: json4s 4.x parse(input, useBigDecimal) requires an AsJsonInput[T]; switch to the single-arg String overload parse(sparkJson).
  • LivyExecuteStatementOperation.scala: explicit res.toSeq since Scala 2.13 no longer widens a mutable Buffer to Seq implicitly.
  • ThriftServerSuites.scala: LIVY-571 assertion now also accepts the Spark 4 error string [SCHEMA_NOT_FOUND] The schema spark_catalog.invalid_database cannot be found.

• InteractiveSessionServlet.scala: logs.asJava -> logs.toSeq.asJava so Scala 2.13 picks the Seq[A] -> java.util.List asJava overload instead of erroring on ambiguity.

Thriftserver libthrift pre-0.16 / 0.16 compatibility The Livy Thrift binary CLI service was written against libthrift 0.9.3 (the version pinned in the root pom.xml). Spark 3 keeps a compatible pre-0.16 libthrift on the runtime classpath (e.g. 0.12.0 in Spark 3.3), but Spark 4 pulls in libthrift 0.16.0 transitively through spark-hive, and 0.16 removed four TThreadPoolServer.Args builder methods that Livy relied on: requestTimeout, requestTimeoutUnit, beBackoffSlotLength and beBackoffSlotLengthUnit. Compiling against 0.9.3 but running against 0.16.0 made the mini-cluster Thrift server abort during startup with a NoSuchMethodError on TThreadPoolServer$Args.requestTimeout(int), which in turn hung JdbcIT on the SASL handshake. ThriftBinaryCLIService now invokes those four builders reflectively via a small applyOptionalArg helper: they run unchanged on Spark 3 classpaths and are silently skipped on Spark 4, where they no longer exist. The functional loss on Spark 4 is limited to the login-timeout / backoff knobs, which have no equivalent in the newer libthrift API.

Test fix: Scala 2.13 REPL output shape changes in InteractiveIT The Scala 2.13 REPL prints results differently from Scala 2.12 in four ways that InteractiveIT was too strict about; the default Spark 4 build (Scala 2.13 only) trips all four. Fixed in place:

• Value results: val res0: Int = 2 (2.13) vs. res0: Int = 2 (2.12). Six verifyResult regexes (across basic interactive session, user jars are properly imported ... and recover interactive session) now accept an optional val prefix.
• Compile errors: error: not found: value abcde (2.13) vs. <console>:12: error: not found: value abcde (2.12). The <source>:<line>: location marker is now optional in the verifyError regex.
• Class definitions: class Item (2.13) vs. defined class Item (2.12). The case class Item(i: Int) check in user jars are properly imported ... now accepts an optional defined prefix.
• Deprecation-warning header: Scala 2.13 prints a warning: 1 deprecation ... line above the value binding when the expression uses a deprecated API. SparkContext.parallelize is deprecated on Spark 4, so val rdd = sc.parallelize(Array.fill(10){...}) prints the warning before the val rdd: ... line. The verifyResult regex now uses (?s)(?:warning:.*\n)?(?:val )?rdd.* to accept the warning prefix and the DOTALL semantics needed to span the newline. Both forms match on the default Scala 2.13 build and -Pscala-2.12.

Test fix: Spark 4 SQLContext / DataFrame runtime class in InteractiveIT Spark 4 moved the runtime SQLContext and DataFrame implementations into org.apache.spark.sql.classic; only the compile-time aliases remain at org.apache.spark.sql.SQLContext / org.apache.spark.sql.DataFrame. The Scala 2.13 REPL surfaces the runtime class name, so Spark 4 prints sql: org.apache.spark.sql.classic.SQLContext = ... and val df: org.apache.spark.sql.classic.DataFrame = ..., where Spark 3 prints the alias form. Two InteractiveIT regexes now allow an optional classic. package qualifier so the same expectations match both the default Spark 4 build and -Pspark3:

• sql: org.apache.spark.sql.(?:classic\.)?SQLContext = ...
• (?:val )?df: org.apache.spark.sql.(?:classic\.)?DataFrame The old Pattern.quote("df: org.apache.spark.sql.DataFrame") literal was too strict for Spark 4 and caused the basic interactive session case to fail.

Cross-version test helpers: ScalaVersionAware trait New test-lib/.../ScalaVersionAware trait exposes the small Scala-2.12 vs 2.13 REPL-output differences as reusable fragments:

• optionalValPrefix ("val " on 2.13, "" on 2.12) for exact-match expected strings.
• optionalValPrefixRegex / optionalDefinedPrefixRegex / optionalWarningPrefixRegex regex fragments for the corresponding verifyResult regex assertions. Detection uses the runtime scala.util.Properties.versionNumberString so the same test class works regardless of which Scala the artifact was compiled against. Mixed into BaseInterpreterSpec, BaseSessionSpec and InteractiveIT. Downstream specs (ScalaInterpreterSpec, SharedSessionSpec, SparkSessionSpec, InteractiveIT) rewrote their expected-value strings against these fragments; some hard-equal assertions were relaxed to include/fullyMatch regex where the 2.13 REPL adds a deprecation banner ahead of the result or renumbers resN slots differently.

Server/unit tests: cross-Scala tolerance in InteractiveSessionSpec The should get scala version case now decomposes the JSON result by hand and asserts text/plain equals either res0: Int = 3\n or val res0: Int = 3\n, plus separate status / execution_count assertions. Straight equality with a decomposed Map broke on the Spark 4 driver where the 2.13 REPL adds the val prefix.

Unit tests: LivySparkUtilsSuite
Added 4.0.0 and 4.1.2 to both the supported-version list and the defaultSparkScalaVersion -> "2.13" expectations.

scala-api tests

• ScalaClientTestUtils.scala: context.sc.parallelize(buffer, ...) -> context.sc.parallelize(buffer.toSeq, ...) (2.13 no longer widens a mutable ArrayBuffer to Seq implicitly for the parallelize signature).
• ScalaJobHandleTest.scala: Duration.Undefined -> Duration.Inf (2.13's Await.ready rejects Undefined with "Cannot wait for Undefined duration of time"); assertion switched from verify(mockJobHandle, times(1)).get() to verify(mockJobHandle, atLeastOnce()).isDone, because 2.13's Await.ready short-circuits when isCompleted is already true and does not call the underlying ready(atMost) -- so .get() is not observed on 2.13. isDone is exercised by both versions.

Repl tests: SparkInterpreterSpec move
SparkInterpreterSpec.scala moved from repl/scala-2.12/src/test/... into the shared repl/src/test/scala/... tree (it was byte-identical between scala-2.12 and scala-2.13). A new should skip leading caret lines in Scala 2.13 error format. test case was added for the caret- first parseError branch, so the shared spec runs three cases under both the default Scala 2.13 build and -Pscala-2.12.

CI matrix

• .github/workflows/unit-tests.yaml and integration-tests.yaml converted the flat maven_profile x jdk_path matrix to an explicit include: list. The default spark4 build (no extra Maven profiles) runs on JDK 17; Spark 3 matrix entries use -Pscala-2.12 -Pspark3.

How was this patch tested?

• Unit tests: mvn verify -Pthriftserver passes all 19 modules on JDK 17 (macOS aarch64) with the default Spark 4 build. Verified modules include livy-rsc (40 tests), livy-repl_2.13 (87 tests), livy-server (205 tests), livy-client-http (17 tests), livy-scala-api_2.13 (17 tests), and the thriftserver integration suite (27 tests: HttpThriftServerSuite + BinaryThriftServerSuite). The SparkInterpreterSpec move added one new caret-first parseError test case; the shared spec now has 3 cases and runs identically under both the default Scala 2.13 build and -Pscala-2.12.
• Integration tests (mvn integration-test -Pspark3 -Pscala-2.12 -pl :livy-integration-test) pass.
• GitHub Actions CI runs both spark3 and spark4 matrix entries for unit and integration test workflows.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 4.7)

[roczei]

The integration test failed because Python 3.11.11 is missing from the test environment:

pyenv: version `3.11.11' not installed
Error: Process completed with exit code 1.

This will be resolved once #540 is merged and the Docker image is updated. Related comment from @gyogal: #540 (comment)