Add Support for Spark 3.3.4 and 3.5.7

This PR adds support for Apache Spark 3.3.4 and 3.5.7 while removing support for older versions (2.4.5, 3.1.2, 3.2.1).

Key Changes

1. Build Configuration (build.sbt)

• Updated sparkVersions map to support Spark 3.3.4 and 3.5.7 for both Scala 2.12 and 2.13
• Default Spark version is now 3.5.7 (can be overridden with SPARK_VERSION env var)
• Updated SHA512 checksums for new Spark versions
• Changed from hadoop2.7 to hadoop3 for Spark 3.3+

val sparkVersions = Map(
  "2.12" -> Seq("3.3.4", "3.5.7"),
  "2.13" -> Seq("3.3.4", "3.5.7")
)

2. CI Workflows (.github/workflows/)

Split CI into 4 separate workflows for better isolation and clarity:

• ci-backend-2.12-spark3.3.yml - Tests Scala 2.12 + Spark 3.3.4
• ci-backend-2.12-spark3.5.yml - Tests Scala 2.12 + Spark 3.5.7
• ci-backend-2.13-spark3.3.yml - Tests Scala 2.13 + Spark 3.3.4
• ci-backend-2.13-spark3.5.yml - Tests Scala 2.13 + Spark 3.5.7

Additional improvements:

• Replaced pip with uv for faster Python dependency installation (test time went from ~7:15m -> ~5:45m)
• Updated dist.yml to build Docker images for Spark 3.3 and 3.5 (removed 2.4 and 3.1)

3. Spark SQL Parser Compatibility

Files: polynote-spark/src/main/scala/polynote/kernel/interpreter/sql/

Changes in Parser.scala:

• Changed from listener pattern to visitor pattern due to Spark 3.3+ ANTLR changes
• Updated class name: SqlBaseBaseListener → SqlBaseParserBaseVisitor

Changes in SparkSqlInterpreter.scala:

• Updated import: SqlBaseBaseVisitor → SqlBaseParserBaseVisitor

Reason: Spark 3.3+ renamed parser classes and only generates visitors (not listeners) in the ANTLR build configuration.

4. Row Encoder Compatibility

File: polynote-spark-runtime/src/main/scala/polynote/runtime/spark/reprs/SparkReprsOf.scala

Implemented reflection-based solution to support both Spark versions at runtime:

// Spark 3.5+: Uses Encoders.row(schema)
// Spark 3.3:  Uses RowEncoder.apply(schema)
val expressionEncoder = try {
  val encodersClass = Class.forName("org.apache.spark.sql.Encoders$")
  val encodersModule = encodersClass.getField("MODULE$").get(null)
  val rowMethod = encodersClass.getMethod("row", classOf[StructType])
  val encoder = rowMethod.invoke(encodersModule, prototype.schema)
  encoder.asInstanceOf[ExpressionEncoder[Row]]
} catch {
  case _: ClassCastException | _: NoSuchMethodException =>
    // Fall back to Spark 3.3 API
    val method = RowEncoder.getClass.getMethod("apply", classOf[StructType])
    method.invoke(RowEncoder, prototype.schema).asInstanceOf[ExpressionEncoder[Row]]
}

Reason: Spark 3.5+ removed RowEncoder.apply() and replaced it with Encoders.row(). Using reflection allows the same compiled binary to work with both versions.

5. Documentation (docs-site/docs/docs/installation.md)

Updated to reflect currently supported Spark versions:

Currently, Polynote supports the following Spark versions:

• Scala 2.12: Spark 3.3.4, 3.5.7 (default: 3.5.7)
• Scala 2.13: Spark 3.3.4, 3.5.7 (default: 3.5.7)

You can override the default Spark version by setting the SPARK_VERSION environment variable when building.

Compatibility

• ✅ Both Spark versions supported in the same build using runtime reflection
• ✅ Users can choose their Spark version at build time via SPARK_VERSION environment variable
• ✅ Default version is 3.5.7 for both Scala 2.12 and 2.13
• ✅ Backward compatible runtime behavior - same binary works with both Spark 3.3.4 and 3.5.7

Testing

Each Scala/Spark version combination is tested independently in CI:

• Scala 2.12 with Spark 3.3.4
• Scala 2.12 with Spark 3.5.7
• Scala 2.13 with Spark 3.3.4
• Scala 2.13 with Spark 3.5.7