Search before asking
Motivation
paimon-spark-4.0 takes the shared Spark test bases from paimon-spark-ut's test-jar, and that jar is compiled against the spark4 profile baseline, currently Spark 4.1.2. Test bytecode built against 4.1.2 can name Spark symbols that a 4.0.3 runtime does not have, and then the class fails to link when the 4.0 suite loads it.
The workaround for that is already in the tree. paimon-spark/paimon-spark-4.0/src/test/scala/org/apache/spark/sql/streaming/StreamTestCheckAnswerWithTimeoutStub.scala declares eight empty classes whose compiled names match StreamTest inner classes that exist only from Spark 4.1 on, so that test discovery can verify the shared classes on a 4.0 runtime. In the current baseline build, 15 classfiles in paimon-spark-ut's test-jar carry references to those eight names, none of them from anything a Paimon test calls: scalac emits them into the constant pool through the mix-in forwarders of StreamTest.
Two things are unsatisfying about it. It is per-symbol, so it has to be extended by hand whenever the baseline moves and the shared sources reach something else that 4.0 lacks. And it only holds while no test actually invokes the missing member, which nothing checks.
Solution
Compile the shared test sources a second time, against 4.0.3, and let paimon-spark-4.0 consume that build:
- add a
paimon-spark-ut-4.0 module that owns no sources of its own, adds ../paimon-spark-ut/src/test/{scala,java} through build-helper, pins every Spark artifact to 4.0.3, and produces a test-jar only;
- point
paimon-spark-4.0 at that test-jar instead of paimon-spark-ut's;
- delete the stub file.
A class compiled against 4.0.3 cannot reference a symbol that 4.0.3 does not have, so there is nothing left for the stub to stand in for. paimon-spark-4.1 needs none of this while the baseline is 4.1.x, and keeps consuming paimon-spark-ut directly.
Anything else?
The same mechanism is what #9265 needs for both 4.0 and 4.1 once the baseline moves to 4.2, but the 4.0 half stands on its own and is worth landing separately: it removes an existing workaround and changes how one CI lane compiles its tests, which is easier to review apart from a baseline bump.
Are you willing to submit a PR?
Search before asking
Motivation
paimon-spark-4.0takes the shared Spark test bases frompaimon-spark-ut's test-jar, and that jar is compiled against thespark4profile baseline, currently Spark 4.1.2. Test bytecode built against 4.1.2 can name Spark symbols that a 4.0.3 runtime does not have, and then the class fails to link when the 4.0 suite loads it.The workaround for that is already in the tree.
paimon-spark/paimon-spark-4.0/src/test/scala/org/apache/spark/sql/streaming/StreamTestCheckAnswerWithTimeoutStub.scaladeclares eight empty classes whose compiled names matchStreamTestinner classes that exist only from Spark 4.1 on, so that test discovery can verify the shared classes on a 4.0 runtime. In the current baseline build, 15 classfiles inpaimon-spark-ut's test-jar carry references to those eight names, none of them from anything a Paimon test calls: scalac emits them into the constant pool through the mix-in forwarders ofStreamTest.Two things are unsatisfying about it. It is per-symbol, so it has to be extended by hand whenever the baseline moves and the shared sources reach something else that 4.0 lacks. And it only holds while no test actually invokes the missing member, which nothing checks.
Solution
Compile the shared test sources a second time, against 4.0.3, and let
paimon-spark-4.0consume that build:paimon-spark-ut-4.0module that owns no sources of its own, adds../paimon-spark-ut/src/test/{scala,java}throughbuild-helper, pins every Spark artifact to 4.0.3, and produces a test-jar only;paimon-spark-4.0at that test-jar instead ofpaimon-spark-ut's;A class compiled against 4.0.3 cannot reference a symbol that 4.0.3 does not have, so there is nothing left for the stub to stand in for.
paimon-spark-4.1needs none of this while the baseline is 4.1.x, and keeps consumingpaimon-spark-utdirectly.Anything else?
The same mechanism is what #9265 needs for both 4.0 and 4.1 once the baseline moves to 4.2, but the 4.0 half stands on its own and is worth landing separately: it removes an existing workaround and changes how one CI lane compiles its tests, which is easier to review apart from a baseline bump.
Are you willing to submit a PR?