[SPARK-44113][BUILD][INFRA][DOCS] Drop support for Scala 2.12

### What changes were proposed in this pull request?
The main purpose of this PR is to remove support for Scala 2.12 in Apache Spark 4.0, the specific work includes:

1. Run `dev/change-scala-version.sh 2.13` to change the Scala version in all `pom.xml` files to 2.13
2. Cleaned up the parent `pom.xml` configuration related to Scala 2.12 and set the Scala 2.13 configuration as the default
3. Cleaned up the Scala 2.12 configuration in `SparkBuild.scala`
4. Cleaned up the support for Scala 2.12 in `python/run-tests.py` and `dev/run-tests.py`
5. Updated the sections related to Scala 2.12 in `docs/building-spark.md`, `docs/index.md`,`docs/spark-connect-overview.md`,`docs/storage-openstack-swift.md`,`docs/_config.yml` to Scala 2.13
6. Updated the parts related to Scala 2.12 in `dev/test-dependencies.sh`, `dev/scalafmt`, `dev/mima`,`dev/lint-scala` to Scala 2.13
7. Removed the support for Scala 2.12 in `dev/change-scala-version.sh` and cleaned up its invocation in Spark code
8. Updated `dev/deps/spark-deps-hadoop-3-hive-2.3` and `LICENSE-binary`
9. Removed task `scala-213` from `build_and_test.yml` because the daily test of other branches will not run this task, and the master branch already uses Scala 2.13 by default.
10. Replaced Scala 2.12 with Scala 2.13 in the `name` of the following `.yml` files: `build_ansi.yml`, `build_coverage.yml`, `build_java11.yml`,`build_java17.yml`, `build_java21.yml`,`build_maven.yml`, and `build_rockdb_as_ui_backend.yml`
11. Removed the support for Scala 2.12 in `benchmark.yml`
12. Moved files from `src/scala-2.13/` directory to `src/main/scala` and deleted all files in the `src/scala-2.12/` directory.
13. Comment out the code related to multiple Scala versions in `load-spark-env.cmd` and `load-spark-env.sh`.
14. Clean up redundant `build-helper-maven-plugin` configurations from `core/pom.xml`, `repl/pom.xml`, `sql/api/pom.xml`, `sql/catalyst/pom.xml`, `sql/core/pom.xml` and `connector/connect/common/pom.xml`

### Why are the changes needed?
The minimum supported Scala version for Apache Spark 4.0 is Scala 2.13.

### Does this PR introduce _any_ user-facing change?
Yes, Apache will no longer support Scala 2.12

### How was this patch tested?
Pass GitHub Actions

### Was this patch authored or co-authored using generative AI tooling?
No

Closes apache#43008 from LuciferYang/SPARK-44113.

Lead-authored-by: yangjie01 <[email protected]>
Co-authored-by: YangJie <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>

Author: LuciferYang

.github/workflows/benchmark.yml

@@ -31,9 +31,9 @@ on:
         required: true
         default: '8'
       scala:
-        description: 'Scala version: 2.12 or 2.13'
+        description: 'Scala version: 2.13'
         required: true
-        default: '2.12'
+        default: '2.13'
       failfast:
         description: 'Failfast: true or false'
         required: true
@@ -170,7 +170,6 @@ jobs:
         key: tpcds-${{ hashFiles('.github/workflows/benchmark.yml', 'sql/core/src/test/scala/org/apache/spark/sql/TPCDSSchema.scala') }}
     - name: Run benchmarks
       run: |
-        dev/change-scala-version.sh ${{ github.event.inputs.scala }}
         ./build/sbt -Pscala-${{ github.event.inputs.scala }} -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Pkinesis-asl -Pspark-ganglia-lgpl Test/package
         # Make less noisy
         cp conf/log4j2.properties.template conf/log4j2.properties
@@ -181,8 +180,6 @@ jobs:
           --jars "`find . -name '*-SNAPSHOT-tests.jar' -o -name '*avro*-SNAPSHOT.jar' | paste -sd ',' -`" \
           "`find . -name 'spark-core*-SNAPSHOT-tests.jar'`" \
           "${{ github.event.inputs.class }}"
-        # Revert to default Scala version to clean up unnecessary git diff
-        dev/change-scala-version.sh 2.12
         # To keep the directory structure and file permissions, tar them
         # See also https://github.com/actions/upload-artifact#maintaining-file-permissions-and-case-sensitive-files
         echo "Preparing the benchmark results:"

.github/workflows/build_and_test.yml

@@ -86,7 +86,7 @@ jobs:
           sparkr=`./dev/is-changed.py -m sparkr`
           tpcds=`./dev/is-changed.py -m sql`
           docker=`./dev/is-changed.py -m docker-integration-tests`
-          # 'build', 'scala-213', and 'java-other-versions' are always true for now.
+          # 'build' and 'java-other-versions' are always true for now.
           # It does not save significant time and most of PRs trigger the build.
           precondition="
             {
@@ -95,7 +95,6 @@ jobs:
               \"sparkr\": \"$sparkr\",
               \"tpcds-1g\": \"$tpcds\",
               \"docker-integration-tests\": \"$docker\",
-              \"scala-213\": \"true\",
               \"java-other-versions\": \"true\",
               \"lint\" : \"true\",
               \"k8s-integration-tests\" : \"true\",
@@ -828,53 +827,6 @@ jobs:
         ./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Djava.version=${JAVA_VERSION/-ea} install
         rm -rf ~/.m2/repository/org/apache/spark
 
-  scala-213:
-    needs: precondition
-    if: fromJson(needs.precondition.outputs.required).scala-213 == 'true'
-    name: Scala 2.13 build with SBT
-    runs-on: ubuntu-22.04
-    timeout-minutes: 300
-    steps:
-    - name: Checkout Spark repository
-      uses: actions/checkout@v3
-      with:
-        fetch-depth: 0
-        repository: apache/spark
-        ref: ${{ inputs.branch }}
-    - name: Sync the current branch with the latest in Apache Spark
-      if: github.repository != 'apache/spark'
-      run: |
-        git fetch https://github.com/$GITHUB_REPOSITORY.git ${GITHUB_REF#refs/heads/}
-        git -c user.name='Apache Spark Test Account' -c user.email='[email protected]' merge --no-commit --progress --squash FETCH_HEAD
-        git -c user.name='Apache Spark Test Account' -c user.email='[email protected]' commit -m "Merged commit" --allow-empty
-    - name: Cache Scala, SBT and Maven
-      uses: actions/cache@v3
-      with:
-        path: |
-          build/apache-maven-*
-          build/scala-*
-          build/*.jar
-          ~/.sbt
-        key: build-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
-        restore-keys: |
-          build-
-    - name: Cache Coursier local repository
-      uses: actions/cache@v3
-      with:
-        path: ~/.cache/coursier
-        key: scala-213-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
-        restore-keys: |
-          scala-213-coursier-
-    - name: Install Java 8
-      uses: actions/setup-java@v3
-      with:
-        distribution: zulu
-        java-version: 8
-    - name: Build with SBT
-      run: |
-        ./dev/change-scala-version.sh 2.13
-        ./build/sbt -Pyarn -Pmesos -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pkinesis-asl -Pdocker-integration-tests -Pkubernetes-integration-tests -Pspark-ganglia-lgpl -Pscala-2.13 compile Test/compile
-
   # Any TPC-DS related updates on this job need to be applied to tpcds-1g-gen job of benchmark.yml as well
   tpcds-1g:
     needs: precondition

.github/workflows/build_ansi.yml

@@ -17,7 +17,7 @@
 # under the License.
 #
 
-name: "Build / ANSI (master, Hadoop 3, JDK 8, Scala 2.12)"
+name: "Build / ANSI (master, Hadoop 3, JDK 8, Scala 2.13)"
 
 on:
   schedule:

.github/workflows/build_coverage.yml

@@ -17,7 +17,7 @@
 # under the License.
 #
 
-name: "Build / Coverage (master, Scala 2.12, Hadoop 3, JDK 8)"
+name: "Build / Coverage (master, Scala 2.13, Hadoop 3, JDK 8)"
 
 on:
   schedule:

.github/workflows/build_java11.yml

@@ -17,7 +17,7 @@
 # under the License.
 #
 
-name: "Build (master, Scala 2.12, Hadoop 3, JDK 11)"
+name: "Build (master, Scala 2.13, Hadoop 3, JDK 11)"
 
 on:
   schedule:

.github/workflows/build_java17.yml

@@ -17,7 +17,7 @@
 # under the License.
 #
 
-name: "Build (master, Scala 2.12, Hadoop 3, JDK 17)"
+name: "Build (master, Scala 2.13, Hadoop 3, JDK 17)"
 
 on:
   schedule:

.github/workflows/build_java21.yml

@@ -17,7 +17,7 @@
 # under the License.
 #
 
-name: "Build (master, Scala 2.12, Hadoop 3, JDK 21)"
+name: "Build (master, Scala 2.13, Hadoop 3, JDK 21)"
 
 on:
   schedule:

.github/workflows/build_maven.yml

@@ -17,7 +17,7 @@
 # under the License.
 #
 
-name: "Build using Maven (master, Scala 2.12, Hadoop 3, JDK 8)"
+name: "Build using Maven (master, Scala 2.13, Hadoop 3, JDK 8)"
 
 on:
   schedule:

.github/workflows/build_rockdb_as_ui_backend.yml

@@ -17,7 +17,7 @@
 # under the License.
 #
 
-name: "Build / RocksDB as UI Backend (master, Hadoop 3, JDK 8, Scala 2.12)"
+name: "Build / RocksDB as UI Backend (master, Hadoop 3, JDK 8, Scala 2.13)"
 
 on:
   schedule:

LICENSE-binary

@@ -209,34 +209,34 @@ org.apache.zookeeper:zookeeper
 oro:oro
 commons-configuration:commons-configuration
 commons-digester:commons-digester
-com.chuusai:shapeless_2.12
+com.chuusai:shapeless_2.13
 com.googlecode.javaewah:JavaEWAH
 com.twitter:chill-java
-com.twitter:chill_2.12
+com.twitter:chill_2.13
 com.univocity:univocity-parsers
 javax.jdo:jdo-api
 joda-time:joda-time
 net.sf.opencsv:opencsv
 org.apache.derby:derby
 org.objenesis:objenesis
 org.roaringbitmap:RoaringBitmap
-org.scalanlp:breeze-macros_2.12
-org.scalanlp:breeze_2.12
-org.typelevel:macro-compat_2.12
+org.scalanlp:breeze-macros_2.13
+org.scalanlp:breeze_2.13
+org.typelevel:macro-compat_2.13
 org.yaml:snakeyaml
 org.apache.xbean:xbean-asm7-shaded
 com.squareup.okhttp3:logging-interceptor
 com.squareup.okhttp3:okhttp
 com.squareup.okio:okio
-org.apache.spark:spark-catalyst_2.12
-org.apache.spark:spark-kvstore_2.12
-org.apache.spark:spark-launcher_2.12
-org.apache.spark:spark-mllib-local_2.12
-org.apache.spark:spark-network-common_2.12
-org.apache.spark:spark-network-shuffle_2.12
-org.apache.spark:spark-sketch_2.12
-org.apache.spark:spark-tags_2.12
-org.apache.spark:spark-unsafe_2.12
+org.apache.spark:spark-catalyst_2.13
+org.apache.spark:spark-kvstore_2.13
+org.apache.spark:spark-launcher_2.13
+org.apache.spark:spark-mllib-local_2.13
+org.apache.spark:spark-network-common_2.13
+org.apache.spark:spark-network-shuffle_2.13
+org.apache.spark:spark-sketch_2.13
+org.apache.spark:spark-tags_2.13
+org.apache.spark:spark-unsafe_2.13
 commons-httpclient:commons-httpclient
 com.vlkan:flatbuffers
 com.ning:compress-lzf
@@ -299,10 +299,10 @@ org.apache.orc:orc-mapreduce
 org.mortbay.jetty:jetty
 org.mortbay.jetty:jetty-util
 com.jolbox:bonecp
-org.json4s:json4s-ast_2.12
-org.json4s:json4s-core_2.12
-org.json4s:json4s-jackson_2.12
-org.json4s:json4s-scalap_2.12
+org.json4s:json4s-ast_2.13
+org.json4s:json4s-core_2.13
+org.json4s:json4s-jackson_2.13
+org.json4s:json4s-scalap_2.13
 com.carrotsearch:hppc
 com.fasterxml.jackson.core:jackson-annotations
 com.fasterxml.jackson.core:jackson-core
@@ -312,7 +312,7 @@ com.fasterxml.jackson.jaxrs:jackson-jaxrs-base
 com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider
 com.fasterxml.jackson.module:jackson-module-jaxb-annotations
 com.fasterxml.jackson.module:jackson-module-paranamer
-com.fasterxml.jackson.module:jackson-module-scala_2.12
+com.fasterxml.jackson.module:jackson-module-scala_2.13
 com.github.mifmif:generex
 com.google.code.findbugs:jsr305
 com.google.code.gson:gson
@@ -385,8 +385,8 @@ org.eclipse.jetty:jetty-xml
 org.scala-lang:scala-compiler
 org.scala-lang:scala-library
 org.scala-lang:scala-reflect
-org.scala-lang.modules:scala-parser-combinators_2.12
-org.scala-lang.modules:scala-xml_2.12
+org.scala-lang.modules:scala-parser-combinators_2.13
+org.scala-lang.modules:scala-xml_2.13
 com.github.joshelser:dropwizard-metrics-hadoop-metrics2-reporter
 com.zaxxer.HikariCP
 org.apache.hive:hive-beeline
@@ -471,19 +471,19 @@ MIT License
 -----------
 
 com.microsoft.sqlserver:mssql-jdbc
-org.typelevel:spire_2.12
-org.typelevel:spire-macros_2.12
-org.typelevel:spire-platform_2.12
-org.typelevel:spire-util_2.12
-org.typelevel:algebra_2.12:jar
-org.typelevel:cats-kernel_2.12
-org.typelevel:machinist_2.12
+org.typelevel:spire_2.13
+org.typelevel:spire-macros_2.13
+org.typelevel:spire-platform_2.13
+org.typelevel:spire-util_2.13
+org.typelevel:algebra_2.13:jar
+org.typelevel:cats-kernel_2.13
+org.typelevel:machinist_2.13
 net.razorvine:pickle
 org.slf4j:jcl-over-slf4j
 org.slf4j:jul-to-slf4j
 org.slf4j:slf4j-api
 org.slf4j:slf4j-log4j12
-com.github.scopt:scopt_2.12
+com.github.scopt:scopt_2.13
 dev.ludovic.netlib:blas
 dev.ludovic.netlib:arpack
 dev.ludovic.netlib:lapack