[build] KUDU-3551 Upgrade gradle to 7.6.4

The current version of gradle is 3+ years old, and the highest Java
version supported by it is Java 13, which is ~5years old. Upgrading
Kudu's gradle version will make it possible to move from Java8 to a more
recently released stable Java version, that would futureproof the Kudu
Java client. Gradle 7.6 makes it possible to use up to Java19.
Additionally Gradle has fixed multiple security vulnerabilities and
issues since 6.8.3 [1]

*   Gradle version has been upgraded to 7.6.4 from 6.8.3, both in the
    dependencies declaration and the actual wrapper code itself
    (/java/gradlew)
*   Removed jcenter from list of repositories [2]
*   Replaced compile and testCompile configurations with implementation
    and testImplementation respectively [3]
*   Replaced extension methods with archiveExtension [4]
*   Removed scopes.gradle and the propdeps plugin as optional and
    provided scopes are no longer supported/working in Gradle7, these
    have been replaced by either compileOnly
*   Upgraded scalafmt version [5]
*   Pinned the scalafmt config to version 1.5.1 to avoid the new
    formatting errors that popped up with the new scalafmt version
*   Upgraded Spark3 version to 3.2.4 to solve Guava related compilation
    errors (some classes have been deprecated and removed, which led to
    compilation failure)
*   Updated the shadow plugin version from 6.1.0 to 7.1.2
*   Added LogCaptor to resolve issues with CapturingLogAppender, which
    stopped working in the Scala tests, fixing was taking too much time
    so this was implemented as a workaround, it only affected 4 tests
    in TestKuduBackup.scala, if the CLA errors are fixed later, the
    dependency can be removed and the test changes reverted.
*   As the legacy maven plugin was removed, the signing and publishing
    has been rewritten using the new maven-publish plugin. I tested the
    signing and publishing with a locally hosted Maven repository,
    however this needs further testing and confirmation from a commiter
    to ensure it works as expected. The new plugin uses a different
    task to publish the artifacts (publish instead of uploadArchives)
    an alias was created to keep backwards compatibility with existing
    docs/scripts.
*   The permissions for assign-location.py script were updated, as
    previously the permissions of it's symlink were copied, but that
    changed in the new gradle version and the original (non executable)
    permissions led to test failures.
*   Two tasks were updated in the shadow.gradle file as well, this
    needs another round of verification after the jars are built
    correctly
*   Dependency scopes were only modified when necessary (compilation
    failure)

Additional manual testing:
* Attila Bukor has tested and confirmed that the java artifact signing
  and publishing works.

[1] https://docs.gradle.org/7.6.3/release-notes.html?_gl=1*n7ayww*_ga*MjQ4NTk5ODI5LjE2ODc0MDc0OTA.*_ga_7W7NC6YNPT*MTcxNzc2MzIzOS44NC4xLjE3MTc3NjQyMTUuMjYuMC4w
[2] https://docs.gradle.org/6.9.2/userguide/upgrading_version_6.html?_gl=1*1xt1kk*_ga*MjQ4NTk5ODI5LjE2ODc0MDc0OTA.*_ga_7W7NC6YNPT*MTcwNzEyMzc1Ni4zLjEuMTcwNzEyODU2MS41NC4wLjA.#jcenter_deprecation
[3] https://docs.gradle.org/6.9.2/userguide/upgrading_version_5.html?_gl=1*1ldn5ls*_ga*MjQ4NTk5ODI5LjE2ODc0MDc0OTA.*_ga_7W7NC6YNPT*MTcwNzEyMzc1Ni4zLjEuMTcwNzEyODY1My40NS4wLjA.#dependencies_should_no_longer_be_declared_using_the_compile_and_runtime_configurations
[4] https://docs.gradle.org/6.9.2/dsl/org.gradle.api.tasks.bundling.AbstractArchiveTask.html?_gl=1*1ldn5ls*_ga*MjQ4NTk5ODI5LjE2ODc0MDc0OTA.*_ga_7W7NC6YNPT*MTcwNzEyMzc1Ni4zLjEuMTcwNzEyODY1My40NS4wLjA.#org.gradle.api.tasks.bundling.AbstractArchiveTask:extension
[5] alenkacz/gradle-scalafmt#39

Change-Id: I915dab011aba633d55a79c72ea6f459d703d7f47
Reviewed-on: http://gerrit.cloudera.org:8080/21030
Tested-by: Kudu Jenkins
Reviewed-by: Attila Bukor <[email protected]>

Author: zchovan

RELEASING.adoc

@@ -210,7 +210,7 @@ Example doc: https://s.apache.org/kudu1.16rn
   # Take a note of the identifier of the key you want to use for signing.
   gpg --list-secret-keys --keyid-format=short
   cd java
-  ./gradlew clean install -PforceSigning -Psigning.gnupg.keyName=<8-character-pgp-key-id>
+  ./gradlew clean assemble  -PforceSigning -Psigning.gnupg.keyName=<8-character-pgp-key-id>
 ----
 
 . Create a new version update commit which removes the -SNAPSHOT suffix (same
@@ -252,19 +252,19 @@ Example doc: https://s.apache.org/kudu1.16rn
   # Run a gpg-agent if you don't normally
   gpg-agent --daemon
   cd java
-  ./gradlew clean assemble
+  ./gradlew clean assemble -PforceSigning -Psigning.gnupg.keyName=<8-character-pgp-key-id>
   # Turn off bash history: this is to avoid exposing the credentials
   # via .bash_history file.
   set +o history
-  ./gradlew --no-parallel uploadArchives \
+  ./gradlew --no-parallel publish \
       -Psigning.gnupg.keyName=<8-character-pgp-key-id> \
       -PmavenUsername='<APACHE-LDAP-USERNAME>' \
       -PmavenPassword='<APACHE-LDAP-PASSWORD>'
   # Turn on bash history.
   set -o history
 ----
 +
-NOTE: If `uploadArchives` is executed without `--no-parallel`, uploading a
+NOTE: If `publish` is executed without `--no-parallel`, uploading a
 number of artifacts fails with "peer not authenticated" errors.
 
 . Build and deploy new binary test JARs for the RC on macOS and Linux. Build

java/.scalafmt.conf

@@ -1,3 +1,4 @@
+version = 1.5.1 // TODO: remove this and fix the formatting errors
 style = defaultWithAlign
 
 align = false

java/build.gradle

@@ -48,7 +48,6 @@ subprojects {
   // These are common to all subprojects. However, subprojects may
   // include their own plugins and scripts as well.
   apply plugin: "java"
-  apply from: "$rootDir/gradle/scopes.gradle"
   apply from: "$rootDir/gradle/compile.gradle"
   apply from: "$rootDir/gradle/tests.gradle"
   apply from: "$rootDir/gradle/quality.gradle"
@@ -58,19 +57,17 @@ subprojects {
   // Ignore the transitive annotations libraries that are
   // not marked as optional in Guava version 22.0+.
   // See https://github.com/google/guava/issues/2824
-  configurations.compile {
+  configurations.implementation {
     exclude group: "com.google.errorprone", module: "error_prone_annotations"
-    exclude group: "com.google.code.findbugs", module: "jsr305"
     exclude group: "com.google.j2objc", module: "j2objc-annotations"
     exclude group: "org.checkerframework", module: "checker-compat-qual"
     exclude group: "org.codehaus.mojo", module: "animal-sniffer-annotations"
   }
 
-  sourceSets {
-    all {
-      configurations.all { conf ->
-        attributes.attribute(Attribute.of("org.gradle.jvm.environment", String), "standard-jvm")
-      }
+  // Force gradle to use the same (declared in dependencies) guava version everywhere
+  configurations.all {
+    resolutionStrategy {
+      force libs.guava
     }
   }
 }
@@ -96,10 +93,7 @@ task javadocAggregate(type: Javadoc, group: "Documentation") {
 task copyDistTestJars(type: Copy) {
   into "$buildDir/jars/"
   from subprojects.collect {
-    it.configurations.testRuntime
-  }
-  from subprojects.collect {
-    it.configurations.provided
+    it.configurations.testRuntimeClasspath
   }
 }
 
@@ -112,3 +106,8 @@ task distTest(type: DistTestTask, dependsOn: copyDistTestJars) {
     }
   }
 }
+
+// Task alias to keep compatibility with previous Gradle version's publishing plugin
+task uploadArchives () {
+  dependsOn 'publish'
+}

java/buildSrc/build.gradle

@@ -19,28 +19,26 @@
 
 repositories {
   mavenCentral()
-  jcenter()
   maven { url "https://clojars.org/repo" } // Only used for the clojure plugin below.
   maven { url "https://plugins.gradle.org/m2/" }
 }
 
 // Manage plugin dependencies since the plugin block can't be used in included build scripts yet.
 // For more details see: https://docs.gradle.org/current/userguide/plugins.html#plugins_dsl_limitations
 dependencies {
-  compile "com.github.ben-manes:gradle-versions-plugin:0.41.0"
-  compile "com.github.jengelman.gradle.plugins:shadow:6.1.0"
-  compile "gradle.plugin.org.barfuin.gradle.jacocolog:gradle-jacoco-log:1.2.4"
-  compile "gradle.plugin.com.google.gradle:osdetector-gradle-plugin:1.7.0"
-  compile "com.google.protobuf:protobuf-gradle-plugin:0.8.18"
-  compile "com.netflix.nebula:nebula-clojure-plugin:10.1.1"
-  compile "gradle.plugin.com.github.spotbugs.snom:spotbugs-gradle-plugin:4.7.0"
-  compile "gradle.plugin.cn.bestwu.gradle:propdeps-plugin:0.0.10"
-  compile "net.ltgt.gradle:gradle-errorprone-plugin:2.0.2"
-  compile "ru.vyarus:gradle-animalsniffer-plugin:1.5.4"
-  compile "com.google.code.gson:gson:2.8.9"
-  compile "cz.alenkacz:gradle-scalafmt:1.14.0"
-  compile "com.google.guava:guava:31.0.1-jre"
-  compile "me.champeau.gradle:jmh-gradle-plugin:0.5.3"
+  implementation "com.github.ben-manes:gradle-versions-plugin:0.41.0"
+  implementation "gradle.plugin.com.github.johnrengelman:shadow:7.1.2"
+  implementation "gradle.plugin.org.barfuin.gradle.jacocolog:gradle-jacoco-log:1.2.4"
+  implementation "gradle.plugin.com.google.gradle:osdetector-gradle-plugin:1.7.0"
+  implementation "com.google.protobuf:protobuf-gradle-plugin:0.8.18"
+  implementation "com.netflix.nebula:nebula-clojure-plugin:10.1.1"
+  implementation "gradle.plugin.com.github.spotbugs.snom:spotbugs-gradle-plugin:4.7.0"
+  implementation "net.ltgt.gradle:gradle-errorprone-plugin:2.0.2"
+  implementation "ru.vyarus:gradle-animalsniffer-plugin:1.5.4"
+  implementation "com.google.code.gson:gson:2.8.9"
+  implementation "gradle.plugin.cz.alenkacz:gradle-scalafmt:1.16.2"
+  implementation "com.google.guava:guava:31.0.1-jre"
+  implementation "me.champeau.gradle:jmh-gradle-plugin:0.5.3"
 }
 
 // Compiler configuration

java/buildSrc/src/main/groovy/org/apache/kudu/gradle/DistTestTask.java

@@ -116,6 +116,10 @@ public FileCollection getInputClasses() {
     return fc;
   }
 
+  public File getOutputDir() {
+    return this.outputDir;
+  }
+
   @TaskAction
   public void doStuff() throws IOException {
     getProject().delete(outputDir);

java/gradle/artifacts.gradle

@@ -23,29 +23,29 @@ configurations.create("test")
 task testJar(type: Jar, dependsOn: testClasses, group: "Build") {
   description = "Assembles a jar archive containing the test classes."
   from sourceSets.test.output
-  classifier = "tests"
-  extension "jar"
+  archiveClassifier = "tests"
+  archiveExtension = "jar"
 }
 
 task sourcesJar(type: Jar, dependsOn: classes, group: "Build") {
   description = "Assembles a jar archive containing the main source."
   from sourceSets.main.allSource
-  classifier "sources"
-  extension "jar"
+  archiveClassifier = "sources"
+  archiveExtension = "jar"
 }
 
 task testSourcesJar(type: Jar, dependsOn: testJar, group: "Build") {
   description = "Assembles a jar archive containing the test source."
   from sourceSets.test.allSource
-  classifier "test-sources"
-  extension "jar"
+  archiveClassifier = "test-sources"
+  archiveExtension = "jar"
 }
 
 task javadocJar(type: Jar, dependsOn: javadoc, group: "Build") {
   description = "Assembles a jar archive containing the javadoc."
   from javadoc.destinationDir
-  classifier "javadoc"
-  extension "jar"
+  archiveClassifier = "javadoc"
+  archiveExtension = "jar"
 }
 
 tasks.withType(Jar) {

java/gradle/dependencies.gradle

@@ -32,7 +32,7 @@ versions += [
     commonsIo      : "2.11.0",
     errorProne     : "2.3.3",
     errorProneJavac: "9+181-r4173-1",
-    gradle         : "6.9.2",
+    gradle         : "7.6.4",
     guava          : "32.1.1-jre",
     hadoop         : "3.3.1",
     hamcrest       : "2.2",
@@ -46,6 +46,7 @@ versions += [
     jsr305         : "3.0.2",
     junit          : "4.13.2",
     log4j          : "2.17.1",
+    logCaptor      : "2.9.2",
     micrometer     : "1.8.2",
     mockito        : "4.2.0",
     murmur         : "1.0.0",
@@ -59,7 +60,7 @@ versions += [
     scopt          : "4.0.1",
     slf4j          : "1.7.33",
     spark2         : "2.4.8",
-    spark          : "3.1.2",
+    spark          : "3.2.4",
     spotBugs       : "4.1.1",
     yetus          : "0.13.0"
 ]
@@ -112,6 +113,7 @@ libs += [
     log4jCompat          : "org.apache.logging.log4j:log4j-1.2-api:$versions.log4j",
     log4jCore            : "org.apache.logging.log4j:log4j-core:$versions.log4j",
     log4jSlf4jImpl       : "org.apache.logging.log4j:log4j-slf4j-impl:$versions.log4j",
+    logCaptor            : "io.github.hakky54:logcaptor:$versions.logCaptor",
     micrometerCore       : "io.micrometer:micrometer-core:$versions.micrometer",
     mockitoCore          : "org.mockito:mockito-core:$versions.mockito",
     murmur               : "com.sangupta:murmur:$versions.murmur",

java/gradle/publishing.gradle

@@ -17,7 +17,7 @@
 
 // This file contains common tasks and configuration for artifact publishing.
 
-apply plugin: "maven"
+apply plugin: "maven-publish"
 apply plugin: "signing"
 
 ext {
@@ -40,51 +40,57 @@ ext {
   mavenPassword = propertyWithDefault("mavenPassword", "")
 }
 
-uploadArchives {
-  repositories {
-    signing {
-      required { shouldSign }
-      // Check if we are going to sign, because CI environments may not have
-      // gpg on their path and useGpgCmd evaluates eagerly.
-      if (shouldSign) {
-        useGpgCmd() // Use gpg-agent to sign
+publishing {
+  publications {
+    maven(MavenPublication) {
+      from components.java
+
+      groupId "org.apache.kudu"
+      version "${project.version}"
+
+      artifact(sourcesJar) {
+        classifier = "sources"
       }
-      sign configurations.archives
-      mavenDeployer {
-        beforeDeployment { MavenDeployment deployment -> signing.signPom(deployment) }
-        repository(url: "${mavenPublishUrl}") {
-          authentication(userName: "${mavenUsername}", password: "${mavenPassword}")
-        }
-        afterEvaluate {
-          pom.project {
-            url = "$url"
-            licenses {
-              license {
-                name = "The Apache Software License, Version 2.0"
-                url = "http://www.apache.org/licenses/LICENSE-2.0.txt"
-                distribution = "repo"
-              }
-            }
+      artifact(testSourcesJar) {
+        classifier = "testSources"
+      }
+      artifact(javadocJar) {
+        classifier = "javadoc"
+      }
+
+      pom {
+        url = "$url"
+        licenses {
+          license {
+            name = "The Apache License, Version 2.0"
+            url = "https://www.apache.org/licenses/LICENSE-2.0.txt"
+            distribution = "repo"
           }
         }
       }
     }
   }
-}
-
-// Add the install task to the "Upload" group so it's visible in the tasks output.
-install.group = "Upload"
-
-// Sort the generated maven dependencies to make pom comparisons easier.
-tasks.withType(Upload) {
-  def installer = install.repositories.mavenInstaller
-  def deployer = uploadArchives.repositories.mavenDeployer
 
-  [installer, deployer]*.pom*.whenConfigured { pom ->
-    pom.dependencies = pom.dependencies.sort { dep ->
-      "$dep.scope:$dep.optional:$dep.groupId:$dep.artifactId"
+  repositories {
+    maven {
+      url "${mavenPublishUrl}"
+      credentials {
+        username "${mavenUsername}"
+        password "${mavenPassword}"
+      }
     }
   }
 }
 
+signing {
+  required { shouldSign }
+  if (shouldSign) {
+    useGpgCmd() // Use gpg-agent to sign
+  }
+
+  sign configurations.archives
+}
+
+// Add the install task to the "Upload" group so it's visible in the tasks output.
+publish.group = "Upload"
 

java/gradle/quality.gradle

@@ -40,7 +40,7 @@ if (!propertyExists("skipCodeStyleChecks")) {
 
   checkstyle {
     toolVersion = versions.checkstyle
-    configDir = file("$rootProject.projectDir/config/checkstyle")
+    configDirectory = file("$rootProject.projectDir/config/checkstyle")
     ignoreFailures = ignoreCheckFailures
     maxWarnings = 0
     showViolations = true