HDDS-15593. Fix recursive Ratis protobuf import rewriting

[peterxcli]

What changes were proposed in this pull request?

Fix protobuf/gRPC generated-source rewriting in hdds-interface-client and hdds-interface-server.

These modules generate Ratis protocol classes that must use Ratis-shaded packages, for example:

org.apache.ratis.thirdparty.io.grpc
org.apache.ratis.thirdparty.com.google.protobuf
org.apache.ratis.thirdparty.com.google.common

The previous Ant <replace dir="..."> configuration did not reliably rewrite nested generated Java files under:

target/generated-sources/proto-java-for-ratis/org/apache/hadoop/...

This patch changes the replacement rules to use explicit recursive <fileset> entries over **/*.java.

Reproduction

A targeted clean build can fail before this patch:

mvn clean
mvn clean test-compile -pl hadoop-ozone/s3gateway -am

Example error:

XceiverClientProtocolServiceGrpc.java:[3,22] package io.grpc does not exist
IntraDatanodeProtocolServiceGrpc.java:[7,26] package io.grpc.stub.annotations does not exist

This may be hidden by a prior full build or local cache, because downstream module-only tests can resolve already-installed snapshot artifacts from ~/.m2.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15593

How was this patch tested?

(Please explain how this patch was tested. Ex: unit tests, manual tests, workflow run on the fork git repo.)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this.)

[adoroszlai]

Reproduction

A targeted clean build can fail before this patch:

mvn -pl :hdds-interface-client clean compile \
  -DskipTests -DskipShade -DskipRecon -DskipDocs

Note: must have dependencies of hdds-interface-client installed:

mvn -am -pl :hdds-interface-client -DskipTests clean install
rm -fr ~/.m2/repository/org/apache/ozone/hdds-interface-client

I cannot reproduce the problem with your command on:

Apache Maven 3.9.11 (3e54c93a704957b63ee3494413a2b544fd3d825b)
Java version: 21.0.11, vendor: Ubuntu, runtime: /usr/lib/jvm/java-21-openjdk-amd64

Do you have different versions? Do you have some AI agent working in parallel on the same source root?

[peterxcli]

@adoroszlai Thanks for checking out!

I cannot reproduce the problem with your command on:

sorry the reproduction command is (updated in PR description):

mvn clean
mvn clean test-compile -pl hadoop-ozone/s3gateway -am

Do you have different versions?

my mvn --version:

Apache Maven 3.9.16 (2bdd9fddda4b155ebf8000e807eb73fd829a51d5)
Maven home: /opt/homebrew/Cellar/maven/3.9.16/libexec
Java version: 21.0.6, vendor: Azul Systems, Inc., runtime: /Users/lixucheng/.sdkman/candidates/java/21.0.6-zulu/zulu-21.jdk/Contents/Home
Default locale: en_TW, platform encoding: UTF-8
OS name: "mac os x", version: "26.5", arch: "aarch64", family: "mac"

Do you have some AI agent working in parallel on the same source root?

Actually I was using zed's UI to directly run single test case, for example, the auto generate command by zed is:

/bin/sh -i -c 'package="org.apache.hadoop.ozone.s3.endpoint"; outer=""; inner="TestObjectDelete"; method="testDelete"; sep="$"; if [ -n "$outer" ]; then c="$outer$sep$inner"; else c="$inner"; fi; if [ -n "$package" ]; then fqc="$package.$c"; else fqc="$c"; fi; f="/Users/lixucheng/Documents/oss/apache/ozone/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectDelete.java"; p="$PWD"; d=$(dirname "${f#$p/}"); if [ -f pom.xml ]; then m="."; md="$d"; while [ "$md" != "." ] && [ "$md" != "/" ]; do if [ -f "$md/pom.xml" ]; then m="$md"; break; fi; md=$(dirname "$md"); done; [ -f ./mvnw ] && CMD="./mvnw" || CMD="mvn"; if [ "$m" = "." ]; then $CMD clean test -Dtest="$fqc#$method"; else $CMD clean test-compile -pl "$m" -am && $CMD test -pl "$m" -Dtest="$fqc#$method"; fi; elif [ -f build.gradle ] || [ -f build.gradle.kts ] || [ -f settings.gradle ] || [ -f settings.gradle.kts ]; then m="."; md="$d"; while [ "$md" != "." ] && [ "$md" != "/" ]; do if [ -f "$md/build.gradle" ] || [ -f "$md/build.gradle.kts" ]; then m="$md"; break; fi; md=$(dirname "$md"); done; if [ "$m" = "." ]; then mp=""; else mp=":$(echo "$m" | tr '/' ':')"; fi; [ -f ./gradlew ] && CMD="./gradlew" || CMD="gradle"; $CMD ${mp}:test --tests "$fqc.$method"; else >&2 echo 'No build tool found'; exit 1; fi;'

[jojochuang]

cc @yandrey321 might be related to #10030

[adoroszlai]

sorry the reproduction command is (updated in PR description):

mvn clean
mvn clean test-compile -pl hadoop-ozone/s3gateway -am

Sorry, that doesn't fail either (even after rm -fr ~/.m2/repository/org/apache/ozone).

Actually I was using zed's UI to directly run single test case, for example, the auto generate command by zed is:

... $CMD clean test-compile -pl "$m" -am && $CMD test -pl "$m" -Dtest="$fqc#$method" ...

mvn test executed separately will look for depedent modules in the Maven repo. It does not pick up code generated by the prior separate clean test-compile. So without prior mvn install and without using released version of Ozone, it will fail, but not with the message you quoted in the PR description.

[yandrey321]

I'm not sure that adding a filter would fix anything here. As far as I understand the fix is to disable incremental build for the parts that do shading/renaming.

[peterxcli]

Sorry, that doesn't fail either (even after rm -fr ~/.m2/repository/org/apache/ozone).

@adoroszlai maybe

mvn clean test-compile -pl hadoop-ozone/s3gateway -am -Ddevelocity.cache.local.enabled=false

[adoroszlai]

maybe

mvn clean test-compile -pl hadoop-ozone/s3gateway -am -Ddevelocity.cache.local.enabled=false

Nope, [INFO] BUILD SUCCESS.

[adoroszlai]

As far as I understand the fix is to disable incremental build for the parts that do shading/renaming.

1. We already have useIncrementalCompilation=false set globally:
   

   ozone/pom.xml

   Line 1830 in 48613a7

   <useIncrementalCompilation>false</useIncrementalCompilation>

2. useIncrementalCompilation behavior may not be what you expect, please see MCOMPILER-209 for long discussion and the documentation added in MCOMPILER-428:

• true (default) in this mode the compiler plugin determines whether any JAR files the current module depends on have changed in the current build run; or any source file was added, removed or changed since the last compilation. If this is the case, the compiler plugin recompiles all sources
• false (not recommended) this only compiles source files which are newer than their corresponding class files, namely which have changed since the last compilation. This does not recompile other classes which use the changed class, potentially leaving them with references to methods that no longer exist, leading to errors at runtime

I don't know if the description is accurate or not, but you might want to experiment with setting it to true.