Skip to content

HADOOP-19616. Add bswap support for RISC-V #7809

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jul 23, 2025

Conversation

leiwen2025
Copy link
Contributor

Description of PR

While building Hadoop native code on the riscv64 architecture, the build fails due to missing support for bswap operations. The standard implementation relies on compiler intrinsics or platform-specific assembly which are not defined for RISC-V targets.

This results in compilation errors such as:

Error: unrecognized opcode `bswap a4'
Error: unrecognized opcode `bswap a3'

Add bswap support for RISC-V can solve the problem.

How was this patch tested?

mvn package -Pdist,native -DskipTests -Dtar

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 45m 31s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
-1 ❌ mvninstall 80m 13s /branch-mvninstall-root.txt root in trunk failed.
+1 💚 compile 1m 1s trunk passed
+1 💚 mvnsite 0m 33s trunk passed
+1 💚 shadedclient 122m 7s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 20s the patch passed
+1 💚 compile 0m 51s the patch passed
+1 💚 cc 0m 51s the patch passed
+1 💚 golang 0m 51s the patch passed
+1 💚 javac 0m 51s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 0m 21s the patch passed
+1 💚 shadedclient 40m 18s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 50s hadoop-mapreduce-client-nativetask in the patch passed.
+1 💚 asflicense 0m 37s The patch does not generate ASF License warnings.
214m 11s
Subsystem Report/Notes
Docker ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7809/1/artifact/out/Dockerfile
GITHUB PR #7809
Optional Tests dupname asflicense compile cc mvnsite javac unit codespell detsecrets golang
uname Linux bb3bde6d0dbd 5.15.0-143-generic #153-Ubuntu SMP Fri Jun 13 19:10:45 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 3c622c7
Default Java Red Hat, Inc.-1.8.0_412-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7809/1/testReport/
Max. process+thread count 533 (vs. ulimit of 5500)
modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7809/1/console
versions git=2.9.5 maven=3.6.3
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 45m 35s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
-1 ❌ mvninstall 45m 25s /branch-mvninstall-root.txt root in trunk failed.
+1 💚 compile 1m 1s trunk passed
+1 💚 mvnsite 0m 34s trunk passed
+1 💚 shadedclient 87m 35s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 20s the patch passed
+1 💚 compile 0m 51s the patch passed
+1 💚 cc 0m 51s the patch passed
+1 💚 golang 0m 51s the patch passed
+1 💚 javac 0m 51s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 0m 22s the patch passed
+1 💚 shadedclient 40m 34s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 48s hadoop-mapreduce-client-nativetask in the patch passed.
+1 💚 asflicense 0m 40s The patch does not generate ASF License warnings.
180m 11s
Subsystem Report/Notes
Docker ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7809/2/artifact/out/Dockerfile
GITHUB PR #7809
Optional Tests dupname asflicense compile cc mvnsite javac unit codespell detsecrets golang
uname Linux 99b205f96d7c 5.15.0-143-generic #153-Ubuntu SMP Fri Jun 13 19:10:45 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / cf3caa6
Default Java Red Hat, Inc.-1.8.0_412-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7809/2/testReport/
Max. process+thread count 535 (vs. ulimit of 5500)
modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7809/2/console
versions git=2.9.5 maven=3.6.3
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@leiwen2025
Copy link
Contributor Author

Hi @ayushtkn,
This PR has been open for a while. Could you please help review it when you get a chance, or suggest someone who might be appropriate? Thanks a lot!

Copy link
Member

@ayushtkn ayushtkn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 52s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
-1 ❌ mvninstall 45m 19s /branch-mvninstall-root.txt root in trunk failed.
+1 💚 compile 1m 1s trunk passed
+1 💚 mvnsite 0m 33s trunk passed
+1 💚 shadedclient 86m 57s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 20s the patch passed
+1 💚 compile 0m 51s the patch passed
+1 💚 cc 0m 51s the patch passed
+1 💚 golang 0m 51s the patch passed
+1 💚 javac 0m 51s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 0m 21s the patch passed
+1 💚 shadedclient 40m 19s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 49s hadoop-mapreduce-client-nativetask in the patch passed.
+1 💚 asflicense 0m 40s The patch does not generate ASF License warnings.
134m 28s
Subsystem Report/Notes
Docker ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7809/3/artifact/out/Dockerfile
GITHUB PR #7809
Optional Tests dupname asflicense compile cc mvnsite javac unit codespell detsecrets golang
uname Linux 3d60d1ddd170 5.15.0-143-generic #153-Ubuntu SMP Fri Jun 13 19:10:45 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / cf3caa6
Default Java Red Hat, Inc.-1.8.0_412-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7809/3/testReport/
Max. process+thread count 606 (vs. ulimit of 5500)
modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7809/3/console
versions git=2.9.5 maven=3.6.3
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@ayushtkn
Copy link
Member

For some reason mvn install is failing in the CI, can u rebase your PR

@leiwen2025
Copy link
Contributor Author

For some reason mvn install is failing in the CI, can u rebase your PR

Hi, I already rebased the PR yesterday.
I noticed that the mvn install error may be due to an env issue with protoc-gen-grpc-java 1.69.0, as discussed in YARN-11794.

[ERROR] PROTOC FAILED: /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-7809/centos-7/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/target/protoc-plugins/protoc-gen-grpc-java-1.69.0-linux-x86_64.exe: /usr/lib64/libstdc++.so.6: version CXXABI_1.3.8 not found (required by /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-7809/centos-7/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/target/protoc-plugins/protoc-gen-grpc-java-1.69.0-linux-x86_64.exe)
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-7809/centos-7/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/target/protoc-plugins/protoc-gen-grpc-java-1.69.0-linux-x86_64.exe: /usr/lib64/libstdc++.so.6: version CXXABI_1.3.9 not found (required by /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-7809/centos-7/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/target/protoc-plugins/protoc-gen-grpc-java-1.69.0-linux-x86_64.exe)
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-7809/centos-7/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/target/protoc-plugins/protoc-gen-grpc-java-1.69.0-linux-x86_64.exe: /usr/lib64/libstdc++.so.6: version GLIBCXX_3.4.21 not found (required by /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-7809/centos-7/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/target/protoc-plugins/protoc-gen-grpc-java-1.69.0-linux-x86_64.exe)
--grpc-java_out: protoc-gen-grpc-java: Plugin failed with status code 1.

Could you help resolve this? Thank you!

@ayushtkn ayushtkn merged commit c357e43 into apache:trunk Jul 23, 2025
1 of 2 checks passed
@leiwen2025
Copy link
Contributor Author

LGTM

Thanks for the review and merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants