HIVE-28421: Iceberg: mvn test can not run UTs in iceberg-cacatlog

[zhangbutao]

What changes were proposed in this pull request?

Added a missing junit5 dependency. Refer to https://stackoverflow.com/questions/46360736/tests-not-running-through-maven

Why are the changes needed?

Maybe this issue was caused by the mixed use of junit4&junit5.
For example if i want to do a mvn test locally:
mvn clean test -Dtest=TestHiveCommits -pl iceberg/iceberg-catalog -Piceberg

will print Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, and acutually the test won't be ran.

Does this PR introduce any user-facing change?

No

Is the change a dependency upgrade?

No

How was this patch tested?

Local mvn test in iceberg-catalog moudle

accidental click

[deniskuzZ]

LGTM +1, pending tests

[zhangbutao]

It's weird that UTs in iceberg-catalog are failed with ClassNotFoundException: org.apache.iceberg.mr.hive.HiveIcebergStorageHandler , bu i can run the UTs on my local develop env by cmd mvn test -Dtest=TestHiveCatalog-pl iceberg/iceberg-catalog -Piceberg

-------------------------------------------------------------------------------
Test set: org.apache.iceberg.hive.TestHiveCatalog
-------------------------------------------------------------------------------
Tests run: 106, Failures: 0, Errors: 32, Skipped: 11, Time elapsed: 41.97 s <<< FAILURE! - in org.apache.iceberg.hive.TestHiveCatalog
org.apache.iceberg.hive.TestHiveCatalog.testConcurrentReplaceTransactionPartitionSpec  Time elapsed: 2.614 s  <<< ERROR!
java.lang.RuntimeException: Error checking storage handler class
	at org.apache.iceberg.hive.HiveOperationsBase.isHiveIcebergStorageHandler(HiveOperationsBase.java:109)
	at org.apache.iceberg.hive.HiveTableOperations.setHmsTableParameters(HiveTableOperations.java:345)
	at org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:222)
	at org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:135)
	at org.apache.iceberg.BaseTransaction.lambda$commitReplaceTransaction$1(BaseTransaction.java:381)
	at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413)
	at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:219)
	at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:203)
	at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196)
	at org.apache.iceberg.BaseTransaction.commitReplaceTransaction(BaseTransaction.java:365)
	at org.apache.iceberg.BaseTransaction.commitTransaction(BaseTransaction.java:310)
	at org.apache.iceberg.catalog.CatalogTests.testConcurrentReplaceTransactionPartitionSpec(CatalogTests.java:2382)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at java.util.ArrayList.forEach(ArrayList.java:1259)
	at java.util.ArrayList.forEach(ArrayList.java:1259)
Caused by: java.lang.ClassNotFoundException: org.apache.iceberg.mr.hive.HiveIcebergStorageHandler
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:264)
	at org.apache.iceberg.hive.HiveOperationsBase.isHiveIcebergStorageHandler(HiveOperationsBase.java:106)
	... 14 more

[deniskuzZ]

It's weird that UTs in iceberg-catalog are failed with ClassNotFoundException: org.apache.iceberg.mr.hive.HiveIcebergStorageHandler , bu i can run the UTs on my local develop env by cmd mvn test -Dtest=TestHiveCatalog-pl iceberg/iceberg-catalog -Piceberg

-------------------------------------------------------------------------------
Test set: org.apache.iceberg.hive.TestHiveCatalog
-------------------------------------------------------------------------------
Tests run: 106, Failures: 0, Errors: 32, Skipped: 11, Time elapsed: 41.97 s <<< FAILURE! - in org.apache.iceberg.hive.TestHiveCatalog
org.apache.iceberg.hive.TestHiveCatalog.testConcurrentReplaceTransactionPartitionSpec  Time elapsed: 2.614 s  <<< ERROR!
java.lang.RuntimeException: Error checking storage handler class
	at org.apache.iceberg.hive.HiveOperationsBase.isHiveIcebergStorageHandler(HiveOperationsBase.java:109)
	at org.apache.iceberg.hive.HiveTableOperations.setHmsTableParameters(HiveTableOperations.java:345)
	at org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:222)
	at org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:135)
	at org.apache.iceberg.BaseTransaction.lambda$commitReplaceTransaction$1(BaseTransaction.java:381)
	at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413)
	at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:219)
	at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:203)
	at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196)
	at org.apache.iceberg.BaseTransaction.commitReplaceTransaction(BaseTransaction.java:365)
	at org.apache.iceberg.BaseTransaction.commitTransaction(BaseTransaction.java:310)
	at org.apache.iceberg.catalog.CatalogTests.testConcurrentReplaceTransactionPartitionSpec(CatalogTests.java:2382)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at java.util.ArrayList.forEach(ArrayList.java:1259)
	at java.util.ArrayList.forEach(ArrayList.java:1259)
Caused by: java.lang.ClassNotFoundException: org.apache.iceberg.mr.hive.HiveIcebergStorageHandler
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:264)
	at org.apache.iceberg.hive.HiveOperationsBase.isHiveIcebergStorageHandler(HiveOperationsBase.java:106)
	... 14 more

HiveIcebergStorageHandler is in iceberg-handler package, I am not sure how was it even working

[zhangbutao]

I have checked previous CI, and found that the CI never ran the unit tests for iceberg-catalog module. So any new added UT in iceberg-catalog won't be ran, and the CI will always dispaly green though the ut in iceberg-catalog may be bad.

This PR can solve the issue and the CI can run the UTs in iceberg-catalog, but some UTs need the dependency iceberg-handler, just like the error Caused by: java.lang.ClassNotFoundException: org.apache.iceberg.mr.hive.HiveIcebergStorageHandler.

Here is my thought:

1. One way is adding iceberg-handler dependency in iceberg-catalog pom, but iceberg-handler has already added iceberg-catalog as its dependncy, so this way will fail with cyclic dependency problem;
2. Another way is migrate all UTs in iceberg-catalog to iceberg-handler moudle, and then write new ut about iceberg-catalog in iceberg-handler. This way may work but looks strange :(

[deniskuzZ]

I have checked previous CI, and found that the CI never ran the unit tests for iceberg-catalog module. So any new added UT in iceberg-catalog won't be ran, and the CI will always dispaly green though the ut in iceberg-catalog may be bad.

This PR can solve the issue and the CI can run the UTs in iceberg-catalog, but some UTs need the dependency iceberg-handler, just like the error Caused by: java.lang.ClassNotFoundException: org.apache.iceberg.mr.hive.HiveIcebergStorageHandler.

Here is my thought:

1. One way is adding iceberg-handler dependency in iceberg-catalog pom, but iceberg-handler has already added iceberg-catalog as its dependncy, so this way will fail with cyclic dependency problem;
2. Another way is migrate all UTs in iceberg-catalog to iceberg-handler moudle, and then write new ut about iceberg-catalog in iceberg-handler. This way may work but looks strange :(

should we migrate all the catalog tests or just a few, also we might disable that isHiveIcebergStorageHandler check by setting 'hiveEngineEnabled'=false or setting SH explicitly in META_TABLE_STORAGE tbl prop

[sonarcloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[zhangbutao]

@deniskuzZ Thanks for your additional change!!!