Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorrect heap usage reporting #270

Open
wenyuzhao opened this issue Mar 1, 2024 · 0 comments
Open

Incorrect heap usage reporting #270

wenyuzhao opened this issue Mar 1, 2024 · 0 comments

Comments

@wenyuzhao
Copy link
Member

MMTk GCs could not run hbase due to the incorrect or missing implementation of heap stat hooks.

Probably related to these two methods: https://github.com/mmtk/mmtk-openjdk/blob/master/openjdk/mmtkHeap.hpp#L93-L94

Steps to run hbase and reproduce the error:

wget https://archive.apache.org/dist/hbase/2.4.14/hbase-2.4.14-bin.tar.gz
tar -zxvf hbase-2.4.14-bin.tar.gz

export MMTK_PLAN=Immix 
export JAVA_HOME="MY-MMTK-JDK-HOME"
export HBASE_OPTS="-Xms1650m -Xmx1650m  -XX:+UseThirdPartyHeap "

bash ./hbase-2.4.14/bin/start-hbase.sh

After the script execution is complete, HBase did not start correctly. By examining the logs at ./hbase-2.4.14/log, the following error messages can be observed.

2024-03-01 02:05:27,911 ERROR [main] impl.MetricsSourceAdapter: Error getting metrics from source JvmMetrics
java.lang.IllegalArgumentException: committed = 9223372036854775807 should be < max = 1730150400
    at java.management/java.lang.management.MemoryUsage.<init>(MemoryUsage.java:166)
    at java.management/sun.management.MemoryImpl.getMemoryUsage0(Native Method)
    at java.management/sun.management.MemoryImpl.getHeapMemoryUsage(MemoryImpl.java:71)
    at org.apache.hadoop.metrics2.source.JvmMetrics.getMemoryUsage(JvmMetrics.java:146)
    at org.apache.hadoop.metrics2.source.JvmMetrics.getMetrics(JvmMetrics.java:138)
    at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:200)
    at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:183)
    at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMBeanInfo(MetricsSourceAdapter.java:156)
    at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getNewMBeanClassName(DefaultMBeanServerInterceptor.java:329)
    at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:315)
    at java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
    at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:66)
    at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(MetricsSourceAdapter.java:223)
    at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSourceAdapter.java:101)
    at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(MetricsSystemImpl.java:268)
    at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:233)
    at org.apache.hadoop.metrics2.source.JvmMetrics.create(JvmMetrics.java:111)
    at org.apache.hadoop.metrics2.source.JvmMetrics$Singleton.init(JvmMetrics.java:61)
    at org.apache.hadoop.metrics2.source.JvmMetrics.initSingleton(JvmMetrics.java:120)
    at org.apache.hadoop.hbase.metrics.BaseSourceImpl$DefaultMetricsSystemInitializer.init(BaseSourceImpl.java:54)
    at org.apache.hadoop.hbase.metrics.BaseSourceImpl.<init>(BaseSourceImpl.java:111)
    at org.apache.hadoop.hbase.io.MetricsIOSourceImpl.<init>(MetricsIOSourceImpl.java:42)
    at org.apache.hadoop.hbase.io.MetricsIOSourceImpl.<init>(MetricsIOSourceImpl.java:37)
    at org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactoryImpl.createIO(MetricsRegionServerSourceFactoryImpl.java:95)
    at org.apache.hadoop.hbase.io.MetricsIO.<init>(MetricsIO.java:32)
    at org.apache.hadoop.hbase.io.hfile.HFile.<clinit>(HFile.java:172)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:591)
    at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:422)
    at org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.<init>(HMasterCommandLine.java:318)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
    at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:122)
    at org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:222)
    at org.apache.hadoop.hbase.LocalHBaseCluster.<init>(LocalHBaseCluster.java:169)
    at org.apache.hadoop.hbase.LocalHBaseCluster.<init>(LocalHBaseCluster.java:112)
    at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:240)
    at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:145)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
    at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:144)
    at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2934)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant