Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make fast-test-all failed:3 FAILED, dekker peterson prio-inv #460

Closed
fubingting opened this issue Dec 13, 2024 · 5 comments
Closed

Make fast-test-all failed:3 FAILED, dekker peterson prio-inv #460

fubingting opened this issue Dec 13, 2024 · 5 comments

Comments

@fubingting
Copy link

cmd:Make fast-test-all
os:centos8,euler22
version:0.18.07
log:

zombie PASSED
350 PASSED
3 FAILED, dekker peterson prio-inv
2 SKIPPED, module quota
make: *** [Makefile:911: fast-test-all] Error 1

ekker at Fri Dec 13 09:49:39 CST 2024
stress-ng: 09:49:39.67 debug: [2009746] invoked with './stress-ng -v -t 1 --dekker 4 --verify --timestamp --metrics --vmstat 1' by user 0 'root'
stress-ng: 09:49:39.67 debug: [2009746] stress-ng 0.18.07
stress-ng: 09:49:39.67 debug: [2009746] system: Linux localhost 5.10.0-131.0.0.72.oe2203sp1.aarch64 #1 SMP Fri Dec 2 00:56:17 CST 2022 aarch64, gcc 10.3.1, glibc 2.34, little endian
stress-ng: 09:49:39.67 debug: [2009746] RAM total: 502.4G, RAM free: 32.2G, swap free: 0.0
stress-ng: 09:49:39.67 debug: [2009746] temporary file path: '/home/stage/root/spack-stage-stress-ng-0.18.07-dujusywvlham3isrppxds5aecadzkkgq/spack-src', filesystem type: ext2 (54109659 blocks available)
stress-ng: 09:49:39.67 debug: [2009746] 128 processors online, 128 processors configured
stress-ng: 09:49:39.67 info:  [2009746] setting to a 1 sec run per stressor
stress-ng: 09:49:39.76 debug: [2009746] CPU data cache: L1: 64K, L2: 512K, L3: 32768K
stress-ng: 09:49:39.84 debug: [2009746] cache allocate: shared cache buffer size: 131072K (LLC size x 4 NUMA nodes)
stress-ng: 09:49:39.84 info:  [2009746] dispatching hogs: 4 dekker
stress-ng: 09:49:39.84 debug: [2009746] starting stressors
stress-ng: 09:49:39.85 debug: [2010299] dekker: [2010299] started (instance 0 on CPU 124)
stress-ng: 09:49:39.85 debug: [2009746] 4 stressors started
stress-ng: 09:49:39.85 debug: [2010300] dekker: [2010300] started (instance 1 on CPU 99)
stress-ng: 09:49:39.85 debug: [2010301] dekker: [2010301] started (instance 2 on CPU 122)
stress-ng: 09:49:39.85 debug: [2010302] dekker: [2010302] started (instance 3 on CPU 115)
stress-ng: 09:49:39.85 fail:  [2010308] dekker p0: dekker mutex check failed 677 vs 676
stress-ng: 09:49:40.03 fail:  [2010309] dekker p0: dekker mutex check failed 185140 vs 185139
stress-ng: 09:49:40.04 fail:  [2010310] dekker p0: dekker mutex check failed 183359 vs 183358
stress-ng: 09:49:40.85 info:  [2010294] vmstat:   r   b      swpd      free      buff     cache   si   so     bi     bo   in   cs us sy id wa st
stress-ng: 09:49:40.85 info:  [2010294] vmstat:  19   0         0  33978932  14327160 469825000    0    0      0     44 19016 5956 14  1 85  0  0
stress-ng: 09:49:40.85 debug: [2010299] dekker: [2010299] exited (instance 0 on CPU 124)
stress-ng: 09:49:40.85 debug: [2010300] dekker: [2010300] exited (instance 1 on CPU 99)
stress-ng: 09:49:40.85 debug: [2010301] dekker: [2010301] exited (instance 2 on CPU 122)
stress-ng: 09:49:40.85 error: [2009746] dekker: [2010299] terminated with an error, exit status=2 (stressor failed)
stress-ng: 09:49:40.85 debug: [2009746] dekker: [2010299] terminated (stressor failed)
stress-ng: 09:49:40.85 debug: [2010302] dekker: [2010302] exited (instance 3 on CPU 116)
stress-ng: 09:49:40.85 error: [2009746] dekker: [2010300] terminated with an error, exit status=2 (stressor failed)
stress-ng: 09:49:40.85 debug: [2009746] dekker: [2010300] terminated (stressor failed)
stress-ng: 09:49:40.85 error: [2009746] dekker: [2010301] terminated with an error, exit status=2 (stressor failed)
stress-ng: 09:49:40.85 debug: [2009746] dekker: [2010301] terminated (stressor failed)
stress-ng: 09:49:40.85 debug: [2009746] dekker: [2010302] terminated (success)
stress-ng: 09:49:40.85 debug: [2009746] metrics-check: all stressor metrics validated and sane
stress-ng: 09:49:40.85 metrc: [2009746] stressor       bogo ops real time  usr time  sys time   bogo ops/s     bogo ops/s CPU used per       RSS Max
stress-ng: 09:49:40.85 metrc: [2009746]                           (secs)    (secs)    (secs)   (real time) (usr+sys time) instance (%)          (KB)
stress-ng: 09:49:40.85 metrc: [2009746] dekker         35438128      1.00      5.35      0.02  35432717.17     6600757.67       134.20          2504
stress-ng: 09:49:40.85 metrc: [2009746] miscellaneous metrics:
stress-ng: 09:49:40.85 metrc: [2009746] dekker                69.81 nanosecs per mutex (harmonic mean of 4 instances)
stress-ng: 09:49:40.85 info:  [2009746] skipped: 0
stress-ng: 09:49:40.85 info:  [2009746] passed: 1: dekker (1)
stress-ng: 09:49:40.85 info:  [2009746] failed: 3: dekker (3)
stress-ng: 09:49:40.85 info:  [2009746] metrics untrustworthy: 0
stress-ng: 09:49:40.85 info:  [2009746] unsuccessful run completed in 1.00 sec
dekker FAILED


peterson at Fri Dec 13 09:56:01 CST 2024
stress-ng: 09:56:01.73 debug: [2187221] invoked with './stress-ng -v -t 1 --peterson 4 --verify --timestamp --metrics --vmstat 1' by user 0 'root'
stress-ng: 09:56:01.73 debug: [2187221] stress-ng 0.18.07
stress-ng: 09:56:01.73 debug: [2187221] system: Linux localhost 5.10.0-131.0.0.72.oe2203sp1.aarch64 #1 SMP Fri Dec 2 00:56:17 CST 2022 aarch64, gcc 10.3.1, glibc 2.34, little endian
stress-ng: 09:56:01.73 debug: [2187221] RAM total: 502.4G, RAM free: 476.9G, swap free: 0.0
stress-ng: 09:56:01.73 debug: [2187221] temporary file path: '/home/stage/root/spack-stage-stress-ng-0.18.07-dujusywvlham3isrppxds5aecadzkkgq/spack-src', filesystem type: ext2 (54109656 blocks available)
stress-ng: 09:56:01.73 debug: [2187221] 128 processors online, 128 processors configured
stress-ng: 09:56:01.73 info:  [2187221] setting to a 1 sec run per stressor
stress-ng: 09:56:01.81 debug: [2187221] CPU data cache: L1: 64K, L2: 512K, L3: 32768K
stress-ng: 09:56:01.89 debug: [2187221] cache allocate: shared cache buffer size: 131072K (LLC size x 4 NUMA nodes)
stress-ng: 09:56:01.89 info:  [2187221] dispatching hogs: 4 peterson
stress-ng: 09:56:01.89 debug: [2187221] starting stressors
stress-ng: 09:56:01.89 debug: [2187223] peterson: [2187223] started (instance 0 on CPU 79)
stress-ng: 09:56:01.89 debug: [2187221] 4 stressors started
stress-ng: 09:56:01.89 debug: [2187224] peterson: [2187224] started (instance 1 on CPU 80)
stress-ng: 09:56:01.89 debug: [2187225] peterson: [2187225] started (instance 2 on CPU 81)
stress-ng: 09:56:01.89 debug: [2187226] peterson: [2187226] started (instance 3 on CPU 82)
stress-ng: 09:56:01.92 fail:  [2187226] peterson p1: peterson mutex check failed 24524 vs 24525
stress-ng: 09:56:01.92 debug: [2187226] peterson: [2187226] exited (instance 3 on CPU 82)
stress-ng: 09:56:01.94 fail:  [2187224] peterson p1: peterson mutex check failed 26536 vs 26537
stress-ng: 09:56:01.94 debug: [2187224] peterson: [2187224] exited (instance 1 on CPU 80)
stress-ng: 09:56:02.89 info:  [2187222] vmstat:   r   b      swpd      free      buff     cache   si   so     bi     bo   in   cs us sy id wa st
stress-ng: 09:56:02.89 info:  [2187222] vmstat:   5   0         0 499910916    106844  20567268    0    0      0      0 1509  433  3  0 97  0  0
stress-ng: 09:56:02.89 debug: [2187223] peterson: [2187223] exited (instance 0 on CPU 0)
stress-ng: 09:56:02.89 debug: [2187225] peterson: [2187225] exited (instance 2 on CPU 33)
stress-ng: 09:56:02.89 debug: [2187221] peterson: [2187223] terminated (success)
stress-ng: 09:56:02.89 error: [2187221] peterson: [2187224] terminated with an error, exit status=2 (stressor failed)
stress-ng: 09:56:02.89 debug: [2187221] peterson: [2187224] terminated (stressor failed)
stress-ng: 09:56:02.89 debug: [2187221] peterson: [2187225] terminated (success)
stress-ng: 09:56:02.89 error: [2187221] peterson: [2187226] terminated with an error, exit status=2 (stressor failed)
stress-ng: 09:56:02.89 debug: [2187221] peterson: [2187226] terminated (stressor failed)
stress-ng: 09:56:02.89 debug: [2187221] metrics-check: all stressor metrics validated and sane
stress-ng: 09:56:02.89 metrc: [2187221] stressor       bogo ops real time  usr time  sys time   bogo ops/s     bogo ops/s CPU used per       RSS Max
stress-ng: 09:56:02.89 metrc: [2187221]                           (secs)    (secs)    (secs)   (real time) (usr+sys time) instance (%)          (KB)
stress-ng: 09:56:02.89 metrc: [2187221] peterson        6651051      0.52      4.14      0.00  12838693.65     1607226.25       199.70          4432
stress-ng: 09:56:02.89 metrc: [2187221] miscellaneous metrics:
stress-ng: 09:56:02.89 metrc: [2187221] peterson             193.97 nanosecs per mutex (harmonic mean of 4 instances)
stress-ng: 09:56:02.89 info:  [2187221] skipped: 0
stress-ng: 09:56:02.89 info:  [2187221] passed: 2: peterson (2)
stress-ng: 09:56:02.89 info:  [2187221] failed: 2: peterson (2)
stress-ng: 09:56:02.89 info:  [2187221] metrics untrustworthy: 0
stress-ng: 09:56:02.89 info:  [2187221] unsuccessful run completed in 1.00 sec
peterson FAILED


prio-inv at Fri Dec 13 09:57:09 CST 2024
stress-ng: 09:57:09.71 debug: [2212168] invoked with './stress-ng -v -t 1 --prio-inv 4 --verify --timestamp --metrics --vmstat 1' by user 0 'root'
stress-ng: 09:57:09.71 debug: [2212168] stress-ng 0.18.07
stress-ng: 09:57:09.71 debug: [2212168] system: Linux localhost 5.10.0-131.0.0.72.oe2203sp1.aarch64 #1 SMP Fri Dec 2 00:56:17 CST 2022 aarch64, gcc 10.3.1, glibc 2.34, little endian
stress-ng: 09:57:09.71 debug: [2212168] RAM total: 502.4G, RAM free: 476.8G, swap free: 0.0
stress-ng: 09:57:09.71 debug: [2212168] temporary file path: '/home/stage/root/spack-stage-stress-ng-0.18.07-dujusywvlham3isrppxds5aecadzkkgq/spack-src', filesystem type: ext2 (54109656 blocks available)
stress-ng: 09:57:09.71 debug: [2212168] 128 processors online, 128 processors configured
stress-ng: 09:57:09.71 info:  [2212168] setting to a 1 sec run per stressor
stress-ng: 09:57:09.78 debug: [2212168] CPU data cache: L1: 64K, L2: 512K, L3: 32768K
stress-ng: 09:57:09.86 debug: [2212168] cache allocate: shared cache buffer size: 131072K (LLC size x 4 NUMA nodes)
stress-ng: 09:57:09.86 info:  [2212168] dispatching hogs: 4 prio-inv
stress-ng: 09:57:09.86 debug: [2212168] starting stressors
stress-ng: 09:57:09.86 debug: [2212168] 4 stressors started
stress-ng: 09:57:09.86 debug: [2212170] prio-inv: [2212170] started (instance 0 on CPU 84)
stress-ng: 09:57:09.86 debug: [2212171] prio-inv: [2212171] started (instance 1 on CPU 85)
stress-ng: 09:57:09.86 debug: [2212172] prio-inv: [2212172] started (instance 2 on CPU 86)
stress-ng: 09:57:09.86 debug: [2212173] prio-inv: [2212173] started (instance 3 on CPU 87)
stress-ng: 09:57:09.86 fail:  [2212175] prio-inv: cannot set scheduling priority to 1 and policy, errno=1 (Operation not permitted)
stress-ng: 09:57:09.86 fail:  [2212176] prio-inv: cannot set scheduling priority to 1 and policy, errno=1 (Operation not permitted)
stress-ng: 09:57:09.86 fail:  [2212177] prio-inv: cannot set scheduling priority to 1 and policy, errno=1 (Operation not permitted)
stress-ng: 09:57:09.86 fail:  [2212171] prio-inv: cannot set scheduling priority to 99 and policy, errno=1 (Operation not permitted)
stress-ng: 09:57:09.86 fail:  [2212170] prio-inv: cannot set scheduling priority to 99 and policy, errno=1 (Operation not permitted)
stress-ng: 09:57:09.86 fail:  [2212172] prio-inv: cannot set scheduling priority to 99 and policy, errno=1 (Operation not permitted)
stress-ng: 09:57:09.86 fail:  [2212180] prio-inv: cannot set scheduling priority to 49 and policy, errno=1 (Operation not permitted)
stress-ng: 09:57:09.86 fail:  [2212173] prio-inv: cannot set scheduling priority to 99 and policy, errno=1 (Operation not permitted)
stress-ng: 09:57:09.86 fail:  [2212181] prio-inv: cannot set scheduling priority to 97 and policy, errno=1 (Operation not permitted)
stress-ng: 09:57:09.86 fail:  [2212184] prio-inv: cannot set scheduling priority to 97 and policy, errno=1 (Operation not permitted)
stress-ng: 09:57:09.86 debug: [2212170] prio-inv: [2212170] exited (instance 0 on CPU 87)
stress-ng: 09:57:09.86 debug: [2212171] prio-inv: [2212171] exited (instance 1 on CPU 82)
stress-ng: 09:57:09.86 fail:  [2212185] prio-inv: cannot set scheduling priority to 97 and policy, errno=1 (Operation not permitted)
stress-ng: 09:57:09.86 debug: [2212172] prio-inv: [2212172] exited (instance 2 on CPU 94)
stress-ng: 09:57:09.86 error: [2212168] prio-inv: [2212170] terminated with an error, exit status=2 (stressor failed)
stress-ng: 09:57:09.86 debug: [2212168] prio-inv: [2212170] terminated (stressor failed)
stress-ng: 09:57:09.86 error: [2212168] prio-inv: [2212171] terminated with an error, exit status=2 (stressor failed)
stress-ng: 09:57:09.86 debug: [2212168] prio-inv: [2212171] terminated (stressor failed)
stress-ng: 09:57:09.86 error: [2212168] prio-inv: [2212172] terminated with an error, exit status=2 (stressor failed)
stress-ng: 09:57:09.86 debug: [2212168] prio-inv: [2212172] terminated (stressor failed)
stress-ng: 09:57:09.86 debug: [2212173] prio-inv: [2212173] exited (instance 3 on CPU 87)
stress-ng: 09:57:09.86 error: [2212168] prio-inv: [2212173] terminated with an error, exit status=2 (stressor failed)
stress-ng: 09:57:09.86 debug: [2212168] prio-inv: [2212173] terminated (stressor failed)
stress-ng: 09:57:09.86 debug: [2212168] metrics-check: all stressor metrics validated and sane
stress-ng: 09:57:09.86 metrc: [2212168] stressor       bogo ops real time  usr time  sys time   bogo ops/s     bogo ops/s CPU used per       RSS Max
stress-ng: 09:57:09.86 metrc: [2212168]                           (secs)    (secs)    (secs)   (real time) (usr+sys time) instance (%)          (KB)
stress-ng: 09:57:09.86 metrc: [2212168] prio-inv              0      0.00      0.01      0.00         0.00           0.00       192.81          2580
stress-ng: 09:57:09.86 info:  [2212168] skipped: 0
stress-ng: 09:57:09.86 info:  [2212168] passed: 0
stress-ng: 09:57:09.86 info:  [2212168] failed: 4: prio-inv (4)
stress-ng: 09:57:09.86 info:  [2212168] metrics untrustworthy: 0
stress-ng: 09:57:09.86 info:  [2212168] unsuccessful run completed in 0 secs
prio-inv FAILED
@ColinIanKing
Copy link
Owner

Hi, do you have any details of the specific ARM cpu being used?

@ColinIanKing
Copy link
Owner

Pushed commit 8c8bbc7 to address the prio-inv issue:

@ColinIanKing
Copy link
Owner

I suspect the NUMA aspect of the ARM system and the fact that memory ordering can't be guaranteed is the root cause to the dekker and peterson failures.

@ColinIanKing
Copy link
Owner

I've pushed commit 0ac281b to use a full system memory barrier, for the dekker and peterson stressors, do you mind testing these?

@ColinIanKing
Copy link
Owner

I'm marking this as closed, if this is still an issue please re-open it or file a new report.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ColinIanKing @fubingting