Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

invalid bkey u64s 8 type accounting: fsck cannot repair [6f03ad7] #735

Open
jpsollie opened this issue Aug 25, 2024 · 18 comments
Open

invalid bkey u64s 8 type accounting: fsck cannot repair [6f03ad7] #735

jpsollie opened this issue Aug 25, 2024 · 18 comments

Comments

@jpsollie
Copy link
Contributor

following up on issue #732 :
mounting / running bcachefs fsck complains about bkey accounting errors:

janpieter@linuxserver ~ $ dmesg | grep bkey
[   36.401288] [ T4322] invalid bkey u64s 8 type accounting 0:1:3 len 0 ver 442687733807186000: dev_data_type dev=0 data_type=free 125945 0 0
[   36.401297] [ T4322] invalid bkey u64s 8 type accounting 0:2:3 len 0 ver 442687733807186075: dev_data_type dev=0 data_type=free 125853 0 0
[   36.401300] [ T4322] invalid bkey u64s 8 type accounting 0:3:3 len 0 ver 442687733807185925: dev_data_type dev=0 data_type=free 125945 0 0
[   36.401303] [ T4322] invalid bkey u64s 8 type accounting 0:4:3 len 0 ver 442687729512218704: dev_data_type dev=0 data_type=free 125749 0 0
[   36.401306] [ T4322] invalid bkey u64s 8 type accounting 0:5:3 len 0 ver 442687725217251486: dev_data_type dev=0 data_type=free 125841 0 0
[   36.401309] [ T4322] invalid bkey u64s 8 type accounting 0:6:3 len 0 ver 442687729512218629: dev_data_type dev=0 data_type=free 125749 0 0
[   36.401311] [ T4322] invalid bkey u64s 8 type accounting 0:7:3 len 0 ver 442653593112169038: dev_data_type dev=0 data_type=free 14955 0 0
[   36.401314] [ T4322] invalid bkey u64s 8 type accounting 0:8:3 len 0 ver 442653593112181959: dev_data_type dev=0 data_type=free 33682 0 0
[   36.401317] [ T4322] invalid bkey u64s 8 type accounting 0:9:3 len 0 ver 442653593112162252: dev_data_type dev=0 data_type=free 6022 0 0
[   36.401320] [ T4322] invalid bkey u64s 8 type accounting 0:10:3 len 0 ver 442653593112162600: dev_data_type dev=0 data_type=free 6022 0 0
[   36.401323] [ T4322] invalid bkey u64s 8 type accounting 0:11:3 len 0 ver 442653593112164949: dev_data_type dev=0 data_type=free 6022 0 0
[   36.401326] [ T4322] invalid bkey u64s 8 type accounting 0:12:3 len 0 ver 442653593112158610: dev_data_type dev=0 data_type=free 35817 0 0
[   36.401329] [ T4322] invalid bkey u64s 8 type accounting 0:13:3 len 0 ver 442653593112174432: dev_data_type dev=0 data_type=free 33447 0 0
[   36.401332] [ T4322] invalid bkey u64s 8 type accounting 0:256:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   36.401335] [ T4322] invalid bkey u64s 8 type accounting 0:257:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   36.401338] [ T4322] invalid bkey u64s 8 type accounting 0:258:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   36.401341] [ T4322] invalid bkey u64s 8 type accounting 0:259:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   36.401343] [ T4322] invalid bkey u64s 8 type accounting 0:260:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   36.401346] [ T4322] invalid bkey u64s 8 type accounting 0:261:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   36.401348] [ T4322] invalid bkey u64s 8 type accounting 0:262:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   36.401351] [ T4322] invalid bkey u64s 8 type accounting 0:263:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   36.401354] [ T4322] invalid bkey u64s 8 type accounting 0:264:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   36.401356] [ T4322] invalid bkey u64s 8 type accounting 0:265:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   36.401359] [ T4322] invalid bkey u64s 8 type accounting 0:266:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   36.401361] [ T4322] invalid bkey u64s 8 type accounting 0:267:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   36.401364] [ T4322] invalid bkey u64s 8 type accounting 0:268:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   36.401367] [ T4322] invalid bkey u64s 8 type accounting 0:269:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   36.401369] [ T4322] invalid bkey u64s 8 type accounting 0:512:3 len 0 ver 0: dev_data_type dev=0 data_type=free 8192 4194304 0
[   36.401372] [ T4322] invalid bkey u64s 8 type accounting 0:513:3 len 0 ver 0: dev_data_type dev=0 data_type=free 1192 610304 0
[   36.401375] [ T4322] invalid bkey u64s 8 type accounting 0:514:3 len 0 ver 0: dev_data_type dev=0 data_type=free 1192 610304 0
[   36.401377] [ T4322] invalid bkey u64s 8 type accounting 0:515:3 len 0 ver 0: dev_data_type dev=0 data_type=free 1192 610304 0
[   36.401380] [ T4322] invalid bkey u64s 8 type accounting 0:516:3 len 0 ver 0: dev_data_type dev=0 data_type=free 1192 610304 0
[   36.401383] [ T4322] invalid bkey u64s 8 type accounting 0:517:3 len 0 ver 0: dev_data_type dev=0 data_type=free 1192 610304 0
[   36.401385] [ T4322] invalid bkey u64s 8 type accounting 0:518:3 len 0 ver 0: dev_data_type dev=0 data_type=free 1192 610304 0
[   36.401388] [ T4322] invalid bkey u64s 8 type accounting 0:519:3 len 0 ver 0: dev_data_type dev=0 data_type=free 2980 1525760 0
[   36.401390] [ T4322] invalid bkey u64s 8 type accounting 0:520:3 len 0 ver 0: dev_data_type dev=0 data_type=free 6725 3443200 0
[   36.401393] [ T4322] invalid bkey u64s 8 type accounting 0:521:3 len 0 ver 0: dev_data_type dev=0 data_type=free 1193 610816 0
[   36.401396] [ T4322] invalid bkey u64s 8 type accounting 0:522:3 len 0 ver 0: dev_data_type dev=0 data_type=free 1193 610816 0
[   36.401398] [ T4322] invalid bkey u64s 8 type accounting 0:523:3 len 0 ver 0: dev_data_type dev=0 data_type=free 1193 610816 0
[   36.401401] [ T4322] invalid bkey u64s 8 type accounting 0:524:3 len 0 ver 0: dev_data_type dev=0 data_type=free 7152 3661824 0
[   36.401404] [ T4322] invalid bkey u64s 8 type accounting 0:525:3 len 0 ver 0: dev_data_type dev=0 data_type=free 6678 3419136 0
[   36.401406] [ T4322] invalid bkey u64s 8 type accounting 0:769:3 len 0 ver 442687600663200205: dev_data_type dev=0 data_type=free 25438 13024256 0
[   36.401409] [ T4322] invalid bkey u64s 8 type accounting 0:770:3 len 0 ver 442687725217251432: dev_data_type dev=0 data_type=free 25530 13071360 0
[   36.401412] [ T4322] invalid bkey u64s 8 type accounting 0:771:3 len 0 ver 442687600663200241: dev_data_type dev=0 data_type=free 25438 13024256 0
[   36.401415] [ T4322] invalid bkey u64s 8 type accounting 0:772:3 len 0 ver 442687600663200250: dev_data_type dev=0 data_type=free 25634 13124608 0
[   36.401418] [ T4322] invalid bkey u64s 8 type accounting 0:773:3 len 0 ver 442687725217251477: dev_data_type dev=0 data_type=free 25542 13077504 0
[   36.401420] [ T4322] invalid bkey u64s 8 type accounting 0:774:3 len 0 ver 442687600663200286: dev_data_type dev=0 data_type=free 25634 13124608 0
[   36.401424] [ T4322] invalid bkey u64s 8 type accounting 0:1024:3 len 0 ver 441305334158476074: dev_data_type dev=0 data_type=free 10619150 5436976800 28000
[   36.401427] [ T4322] invalid bkey u64s 8 type accounting 0:1031:3 len 0 ver 441305334158475880: dev_data_type dev=0 data_type=free 0 0 0
[   36.401429] [ T4322] invalid bkey u64s 8 type accounting 0:1032:3 len 0 ver 441305334158476104: dev_data_type dev=0 data_type=free 0 0 0
[   36.401432] [ T4322] invalid bkey u64s 8 type accounting 0:1033:3 len 0 ver 441305334158461631: dev_data_type dev=0 data_type=free 0 0 0
[   36.401435] [ T4322] invalid bkey u64s 8 type accounting 0:1034:3 len 0 ver 441305334158469555: dev_data_type dev=0 data_type=free 0 0 0
[   36.401438] [ T4322] invalid bkey u64s 8 type accounting 0:1035:3 len 0 ver 441305329863523651: dev_data_type dev=0 data_type=free 0 0 0
[   36.401440] [ T4322] invalid bkey u64s 8 type accounting 0:1036:3 len 0 ver 441305334158474919: dev_data_type dev=0 data_type=free 0 0 0
[   36.401443] [ T4322] invalid bkey u64s 8 type accounting 0:1037:3 len 0 ver 441305334158475109: dev_data_type dev=0 data_type=free 0 0 0
[   36.401446] [ T4322] invalid bkey u64s 8 type accounting 0:1287:3 len 0 ver 442653567342376952: dev_data_type dev=0 data_type=free 363524 186124120 168
[   36.401449] [ T4322] invalid bkey u64s 8 type accounting 0:1288:3 len 0 ver 442653584522246572: dev_data_type dev=0 data_type=free 820468 420079280 336
[   36.401452] [ T4322] invalid bkey u64s 8 type accounting 0:1289:3 len 0 ver 442653532982608524: dev_data_type dev=0 data_type=free 145548 74520576 0
[   36.401455] [ T4322] invalid bkey u64s 8 type accounting 0:1290:3 len 0 ver 442653588817206817: dev_data_type dev=0 data_type=free 145548 74520336 240
[   36.401458] [ T4322] invalid bkey u64s 8 type accounting 0:1291:3 len 0 ver 442653588817184454: dev_data_type dev=0 data_type=free 145576 74534912 0
[   36.401461] [ T4322] invalid bkey u64s 8 type accounting 0:1292:3 len 0 ver 442653584522243428: dev_data_type dev=0 data_type=free 872542 446740664 840
[   36.401464] [ T4322] invalid bkey u64s 8 type accounting 0:1293:3 len 0 ver 442653584522244635: dev_data_type dev=0 data_type=free 814746 417149736 216
[   36.401467] [ T4322] invalid bkey u64s 8 type accounting 0:2049:3 len 0 ver 2885698331872639: dev_data_type dev=0 data_type=free 0 0 0
[   36.401469] [ T4322] invalid bkey u64s 8 type accounting 0:2051:3 len 0 ver 2885698331872735: dev_data_type dev=0 data_type=free 0 0 0
[   36.401472] [ T4322] invalid bkey u64s 8 type accounting 0:2305:3 len 0 ver 442687733807186009: dev_data_type dev=0 data_type=free 0 0 0
[   36.401475] [ T4322] invalid bkey u64s 8 type accounting 0:2306:3 len 0 ver 442687733807186084: dev_data_type dev=0 data_type=free 0 0 0
[   36.401478] [ T4322] invalid bkey u64s 8 type accounting 0:2307:3 len 0 ver 442687733807185934: dev_data_type dev=0 data_type=free 0 0 0
[   36.401480] [ T4322] invalid bkey u64s 8 type accounting 0:2308:3 len 0 ver 442687729512218713: dev_data_type dev=0 data_type=free 0 0 0
[   36.401483] [ T4322] invalid bkey u64s 8 type accounting 0:2309:3 len 0 ver 442687600663199908: dev_data_type dev=0 data_type=free 0 0 0
[   36.401486] [ T4322] invalid bkey u64s 8 type accounting 0:2310:3 len 0 ver 442687729512218638: dev_data_type dev=0 data_type=free 0 0 0
[   36.401489] [ T4322] invalid bkey u64s 8 type accounting 0:2311:3 len 0 ver 442653593112169047: dev_data_type dev=0 data_type=free 0 0 0
[   36.401491] [ T4322] invalid bkey u64s 8 type accounting 0:2312:3 len 0 ver 442653593112181968: dev_data_type dev=0 data_type=free 0 0 0
[   36.401494] [ T4322] invalid bkey u64s 8 type accounting 0:2313:3 len 0 ver 442653593112162261: dev_data_type dev=0 data_type=free 0 0 0
[   36.401497] [ T4322] invalid bkey u64s 8 type accounting 0:2314:3 len 0 ver 442653593112162609: dev_data_type dev=0 data_type=free 0 0 0
[   36.401500] [ T4322] invalid bkey u64s 8 type accounting 0:2315:3 len 0 ver 442653593112164958: dev_data_type dev=0 data_type=free 0 0 0
[   36.401502] [ T4322] invalid bkey u64s 8 type accounting 0:2316:3 len 0 ver 442653593112158619: dev_data_type dev=0 data_type=free 0 0 0
[   36.401505] [ T4322] invalid bkey u64s 8 type accounting 0:2317:3 len 0 ver 442653593112174441: dev_data_type dev=0 data_type=free 0 0 0
[   36.401508] [ T4322] invalid bkey u64s 6 type accounting 0:65796:2 len 0 ver 441305334158476048: replicas free: 0/0 [] 5436976800
[   36.401511] [ T4322] invalid bkey u64s 6 type accounting 0:117506308:2 len 0 ver 441305334158475807: replicas free: 0/0 [] 0
[   36.401514] [ T4322] invalid bkey u64s 6 type accounting 0:117506309:2 len 0 ver 442653567342376919: replicas free: 0/0 [] 186124120
[   36.401517] [ T4322] invalid bkey u64s 6 type accounting 0:134283524:2 len 0 ver 441305334158476010: replicas free: 0/0 [] 0
[   36.401520] [ T4322] invalid bkey u64s 6 type accounting 0:134283525:2 len 0 ver 442653584522246541: replicas free: 0/0 [] 420079280
[   36.401523] [ T4322] invalid bkey u64s 6 type accounting 0:151060740:2 len 0 ver 441305334158461537: replicas free: 0/0 [] 0
[   36.401526] [ T4322] invalid bkey u64s 6 type accounting 0:151060741:2 len 0 ver 442653532982608503: replicas free: 0/0 [] 74520576
[   36.401528] [ T4322] invalid bkey u64s 6 type accounting 0:167837956:2 len 0 ver 441305334158469467: replicas free: 0/0 [] 0
[   36.401531] [ T4322] invalid bkey u64s 6 type accounting 0:167837957:2 len 0 ver 442653588817206786: replicas free: 0/0 [] 74520336
[   36.401534] [ T4322] invalid bkey u64s 6 type accounting 0:184615172:2 len 0 ver 441305329863523557: replicas free: 0/0 [] 0
[   36.401537] [ T4322] invalid bkey u64s 6 type accounting 0:184615173:2 len 0 ver 442653588817184421: replicas free: 0/0 [] 74534912
[   36.401540] [ T4322] invalid bkey u64s 6 type accounting 0:201392388:2 len 0 ver 441305334158474831: replicas free: 0/0 [] 0
[   36.401542] [ T4322] invalid bkey u64s 6 type accounting 0:201392389:2 len 0 ver 442653584522243397: replicas free: 0/0 [] 446740664
[   36.401545] [ T4322] invalid bkey u64s 6 type accounting 0:218169604:2 len 0 ver 441305334158475021: replicas free: 0/0 [] 0
[   36.401548] [ T4322] invalid bkey u64s 6 type accounting 0:218169605:2 len 0 ver 442653584522244604: replicas free: 0/0 [] 417149736
[   36.401551] [ T4322] invalid bkey u64s 6 type accounting 0:4294967211:5 len 0 ver 441304518114696243: snapshot id=2868903936 3394160
[   36.401553] [ T4322] invalid bkey u64s 6 type accounting 0:4294967235:5 len 0 ver 0: snapshot id=3271557120 25002112
[   36.401556] [ T4322] invalid bkey u64s 6 type accounting 0:4294967253:5 len 0 ver 0: snapshot id=3573547008 12404680
[   36.401558] [ T4322] invalid bkey u64s 6 type accounting 0:4294967269:5 len 0 ver 0: snapshot id=3841982464 80738408
[   36.401561] [ T4322] invalid bkey u64s 6 type accounting 0:4294967271:5 len 0 ver 0: snapshot id=3875536896 1638808
[   36.401563] [ T4322] invalid bkey u64s 6 type accounting 0:4294967273:5 len 0 ver 0: snapshot id=3909091328 4546552
[   36.401566] [ T4322] invalid bkey u64s 6 type accounting 0:4294967295:5 len 0 ver 441272743946630842: snapshot id=4278190080 5298850368
[   36.401569] [ T4322] invalid bkey u64s 6 type accounting 0:3307141661443:2 len 0 ver 442687600663200138: replicas free: 0/0 [] 19106304
[   36.401571] [ T4322] invalid bkey u64s 6 type accounting 0:5510459884291:2 len 0 ver 442687493289017381: replicas free: 0/0 [] 19966464
[   36.401574] [ T4322] invalid bkey u64s 6 type accounting 0:6614283256579:2 len 0 ver 442687725217251365: replicas free: 0/0 [] 20107776
[   36.401577] [ T4322] invalid bkey u64s 6 type accounting 0:6618611778307:2 len 0 ver 442687725217251409: replicas free: 0/0 [] 19266048
[   37.029452] [ T4364] invalid bkey u64s 8 type accounting 0:1:3 len 0 ver 374038230678372432: dev_data_type dev=0 data_type=free 141622 0 0
[   37.029461] [ T4364] invalid bkey u64s 8 type accounting 0:2:3 len 0 ver 374038243563274451: dev_data_type dev=0 data_type=free 141682 0 0
[   37.029464] [ T4364] invalid bkey u64s 8 type accounting 0:3:3 len 0 ver 374038226383405285: dev_data_type dev=0 data_type=free 141599 0 0
[   37.029468] [ T4364] invalid bkey u64s 8 type accounting 0:4:3 len 0 ver 374038230678372357: dev_data_type dev=0 data_type=free 141748 0 0
[   37.029471] [ T4364] invalid bkey u64s 8 type accounting 0:5:3 len 0 ver 374038247858241616: dev_data_type dev=0 data_type=free 141708 0 0
[   37.029473] [ T4364] invalid bkey u64s 8 type accounting 0:6:3 len 0 ver 374038247858241541: dev_data_type dev=0 data_type=free 141827 0 0
[   37.029476] [ T4364] invalid bkey u64s 8 type accounting 0:7:3 len 0 ver 374021042219253852: dev_data_type dev=0 data_type=free 14956 0 0
[   37.029479] [ T4364] invalid bkey u64s 8 type accounting 0:8:3 len 0 ver 374021029334351877: dev_data_type dev=0 data_type=free 11500 0 0
[   37.029482] [ T4364] invalid bkey u64s 8 type accounting 0:9:3 len 0 ver 374021033629319173: dev_data_type dev=0 data_type=free 6015 0 0
[   37.029485] [ T4364] invalid bkey u64s 8 type accounting 0:10:3 len 0 ver 374021037924286761: dev_data_type dev=0 data_type=free 6015 0 0
[   37.029488] [ T4364] invalid bkey u64s 8 type accounting 0:11:3 len 0 ver 374021037924287022: dev_data_type dev=0 data_type=free 6015 0 0
[   37.029491] [ T4364] invalid bkey u64s 8 type accounting 0:12:3 len 0 ver 374021037924286674: dev_data_type dev=0 data_type=free 35817 0 0
[   37.029494] [ T4364] invalid bkey u64s 8 type accounting 0:13:3 len 0 ver 374021037924286935: dev_data_type dev=0 data_type=free 23300 0 0
[   37.029497] [ T4364] invalid bkey u64s 8 type accounting 0:256:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   37.029500] [ T4364] invalid bkey u64s 8 type accounting 0:257:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   37.029502] [ T4364] invalid bkey u64s 8 type accounting 0:258:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   37.029505] [ T4364] invalid bkey u64s 8 type accounting 0:259:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   37.029508] [ T4364] invalid bkey u64s 8 type accounting 0:260:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   37.029510] [ T4364] invalid bkey u64s 8 type accounting 0:261:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   37.029513] [ T4364] invalid bkey u64s 8 type accounting 0:262:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   37.029516] [ T4364] invalid bkey u64s 8 type accounting 0:263:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   37.029518] [ T4364] invalid bkey u64s 8 type accounting 0:264:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   37.029521] [ T4364] invalid bkey u64s 8 type accounting 0:265:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   37.029523] [ T4364] invalid bkey u64s 8 type accounting 0:266:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   37.029526] [ T4364] invalid bkey u64s 8 type accounting 0:267:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   37.029529] [ T4364] invalid bkey u64s 8 type accounting 0:268:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   37.029531] [ T4364] invalid bkey u64s 8 type accounting 0:269:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   37.029534] [ T4364] invalid bkey u64s 8 type accounting 0:512:3 len 0 ver 0: dev_data_type dev=0 data_type=free 8192 4194304 0
[   37.029536] [ T4364] invalid bkey u64s 8 type accounting 0:513:3 len 0 ver 0: dev_data_type dev=0 data_type=free 1192 610304 0
[   37.029539] [ T4364] invalid bkey u64s 8 type accounting 0:514:3 len 0 ver 0: dev_data_type dev=0 data_type=free 1192 610304 0
[   37.029542] [ T4364] invalid bkey u64s 8 type accounting 0:515:3 len 0 ver 0: dev_data_type dev=0 data_type=free 1192 610304 0
[   37.029545] [ T4364] invalid bkey u64s 8 type accounting 0:516:3 len 0 ver 0: dev_data_type dev=0 data_type=free 1192 610304 0
[   37.029547] [ T4364] invalid bkey u64s 8 type accounting 0:517:3 len 0 ver 0: dev_data_type dev=0 data_type=free 1192 610304 0
[   37.029550] [ T4364] invalid bkey u64s 8 type accounting 0:518:3 len 0 ver 0: dev_data_type dev=0 data_type=free 1192 610304 0
[   37.029552] [ T4364] invalid bkey u64s 8 type accounting 0:519:3 len 0 ver 0: dev_data_type dev=0 data_type=free 2980 1525760 0
[   37.029555] [ T4364] invalid bkey u64s 8 type accounting 0:520:3 len 0 ver 0: dev_data_type dev=0 data_type=free 2289 1171968 0
[   37.029558] [ T4364] invalid bkey u64s 8 type accounting 0:521:3 len 0 ver 0: dev_data_type dev=0 data_type=free 1192 610304 0
[   37.029560] [ T4364] invalid bkey u64s 8 type accounting 0:522:3 len 0 ver 0: dev_data_type dev=0 data_type=free 1192 610304 0
[   37.029563] [ T4364] invalid bkey u64s 8 type accounting 0:523:3 len 0 ver 0: dev_data_type dev=0 data_type=free 1192 610304 0
[   37.029565] [ T4364] invalid bkey u64s 8 type accounting 0:524:3 len 0 ver 0: dev_data_type dev=0 data_type=free 7152 3661824 0
[   37.029568] [ T4364] invalid bkey u64s 8 type accounting 0:525:3 len 0 ver 0: dev_data_type dev=0 data_type=free 4649 2380288 0
[   37.029571] [ T4364] invalid bkey u64s 8 type accounting 0:769:3 len 0 ver 374038226383405267: dev_data_type dev=0 data_type=free 9761 4997632 0
[   37.029574] [ T4364] invalid bkey u64s 8 type accounting 0:770:3 len 0 ver 374038243563274442: dev_data_type dev=0 data_type=free 9689 4960768 0
[   37.029577] [ T4364] invalid bkey u64s 8 type accounting 0:771:3 len 0 ver 374038226383405276: dev_data_type dev=0 data_type=free 9784 5009408 0
[   37.029580] [ T4364] invalid bkey u64s 8 type accounting 0:772:3 len 0 ver 374038226383405303: dev_data_type dev=0 data_type=free 9635 4933120 0
[   37.029582] [ T4364] invalid bkey u64s 8 type accounting 0:773:3 len 0 ver 374038243563274469: dev_data_type dev=0 data_type=free 9675 4953600 0
[   37.029585] [ T4364] invalid bkey u64s 8 type accounting 0:774:3 len 0 ver 374038007340073665: dev_data_type dev=0 data_type=free 9556 4892672 0
[   37.029588] [ T4364] invalid bkey u64s 8 type accounting 0:1024:3 len 0 ver 373299839900845053: dev_data_type dev=0 data_type=free 5290912 2708693226 253718
[   37.029591] [ T4364] invalid bkey u64s 8 type accounting 0:1031:3 len 0 ver 373297640877588625: dev_data_type dev=0 data_type=free 0 0 0
[   37.029594] [ T4364] invalid bkey u64s 8 type accounting 0:1032:3 len 0 ver 373219313559021630: dev_data_type dev=0 data_type=free 0 0 0
[   37.029597] [ T4364] invalid bkey u64s 8 type accounting 0:1033:3 len 0 ver 373219313559023909: dev_data_type dev=0 data_type=free 0 0 0
[   37.029599] [ T4364] invalid bkey u64s 8 type accounting 0:1034:3 len 0 ver 373219313559023485: dev_data_type dev=0 data_type=free 0 0 0
[   37.029602] [ T4364] invalid bkey u64s 8 type accounting 0:1035:3 len 0 ver 373219313559020794: dev_data_type dev=0 data_type=free 0 0 0
[   37.029605] [ T4364] invalid bkey u64s 8 type accounting 0:1036:3 len 0 ver 373219313559019166: dev_data_type dev=0 data_type=free 0 0 0
[   37.029608] [ T4364] invalid bkey u64s 8 type accounting 0:1037:3 len 0 ver 373219313559022042: dev_data_type dev=0 data_type=free 0 0 0
[   37.029611] [ T4364] invalid bkey u64s 8 type accounting 0:1287:3 len 0 ver 374021037924289465: dev_data_type dev=0 data_type=free 363519 186117472 4256
[   37.029614] [ T4364] invalid bkey u64s 8 type accounting 0:1288:3 len 0 ver 374021029334354480: dev_data_type dev=0 data_type=free 279230 142963902 1858
[   37.029617] [ T4364] invalid bkey u64s 8 type accounting 0:1289:3 len 0 ver 374021033629319755: dev_data_type dev=0 data_type=free 145368 74426706 1710
[   37.029620] [ T4364] invalid bkey u64s 8 type accounting 0:1290:3 len 0 ver 373986648121177026: dev_data_type dev=0 data_type=free 145368 74425663 2753
[   37.029623] [ T4364] invalid bkey u64s 8 type accounting 0:1291:3 len 0 ver 373986630941309102: dev_data_type dev=0 data_type=free 145368 74426247 2169
[   37.029626] [ T4364] invalid bkey u64s 8 type accounting 0:1292:3 len 0 ver 374021037924288436: dev_data_type dev=0 data_type=free 872546 446716703 26849
[   37.029629] [ T4364] invalid bkey u64s 8 type accounting 0:1293:3 len 0 ver 373986622351374795: dev_data_type dev=0 data_type=free 567134 290366840 5768
[   37.029632] [ T4364] invalid bkey u64s 8 type accounting 0:2050:3 len 0 ver 372513646137332366: dev_data_type dev=0 data_type=free 0 0 0
[   37.029635] [ T4364] invalid bkey u64s 8 type accounting 0:2051:3 len 0 ver 372513676202087631: dev_data_type dev=0 data_type=free 0 0 0
[   37.029638] [ T4364] invalid bkey u64s 8 type accounting 0:2052:3 len 0 ver 372513701971913946: dev_data_type dev=0 data_type=free 0 0 0
[   37.029640] [ T4364] invalid bkey u64s 8 type accounting 0:2054:3 len 0 ver 372513762101436092: dev_data_type dev=0 data_type=free 0 0 0
[   37.029643] [ T4364] invalid bkey u64s 8 type accounting 0:2304:3 len 0 ver 373299844195811342: dev_data_type dev=0 data_type=free 0 0 0
[   37.029646] [ T4364] invalid bkey u64s 8 type accounting 0:2305:3 len 0 ver 374038230678372441: dev_data_type dev=0 data_type=free 0 0 0
[   37.029649] [ T4364] invalid bkey u64s 8 type accounting 0:2306:3 len 0 ver 374037341620142200: dev_data_type dev=0 data_type=free 0 0 0
[   37.029652] [ T4364] invalid bkey u64s 8 type accounting 0:2307:3 len 0 ver 374038011635040345: dev_data_type dev=0 data_type=free 0 0 0
[   37.029655] [ T4364] invalid bkey u64s 8 type accounting 0:2308:3 len 0 ver 374038230678372366: dev_data_type dev=0 data_type=free 0 0 0
[   37.029657] [ T4364] invalid bkey u64s 8 type accounting 0:2309:3 len 0 ver 374038247858241625: dev_data_type dev=0 data_type=free 0 0 0
[   37.029660] [ T4364] invalid bkey u64s 8 type accounting 0:2310:3 len 0 ver 374038247858241550: dev_data_type dev=0 data_type=free 0 0 0
[   37.029663] [ T4364] invalid bkey u64s 8 type accounting 0:2311:3 len 0 ver 374021042219253861: dev_data_type dev=0 data_type=free 0 0 0
[   37.029666] [ T4364] invalid bkey u64s 8 type accounting 0:2312:3 len 0 ver 374021029334351886: dev_data_type dev=0 data_type=free 0 0 0
[   37.029669] [ T4364] invalid bkey u64s 8 type accounting 0:2313:3 len 0 ver 374021033629319182: dev_data_type dev=0 data_type=free 0 0 0
[   37.029672] [ T4364] invalid bkey u64s 8 type accounting 0:2314:3 len 0 ver 374021037924286770: dev_data_type dev=0 data_type=free 0 0 0
[   37.029675] [ T4364] invalid bkey u64s 8 type accounting 0:2315:3 len 0 ver 374021037924287031: dev_data_type dev=0 data_type=free 0 0 0
[   37.029677] [ T4364] invalid bkey u64s 8 type accounting 0:2316:3 len 0 ver 374021037924286683: dev_data_type dev=0 data_type=free 0 0 0
[   37.029680] [ T4364] invalid bkey u64s 8 type accounting 0:2317:3 len 0 ver 374021037924286944: dev_data_type dev=0 data_type=free 0 0 0
[   37.029684] [ T4364] invalid bkey u64s 6 type accounting 0:65796:2 len 0 ver 373299839900845024: replicas free: 0/0 [] 2708693226
[   37.029687] [ T4364] invalid bkey u64s 6 type accounting 0:65797:2 len 0 ver 1939319468064800: replicas free: 0/0 [] 0
[   37.029690] [ T4364] invalid bkey u64s 6 type accounting 0:83951875:2 len 0 ver 372700631833520782: replicas free: 0/0 [] 0
[   37.029693] [ T4364] invalid bkey u64s 6 type accounting 0:117506308:2 len 0 ver 373297640877588528: replicas free: 0/0 [] 0
[   37.029696] [ T4364] invalid bkey u64s 6 type accounting 0:117506309:2 len 0 ver 374021037924289422: replicas free: 0/0 [] 186117472
[   37.029699] [ T4364] invalid bkey u64s 6 type accounting 0:134283524:2 len 0 ver 373219313559021542: replicas free: 0/0 [] 0
[   37.029702] [ T4364] invalid bkey u64s 6 type accounting 0:134283525:2 len 0 ver 374021029334354447: replicas free: 0/0 [] 142963902
[   37.029705] [ T4364] invalid bkey u64s 6 type accounting 0:151060740:2 len 0 ver 373219313559023815: replicas free: 0/0 [] 0
[   37.029708] [ T4364] invalid bkey u64s 6 type accounting 0:151060741:2 len 0 ver 374021033629319722: replicas free: 0/0 [] 74426706
[   37.029711] [ T4364] invalid bkey u64s 6 type accounting 0:167837956:2 len 0 ver 373219313559023391: replicas free: 0/0 [] 0
[   37.029714] [ T4364] invalid bkey u64s 6 type accounting 0:167837957:2 len 0 ver 373986648121176995: replicas free: 0/0 [] 74425663
[   37.029717] [ T4364] invalid bkey u64s 6 type accounting 0:184615172:2 len 0 ver 373219313559020706: replicas free: 0/0 [] 0
[   37.029720] [ T4364] invalid bkey u64s 6 type accounting 0:184615173:2 len 0 ver 373986630941309081: replicas free: 0/0 [] 74426247
[   37.029723] [ T4364] invalid bkey u64s 6 type accounting 0:201392388:2 len 0 ver 373219313559019078: replicas free: 0/0 [] 0
[   37.029725] [ T4364] invalid bkey u64s 6 type accounting 0:201392389:2 len 0 ver 374021037924288403: replicas free: 0/0 [] 446716703
[   37.029728] [ T4364] invalid bkey u64s 6 type accounting 0:218169604:2 len 0 ver 373219313559021954: replicas free: 0/0 [] 0
[   37.029731] [ T4364] invalid bkey u64s 6 type accounting 0:218169605:2 len 0 ver 373986622351374764: replicas free: 0/0 [] 290366840
[   37.029734] [ T4364] invalid bkey u64s 6 type accounting 0:4294967219:5 len 0 ver 373299839900845031: snapshot id=3003121664 44280952
[   37.029737] [ T4364] invalid bkey u64s 6 type accounting 0:4294967221:5 len 0 ver 1951710448745927: snapshot id=3036676096 21248205
[   37.029740] [ T4364] invalid bkey u64s 6 type accounting 0:4294967227:5 len 0 ver 373291009448102842: snapshot id=3137339392 19776835
[   37.029743] [ T4364] invalid bkey u64s 6 type accounting 0:4294967257:5 len 0 ver 0: snapshot id=3640655872 76717129
[   37.029746] [ T4364] invalid bkey u64s 6 type accounting 0:4294967261:5 len 0 ver 374021179658220205: snapshot id=3707764736 82952572
[   37.029748] [ T4364] invalid bkey u64s 6 type accounting 0:4294967289:5 len 0 ver 373292559931305213: snapshot id=4177526784 90973371
[   37.029751] [ T4364] invalid bkey u64s 6 type accounting 0:4294967291:5 len 0 ver 373291851261698349: snapshot id=4211081216 88542598
[   37.029754] [ T4364] invalid bkey u64s 6 type accounting 0:4294967295:5 len 0 ver 373292358067826468: snapshot id=4278190080 1981147847
[   37.029757] [ T4364] invalid bkey u64s 6 type accounting 0:8606777859:2 len 0 ver 374030937823904378: replicas free: 0/0 [] 4959232
[   37.029760] [ T4364] invalid bkey u64s 6 type accounting 0:12901745155:2 len 0 ver 374024585567277787: replicas free: 0/0 [] 1812480
[   37.029763] [ T4364] invalid bkey u64s 6 type accounting 0:12918522371:2 len 0 ver 374033416020033628: replicas free: 0/0 [] 1013760
[   37.029766] [ T4364] invalid bkey u64s 6 type accounting 0:17196712451:2 len 0 ver 374038226383405210: replicas free: 0/0 [] 1582080
[   37.029769] [ T4364] invalid bkey u64s 6 type accounting 0:17213489667:2 len 0 ver 374035572093616220: replicas free: 0/0 [] 1200128
[   37.029772] [ T4364] invalid bkey u64s 6 type accounting 0:17230266883:2 len 0 ver 374024409473623422: replicas free: 0/0 [] 2609152
[   37.029775] [ T4364] invalid bkey u64s 6 type accounting 0:21491679747:2 len 0 ver 374013654875504794: replicas free: 0/0 [] 227328
[   37.029778] [ T4364] invalid bkey u64s 6 type accounting 0:21508456963:2 len 0 ver 374024340754147107: replicas free: 0/0 [] 1460224
[   37.029781] [ T4364] invalid bkey u64s 6 type accounting 0:21525234179:2 len 0 ver 374038226383405244: replicas free: 0/0 [] 3363840
[   37.029784] [ T4364] invalid bkey u64s 6 type accounting 0:21542011395:2 len 0 ver 374017365727248507: replicas free: 0/0 [] 1733632
[   37.029787] [ T4364] invalid bkey u64s 6 type accounting 0:25786647043:2 len 0 ver 374018929095344282: replicas free: 0/0 [] 1414144
[   37.029790] [ T4364] invalid bkey u64s 6 type accounting 0:25803424259:2 len 0 ver 374038243563274428: replicas free: 0/0 [] 1288192
[   37.029793] [ T4364] invalid bkey u64s 6 type accounting 0:25820201475:2 len 0 ver 374021123823632621: replicas free: 0/0 [] 1219584
[   37.029796] [ T4364] invalid bkey u64s 6 type accounting 0:25836978691:2 len 0 ver 374032230609059994: replicas free: 0/0 [] 2741248
[   37.029798] [ T4364] invalid bkey u64s 6 type accounting 0:25853755907:2 len 0 ver 374038243563274394: replicas free: 0/0 [] 3122176
[   37.645614] [ T4402] invalid bkey u64s 8 type accounting 0:1:3 len 0 ver 30846738587713647: dev_data_type dev=0 data_type=free 149806 0 0
[   37.645622] [ T4402] invalid bkey u64s 8 type accounting 0:2:3 len 0 ver 30846725702813295: dev_data_type dev=0 data_type=free 149763 0 0
[   37.645625] [ T4402] invalid bkey u64s 8 type accounting 0:3:3 len 0 ver 30846738587713572: dev_data_type dev=0 data_type=free 149767 0 0
[   37.645628] [ T4402] invalid bkey u64s 8 type accounting 0:4:3 len 0 ver 30846678458171515: dev_data_type dev=0 data_type=free 149804 0 0
[   37.645631] [ T4402] invalid bkey u64s 8 type accounting 0:5:3 len 0 ver 30846734292746412: dev_data_type dev=0 data_type=free 149792 0 0
[   37.645634] [ T4402] invalid bkey u64s 8 type accounting 0:6:3 len 0 ver 30846678458172026: dev_data_type dev=0 data_type=free 149779 0 0
[   37.645637] [ T4402] invalid bkey u64s 8 type accounting 0:7:3 len 0 ver 30846729997778949: dev_data_type dev=0 data_type=free 149778 0 0
[   37.645640] [ T4402] invalid bkey u64s 8 type accounting 0:8:3 len 0 ver 30818318789147988: dev_data_type dev=0 data_type=free 57817 0 0
[   37.645643] [ T4402] invalid bkey u64s 8 type accounting 0:9:3 len 0 ver 30818327379062585: dev_data_type dev=0 data_type=free 74017 0 0
[   37.645646] [ T4402] invalid bkey u64s 8 type accounting 0:10:3 len 0 ver 30818323084084385: dev_data_type dev=0 data_type=free 57817 0 0
[   37.645649] [ T4402] invalid bkey u64s 8 type accounting 0:11:3 len 0 ver 30818327379064042: dev_data_type dev=0 data_type=free 346806 0 0
[   37.645652] [ T4402] invalid bkey u64s 8 type accounting 0:12:3 len 0 ver 30818327379055641: dev_data_type dev=0 data_type=free 86722 0 0
[   37.645655] [ T4402] invalid bkey u64s 8 type accounting 0:256:3 len 0 ver 0: dev_data_type dev=0 data_type=free 7 6152 1016
[   37.645658] [ T4402] invalid bkey u64s 8 type accounting 0:257:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   37.645661] [ T4402] invalid bkey u64s 8 type accounting 0:258:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   37.645663] [ T4402] invalid bkey u64s 8 type accounting 0:259:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   37.645666] [ T4402] invalid bkey u64s 8 type accounting 0:260:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   37.645669] [ T4402] invalid bkey u64s 8 type accounting 0:261:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   37.645671] [ T4402] invalid bkey u64s 8 type accounting 0:262:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   37.645674] [ T4402] invalid bkey u64s 8 type accounting 0:263:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   37.645676] [ T4402] invalid bkey u64s 8 type accounting 0:264:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   37.645679] [ T4402] invalid bkey u64s 8 type accounting 0:265:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   37.645682] [ T4402] invalid bkey u64s 8 type accounting 0:266:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   37.645684] [ T4402] invalid bkey u64s 8 type accounting 0:267:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   37.645687] [ T4402] invalid bkey u64s 8 type accounting 0:268:3 len 0 ver 0: dev_data_type dev=0 data_type=free 13 6152 504
[   37.645689] [ T4402] invalid bkey u64s 8 type accounting 0:512:3 len 0 ver 0: dev_data_type dev=0 data_type=free 8192 8388608 0
[   37.645692] [ T4402] invalid bkey u64s 8 type accounting 0:513:3 len 0 ver 0: dev_data_type dev=0 data_type=free 1192 610304 0
[   37.645695] [ T4402] invalid bkey u64s 8 type accounting 0:514:3 len 0 ver 0: dev_data_type dev=0 data_type=free 1192 610304 0
[   37.645698] [ T4402] invalid bkey u64s 8 type accounting 0:515:3 len 0 ver 0: dev_data_type dev=0 data_type=free 1192 610304 0
[   37.645700] [ T4402] invalid bkey u64s 8 type accounting 0:516:3 len 0 ver 0: dev_data_type dev=0 data_type=free 1192 610304 0
[   37.645703] [ T4402] invalid bkey u64s 8 type accounting 0:517:3 len 0 ver 0: dev_data_type dev=0 data_type=free 1192 610304 0
[   37.645705] [ T4402] invalid bkey u64s 8 type accounting 0:518:3 len 0 ver 0: dev_data_type dev=0 data_type=free 1192 610304 0
[   37.645708] [ T4402] invalid bkey u64s 8 type accounting 0:519:3 len 0 ver 0: dev_data_type dev=0 data_type=free 1192 610304 0
[   37.645711] [ T4402] invalid bkey u64s 8 type accounting 0:520:3 len 0 ver 0: dev_data_type dev=0 data_type=free 1192 610304 0
[   37.645713] [ T4402] invalid bkey u64s 8 type accounting 0:521:3 len 0 ver 0: dev_data_type dev=0 data_type=free 1526 781312 0
[   37.645717] [ T4402] invalid bkey u64s 8 type accounting 0:522:3 len 0 ver 0: dev_data_type dev=0 data_type=free 1192 610304 0
[   37.645721] [ T4402] invalid bkey u64s 8 type accounting 0:523:3 len 0 ver 0: dev_data_type dev=0 data_type=free 7152 3661824 0
[   37.645725] [ T4402] invalid bkey u64s 8 type accounting 0:524:3 len 0 ver 0: dev_data_type dev=0 data_type=free 1788 915456 0
[   37.645728] [ T4402] invalid bkey u64s 8 type accounting 0:769:3 len 0 ver 30846725702813277: dev_data_type dev=0 data_type=free 1573 805376 0
[   37.645730] [ T4402] invalid bkey u64s 8 type accounting 0:770:3 len 0 ver 30846725702813286: dev_data_type dev=0 data_type=free 1620 829440 0
[   37.645733] [ T4402] invalid bkey u64s 8 type accounting 0:771:3 len 0 ver 30846734292746394: dev_data_type dev=0 data_type=free 1612 825344 0
[   37.645736] [ T4402] invalid bkey u64s 8 type accounting 0:772:3 len 0 ver 30846678458171506: dev_data_type dev=0 data_type=free 1575 806400 0
[   37.645739] [ T4402] invalid bkey u64s 8 type accounting 0:773:3 len 0 ver 30846734292746403: dev_data_type dev=0 data_type=free 1587 812544 0
[   37.645742] [ T4402] invalid bkey u64s 8 type accounting 0:774:3 len 0 ver 30846678458172017: dev_data_type dev=0 data_type=free 1600 819200 0
[   37.645745] [ T4402] invalid bkey u64s 8 type accounting 0:775:3 len 0 ver 30846725702813331: dev_data_type dev=0 data_type=free 1601 819712 0
[   37.645748] [ T4402] invalid bkey u64s 8 type accounting 0:1024:3 len 0 ver 30607788082223531: dev_data_type dev=0 data_type=free 524896 537488008 5496
[   37.645751] [ T4402] invalid bkey u64s 8 type accounting 0:1032:3 len 0 ver 30607788082220561: dev_data_type dev=0 data_type=free 0 0 0
[   37.645753] [ T4402] invalid bkey u64s 8 type accounting 0:1033:3 len 0 ver 30607788082220953: dev_data_type dev=0 data_type=free 0 0 0
[   37.645756] [ T4402] invalid bkey u64s 8 type accounting 0:1034:3 len 0 ver 30607788082220757: dev_data_type dev=0 data_type=free 0 0 0
[   37.645759] [ T4402] invalid bkey u64s 8 type accounting 0:1035:3 len 0 ver 30607788082223561: dev_data_type dev=0 data_type=free 0 0 0
[   37.645762] [ T4402] invalid bkey u64s 8 type accounting 0:1036:3 len 0 ver 30607788082222287: dev_data_type dev=0 data_type=free 0 0 0
[   37.645765] [ T4402] invalid bkey u64s 8 type accounting 0:1288:3 len 0 ver 30818323084084215: dev_data_type dev=0 data_type=free 93566 47904832 960
[   37.645768] [ T4402] invalid bkey u64s 8 type accounting 0:1289:3 len 0 ver 30818327379063872: dev_data_type dev=0 data_type=free 119800 61335872 1728
[   37.645771] [ T4402] invalid bkey u64s 8 type accounting 0:1290:3 len 0 ver 30818323084086336: dev_data_type dev=0 data_type=free 93566 47904840 952
[   37.645773] [ T4402] invalid bkey u64s 8 type accounting 0:1291:3 len 0 ver 30818327379066786: dev_data_type dev=0 data_type=free 561557 287511144 6040
[   37.645776] [ T4402] invalid bkey u64s 8 type accounting 0:1292:3 len 0 ver 30818327379056884: dev_data_type dev=0 data_type=free 140357 71862480 304
[   37.645780] [ T4402] invalid bkey u64s 8 type accounting 0:2305:3 len 0 ver 30846738587713656: dev_data_type dev=0 data_type=free 0 0 0
[   37.645782] [ T4402] invalid bkey u64s 8 type accounting 0:2306:3 len 0 ver 30846407875231833: dev_data_type dev=0 data_type=free 0 0 0
[   37.645785] [ T4402] invalid bkey u64s 8 type accounting 0:2307:3 len 0 ver 30846738587713581: dev_data_type dev=0 data_type=free 0 0 0
[   37.645788] [ T4402] invalid bkey u64s 8 type accounting 0:2308:3 len 0 ver 30846614033662072: dev_data_type dev=0 data_type=free 0 0 0
[   37.645791] [ T4402] invalid bkey u64s 8 type accounting 0:2309:3 len 0 ver 30846682753138702: dev_data_type dev=0 data_type=free 0 0 0
[   37.645793] [ T4402] invalid bkey u64s 8 type accounting 0:2310:3 len 0 ver 30846665573269518: dev_data_type dev=0 data_type=free 0 0 0
[   37.645796] [ T4402] invalid bkey u64s 8 type accounting 0:2311:3 len 0 ver 30846729997778958: dev_data_type dev=0 data_type=free 0 0 0
[   37.645799] [ T4402] invalid bkey u64s 8 type accounting 0:2315:3 len 0 ver 403400508313612: dev_data_type dev=0 data_type=free 0 0 0
[   37.645802] [ T4402] invalid bkey u64s 8 type accounting 0:2316:3 len 0 ver 403348968702233: dev_data_type dev=0 data_type=free 0 0 0
[   37.645805] [ T4402] invalid bkey u64s 6 type accounting 0:65796:2 len 0 ver 30607788082232793: replicas free: 0/0 [] 537488008
[   37.645808] [ T4402] invalid bkey u64s 6 type accounting 0:134283524:2 len 0 ver 30607788082220467: replicas free: 0/0 [] 0
[   37.645811] [ T4402] invalid bkey u64s 6 type accounting 0:134283525:2 len 0 ver 30818323084084182: replicas free: 0/0 [] 47904832
[   37.645814] [ T4402] invalid bkey u64s 6 type accounting 0:151060740:2 len 0 ver 30607788082220859: replicas free: 0/0 [] 0
[   37.645818] [ T4402] invalid bkey u64s 6 type accounting 0:151060741:2 len 0 ver 30818327379063851: replicas free: 0/0 [] 61335872
[   37.645821] [ T4402] invalid bkey u64s 6 type accounting 0:167837956:2 len 0 ver 30607788082220663: replicas free: 0/0 [] 0
[   37.645824] [ T4402] invalid bkey u64s 6 type accounting 0:167837957:2 len 0 ver 30818323084086303: replicas free: 0/0 [] 47904840
[   37.645826] [ T4402] invalid bkey u64s 6 type accounting 0:184615172:2 len 0 ver 30607788082223473: replicas free: 0/0 [] 0
[   37.645829] [ T4402] invalid bkey u64s 6 type accounting 0:184615173:2 len 0 ver 30818327379066743: replicas free: 0/0 [] 287511144
[   37.645832] [ T4402] invalid bkey u64s 6 type accounting 0:201392388:2 len 0 ver 30607788082222193: replicas free: 0/0 [] 0
[   37.645835] [ T4402] invalid bkey u64s 6 type accounting 0:201392389:2 len 0 ver 30818327379056851: replicas free: 0/0 [] 71862480
[   37.645838] [ T4402] invalid bkey u64s 6 type accounting 0:4294967219:5 len 0 ver 30607427304952260: snapshot id=3003121664 72664
[   37.645840] [ T4402] invalid bkey u64s 6 type accounting 0:4294967287:5 len 0 ver 30606851779330097: snapshot id=4143972352 310744
[   37.645843] [ T4402] invalid bkey u64s 6 type accounting 0:4294967295:5 len 0 ver 0: snapshot id=4278190080 537104600
[   37.645846] [ T4402] invalid bkey u64s 6 type accounting 0:8606777859:2 len 0 ver 30819130537935002: replicas free: 0/0 [] 152576
[   37.645848] [ T4402] invalid bkey u64s 6 type accounting 0:12901745155:2 len 0 ver 30846734292746301: replicas free: 0/0 [] 315392
[   37.645851] [ T4402] invalid bkey u64s 6 type accounting 0:12918522371:2 len 0 ver 30846725702813254: replicas free: 0/0 [] 791552
[   37.645854] [ T4402] invalid bkey u64s 6 type accounting 0:17196712451:2 len 0 ver 30846678458171456: replicas free: 0/0 [] 663552
[   37.645857] [ T4402] invalid bkey u64s 6 type accounting 0:17213489667:2 len 0 ver 30773187272785588: replicas free: 0/0 [] 77824
[   37.645859] [ T4402] invalid bkey u64s 6 type accounting 0:17230266883:2 len 0 ver 30817464090639514: replicas free: 0/0 [] 171008
[   37.645862] [ T4402] invalid bkey u64s 6 type accounting 0:21491679747:2 len 0 ver 30846734292746335: replicas free: 0/0 [] 101376
[   37.645865] [ T4402] invalid bkey u64s 6 type accounting 0:21508456963:2 len 0 ver 30846661278302334: replicas free: 0/0 [] 575488
[   37.645868] [ T4402] invalid bkey u64s 6 type accounting 0:21525234179:2 len 0 ver 30818314494148925: replicas free: 0/0 [] 125952
[   37.645870] [ T4402] invalid bkey u64s 6 type accounting 0:21542011395:2 len 0 ver 30821948036481054: replicas free: 0/0 [] 113664
[   37.645873] [ T4402] invalid bkey u64s 6 type accounting 0:25786647043:2 len 0 ver 30829459934281820: replicas free: 0/0 [] 234496
[   37.645876] [ T4402] invalid bkey u64s 6 type accounting 0:25803424259:2 len 0 ver 30810330149946331: replicas free: 0/0 [] 7168
[   37.645879] [ T4402] invalid bkey u64s 6 type accounting 0:25820201475:2 len 0 ver 30846609738694720: replicas free: 0/0 [] 115712
[   37.645882] [ T4402] invalid bkey u64s 6 type accounting 0:25836978691:2 len 0 ver 30846609738694686: replicas free: 0/0 [] 534528
[   37.645884] [ T4402] invalid bkey u64s 6 type accounting 0:25853755907:2 len 0 ver 30819207847346206: replicas free: 0/0 [] 98304
[   37.645887] [ T4402] invalid bkey u64s 6 type accounting 0:30081614339:2 len 0 ver 30846725702813220: replicas free: 0/0 [] 143360
[   37.645890] [ T4402] invalid bkey u64s 6 type accounting 0:30098391555:2 len 0 ver 30817730378624811: replicas free: 0/0 [] 54272
[   37.645892] [ T4402] invalid bkey u64s 6 type accounting 0:30115168771:2 len 0 ver 30823713268039803: replicas free: 0/0 [] 131072
[   37.645895] [ T4402] invalid bkey u64s 6 type accounting 0:30131945987:2 len 0 ver 30846304796017066: replicas free: 0/0 [] 52224
[   37.645898] [ T4402] invalid bkey u64s 6 type accounting 0:30148723203:2 len 0 ver 30839218099978394: replicas free: 0/0 [] 610304
[   37.645901] [ T4402] invalid bkey u64s 6 type accounting 0:30165500419:2 len 0 ver 30846678458171967: replicas free: 0/0 [] 648192
[  386.422466] [ T4594] invalid bkey u64s 8 type accounting 0:1:3 len 0 ver 374038230678372432: dev_data_type dev=0 data_type=free 141622 0 0
[  386.422472] [ T4594] invalid bkey u64s 8 type accounting 0:2:3 len 0 ver 374038243563274451: dev_data_type dev=0 data_type=free 141682 0 0
[  386.422475] [ T4594] invalid bkey u64s 8 type accounting 0:3:3 len 0 ver 374038226383405285: dev_data_type dev=0 data_type=free 141599 0 0
[  386.422478] [ T4594] invalid bkey u64s 8 type accounting 0:4:3 len 0 ver 374038230678372357: dev_data_type dev=0 data_type=free 141748 0 0
[  386.422481] [ T4594] invalid bkey u64s 8 type accounting 0:5:3 len 0 ver 374038247858241616: dev_data_type dev=0 data_type=free 141708 0 0
[  386.422484] [ T4594] invalid bkey u64s 8 type accounting 0:6:3 len 0 ver 374038247858241541: dev_data_type dev=0 data_type=free 141827 0 0
[  386.422487] [ T4594] invalid bkey u64s 8 type accounting 0:7:3 len 0 ver 374021042219253852: dev_data_type dev=0 data_type=free 14956 0 0
[  386.422490] [ T4594] invalid bkey u64s 8 type accounting 0:8:3 len 0 ver 374021029334351877: dev_data_type dev=0 data_type=free 11500 0 0
[  386.422493] [ T4594] invalid bkey u64s 8 type accounting 0:9:3 len 0 ver 374021033629319173: dev_data_type dev=0 data_type=free 6015 0 0
[  386.422496] [ T4594] invalid bkey u64s 8 type accounting 0:10:3 len 0 ver 374021037924286761: dev_data_type dev=0 data_type=free 6015 0 0
[  386.422499] [ T4594] invalid bkey u64s 8 type accounting 0:11:3 len 0 ver 374021037924287022: dev_data_type dev=0 data_type=free 6015 0 0
[  586.605247] [ T4680] invalid bkey u64s 8 type accounting 0:1:3 len 0 ver 374038230678372432: dev_data_type dev=0 data_type=free 141622 0 0
[  586.605253] [ T4680] invalid bkey u64s 8 type accounting 0:2:3 len 0 ver 374038243563274451: dev_data_type dev=0 data_type=free 141682 0 0
[  586.605257] [ T4680] invalid bkey u64s 8 type accounting 0:3:3 len 0 ver 374038226383405285: dev_data_type dev=0 data_type=free 141599 0 0
[  586.605260] [ T4680] invalid bkey u64s 8 type accounting 0:4:3 len 0 ver 374038230678372357: dev_data_type dev=0 data_type=free 141748 0 0
[  586.605263] [ T4680] invalid bkey u64s 8 type accounting 0:5:3 len 0 ver 374038247858241616: dev_data_type dev=0 data_type=free 141708 0 0
[  586.605266] [ T4680] invalid bkey u64s 8 type accounting 0:6:3 len 0 ver 374038247858241541: dev_data_type dev=0 data_type=free 141827 0 0
[  586.605269] [ T4680] invalid bkey u64s 8 type accounting 0:7:3 len 0 ver 374021042219253852: dev_data_type dev=0 data_type=free 14956 0 0
[  586.605272] [ T4680] invalid bkey u64s 8 type accounting 0:8:3 len 0 ver 374021029334351877: dev_data_type dev=0 data_type=free 11500 0 0
[  586.605275] [ T4680] invalid bkey u64s 8 type accounting 0:9:3 len 0 ver 374021033629319173: dev_data_type dev=0 data_type=free 6015 0 0
[  586.605278] [ T4680] invalid bkey u64s 8 type accounting 0:10:3 len 0 ver 374021037924286761: dev_data_type dev=0 data_type=free 6015 0 0
[  586.605281] [ T4680] invalid bkey u64s 8 type accounting 0:11:3 len 0 ver 374021037924287022: dev_data_type dev=0 data_type=free 6015 0 0

these "invalid bkey" errors happen at every mount / umount on every fs I have (currently I got 3 of them).
A similar issue is happening at bcachefs fsck.
I tried using GDB to see what happened, and I saw bch2_bkey_val_validate returns BCH_ERR_fsck_delete_bkey at btree_io.c:1223, so I assumed it would be deleted. This seems not true.
Any suggestions?
@jpsollie jpsollie changed the title invalid bkey u64s 8 type accounting: fsck cannot repair invalid bkey u64s 8 type accounting: fsck cannot repair [6f03ad7] Aug 25, 2024
@K900
Copy link

K900 commented Aug 25, 2024

Same thing here, also on 6.11-rc4, didn't get those on 6.11-rc1, so presumably something broke between those?

@jpsollie
Copy link
Contributor Author

Same thing here, also on 6.11-rc4, didn't get those on 6.11-rc1, so presumably something broke between those?

okay, that's something ...
I found a way which seems to work (though I do not know why):

  • run mount -o fsck,fix_errors; sleep 10; umount in a loop for 17 times
  • run mount -o fsck,fix_errors
  • run bcachefs rewrite_old_inodes
  • umount & remount with fsck again

but ... as I said ... I don't know why.

@K900
Copy link

K900 commented Aug 25, 2024

Running bcachefs data job rewrite_old_nodes gives me

[  980.251423] bcachefs (e0d3a180-f858-46c8-9186-563c88cbd165): bch2_btree_update_start(): error EIO
[  980.252748] bcachefs (e0d3a180-f858-46c8-9186-563c88cbd165): bch2_btree_update_start(): error EIO
[  980.253541] bcachefs (e0d3a180-f858-46c8-9186-563c88cbd165): bch2_btree_update_start(): error EIO
[  980.254332] bcachefs (e0d3a180-f858-46c8-9186-563c88cbd165): bch2_btree_update_start(): error EIO
[  980.255127] bcachefs (e0d3a180-f858-46c8-9186-563c88cbd165): bch2_btree_update_start(): error EIO
[  980.255944] bcachefs (e0d3a180-f858-46c8-9186-563c88cbd165): bch2_btree_update_start(): error EIO
[  980.256730] bcachefs (e0d3a180-f858-46c8-9186-563c88cbd165): bch2_btree_update_start(): error EIO
[  980.257514] bcachefs (e0d3a180-f858-46c8-9186-563c88cbd165): bch2_btree_update_start(): error EIO
[  980.258298] bcachefs (e0d3a180-f858-46c8-9186-563c88cbd165): bch2_btree_update_start(): error EIO
[  980.259082] bcachefs (e0d3a180-f858-46c8-9186-563c88cbd165): bch2_btree_update_start(): error EIO
[  980.259962] bcachefs (e0d3a180-f858-46c8-9186-563c88cbd165): bch2_move_btree(): error EIO
[  980.260685] bcachefs (e0d3a180-f858-46c8-9186-563c88cbd165): bch2_scan_old_btree_nodes(): error EIO

in dmesg and exits.

@jpsollie
Copy link
Contributor Author

Running bcachefs data job rewrite_old_nodes gives me

[  980.251423] bcachefs (e0d3a180-f858-46c8-9186-563c88cbd165): bch2_btree_update_start(): error EIO
[  980.252748] bcachefs (e0d3a180-f858-46c8-9186-563c88cbd165): bch2_btree_update_start(): error EIO
[  980.253541] bcachefs (e0d3a180-f858-46c8-9186-563c88cbd165): bch2_btree_update_start(): error EIO
[  980.254332] bcachefs (e0d3a180-f858-46c8-9186-563c88cbd165): bch2_btree_update_start(): error EIO
[  980.255127] bcachefs (e0d3a180-f858-46c8-9186-563c88cbd165): bch2_btree_update_start(): error EIO
[  980.255944] bcachefs (e0d3a180-f858-46c8-9186-563c88cbd165): bch2_btree_update_start(): error EIO
[  980.256730] bcachefs (e0d3a180-f858-46c8-9186-563c88cbd165): bch2_btree_update_start(): error EIO
[  980.257514] bcachefs (e0d3a180-f858-46c8-9186-563c88cbd165): bch2_btree_update_start(): error EIO
[  980.258298] bcachefs (e0d3a180-f858-46c8-9186-563c88cbd165): bch2_btree_update_start(): error EIO
[  980.259082] bcachefs (e0d3a180-f858-46c8-9186-563c88cbd165): bch2_btree_update_start(): error EIO
[  980.259962] bcachefs (e0d3a180-f858-46c8-9186-563c88cbd165): bch2_move_btree(): error EIO
[  980.260685] bcachefs (e0d3a180-f858-46c8-9186-563c88cbd165): bch2_scan_old_btree_nodes(): error EIO

in dmesg and exits.

yes, I assume it's a caching error. running bcachefs job without any fsck gives this error here as well, which is why I recommend all fscks even though not useful

@K900
Copy link

K900 commented Aug 25, 2024

fsck does not seem to help here :(

@jpsollie
Copy link
Contributor Author

jpsollie commented Aug 25, 2024 •
edited
Loading

@koverstreet : any ideas?

EDIT: to show exactly what I did which fixed all bcachefs volumes for me:

linuxserver /home/janpieter # echo $i
17
linuxserver /home/janpieter # while [[ $i -lt 40 ]]; do mount -o fsck,fix_errors /data/dir3; sleep 10; umount /data/dir3; i=$(($i+1)); done
Error: Device or resource busy

(I'm waiting a bit here, but I didn't have the patience for i to become 40, so I pressed ctrl-c during "sleep 10" after a while)

^C
linuxserver /home/janpieter # bcachefs data job rewrite_old_nodes /data/dir3
0% complete: current position btree accounting:18446744073709551615:18446744073709551615
Done
linuxserver /home/janpieter # umount /data/dir3
linuxserver /home/janpieter # mount -o fsck,fix_errors /data/dir3
linuxserver /home/janpieter # echo $i
24
linuxserver /home/janpieter #

@K900
Copy link

K900 commented Aug 25, 2024

OK, I think I figured out what's happening with the fix at least: every offline fsck tries to reapply the journal, runs into the broken journal entries, fixes one of them, then exits. My first offline fsck (well not really first, but first in this batch) started replaying at 243662, the following one at 243664, and seemingly so on. This is definitely an fsck bug, at the very least. Now, what is actually causing those entries to be wrong, I don't know.

@K900
Copy link

K900 commented Aug 25, 2024

Actually, never mind, looks like the fsck is what's creating the journal entries, because there's always only one of them according to bcachefs list_journal.

@K900
Copy link

K900 commented Aug 25, 2024

Also, I just tried updating to 6.11-rc5 and that doesn't seem to change anything.

@K900
Copy link

K900 commented Aug 25, 2024

OK so I started looking at what changed and found d97de0d and 8ed823b, which seem to touch these code paths. Reverting those two gets things back to normal again. I'm now going to try only reverting the second one.

@K900
Copy link

K900 commented Aug 25, 2024

Confirmed that just reverting 8ed823b does not fix it, and I'm not qualified enough to revert specifically d97de0d.

@kode54
Copy link
Contributor

kode54 commented Aug 27, 2024

I had this problem with bcachefs-testing prior to rc4's release, and downgraded to rc3 each time I had the problem. Upgrading to rc4 didn't provoke it again.

By the way, I changed my upgrade methodology when I moved to rc4:

Before, I was running: bcachefs fsck <full device list> from a rescue environment, because the mount time fsck was OOMing on my actual OS.

Now, I run: mount <single device> <path> from an updated rescue environment. Or mount.bcachefs <full device list> <path> for my multi-device volume.

I still run it from a rescue environment, because I don't trust the fsck not to OOM yet. I may blindly try booting into the OS next time, whenever that is, since it seems to either finish or not in only 20 minutes now, instead of the previous two hours.

Another difference is I'm also running bcachefs-tools 1.11.0 now.

@koverstreet
Copy link
Owner

Are you guys mounting with -o fsck,fix_errors?

@K900
Copy link

K900 commented Sep 1, 2024

I wasn't initially but I am now.

@koverstreet
Copy link
Owner

Result?

And are you on IRC?

@K900
Copy link

K900 commented Sep 1, 2024

Same errors in dmesg every time. It's currently 4AM for me (don't ask), but I can hop on IRC in maybe 3-4 hours.

@koverstreet
Copy link
Owner

Ok; it's sounding like something must be wrong with the repair dump, so if I can get a metadata dump that should make it quick for me to debug

@K900
Copy link

K900 commented Sep 1, 2024 •
edited
Loading

Some poking on IRC later, it looks like what actually happened here was:

  • the buggy journal was written by an older -rc kernel (bug 1, fixed somewhere between 6.11-rc1 and -rc4)
  • newer kernels would only fix/apply one journal entry at a time (bug 2, not fixed?)
  • reverting the change made the new kernels not notice the error

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@K900 @kode54 @koverstreet @jpsollie