Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fsck --reconstruct_alloc segfaults after interrupted run #240

Open
Architector4 opened this issue Feb 20, 2024 · 9 comments
Open

fsck --reconstruct_alloc segfaults after interrupted run #240

Architector4 opened this issue Feb 20, 2024 · 9 comments

Comments

@Architector4
Copy link

Architector4 commented Feb 20, 2024

Currently in a bit of a pickle here. I ran bcachefs fsck --reconstruct_alloc -pf /dev/myroot out of boredom, saw it printing a load of messages quickly, assumed that's normal part of operation of this mode, and decided to ctrl+c it and run it again with -r.

Now bcachefs tool segfaults when I try to do it again and it won't mount. Both latest stable in Arch Linux repos (3:1.6.2-1) and latest master commit 6ff5313cbe0432 segfault.

Writing from my phone and operating from Arch Linux installer ISO environment at the second, as I don't have another machine to do stuff with.

Here's a run with gdb, with thr apply all bt

Starting program: /mnt/home/architector4/bcachefs-tools/bcachefs-tools/target/release/bcachefs fsck --reconstruct_alloc /dev/mapper/agogusssd

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.archlinux.org>
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
[New Thread 0x7ffff7b95680 (LWP 2554)]
[New Thread 0x7ffff7b8c680 (LWP 2555)]
[New Thread 0x7ffff7b83680 (LWP 2556)]
[New Thread 0x7ffff7b7a680 (LWP 2557)]
[New Thread 0x7ffff7b71680 (LWP 2558)]
[New Thread 0x7ffff7b68680 (LWP 2559)]
[New Thread 0x7ffff7b5f680 (LWP 2560)]
[New Thread 0x7ffff7b56680 (LWP 2561)]
[New Thread 0x7ffff7afa680 (LWP 2562)]
[New Thread 0x7ffff7af1680 (LWP 2563)]
[New Thread 0x7ffff7ae8680 (LWP 2564)]
[New Thread 0x7ffff7adf680 (LWP 2565)]
[New Thread 0x7ffff7ad6680 (LWP 2566)]
[New Thread 0x7ffff7a8b680 (LWP 2567)]
[New Thread 0x7ffff715b680 (LWP 2568)]
[Thread 0x7ffff715b680 (LWP 2568) exited]
[Thread 0x7ffff7a8b680 (LWP 2567) exited]
[New Thread 0x7ffff7afa680 (LWP 2569)]
[Thread 0x7ffff7ad6680 (LWP 2566) exited]
[Thread 0x7ffff7adf680 (LWP 2565) exited]
[Thread 0x7ffff7ae8680 (LWP 2564) exited]
[Thread 0x7ffff7af1680 (LWP 2563) exited]
[Thread 0x7ffff7afa680 (LWP 2562) exited]
[New Thread 0x7ffff7af1680 (LWP 2570)]
[New Thread 0x7ffff7ae8680 (LWP 2571)]
[New Thread 0x7ffff7adf680 (LWP 2572)]
[New Thread 0x7ffff7ad6680 (LWP 2573)]
[New Thread 0x7ffff7a8b680 (LWP 2574)]
[New Thread 0x7ffff715b680 (LWP 2575)]
[New Thread 0x7ffff7b1f680 (LWP 2576)]
[New Thread 0x7ffff5c00680 (LWP 2577)]

Thread 1 "bcachefs" received signal SIGSEGV, Segmentation fault.
0x00007ffff7d043f7 in ?? () from /usr/lib/libc.so.6

Thread 25 (Thread 0x7ffff5c00680 (LWP 2577) "events"):
#0  0x00007ffff7cba74d in syscall () from /usr/lib/libc.so.6
#1  0x00007ffff7fb1596 in ?? () from /usr/lib/liburcu.so.8
#2  0x00007ffff7c389eb in ?? () from /usr/lib/libc.so.6
#3  0x00007ffff7cbc8ac in ?? () from /usr/lib/libc.so.6

Thread 24 (Thread 0x7ffff7b1f680 (LWP 2576) "bch-reclaim//de"):
#0  0x00007ffff7cba74d in syscall () from /usr/lib/libc.so.6
#1  0x000055555569c5b1 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=<optimized out>, op=128, uaddr=<optimized out>) at /usr/include/urcu/futex.h:81
#2  schedule () at linux/sched.c:41
#3  0x000055555569c692 in schedule_timeout (timeout=<optimized out>) at linux/sched.c:96
#4  0x000055555565d622 in bch2_journal_reclaim_thread (arg=0x5555559d8840) at libbcachefs/journal_reclaim.c:749
#5  0x0000555555699863 in kthread_start_fn (data=0x555555998500) at linux/kthread.c:25
#6  0x00007ffff7c389eb in ?? () from /usr/lib/libc.so.6
#7  0x00007ffff7cbc8ac in ?? () from /usr/lib/libc.so.6

Thread 23 (Thread 0x7ffff715b680 (LWP 2575) "btree_update"):
#0  0x00007ffff7cba74d in syscall () from /usr/lib/libc.so.6
#1  0x000055555569c5b1 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=<optimized out>, op=128, uaddr=<optimized out>) at /usr/include/urcu/futex.h:81
#2  schedule () at linux/sched.c:41
#3  0x000055555569fcdb in worker_thread (arg=0x555555922440) at linux/workqueue.c:252
#4  0x0000555555699863 in kthread_start_fn (data=0x555555925010) at linux/kthread.c:25
#5  0x00007ffff7c389eb in ?? () from /usr/lib/libc.so.6
#6  0x00007ffff7cbc8ac in ?? () from /usr/lib/libc.so.6

Thread 22 (Thread 0x7ffff7a8b680 (LWP 2574) "bcachefs_journa"):
#0  0x00007ffff7cba74d in syscall () from /usr/lib/libc.so.6
#1  0x000055555569c5b1 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=<optimized out>, op=128, uaddr=<optimized out>) at /usr/include/urcu/futex.h:81
#2  schedule () at linux/sched.c:41
#3  0x000055555569fcdb in worker_thread (arg=0x55555593ea40) at linux/workqueue.c:252
#4  0x0000555555699863 in kthread_start_fn (data=0x55555593ea90) at linux/kthread.c:25
#5  0x00007ffff7c389eb in ?? () from /usr/lib/libc.so.6
#6  0x00007ffff7cbc8ac in ?? () from /usr/lib/libc.so.6

Thread 21 (Thread 0x7ffff7ad6680 (LWP 2573) "bcachefs_write_"):
#0  0x00007ffff7cba74d in syscall () from /usr/lib/libc.so.6
#1  0x000055555569c5b1 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=<optimized out>, op=128, uaddr=<optimized out>) at /usr/include/urcu/futex.h:81
#2  schedule () at linux/sched.c:41
#3  0x000055555569fcdb in worker_thread (arg=0x555555927980) at linux/workqueue.c:252
#4  0x0000555555699863 in kthread_start_fn (data=0x555555922360) at linux/kthread.c:25
#5  0x00007ffff7c389eb in ?? () from /usr/lib/libc.so.6
#6  0x00007ffff7cbc8ac in ?? () from /usr/lib/libc.so.6

Thread 20 (Thread 0x7ffff7adf680 (LWP 2572) "bcachefs_io"):
#0  0x00007ffff7cba74d in syscall () from /usr/lib/libc.so.6
#1  0x000055555569c5b1 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=<optimized out>, op=128, uaddr=<optimized out>) at /usr/include/urcu/futex.h:81
#2  schedule () at linux/sched.c:41
#3  0x000055555569fcdb in worker_thread (arg=0x555555927800) at linux/workqueue.c:252
#4  0x0000555555699863 in kthread_start_fn (data=0x555555927850) at linux/kthread.c:25
#5  0x00007ffff7c389eb in ?? () from /usr/lib/libc.so.6
#6  0x00007ffff7cbc8ac in ?? () from /usr/lib/libc.so.6

Thread 19 (Thread 0x7ffff7ae8680 (LWP 2571) "bcachefs_copygc"):
#0  0x00007ffff7cba74d in syscall () from /usr/lib/libc.so.6
#1  0x000055555569c5b1 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=<optimized out>, op=128, uaddr=<optimized out>) at /usr/include/urcu/futex.h:81
#2  schedule () at linux/sched.c:41
#3  0x000055555569fcdb in worker_thread (arg=0x555555927680) at linux/workqueue.c:252
#4  0x0000555555699863 in kthread_start_fn (data=0x5555559276d0) at linux/kthread.c:25
#5  0x00007ffff7c389eb in ?? () from /usr/lib/libc.so.6
#6  0x00007ffff7cbc8ac in ?? () from /usr/lib/libc.so.6

Thread 18 (Thread 0x7ffff7af1680 (LWP 2570) "bcachefs_btree_"):
#0  0x00007ffff7cba74d in syscall () from /usr/lib/libc.so.6
#1  0x000055555569c5b1 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=<optimized out>, op=128, uaddr=<optimized out>) at /usr/include/urcu/futex.h:81
#2  schedule () at linux/sched.c:41
#3  0x000055555569fcdb in worker_thread (arg=0x55555591e980) at linux/workqueue.c:252
#4  0x0000555555699863 in kthread_start_fn (data=0x555555927570) at linux/kthread.c:25
#5  0x00007ffff7c389eb in ?? () from /usr/lib/libc.so.6
#6  0x00007ffff7cbc8ac in ?? () from /usr/lib/libc.so.6

Thread 17 (Thread 0x7ffff7afa680 (LWP 2569) "bcachefs"):
#0  0x00007ffff7cba74d in syscall () from /usr/lib/libc.so.6
#1  0x000055555569c5b1 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=<optimized out>, op=128, uaddr=<optimized out>) at /usr/include/urcu/futex.h:81
#2  schedule () at linux/sched.c:41
#3  0x000055555569fcdb in worker_thread (arg=0x55555592d040) at linux/workqueue.c:252
#4  0x0000555555699863 in kthread_start_fn (data=0x555555927490) at linux/kthread.c:25
#5  0x00007ffff7c389eb in ?? () from /usr/lib/libc.so.6
#6  0x00007ffff7cbc8ac in ?? () from /usr/lib/libc.so.6

Thread 9 (Thread 0x7ffff7b56680 (LWP 2561) "timers"):
#0  0x00007ffff7c354ae in ?? () from /usr/lib/libc.so.6
#1  0x00007ffff7c38055 in pthread_cond_timedwait () from /usr/lib/libc.so.6
#2  0x000055555569ecee in timer_thread (arg=<optimized out>) at linux/timer.c:294
#3  0x0000555555699863 in kthread_start_fn (data=0x55555591da00) at linux/kthread.c:25
#4  0x00007ffff7c389eb in ?? () from /usr/lib/libc.so.6
#5  0x00007ffff7cbc8ac in ?? () from /usr/lib/libc.so.6

Thread 8 (Thread 0x7ffff7b5f680 (LWP 2560) "shrinkers"):
#0  0x00007ffff7cba74d in syscall () from /usr/lib/libc.so.6
#1  0x000055555569cd41 in futex (val3=-1, uaddr2=0x0, timeout=0x7ffff7b5e940, val=<optimized out>, op=137, uaddr=<optimized out>) at /usr/include/urcu/futex.h:81
#2  shrinker_thread (arg=<optimized out>) at linux/shrinker.c:106
#3  0x0000555555699863 in kthread_start_fn (data=0x55555591d3a0) at linux/kthread.c:25
#4  0x00007ffff7c389eb in ?? () from /usr/lib/libc.so.6
#5  0x00007ffff7cbc8ac in ?? () from /usr/lib/libc.so.6

Thread 7 (Thread 0x7ffff7b68680 (LWP 2559) "events_freezabl"):
#0  0x00007ffff7cba74d in syscall () from /usr/lib/libc.so.6
#1  0x000055555569c5b1 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=<optimized out>, op=128, uaddr=<optimized out>) at /usr/include/urcu/futex.h:81
#2  schedule () at linux/sched.c:41
#3  0x000055555569fcdb in worker_thread (arg=0x55555591d100) at linux/workqueue.c:252
#4  0x0000555555699863 in kthread_start_fn (data=0x55555591d150) at linux/kthread.c:25
#5  0x00007ffff7c389eb in ?? () from /usr/lib/libc.so.6
#6  0x00007ffff7cbc8ac in ?? () from /usr/lib/libc.so.6

Thread 6 (Thread 0x7ffff7b71680 (LWP 2558) "events_unbound"):
#0  0x00007ffff7cba74d in syscall () from /usr/lib/libc.so.6
#1  0x000055555569c5b1 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=<optimized out>, op=128, uaddr=<optimized out>) at /usr/include/urcu/futex.h:81
#2  schedule () at linux/sched.c:41
#3  0x000055555569fcdb in worker_thread (arg=0x55555591ce40) at linux/workqueue.c:252
#4  0x0000555555699863 in kthread_start_fn (data=0x55555591ce90) at linux/kthread.c:25
#5  0x00007ffff7c389eb in ?? () from /usr/lib/libc.so.6
#6  0x00007ffff7cbc8ac in ?? () from /usr/lib/libc.so.6

Thread 5 (Thread 0x7ffff7b7a680 (LWP 2557) "events_long"):
#0  0x00007ffff7cba74d in syscall () from /usr/lib/libc.so.6
#1  0x000055555569c5b1 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=<optimized out>, op=128, uaddr=<optimized out>) at /usr/include/urcu/futex.h:81
#2  schedule () at linux/sched.c:41
#3  0x000055555569fcdb in worker_thread (arg=0x55555591cb80) at linux/workqueue.c:252
#4  0x0000555555699863 in kthread_start_fn (data=0x55555591cbd0) at linux/kthread.c:25
#5  0x00007ffff7c389eb in ?? () from /usr/lib/libc.so.6
#6  0x00007ffff7cbc8ac in ?? () from /usr/lib/libc.so.6

Thread 4 (Thread 0x7ffff7b83680 (LWP 2556) "events_highpri"):
#0  0x00007ffff7cba74d in syscall () from /usr/lib/libc.so.6
#1  0x000055555569c5b1 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=<optimized out>, op=128, uaddr=<optimized out>) at /usr/include/urcu/futex.h:81
#2  schedule () at linux/sched.c:41
#3  0x000055555569fcdb in worker_thread (arg=0x55555591c8c0) at linux/workqueue.c:252
#4  0x0000555555699863 in kthread_start_fn (data=0x55555591c910) at linux/kthread.c:25
#5  0x00007ffff7c389eb in ?? () from /usr/lib/libc.so.6
#6  0x00007ffff7cbc8ac in ?? () from /usr/lib/libc.so.6

Thread 3 (Thread 0x7ffff7b8c680 (LWP 2555) "events"):
#0  0x00007ffff7cba74d in syscall () from /usr/lib/libc.so.6
#1  0x000055555569c5b1 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=<optimized out>, op=128, uaddr=<optimized out>) at /usr/include/urcu/futex.h:81
#2  schedule () at linux/sched.c:41
#3  0x000055555569fcdb in worker_thread (arg=0x55555591c600) at linux/workqueue.c:252
#4  0x0000555555699863 in kthread_start_fn (data=0x55555591c650) at linux/kthread.c:25
#5  0x00007ffff7c389eb in ?? () from /usr/lib/libc.so.6
#6  0x00007ffff7cbc8ac in ?? () from /usr/lib/libc.so.6

Thread 2 (Thread 0x7ffff7b95680 (LWP 2554) "aio_completion"):
#0  0x00007ffff7cba74d in syscall () from /usr/lib/libc.so.6
#1  0x00007ffff7db4277 in ?? () from /usr/lib/libaio.so.1
#2  0x000055555569722c in aio_completion_thread (arg=<optimized out>) at linux/blkdev.c:276
#3  0x0000555555699863 in kthread_start_fn (data=0x55555591c380) at linux/kthread.c:25
#4  0x00007ffff7c389eb in ?? () from /usr/lib/libc.so.6
#5  0x00007ffff7cbc8ac in ?? () from /usr/lib/libc.so.6

Thread 1 (Thread 0x7ffff7b96c00 (LWP 2549) "bcachefs"):
#0  0x00007ffff7d043f7 in ?? () from /usr/lib/libc.so.6
#1  0x00005555555f3aeb in __move_gap (element_size=24, new_gap=0, old_gap=<optimized out>, size=<optimized out>, nr=<optimized out>, array=<optimized out>) at libbcachefs/util.h:641
#2  bch2_journal_key_insert_take (c=c@entry=0x555555999000, id=id@entry=BTREE_ID_alloc, level=level@entry=0, k=k@entry=0x5555568ef840) at libbcachefs/btree_journal_iter.c:221
#3  0x00005555555f3bfb in bch2_journal_key_insert (c=c@entry=0x555555999000, id=BTREE_ID_alloc, level=0, k=0x55555593ec00) at libbcachefs/btree_journal_iter.c:248
#4  0x00005555555fbdfa in do_bch2_trans_commit_to_journal_replay (trans=0x5555576df000) at libbcachefs/btree_trans_commit.c:974
#5  __bch2_trans_commit (trans=trans@entry=0x5555576df000, flags=flags@entry=64) at libbcachefs/btree_trans_commit.c:1033
#6  0x00005555555deac2 in bch2_trans_commit (flags=64, journal_seq=0x0, disk_res=0x0, trans=0x5555576df000) at libbcachefs/btree_update.h:168
#7  bch2_gc_alloc_done (metadata_only=<optimized out>, c=<optimized out>) at libbcachefs/btree_gc.c:1475
#8  bch2_gc (c=<optimized out>, initial=<optimized out>, metadata_only=<optimized out>) at libbcachefs/btree_gc.c:1823
#9  0x000055555566d645 in bch2_run_recovery_pass (c=c@entry=0x555555999000, pass=pass@entry=BCH_RECOVERY_PASS_check_allocations) at libbcachefs/recovery.c:685
#10 0x000055555566fee7 in bch2_run_recovery_passes (c=<optimized out>) at libbcachefs/recovery.c:702
#11 bch2_fs_recovery (c=0x555555999000) at libbcachefs/recovery.c:978
#12 0x000055555568e2b5 in bch2_fs_start (c=c@entry=0x555555999000) at libbcachefs/super.c:1024
#13 0x00005555556919b0 in bch2_fs_open (devices=devices@entry=0x55555591e8d0, nr_devices=nr_devices@entry=1, opts=...) at libbcachefs/super.c:2074
#14 0x00005555555b0743 in cmd_fsck (argc=<optimized out>, argv=<optimized out>) at c_src/cmd_fsck.c:256
#15 0x0000555555590be6 in bcachefs::main ()
#16 0x000055555559be63 in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#17 0x00005555555a4609 in std::rt::lang_start::{{closure}} ()
#18 0x00005555557389d7 in std::rt::lang_start_internal ()
#19 0x0000555555590f95 in main ()

Or building in debug mode and running the same thing (random output messages from the tool itself are missing, I guess gdb logging doesn't capture stderr):

Starting program: /mnt/home/architector4/bcachefs-tools/bcachefs-tools/target/debug/bcachefs fsck --reconstruct_alloc /dev/mapper/agogusssd

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                
[New Thread 0x7ffff7b95680 (LWP 5350)]
[New Thread 0x7ffff7b8c680 (LWP 5351)]
[New Thread 0x7ffff7b83680 (LWP 5352)]
[New Thread 0x7ffff7b7a680 (LWP 5353)]
[New Thread 0x7ffff7b71680 (LWP 5354)]
[New Thread 0x7ffff7b68680 (LWP 5355)]
[New Thread 0x7ffff7b5f680 (LWP 5356)]
[New Thread 0x7ffff7b56680 (LWP 5357)]
[New Thread 0x7ffff7afa680 (LWP 5358)]
[New Thread 0x7ffff7af1680 (LWP 5359)]
[New Thread 0x7ffff7ae8680 (LWP 5360)]
[New Thread 0x7ffff7adf680 (LWP 5361)]
[New Thread 0x7ffff7ad6680 (LWP 5362)]
[New Thread 0x7ffff7a8b680 (LWP 5363)]
[New Thread 0x7ffff715b680 (LWP 5364)]
[Thread 0x7ffff715b680 (LWP 5364) exited]
[Thread 0x7ffff7a8b680 (LWP 5363) exited]
[Thread 0x7ffff7ad6680 (LWP 5362) exited]
[New Thread 0x7ffff7afa680 (LWP 5365)]
[Thread 0x7ffff7adf680 (LWP 5361) exited]
[Thread 0x7ffff7ae8680 (LWP 5360) exited]
[Thread 0x7ffff7af1680 (LWP 5359) exited]
[Thread 0x7ffff7afa680 (LWP 5358) exited]
[New Thread 0x7ffff7af1680 (LWP 5366)]
[New Thread 0x7ffff7ae8680 (LWP 5367)]
[New Thread 0x7ffff7adf680 (LWP 5368)]
[New Thread 0x7ffff7ad6680 (LWP 5369)]
[New Thread 0x7ffff7a8b680 (LWP 5370)]
[New Thread 0x7ffff715b680 (LWP 5371)]
[New Thread 0x7ffff7b1f680 (LWP 5374)]
[New Thread 0x7ffff5c00680 (LWP 5375)]

Thread 1 "bcachefs" received signal SIGSEGV, Segmentation fault.
0x00007ffff7d043f7 in ?? () from /usr/lib/libc.so.6
Invalid thread ID: all apply bt

Thread 25 (Thread 0x7ffff5c00680 (LWP 5375) "events"):
#0  0x00007ffff7cba74d in syscall () from /usr/lib/libc.so.6
#1  0x00007ffff7fb1596 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=-1, op=0, uaddr=0x7fff90000bb0) at ../include/urcu/futex.h:85
#2  futex_noasync (timeout=0x0, uaddr2=0x0, val3=0, val=-1, op=0, uaddr=0x7fff90000bb0) at ../include/urcu/futex.h:90
#3  call_rcu_wait (crdp=0x7fff90000b70) at /usr/src/debug/liburcu/userspace-rcu-0.14.0/src/urcu-call-rcu-impl.h:248
#4  call_rcu_thread (arg=0x7fff90000b70) at /usr/src/debug/liburcu/userspace-rcu-0.14.0/src/urcu-call-rcu-impl.h:400
#5  0x00007ffff7c389eb in ?? () from /usr/lib/libc.so.6
#6  0x00007ffff7cbc8ac in ?? () from /usr/lib/libc.so.6

Thread 24 (Thread 0x7ffff7b1f680 (LWP 5374) "bch-reclaim//de"):
#0  0x00007ffff7cba74d in syscall () from /usr/lib/libc.so.6
#1  0x00005555556dcb41 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=<optimized out>, op=128, uaddr=<optimized out>) at /usr/include/urcu/futex.h:81
#2  schedule () at linux/sched.c:41
#3  0x00005555556dcc22 in schedule_timeout (timeout=<optimized out>) at linux/sched.c:96
#4  0x000055555569c482 in bch2_journal_reclaim_thread (arg=0x555555b9a880) at libbcachefs/journal_reclaim.c:749
#5  0x00005555556d9df3 in kthread_start_fn (data=0x555555aeff10) at linux/kthread.c:25
#6  0x00007ffff7c389eb in ?? () from /usr/lib/libc.so.6
#7  0x00007ffff7cbc8ac in ?? () from /usr/lib/libc.so.6

Thread 23 (Thread 0x7ffff715b680 (LWP 5371) "btree_update"):
#0  0x00007ffff7cba74d in syscall () from /usr/lib/libc.so.6
#1  0x00005555556dcb41 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=<optimized out>, op=128, uaddr=<optimized out>) at /usr/include/urcu/futex.h:81
#2  schedule () at linux/sched.c:41
#3  0x00005555556e026b in worker_thread (arg=0x555555ae5440) at linux/workqueue.c:252
#4  0x00005555556d9df3 in kthread_start_fn (data=0x555555ae4a10) at linux/kthread.c:25
#5  0x00007ffff7c389eb in ?? () from /usr/lib/libc.so.6
#6  0x00007ffff7cbc8ac in ?? () from /usr/lib/libc.so.6

Thread 22 (Thread 0x7ffff7a8b680 (LWP 5370) "bcachefs_journa"):
#0  0x00007ffff7cba74d in syscall () from /usr/lib/libc.so.6
#1  0x00005555556dcb41 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=<optimized out>, op=128, uaddr=<optimized out>) at /usr/include/urcu/futex.h:81
#2  schedule () at linux/sched.c:41
#3  0x00005555556e026b in worker_thread (arg=0x555555b01a40) at linux/workqueue.c:252
#4  0x00005555556d9df3 in kthread_start_fn (data=0x555555b01a90) at linux/kthread.c:25
#5  0x00007ffff7c389eb in ?? () from /usr/lib/libc.so.6
#6  0x00007ffff7cbc8ac in ?? () from /usr/lib/libc.so.6

Thread 21 (Thread 0x7ffff7ad6680 (LWP 5369) "bcachefs_write_"):
#0  0x00007ffff7cba74d in syscall () from /usr/lib/libc.so.6
#1  0x00005555556dcb41 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=<optimized out>, op=128, uaddr=<optimized out>) at /usr/include/urcu/futex.h:81
#2  schedule () at linux/sched.c:41
#3  0x00005555556e026b in worker_thread (arg=0x555555aea980) at linux/workqueue.c:252
#4  0x00005555556d9df3 in kthread_start_fn (data=0x555555ae5360) at linux/kthread.c:25
#5  0x00007ffff7c389eb in ?? () from /usr/lib/libc.so.6
#6  0x00007ffff7cbc8ac in ?? () from /usr/lib/libc.so.6

Thread 20 (Thread 0x7ffff7adf680 (LWP 5368) "bcachefs_io"):
#0  0x00007ffff7cba74d in syscall () from /usr/lib/libc.so.6
#1  0x00005555556dcb41 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=<optimized out>, op=128, uaddr=<optimized out>) at /usr/include/urcu/futex.h:81
#2  schedule () at linux/sched.c:41
#3  0x00005555556e026b in worker_thread (arg=0x555555aea800) at linux/workqueue.c:252
#4  0x00005555556d9df3 in kthread_start_fn (data=0x555555aea850) at linux/kthread.c:25
#5  0x00007ffff7c389eb in ?? () from /usr/lib/libc.so.6
#6  0x00007ffff7cbc8ac in ?? () from /usr/lib/libc.so.6

Thread 19 (Thread 0x7ffff7ae8680 (LWP 5367) "bcachefs_copygc"):
#0  0x00007ffff7cba74d in syscall () from /usr/lib/libc.so.6
#1  0x00005555556dcb41 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=<optimized out>, op=128, uaddr=<optimized out>) at /usr/include/urcu/futex.h:81
#2  schedule () at linux/sched.c:41
#3  0x00005555556e026b in worker_thread (arg=0x555555aea680) at linux/workqueue.c:252
#4  0x00005555556d9df3 in kthread_start_fn (data=0x555555aea6d0) at linux/kthread.c:25
#5  0x00007ffff7c389eb in ?? () from /usr/lib/libc.so.6
#6  0x00007ffff7cbc8ac in ?? () from /usr/lib/libc.so.6

Thread 18 (Thread 0x7ffff7af1680 (LWP 5366) "bcachefs_btree_"):
#0  0x00007ffff7cba74d in syscall () from /usr/lib/libc.so.6
#1  0x00005555556dcb41 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=<optimized out>, op=128, uaddr=<optimized out>) at /usr/include/urcu/futex.h:81
#2  schedule () at linux/sched.c:41
#3  0x00005555556e026b in worker_thread (arg=0x555555ae1980) at linux/workqueue.c:252
#4  0x00005555556d9df3 in kthread_start_fn (data=0x555555aea570) at linux/kthread.c:25
#5  0x00007ffff7c389eb in ?? () from /usr/lib/libc.so.6
#6  0x00007ffff7cbc8ac in ?? () from /usr/lib/libc.so.6

Thread 17 (Thread 0x7ffff7afa680 (LWP 5365) "bcachefs"):
#0  0x00007ffff7cba74d in syscall () from /usr/lib/libc.so.6
#1  0x00005555556dcb41 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=<optimized out>, op=128, uaddr=<optimized out>) at /usr/include/urcu/futex.h:81
#2  schedule () at linux/sched.c:41
#3  0x00005555556e026b in worker_thread (arg=0x555555af0040) at linux/workqueue.c:252
#4  0x00005555556d9df3 in kthread_start_fn (data=0x555555aea490) at linux/kthread.c:25
#5  0x00007ffff7c389eb in ?? () from /usr/lib/libc.so.6
#6  0x00007ffff7cbc8ac in ?? () from /usr/lib/libc.so.6

Thread 9 (Thread 0x7ffff7b56680 (LWP 5357) "timers"):
#0  0x00007ffff7c354ae in ?? () from /usr/lib/libc.so.6
#1  0x00007ffff7c38055 in pthread_cond_timedwait () from /usr/lib/libc.so.6
#2  0x00005555556df27e in timer_thread (arg=<optimized out>) at linux/timer.c:294
#3  0x00005555556d9df3 in kthread_start_fn (data=0x555555ae0a00) at linux/kthread.c:25
#4  0x00007ffff7c389eb in ?? () from /usr/lib/libc.so.6
#5  0x00007ffff7cbc8ac in ?? () from /usr/lib/libc.so.6

Thread 8 (Thread 0x7ffff7b5f680 (LWP 5356) "shrinkers"):
#0  0x00007ffff7cba74d in syscall () from /usr/lib/libc.so.6
#1  0x00005555556dd2d1 in futex (val3=-1, uaddr2=0x0, timeout=0x7ffff7b5e940, val=<optimized out>, op=137, uaddr=<optimized out>) at /usr/include/urcu/futex.h:81
#2  shrinker_thread (arg=<optimized out>) at linux/shrinker.c:106
#3  0x00005555556d9df3 in kthread_start_fn (data=0x555555ae03a0) at linux/kthread.c:25
#4  0x00007ffff7c389eb in ?? () from /usr/lib/libc.so.6
#5  0x00007ffff7cbc8ac in ?? () from /usr/lib/libc.so.6

Thread 7 (Thread 0x7ffff7b68680 (LWP 5355) "events_freezabl"):
#0  0x00007ffff7cba74d in syscall () from /usr/lib/libc.so.6
#1  0x00005555556dcb41 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=<optimized out>, op=128, uaddr=<optimized out>) at /usr/include/urcu/futex.h:81
#2  schedule () at linux/sched.c:41
#3  0x00005555556e026b in worker_thread (arg=0x555555ae0100) at linux/workqueue.c:252
#4  0x00005555556d9df3 in kthread_start_fn (data=0x555555ae0150) at linux/kthread.c:25
#5  0x00007ffff7c389eb in ?? () from /usr/lib/libc.so.6
#6  0x00007ffff7cbc8ac in ?? () from /usr/lib/libc.so.6

Thread 6 (Thread 0x7ffff7b71680 (LWP 5354) "events_unbound"):
#0  0x00007ffff7cba74d in syscall () from /usr/lib/libc.so.6
#1  0x00005555556dcb41 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=<optimized out>, op=128, uaddr=<optimized out>) at /usr/include/urcu/futex.h:81
#2  schedule () at linux/sched.c:41
#3  0x00005555556e026b in worker_thread (arg=0x555555adfe40) at linux/workqueue.c:252
#4  0x00005555556d9df3 in kthread_start_fn (data=0x555555adfe90) at linux/kthread.c:25
#5  0x00007ffff7c389eb in ?? () from /usr/lib/libc.so.6
#6  0x00007ffff7cbc8ac in ?? () from /usr/lib/libc.so.6

Thread 5 (Thread 0x7ffff7b7a680 (LWP 5353) "events_long"):
#0  0x00007ffff7cba74d in syscall () from /usr/lib/libc.so.6
#1  0x00005555556dcb41 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=<optimized out>, op=128, uaddr=<optimized out>) at /usr/include/urcu/futex.h:81
#2  schedule () at linux/sched.c:41
#3  0x00005555556e026b in worker_thread (arg=0x555555adfb80) at linux/workqueue.c:252
#4  0x00005555556d9df3 in kthread_start_fn (data=0x555555adfbd0) at linux/kthread.c:25
#5  0x00007ffff7c389eb in ?? () from /usr/lib/libc.so.6
#6  0x00007ffff7cbc8ac in ?? () from /usr/lib/libc.so.6

Thread 4 (Thread 0x7ffff7b83680 (LWP 5352) "events_highpri"):
#0  0x00007ffff7cba74d in syscall () from /usr/lib/libc.so.6
#1  0x00005555556dcb41 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=<optimized out>, op=128, uaddr=<optimized out>) at /usr/include/urcu/futex.h:81
#2  schedule () at linux/sched.c:41
#3  0x00005555556e026b in worker_thread (arg=0x555555adf8c0) at linux/workqueue.c:252
#4  0x00005555556d9df3 in kthread_start_fn (data=0x555555adf910) at linux/kthread.c:25
#5  0x00007ffff7c389eb in ?? () from /usr/lib/libc.so.6
#6  0x00007ffff7cbc8ac in ?? () from /usr/lib/libc.so.6

Thread 3 (Thread 0x7ffff7b8c680 (LWP 5351) "events"):
#0  0x00007ffff7cba74d in syscall () from /usr/lib/libc.so.6
#1  0x00005555556dcb41 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=<optimized out>, op=128, uaddr=<optimized out>) at /usr/include/urcu/futex.h:81
#2  schedule () at linux/sched.c:41
#3  0x00005555556e026b in worker_thread (arg=0x555555adf600) at linux/workqueue.c:252
#4  0x00005555556d9df3 in kthread_start_fn (data=0x555555adf650) at linux/kthread.c:25
#5  0x00007ffff7c389eb in ?? () from /usr/lib/libc.so.6
#6  0x00007ffff7cbc8ac in ?? () from /usr/lib/libc.so.6

Thread 2 (Thread 0x7ffff7b95680 (LWP 5350) "aio_completion"):
#0  0x00007ffff7cba74d in syscall () from /usr/lib/libc.so.6
#1  0x00007ffff7db4277 in __io_getevents_0_4 () from /usr/lib/libaio.so.1
#2  0x00005555556d76bc in aio_completion_thread (arg=<optimized out>) at linux/blkdev.c:276
#3  0x00005555556d9df3 in kthread_start_fn (data=0x555555adf380) at linux/kthread.c:25
#4  0x00007ffff7c389eb in ?? () from /usr/lib/libc.so.6
#5  0x00007ffff7cbc8ac in ?? () from /usr/lib/libc.so.6

Thread 1 (Thread 0x7ffff7b96c00 (LWP 5349) "bcachefs"):
#0  0x00007ffff7d043f7 in ?? () from /usr/lib/libc.so.6
#1  0x000055555562fcfb in __move_gap (element_size=24, new_gap=0, old_gap=<optimized out>, size=<optimized out>, nr=<optimized out>, array=<optimized out>) at libbcachefs/util.h:641
#2  bch2_journal_key_insert_take (c=c@entry=0x555555b5b000, id=id@entry=BTREE_ID_alloc, level=level@entry=0, k=k@entry=0x555555c0f040) at libbcachefs/btree_journal_iter.c:221
#3  0x000055555562fe0b in bch2_journal_key_insert (c=c@entry=0x555555b5b000, id=BTREE_ID_alloc, level=0, k=0x555555b5ae00) at libbcachefs/btree_journal_iter.c:248
#4  0x0000555555639644 in do_bch2_trans_commit_to_journal_replay (trans=0x555557df9000) at libbcachefs/btree_trans_commit.c:974
#5  __bch2_trans_commit (trans=trans@entry=0x555557df9000, flags=flags@entry=64) at libbcachefs/btree_trans_commit.c:1033
#6  0x00005555556174f1 in bch2_trans_commit (flags=64, journal_seq=0x0, disk_res=0x0, trans=0x555557df9000) at libbcachefs/btree_update.h:168
#7  bch2_gc_alloc_done (metadata_only=<optimized out>, c=<optimized out>) at libbcachefs/btree_gc.c:1475
#8  bch2_gc (c=<optimized out>, initial=<optimized out>, metadata_only=<optimized out>) at libbcachefs/btree_gc.c:1823
#9  0x00005555556ac635 in bch2_run_recovery_pass (c=c@entry=0x555555b5b000, pass=pass@entry=BCH_RECOVERY_PASS_check_allocations) at libbcachefs/recovery.c:685
#10 0x00005555556aef77 in bch2_run_recovery_passes (c=<optimized out>) at libbcachefs/recovery.c:702
#11 bch2_fs_recovery (c=0x555555b5b000) at libbcachefs/recovery.c:978
#12 0x00005555556ce585 in bch2_fs_start (c=c@entry=0x555555b5b000) at libbcachefs/super.c:1024
#13 0x00005555556d1cb0 in bch2_fs_open (devices=devices@entry=0x555555ae18d0, nr_devices=nr_devices@entry=1, opts=...) at libbcachefs/super.c:2074
#14 0x00005555555e3b13 in cmd_fsck (argc=<optimized out>, argv=<optimized out>) at c_src/cmd_fsck.c:256
#15 0x00005555555b1016 in bcachefs::handle_c_command (args=..., symlink_cmd=...) at src/bcachefs.rs:57
#16 0x00005555555b1d34 in bcachefs::main () at src/bcachefs.rs:120

Here's output (stdout+stderr) with just -p (it doesn't segfault but may still be of note)

mounting version 1.6: btree_subvolume_children opts=ro,compression=zstd,root_reserve_percent=1,degraded,fsck,fix_errors=yes,read_only
recovering from unclean shutdown
superblock requires following recovery passes to be run:
  check_topology,check_subvols,check_dirents
Version upgrade from 1.3: rebalance_work to 1.6: btree_subvolume_children incomplete
Doing compatible version upgrade from 1.3: rebalance_work to 1.6: btree_subvolume_children

journal read done, replaying entries 853340-854796
dropped unflushed entries 854797-854954
error validating btree node on /dev/mapper/agogusssd at btree alloc level 1/1
  u64s 11 type btree_ptr_v2 SPOS_MAX len 0 ver 0: seq abd894c6f1ab0d10 written 30 min_key POS_MIN durability: 1 ptr: 0:8200:0 gen 0
  node offset 0/30: got wrong btree node (want abd894c6f1ab0d10 got 4955798853689e9f)
got btree alloc level 1 pos POS_MIN-SPOS_MAX
error reading btree root alloc, fixing
error validating btree node on /dev/mapper/agogusssd at btree freespace level 0/0
  u64s 11 type btree_ptr_v2 SPOS_MAX len 0 ver 0: seq 735f83ed774f0587 written 1 min_key POS_MIN durability: 1 ptr: 0:12476:512 gen 0
  node offset 0/1: got wrong btree node (want 735f83ed774f0587 got 104181df4d336ebd)
got btree alloc level 0 pos 0:66086:1-SPOS_MAX
error reading btree root freespace, fixing
error validating btree node on /dev/mapper/agogusssd at btree backpointers level 0/0
  u64s 11 type btree_ptr_v2 SPOS_MAX len 0 ver 0: seq 22436ae52add42f written 1 min_key POS_MIN durability: 1 ptr: 0:8197:0 gen 0
  node offset 0/1: got wrong btree node (want 22436ae52add42f got f77d47548e1a69a6)
got btree backpointers level 0 pos POS_MIN-SPOS_MAX
error reading btree root backpointers, fixing
error validating btree node on /dev/mapper/agogusssd at btree bucket_gens level 0/0
  u64s 11 type btree_ptr_v2 SPOS_MAX len 0 ver 0: seq 94dbe2d579f0602c written 1 min_key POS_MIN durability: 1 ptr: 0:12479:0 gen 0
  node offset 0/1: got wrong btree node (want 94dbe2d579f0602c got 47692375be5dcf7b)
got btree alloc level 0 pos 0:63971:1-0:66086:0
error reading btree root bucket_gens, fixing
alloc_read... done
stripes_read... done
snapshots_read... done
check_topology... done
check_allocations...bucket 0:8197 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:8197 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:8199 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:8199 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:8200 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:8200 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:8854 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:8854 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:9281 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:9281 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:9457 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:9457 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:9813 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:9813 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:10826 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:10826 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:10888 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:10888 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:10894 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:10894 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:10934 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:10934 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:11039 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:11039 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:11150 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:11150 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:11168 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:11168 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:11485 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:11485 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:11515 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:11515 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:11534 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:11534 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:11600 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:11600 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:11767 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:11767 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:11864 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:11864 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:12065 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:12065 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:12088 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:12088 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:12136 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:12136 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:12141 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:12141 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:12258 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:12258 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:12318 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:12318 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:12340 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:12340 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:12394 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:12394 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:12455 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:12455 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:12456 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:12456 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:12465 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:12465 gen 0 data type need_discard has wrong dirty_sectors: got 1024, should be 0, fixing
bucket 0:12476 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:12476 gen 0 data type need_discard has wrong dirty_sectors: got 1024, should be 0, fixing
bucket 0:12479 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:12479 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:12634 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:12634 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:12636 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:12636 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:12638 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:12638 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:12693 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:12693 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:12696 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:12696 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:12713 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:12713 gen 0 data type need_discard has wrong dirty_sectors: got 1024, should be 0, fixing
bucket 0:12837 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:12837 gen 0 data type need_discard has wrong dirty_sectors: got 1024, should be 0, fixing
bucket 0:12919 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:12919 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:13089 gen 1 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:13089 gen 1 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:13117 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:13117 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:13383 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:13383 gen 0 data type need_discard has wrong dirty_sectors: got 1024, should be 0, fixing
bucket 0:13417 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:13417 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:13453 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:13453 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:13471 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:13471 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:13591 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:13591 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:13594 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:13594 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:13603 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:13603 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:13679 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:13679 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:13844 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:13844 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:14163 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:14163 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:14322 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:14322 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:14469 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:14469 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:14544 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:14544 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:14545 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:14545 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:14565 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:14565 gen 0 data type need_discard has wrong dirty_sectors: got 512, should be 0, fixing
bucket 0:14580 gen 0 has wrong data_type: got btree, should be need_discard, fixing
bucket 0:14580 gen 0 data type need_discard has wrong dirty_sectors: got 1024, should be 0, fixing
 done
going read-write
journal_replay...error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:10159:0 len 0 ver 0: seq c3e86749c46d85bb written 245 min_key 0:8055:1 durability: 1 ptr: 0:9813:0 gen 0
  node offset 0/245: got wrong btree node (want c3e86749c46d85bb got 3c2dd10038c8731a)
got btree alloc level 0 pos 0:8055:1-0:10159:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:3896:0 len 0 ver 0: seq c8661f35c521d808 written 245 min_key 0:1943:1 durability: 1 ptr: 0:8854:0 gen 0
  node offset 0/245: got wrong btree node (want c8661f35c521d808 got e4d24e8bd4f2e8f8)
got btree alloc level 0 pos 0:1943:1-0:3896:0
error flushing key cache: btree_node_read_error
fatal error - emergency read only
unable to write journal to sufficient devices
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:16537:0 len 0 ver 0: seq eaf5de7b8e75b724 written 245 min_key 0:14411:1 durability: 1 ptr: 0:10894:0 gen 0
  node offset 0/245: got wrong btree node (want eaf5de7b8e75b724 got 49358e14a38af548)
got btree alloc level 0 pos 0:14411:1-0:16537:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:12279:0 len 0 ver 0: seq 9299fefd503c9cc1 written 245 min_key 0:10159:1 durability: 1 ptr: 0:10826:0 gen 0
  node offset 0/245: got wrong btree node (want 9299fefd503c9cc1 got e81c689f71f6ea97)
got btree alloc level 0 pos 0:10159:1-0:12279:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:14411:0 len 0 ver 0: seq 12ebd775715ecf1c written 245 min_key 0:12279:1 durability: 1 ptr: 0:10888:0 gen 0
  node offset 0/245: got wrong btree node (want 12ebd775715ecf1c got d0060688fa7b96be)
got btree alloc level 0 pos 0:12279:1-0:14411:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:18650:0 len 0 ver 0: seq 8fcd556a9116b40e written 245 min_key 0:16537:1 durability: 1 ptr: 0:10934:0 gen 0
  node offset 0/245: got wrong btree node (want 8fcd556a9116b40e got 15ed3b9776b3c76d)
got btree alloc level 0 pos 0:16537:1-0:18650:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:20764:0 len 0 ver 0: seq 17c2883b50587117 written 245 min_key 0:18650:1 durability: 1 ptr: 0:11039:0 gen 0
  node offset 0/245: got wrong btree node (want 17c2883b50587117 got adacd12c484f0fff)
got btree alloc level 0 pos 0:18650:1-0:20764:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:22876:0 len 0 ver 0: seq 79870e95c039ce7a written 245 min_key 0:20764:1 durability: 1 ptr: 0:11150:0 gen 0
  node offset 0/245: got wrong btree node (want 79870e95c039ce7a got 15125cfea7916c86)
got btree alloc level 0 pos 0:20764:1-0:22876:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:24994:0 len 0 ver 0: seq a23759d5b138663e written 245 min_key 0:22876:1 durability: 1 ptr: 0:11168:0 gen 0
  node offset 0/245: got wrong btree node (want a23759d5b138663e got d50cbef902277770)
got btree alloc level 0 pos 0:22876:1-0:24994:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:27113:0 len 0 ver 0: seq c960dd8778ef878a written 245 min_key 0:24994:1 durability: 1 ptr: 0:11485:0 gen 0
  node offset 0/245: got wrong btree node (want c960dd8778ef878a got a54024f5ae3b8c78)
got btree alloc level 0 pos 0:24994:1-0:27113:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:29228:0 len 0 ver 0: seq 935939ea4b2a04cb written 245 min_key 0:27113:1 durability: 1 ptr: 0:11515:0 gen 0
  node offset 0/245: got wrong btree node (want 935939ea4b2a04cb got 229b509da50bfd05)
got btree alloc level 0 pos 0:27113:1-0:29228:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:31340:0 len 0 ver 0: seq 11a72d0902e90f15 written 245 min_key 0:29228:1 durability: 1 ptr: 0:11534:0 gen 0
  node offset 0/245: got wrong btree node (want 11a72d0902e90f15 got 3c80965a73ae66ca)
got btree alloc level 0 pos 0:29228:1-0:31340:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:33448:0 len 0 ver 0: seq 672ff476ffbf7759 written 245 min_key 0:31340:1 durability: 1 ptr: 0:11600:0 gen 0
  node offset 0/245: got wrong btree node (want 672ff476ffbf7759 got 9b5a4b996fddd2e2)
got btree alloc level 0 pos 0:31340:1-0:33448:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:35556:0 len 0 ver 0: seq 97dadd0a4e802ce4 written 245 min_key 0:33448:1 durability: 1 ptr: 0:11767:0 gen 0
  node offset 0/245: got wrong btree node (want 97dadd0a4e802ce4 got 171d99d86b317446)
got btree alloc level 0 pos 0:33448:1-0:35556:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:37665:0 len 0 ver 0: seq f6386b9b6205fc84 written 245 min_key 0:35556:1 durability: 1 ptr: 0:11864:0 gen 0
  node offset 0/245: got wrong btree node (want f6386b9b6205fc84 got 14567796d259e0aa)
got btree alloc level 0 pos 0:35556:1-0:37665:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:39773:0 len 0 ver 0: seq 936114abca45857 written 245 min_key 0:37665:1 durability: 1 ptr: 0:12065:0 gen 0
  node offset 0/245: got wrong btree node (want 936114abca45857 got 30167104e1032008)
got btree alloc level 0 pos 0:37665:1-0:39773:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:40293:0 len 0 ver 0: seq 579fd2d041a9f811 written 245 min_key 0:38184:1 durability: 1 ptr: 0:12065:0 gen 0
  node offset 0/245: got wrong btree node (want 579fd2d041a9f811 got 30167104e1032008)
got btree alloc level 0 pos 0:37665:1-0:39773:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:41883:0 len 0 ver 0: seq 423622b1eb21ec54 written 245 min_key 0:39773:1 durability: 1 ptr: 0:12088:0 gen 0
  node offset 0/245: got wrong btree node (want 423622b1eb21ec54 got 2f29d83192e481f2)
got btree alloc level 0 pos 0:39773:1-0:41883:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:42401:0 len 0 ver 0: seq 7081da7a5f4528aa written 245 min_key 0:40293:1 durability: 1 ptr: 0:12088:0 gen 0
  node offset 0/245: got wrong btree node (want 7081da7a5f4528aa got 2f29d83192e481f2)
got btree alloc level 0 pos 0:39773:1-0:41883:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:43992:0 len 0 ver 0: seq cde85262719805c9 written 245 min_key 0:41883:1 durability: 1 ptr: 0:12136:0 gen 0
  node offset 0/245: got wrong btree node (want cde85262719805c9 got d33799eebd610631)
got btree alloc level 0 pos 0:41883:1-0:43992:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:44511:0 len 0 ver 0: seq eaab2542d552ce0f written 245 min_key 0:42401:1 durability: 1 ptr: 0:12136:0 gen 0
  node offset 0/245: got wrong btree node (want eaab2542d552ce0f got d33799eebd610631)
got btree alloc level 0 pos 0:41883:1-0:43992:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:46102:0 len 0 ver 0: seq bb0d6189dc255b41 written 245 min_key 0:43992:1 durability: 1 ptr: 0:12141:0 gen 0
  node offset 0/245: got wrong btree node (want bb0d6189dc255b41 got 3224cf6bb2fd7934)
got btree alloc level 0 pos 0:43992:1-0:46102:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:46620:0 len 0 ver 0: seq 5d0eb67b0eab1ff4 written 245 min_key 0:44511:1 durability: 1 ptr: 0:12141:0 gen 0
  node offset 0/245: got wrong btree node (want 5d0eb67b0eab1ff4 got 3224cf6bb2fd7934)
got btree alloc level 0 pos 0:43992:1-0:46102:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:48212:0 len 0 ver 0: seq 6e2f8f719d0950cb written 245 min_key 0:46102:1 durability: 1 ptr: 0:12258:0 gen 0
  node offset 0/245: got wrong btree node (want 6e2f8f719d0950cb got f34a4280c532db94)
got btree alloc level 0 pos 0:46102:1-0:48212:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:48732:0 len 0 ver 0: seq df1bb5885c71b735 written 245 min_key 0:46620:1 durability: 1 ptr: 0:12258:0 gen 0
  node offset 0/245: got wrong btree node (want df1bb5885c71b735 got f34a4280c532db94)
got btree alloc level 0 pos 0:46102:1-0:48212:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:50325:0 len 0 ver 0: seq 5d52fad281bc4b71 written 245 min_key 0:48212:1 durability: 1 ptr: 0:12318:0 gen 0
  node offset 0/245: got wrong btree node (want 5d52fad281bc4b71 got a40fa1d721c2d94f)
got btree alloc level 0 pos 0:48212:1-0:50325:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:50844:0 len 0 ver 0: seq 3aae33a503d21a4d written 245 min_key 0:48732:1 durability: 1 ptr: 0:12318:0 gen 0
  node offset 0/245: got wrong btree node (want 3aae33a503d21a4d got a40fa1d721c2d94f)
got btree alloc level 0 pos 0:48212:1-0:50325:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:52437:0 len 0 ver 0: seq a21c7ff71b75bc39 written 245 min_key 0:50325:1 durability: 1 ptr: 0:12340:0 gen 0
  node offset 0/245: got wrong btree node (want a21c7ff71b75bc39 got 4fa3268a730eca22)
got btree alloc level 0 pos 0:50325:1-0:52437:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:52956:0 len 0 ver 0: seq 5b27f924282827b6 written 245 min_key 0:50844:1 durability: 1 ptr: 0:12340:0 gen 0
  node offset 0/245: got wrong btree node (want 5b27f924282827b6 got 4fa3268a730eca22)
got btree alloc level 0 pos 0:50325:1-0:52437:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:54547:0 len 0 ver 0: seq 1ef24f5c209afa8f written 245 min_key 0:52437:1 durability: 1 ptr: 0:12394:0 gen 0
  node offset 0/245: got wrong btree node (want 1ef24f5c209afa8f got 52b93e2e1a623a9d)
got btree alloc level 0 pos 0:52437:1-0:54547:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:55066:0 len 0 ver 0: seq 1f9289b7771f600b written 245 min_key 0:52956:1 durability: 1 ptr: 0:12394:0 gen 0
  node offset 0/245: got wrong btree node (want 1f9289b7771f600b got 52b93e2e1a623a9d)
got btree alloc level 0 pos 0:52437:1-0:54547:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:56658:0 len 0 ver 0: seq cc4bef4e63ac969c written 245 min_key 0:54547:1 durability: 1 ptr: 0:12455:0 gen 0
  node offset 0/245: got wrong btree node (want cc4bef4e63ac969c got e3e65204f68d6e9b)
got btree alloc level 0 pos 0:54547:1-0:56658:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:57177:0 len 0 ver 0: seq 5e7d9045f24479b written 245 min_key 0:55066:1 durability: 1 ptr: 0:12455:0 gen 0
  node offset 0/245: got wrong btree node (want 5e7d9045f24479b got e3e65204f68d6e9b)
got btree alloc level 0 pos 0:54547:1-0:56658:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:58770:0 len 0 ver 0: seq 3a140b0044c6c28f written 245 min_key 0:56658:1 durability: 1 ptr: 0:12456:0 gen 0
  node offset 0/245: got wrong btree node (want 3a140b0044c6c28f got d4e636bbdebc3d43)
got btree alloc level 0 pos 0:56658:1-0:58770:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:59288:0 len 0 ver 0: seq 2da6e3271b3a6827 written 245 min_key 0:57177:1 durability: 1 ptr: 0:12456:0 gen 0
  node offset 0/245: got wrong btree node (want 2da6e3271b3a6827 got d4e636bbdebc3d43)
got btree alloc level 0 pos 0:56658:1-0:58770:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:60879:0 len 0 ver 0: seq 90ec2bf7c2160abe written 245 min_key 0:58770:1 durability: 1 ptr: 0:12465:0 gen 0
  node offset 0/245: got wrong btree node (want 90ec2bf7c2160abe got 66fe9ef5605f3ef8)
got btree alloc level 0 pos 0:58770:1-0:60879:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:61398:0 len 0 ver 0: seq 8e5a336c9fcc0ffc written 245 min_key 0:59288:1 durability: 1 ptr: 0:12465:0 gen 0
  node offset 0/245: got wrong btree node (want 8e5a336c9fcc0ffc got 66fe9ef5605f3ef8)
got btree alloc level 0 pos 0:58770:1-0:60879:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:62990:0 len 0 ver 0: seq f7e8a29d5d442378 written 245 min_key 0:60879:1 durability: 1 ptr: 0:12476:0 gen 0
  node offset 0/245: got wrong btree node (want f7e8a29d5d442378 got c36205516b16ffa6)
got btree alloc level 0 pos 0:61861:1-0:63971:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:63508:0 len 0 ver 0: seq fd91bbe9977a60d0 written 245 min_key 0:61398:1 durability: 1 ptr: 0:12476:0 gen 0
  node offset 0/245: got wrong btree node (want fd91bbe9977a60d0 got c36205516b16ffa6)
got btree alloc level 0 pos 0:61861:1-0:63971:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:65621:0 len 0 ver 0: seq fc616c605123cc8e written 245 min_key 0:63508:1 durability: 1 ptr: 0:12479:0 gen 0
  node offset 0/245: got wrong btree node (want fc616c605123cc8e got 47692375be5dcf7b)
got btree alloc level 0 pos 0:63971:1-0:66086:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:67743:0 len 0 ver 0: seq 8fa8605c05b92c7b written 245 min_key 0:65621:1 durability: 1 ptr: 0:12634:0 gen 0
  node offset 0/245: got wrong btree node (want 8fa8605c05b92c7b got 6a822f2e4770ab2)
got btree alloc level 0 pos 0:66086:1-0:68209:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:69868:0 len 0 ver 0: seq dbd2621b43779abc written 245 min_key 0:67743:1 durability: 1 ptr: 0:12636:0 gen 0
  node offset 0/245: got wrong btree node (want dbd2621b43779abc got d29adb33cc845ef5)
got btree alloc level 0 pos 0:68209:1-0:70331:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:71981:0 len 0 ver 0: seq 4ccb9d2e8042f724 written 245 min_key 0:69868:1 durability: 1 ptr: 0:12638:0 gen 0
  node offset 0/245: got wrong btree node (want 4ccb9d2e8042f724 got 5d1f4af6eea2a870)
got btree alloc level 0 pos 0:70331:1-0:72444:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:74099:0 len 0 ver 0: seq ddbae044af6ed2c2 written 245 min_key 0:71981:1 durability: 1 ptr: 0:12693:0 gen 0
  node offset 0/245: got wrong btree node (want ddbae044af6ed2c2 got a9e164280db44685)
got btree alloc level 0 pos 0:72444:1-0:74561:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:76217:0 len 0 ver 0: seq aa074e817740af63 written 245 min_key 0:74099:1 durability: 1 ptr: 0:12696:0 gen 0
  node offset 0/245: got wrong btree node (want aa074e817740af63 got fe1063f0d941723e)
got btree alloc level 0 pos 0:74561:1-0:76680:0
error validating btree node on /dev/mapper/agogusssd at btree alloc level 0/0
  u64s 11 type btree_ptr_v2 0:78342:0 len 0 ver 0: seq bc5e96759ef75864 written 245 min_key 0:76217:1 durability: 1 ptr: 0:12713:0 gen 0
  node offset 0/245: got wrong btree node (want bc5e96759ef75864 got 620be310e8a13abe)
got btree alloc level 0 pos 0:76680:1-0:78804:0
bch2_btree_update_start(): error Input/output error
bch2_btree_update_start(): error Input/output error
bch2_btree_update_start(): error Input/output error
bch2_btree_update_start(): error Input/output error
bch2_btree_update_start(): error Input/output error
bch2_btree_update_start(): error Input/output error
bch2_btree_update_start(): error Input/output error
bch2_btree_update_start(): error Input/output error
bch2_btree_update_start(): error Input/output error
bch2_btree_update_start(): error Input/output error
bch2_journal_replay(): error while replaying key at btree bucket_gens level 0: Input/output error
bch2_journal_replay(): error Input/output error
bch2_fs_recovery(): error Input/output error
bch2_fs_start(): error starting filesystem Input/output error

I really hope I can get this data un-eaten lol

@Architector4
Copy link
Author

Architector4 commented Feb 20, 2024

This is all running on Linux 6.7.2; let me try to build archiso with Linux 6.7.5 and run off that...

edit: no meaningful change as far as i can tell; though a run without --reconstruct_alloc now hangs forever instead of making a loads of Input/output error messages and quitting lol

@Architector4
Copy link
Author

Architector4 commented Feb 20, 2024

here is running latest git built in debug mode and optimizations disabled and with latest Arch packages and kernel; seems to be more populated with proper debuginfo

Starting program: /mnt/home/architector4/bcachefs-tools/bcachefs-tools/target/debug/bcachefs fsck --reconstruct_alloc /dev/mapper/agogusssd

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                
[New Thread 0x7ffff7b8f680 (LWP 5936)]
[New Thread 0x7ffff7b86680 (LWP 5937)]
[New Thread 0x7ffff7b7d680 (LWP 5938)]
[New Thread 0x7ffff7b74680 (LWP 5939)]
[New Thread 0x7ffff7b6b680 (LWP 5940)]
[New Thread 0x7ffff7b62680 (LWP 5941)]
[New Thread 0x7ffff7b59680 (LWP 5942)]
[New Thread 0x7ffff7b50680 (LWP 5943)]
[New Thread 0x7ffff7af4680 (LWP 5944)]
[New Thread 0x7ffff7aeb680 (LWP 5945)]
[New Thread 0x7ffff7ae2680 (LWP 5946)]
[New Thread 0x7ffff7ad9680 (LWP 5948)]
[New Thread 0x7ffff7ad0680 (LWP 5949)]
[New Thread 0x7ffff7a85680 (LWP 5950)]
[New Thread 0x7ffff715b680 (LWP 5951)]
[Thread 0x7ffff715b680 (LWP 5951) exited]
[Thread 0x7ffff7a85680 (LWP 5950) exited]
[Thread 0x7ffff7ad0680 (LWP 5949) exited]
[Thread 0x7ffff7ad9680 (LWP 5948) exited]
[Thread 0x7ffff7ae2680 (LWP 5946) exited]
[Thread 0x7ffff7aeb680 (LWP 5945) exited]
[Thread 0x7ffff7af4680 (LWP 5944) exited]
[New Thread 0x7ffff7af4680 (LWP 5952)]
[New Thread 0x7ffff7aeb680 (LWP 5953)]
[New Thread 0x7ffff7ae2680 (LWP 5954)]
[New Thread 0x7ffff7ad9680 (LWP 5955)]
[New Thread 0x7ffff7ad0680 (LWP 5956)]
[New Thread 0x7ffff7a85680 (LWP 5957)]
[New Thread 0x7ffff715b680 (LWP 5958)]
[New Thread 0x7ffff7b19680 (LWP 5960)]
[New Thread 0x7ffff6a00680 (LWP 5961)]

Thread 1 "bcachefs" received signal SIGSEGV, Segmentation fault.
__memcpy_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:489

                                                                                                                                                                                                                                                
489		VMOVA	%VMM(2), VEC_SIZE(%rdi)

Thread 25 (Thread 0x7ffff6a00680 (LWP 5961) "events"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007ffff7faf596 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=-1, op=0, uaddr=0x7fff0c000bb0) at ../include/urcu/futex.h:85
#2  futex_noasync (timeout=0x0, uaddr2=0x0, val3=0, val=-1, op=0, uaddr=0x7fff0c000bb0) at ../include/urcu/futex.h:90
#3  call_rcu_wait (crdp=0x7fff0c000b70) at /usr/src/debug/liburcu/userspace-rcu-0.14.0/src/urcu-call-rcu-impl.h:248
#4  call_rcu_thread (arg=0x7fff0c000b70) at /usr/src/debug/liburcu/userspace-rcu-0.14.0/src/urcu-call-rcu-impl.h:400
#5  0x00007ffff7c3555a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007ffff7cb2a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 24 (Thread 0x7ffff7b19680 (LWP 5960) "bch-reclaim//de"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000555555784aa5 in futex (uaddr=0x555555cd902c, op=128, val=16385, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x0000555555784f49 in schedule () at linux/sched.c:41
#3  0x000055555578512a in schedule_timeout (timeout=100) at linux/sched.c:96
#4  0x00005555557185be in bch2_journal_reclaim_thread (arg=0x555555d19880) at libbcachefs/journal_reclaim.c:749
#5  0x000055555577d744 in kthread_start_fn (data=0x555555cd9010) at linux/kthread.c:25
#6  0x00007ffff7c3555a in start_thread (arg=<optimized out>) at pthread_create.c:447
#7  0x00007ffff7cb2a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 23 (Thread 0x7ffff715b680 (LWP 5958) "btree_update"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000555555784aa5 in futex (uaddr=0x555555dde03c, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x0000555555784f49 in schedule () at linux/sched.c:41
#3  0x000055555578d360 in worker_thread (arg=0x555555c63080) at linux/workqueue.c:252
#4  0x000055555577d744 in kthread_start_fn (data=0x555555dde020) at linux/kthread.c:25
#5  0x00007ffff7c3555a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007ffff7cb2a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 22 (Thread 0x7ffff7a85680 (LWP 5957) "bcachefs_journa"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000555555784aa5 in futex (uaddr=0x555555c692ac, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x0000555555784f49 in schedule () at linux/sched.c:41
#3  0x000055555578d360 in worker_thread (arg=0x555555c69840) at linux/workqueue.c:252
#4  0x000055555577d744 in kthread_start_fn (data=0x555555c69290) at linux/kthread.c:25
#5  0x00007ffff7c3555a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007ffff7cb2a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 21 (Thread 0x7ffff7ad0680 (LWP 5956) "bcachefs_write_"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000555555784aa5 in futex (uaddr=0x555555c61eec, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x0000555555784f49 in schedule () at linux/sched.c:41
#3  0x000055555578d360 in worker_thread (arg=0x555555c69500) at linux/workqueue.c:252
#4  0x000055555577d744 in kthread_start_fn (data=0x555555c61ed0) at linux/kthread.c:25
#5  0x00007ffff7c3555a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007ffff7cb2a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 20 (Thread 0x7ffff7ad9680 (LWP 5955) "bcachefs_io"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000555555784aa5 in futex (uaddr=0x555555c61c2c, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x0000555555784f49 in schedule () at linux/sched.c:41
#3  0x000055555578d360 in worker_thread (arg=0x555555c69480) at linux/workqueue.c:252
#4  0x000055555577d744 in kthread_start_fn (data=0x555555c61c10) at linux/kthread.c:25
#5  0x00007ffff7c3555a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007ffff7cb2a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 19 (Thread 0x7ffff7ae2680 (LWP 5954) "bcachefs_copygc"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000555555784aa5 in futex (uaddr=0x555555c6196c, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x0000555555784f49 in schedule () at linux/sched.c:41
#3  0x000055555578d360 in worker_thread (arg=0x555555c69400) at linux/workqueue.c:252
#4  0x000055555577d744 in kthread_start_fn (data=0x555555c61950) at linux/kthread.c:25
#5  0x00007ffff7c3555a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007ffff7cb2a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 18 (Thread 0x7ffff7aeb680 (LWP 5953) "bcachefs_btree_"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000555555784aa5 in futex (uaddr=0x555555c616ac, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x0000555555784f49 in schedule () at linux/sched.c:41
#3  0x000055555578d360 in worker_thread (arg=0x555555c60980) at linux/workqueue.c:252
#4  0x000055555577d744 in kthread_start_fn (data=0x555555c61690) at linux/kthread.c:25
#5  0x00007ffff7c3555a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007ffff7cb2a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 17 (Thread 0x7ffff7af4680 (LWP 5952) "bcachefs"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000555555784aa5 in futex (uaddr=0x555555c613ec, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x0000555555784f49 in schedule () at linux/sched.c:41
#3  0x000055555578d360 in worker_thread (arg=0x555555c6f040) at linux/workqueue.c:252
#4  0x000055555577d744 in kthread_start_fn (data=0x555555c613d0) at linux/kthread.c:25
#5  0x00007ffff7c3555a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007ffff7cb2a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 9 (Thread 0x7ffff7b50680 (LWP 5943) "timers"):
#0  0x00007ffff7c31ebe in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7ffff7b4f930, op=393, expected=0, futex_word=0x555555c5d3e8 <timer_cond+40>) at futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x555555c5d3e8 <timer_cond+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7ffff7b4f930, private=private@entry=0, cancel=cancel@entry=true) at futex-internal.c:87
#2  0x00007ffff7c31f3f in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x555555c5d3e8 <timer_cond+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7ffff7b4f930, private=private@entry=0) at futex-internal.c:139
#3  0x00007ffff7c34a65 in __pthread_cond_wait_common (abstime=0x7ffff7b4f930, clockid=0, mutex=0x555555c5d380 <timer_lock>, cond=0x555555c5d3c0 <timer_cond>) at pthread_cond_wait.c:503
#4  ___pthread_cond_timedwait64 (cond=0x555555c5d3c0 <timer_cond>, mutex=0x555555c5d380 <timer_lock>, abstime=0x7ffff7b4f930) at pthread_cond_wait.c:643
#5  0x000055555578b220 in timer_thread (arg=0x0) at linux/timer.c:294
#6  0x000055555577d744 in kthread_start_fn (data=0x555555c5fa00) at linux/kthread.c:25
#7  0x00007ffff7c3555a in start_thread (arg=<optimized out>) at pthread_create.c:447
#8  0x00007ffff7cb2a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 8 (Thread 0x7ffff7b59680 (LWP 5942) "shrinkers"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000555555785ca0 in futex (uaddr=0x555555c5f3bc, op=137, val=1, timeout=0x7ffff7b58940, uaddr2=0x0, val3=-1) at /usr/include/urcu/futex.h:81
#2  0x00005555557860bf in shrinker_thread (arg=0x0) at linux/shrinker.c:106
#3  0x000055555577d744 in kthread_start_fn (data=0x555555c5f3a0) at linux/kthread.c:25
#4  0x00007ffff7c3555a in start_thread (arg=<optimized out>) at pthread_create.c:447
#5  0x00007ffff7cb2a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 7 (Thread 0x7ffff7b62680 (LWP 5941) "events_freezabl"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000555555784aa5 in futex (uaddr=0x555555c5f16c, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x0000555555784f49 in schedule () at linux/sched.c:41
#3  0x000055555578d360 in worker_thread (arg=0x555555c5f100) at linux/workqueue.c:252
#4  0x000055555577d744 in kthread_start_fn (data=0x555555c5f150) at linux/kthread.c:25
#5  0x00007ffff7c3555a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007ffff7cb2a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 6 (Thread 0x7ffff7b6b680 (LWP 5940) "events_unbound"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000555555784aa5 in futex (uaddr=0x555555c5eeac, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x0000555555784f49 in schedule () at linux/sched.c:41
#3  0x000055555578d360 in worker_thread (arg=0x555555c5ee40) at linux/workqueue.c:252
#4  0x000055555577d744 in kthread_start_fn (data=0x555555c5ee90) at linux/kthread.c:25
#5  0x00007ffff7c3555a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007ffff7cb2a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 5 (Thread 0x7ffff7b74680 (LWP 5939) "events_long"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000555555784aa5 in futex (uaddr=0x555555c5ebec, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x0000555555784f49 in schedule () at linux/sched.c:41
#3  0x000055555578d360 in worker_thread (arg=0x555555c5eb80) at linux/workqueue.c:252
#4  0x000055555577d744 in kthread_start_fn (data=0x555555c5ebd0) at linux/kthread.c:25
#5  0x00007ffff7c3555a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007ffff7cb2a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 4 (Thread 0x7ffff7b7d680 (LWP 5938) "events_highpri"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000555555784aa5 in futex (uaddr=0x555555c5e92c, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x0000555555784f49 in schedule () at linux/sched.c:41
#3  0x000055555578d360 in worker_thread (arg=0x555555c5e8c0) at linux/workqueue.c:252
#4  0x000055555577d744 in kthread_start_fn (data=0x555555c5e910) at linux/kthread.c:25
#5  0x00007ffff7c3555a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007ffff7cb2a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 3 (Thread 0x7ffff7b86680 (LWP 5937) "events"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000555555784aa5 in futex (uaddr=0x555555c5e66c, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x0000555555784f49 in schedule () at linux/sched.c:41
#3  0x000055555578d360 in worker_thread (arg=0x555555c5e600) at linux/workqueue.c:252
#4  0x000055555577d744 in kthread_start_fn (data=0x555555c5e650) at linux/kthread.c:25
#5  0x00007ffff7c3555a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007ffff7cb2a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 2 (Thread 0x7ffff7b8f680 (LWP 5936) "aio_completion"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007ffff7db2277 in __io_getevents_0_4 () from /usr/lib/libaio.so.1
#2  0x0000555555777c06 in aio_completion_thread (arg=0x0) at linux/blkdev.c:276
#3  0x000055555577d744 in kthread_start_fn (data=0x555555c5e380) at linux/kthread.c:25
#4  0x00007ffff7c3555a in start_thread (arg=<optimized out>) at pthread_create.c:447
#5  0x00007ffff7cb2a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 1 (Thread 0x7ffff7b90c00 (LWP 5935) "bcachefs"):
#0  __memcpy_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:489
#1  0x0000555555661285 in __move_gap (array=0x7fff19e01000, element_size=24, nr=71, size=4194304, old_gap=319544, new_gap=0) at libbcachefs/util.h:641
#2  0x0000555555662453 in bch2_journal_key_insert_take (c=0x555555cda000, id=BTREE_ID_alloc, level=0, k=0x55555609f040) at libbcachefs/btree_journal_iter.c:221
#3  0x0000555555662549 in bch2_journal_key_insert (c=0x555555cda000, id=BTREE_ID_alloc, level=0, k=0x555555c6ef00) at libbcachefs/btree_journal_iter.c:248
#4  0x0000555555676bed in do_bch2_trans_commit_to_journal_replay (trans=0x555557591000) at libbcachefs/btree_trans_commit.c:974
#5  0x0000555555676f2c in __bch2_trans_commit (trans=0x555557591000, flags=64) at libbcachefs/btree_trans_commit.c:1033
#6  0x000055555563521a in bch2_trans_commit (trans=0x555557591000, disk_res=0x0, journal_seq=0x0, flags=64) at libbcachefs/btree_update.h:168
#7  0x000055555563d929 in bch2_gc_alloc_done (c=0x555555cda000, metadata_only=false) at libbcachefs/btree_gc.c:1475
#8  0x000055555563f31f in bch2_gc (c=0x555555cda000, initial=true, metadata_only=false) at libbcachefs/btree_gc.c:1823
#9  0x000055555573a948 in bch2_check_allocations (c=0x555555cda000) at libbcachefs/recovery.c:515
#10 0x000055555573afc9 in bch2_run_recovery_pass (c=0x555555cda000, pass=BCH_RECOVERY_PASS_check_allocations) at libbcachefs/recovery.c:685
#11 0x000055555573b067 in bch2_run_recovery_passes (c=0x555555cda000) at libbcachefs/recovery.c:702
#12 0x000055555573bea1 in bch2_fs_recovery (c=0x555555cda000) at libbcachefs/recovery.c:978
#13 0x000055555576a689 in bch2_fs_start (c=0x555555cda000) at libbcachefs/super.c:1024
#14 0x000055555576e07e in bch2_fs_open (devices=0x555555c608d0, nr_devices=1, opts=...) at libbcachefs/super.c:2074
#15 0x00005555555e8e6f in cmd_fsck (argc=1, argv=0x555555c60840) at c_src/cmd_fsck.c:256
#16 0x00005555555b0b96 in bcachefs::handle_c_command (args=..., symlink_cmd=...) at src/bcachefs.rs:57
#17 0x00005555555b18b4 in bcachefs::main () at src/bcachefs.rs:120
Continuing.

Thread 1 "bcachefs" received signal SIGSEGV, Segmentation fault.
__memcpy_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:489
489		VMOVA	%VMM(2), VEC_SIZE(%rdi)
Continuing.
Couldn't get registers: No such process.
[Thread 0x7ffff6a00680 (LWP 5961) exited]
[Thread 0x7ffff7b19680 (LWP 5960) exited]
[Thread 0x7ffff715b680 (LWP 5958) exited]
[Thread 0x7ffff7a85680 (LWP 5957) exited]
[Thread 0x7ffff7ad0680 (LWP 5956) exited]
[Thread 0x7ffff7ad9680 (LWP 5955) exited]
[Thread 0x7ffff7ae2680 (LWP 5954) exited]
[Thread 0x7ffff7aeb680 (LWP 5953) exited]
[Thread 0x7ffff7af4680 (LWP 5952) exited]
[Thread 0x7ffff7b50680 (LWP 5943) exited]
[Thread 0x7ffff7b62680 (LWP 5941) exited]
[Thread 0x7ffff7b6b680 (LWP 5940) exited]
[Thread 0x7ffff7b74680 (LWP 5939) exited]
[Thread 0x7ffff7b7d680 (LWP 5938) exited]
[Thread 0x7ffff7b86680 (LWP 5937) exited]
[Thread 0x7ffff7b8f680 (LWP 5936) exited]
[Thread 0x7ffff7b90c00 (LWP 5935) exited]
[Thread 0x7ffff7b59680 (LWP 5942) exited]
[New process 5935]

Program terminated with signal SIGSEGV, Segmentation fault.
The program no longer exists.
The program is not being run.

And stdout/stderr:

# home/architector4/bcachefs-tools/bcachefs-tools/target/debug/bcachefs fsck --reconstruct_alloc -v /dev/mapper/agogusssd
shutting down
shutdown complete
mounting version 1.6: btree_subvolume_children opts=ro,compression=zstd,root_reserve_percent=1,degraded,fsck,fix_errors=ask,read_only,reconstruct_alloc
recovering from unclean shutdown
superblock requires following recovery passes to be run:
  check_topology,check_subvols,check_dirents
Version upgrade from 1.3: rebalance_work to 1.6: btree_subvolume_children incomplete
Doing compatible version upgrade from 1.3: rebalance_work to 1.6: btree_subvolume_children

starting journal read
journal read done on device /dev/mapper/agogusssd, ret 0
journal read done, replaying entries 853351-857227
dropped unflushed entries 857228-857234
Journal keys: 2174273 read, 319544 after sorting and compacting
alloc_read... done
stripes_read... done
snapshots_read... done
WARNING at libbcachefs/btree_iter.c:2827: btree trans held srcu lock (delaying memory reclaim) for 13 seconds
check_topology... done
[1]    6581 segmentation fault (core dumped)  /home/architector4/bcachefs-tools/bcachefs-tools/target/debug/bcachefs fsck  
/home/architector4/bcachefs-tools/bcachefs-tools/target/debug/bcachefs fsck    23.81s user 2.70s system 88% cpu 29.839 total

@Architector4 Architector4 changed the title fsck --reconstruct_alloc segfaults fsck --reconstruct_alloc segfaults after interrupted run Feb 20, 2024
@Architector4
Copy link
Author

Architector4 commented Feb 20, 2024

...in my case here, i wonder if running reset-counters would help, but i don't think i want to do something stupid that may cause even more damage, so please tell me if that would be a good idea or a bad one or irrelevant

i don't mind losing some data if i can at least restore a meaningful amount of it to be honest

edit: nevermind, i made a full copy of the filesystem partition on another drive and did bcachefs reset-counters and it had no effect

@Architector4
Copy link
Author

Architector4 commented Feb 21, 2024

the segfault seems to be at inserting a key into the journal, which suggests to me the journal itself might have gotten messed up

is there a way to just delete the journal entries without committing them or doing anything with them? i don't care much if this messes up some filesystem structure if i can get this thing to run and make it at least mountable; and if anything i made a full copy of the filesystem partition to test on lol

@Architector4
Copy link
Author

Issue persists with latest commit 25e84a9

stdout/stderr:

# /home/architector4/bcachefs-tools/bcachefs-tools/target/debug/bcachefs fsck --reconstruct_alloc /dev/mapper/agogusssd
shutting down
shutdown complete
mounting version 1.6: btree_subvolume_children opts=ro,compression=zstd,root_reserve_percent=1,degraded,fsck,fix_errors=ask,read_only,reconstruct_alloc
recovering from unclean shutdown
superblock requires following recovery passes to be run:
  check_topology,check_subvols,check_dirents
Version upgrade from 1.3: rebalance_work to 1.6: btree_subvolume_children incomplete
Doing compatible version upgrade from 1.3: rebalance_work to 1.6: btree_subvolume_children

starting journal read
journal read done on device /dev/mapper/agogusssd, ret 0
journal read done, replaying entries 853351-857227
dropped unflushed entries 857228-857234
Journal keys: 2174273 read, 319544 after sorting and compacting
alloc_read... done
stripes_read... done
snapshots_read... done
WARNING at libbcachefs/btree_iter.c:2827: btree trans held srcu lock (delaying memory reclaim) for 12 seconds
check_topology... done
[1]    17113 segmentation fault (core dumped)  /home/architector4/bcachefs-tools/bcachefs-tools/target/debug/bcachefs fsck 

gdb:

Starting program: /mnt/home/architector4/bcachefs-tools/bcachefs-tools/target/debug/bcachefs fsck --reconstruct_alloc /dev/mapper/agogusssd

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".

                                                                                                                                                                                                                                                

                                                                                                                                                                                                                                                
[New Thread 0x7ffff7b8f680 (LWP 17054)]
[New Thread 0x7ffff7b86680 (LWP 17055)]
[New Thread 0x7ffff7b7d680 (LWP 17056)]
[New Thread 0x7ffff7b74680 (LWP 17057)]
[New Thread 0x7ffff7b6b680 (LWP 17058)]
[New Thread 0x7ffff7b62680 (LWP 17059)]
[New Thread 0x7ffff7b59680 (LWP 17060)]
[New Thread 0x7ffff7b50680 (LWP 17061)]
[New Thread 0x7ffff7af4680 (LWP 17062)]
[New Thread 0x7ffff7aeb680 (LWP 17063)]
[New Thread 0x7ffff7ae2680 (LWP 17064)]
[New Thread 0x7ffff7ad9680 (LWP 17065)]
[New Thread 0x7ffff7ad0680 (LWP 17066)]
[New Thread 0x7ffff7a85680 (LWP 17067)]
[New Thread 0x7ffff715b680 (LWP 17068)]
[Thread 0x7ffff715b680 (LWP 17068) exited]
[Thread 0x7ffff7a85680 (LWP 17067) exited]
[New Thread 0x7ffff7af4680 (LWP 17069)]
[Thread 0x7ffff7ad0680 (LWP 17066) exited]
[Thread 0x7ffff7ad9680 (LWP 17065) exited]
[Thread 0x7ffff7ae2680 (LWP 17064) exited]
[Thread 0x7ffff7aeb680 (LWP 17063) exited]
[Thread 0x7ffff7af4680 (LWP 17062) exited]
[New Thread 0x7ffff7aeb680 (LWP 17070)]
[New Thread 0x7ffff7ae2680 (LWP 17071)]
[New Thread 0x7ffff7ad9680 (LWP 17072)]
[New Thread 0x7ffff7ad0680 (LWP 17073)]
[New Thread 0x7ffff7a85680 (LWP 17074)]
[New Thread 0x7ffff715b680 (LWP 17075)]
[New Thread 0x7ffff7b19680 (LWP 17076)]
[New Thread 0x7ffff6a00680 (LWP 17077)]

Thread 1 "bcachefs" received signal SIGSEGV, Segmentation fault.
__memcpy_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:489

                                                                                                                                                                                                                                                
489		VMOVA	%VMM(2), VEC_SIZE(%rdi)

Thread 25 (Thread 0x7ffff6a00680 (LWP 17077) "events"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007ffff7faf596 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=-1, op=0, uaddr=0x7fff0c000bb0) at ../include/urcu/futex.h:85
#2  futex_noasync (timeout=0x0, uaddr2=0x0, val3=0, val=-1, op=0, uaddr=0x7fff0c000bb0) at ../include/urcu/futex.h:90
#3  call_rcu_wait (crdp=0x7fff0c000b70) at /usr/src/debug/liburcu/userspace-rcu-0.14.0/src/urcu-call-rcu-impl.h:248
#4  call_rcu_thread (arg=0x7fff0c000b70) at /usr/src/debug/liburcu/userspace-rcu-0.14.0/src/urcu-call-rcu-impl.h:400
#5  0x00007ffff7c3555a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007ffff7cb2a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 24 (Thread 0x7ffff7b19680 (LWP 17076) "bch-reclaim//de"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00005555557863d7 in futex (uaddr=0x555555cdb02c, op=128, val=16385, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x000055555578687b in schedule () at linux/sched.c:41
#3  0x0000555555786a5c in schedule_timeout (timeout=100) at linux/sched.c:96
#4  0x0000555555719e9b in bch2_journal_reclaim_thread (arg=0x555555d1b8c0) at libbcachefs/journal_reclaim.c:749
#5  0x000055555577f076 in kthread_start_fn (data=0x555555cdb010) at linux/kthread.c:25
#6  0x00007ffff7c3555a in start_thread (arg=<optimized out>) at pthread_create.c:447
#7  0x00007ffff7cb2a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 23 (Thread 0x7ffff715b680 (LWP 17075) "btree_update"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00005555557863d7 in futex (uaddr=0x555555de003c, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x000055555578687b in schedule () at linux/sched.c:41
#3  0x000055555578ec92 in worker_thread (arg=0x555555c65080) at linux/workqueue.c:252
#4  0x000055555577f076 in kthread_start_fn (data=0x555555de0020) at linux/kthread.c:25
#5  0x00007ffff7c3555a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007ffff7cb2a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 22 (Thread 0x7ffff7a85680 (LWP 17074) "bcachefs_journa"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00005555557863d7 in futex (uaddr=0x555555c6b2ac, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x000055555578687b in schedule () at linux/sched.c:41
#3  0x000055555578ec92 in worker_thread (arg=0x555555c6b840) at linux/workqueue.c:252
#4  0x000055555577f076 in kthread_start_fn (data=0x555555c6b290) at linux/kthread.c:25
#5  0x00007ffff7c3555a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007ffff7cb2a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 21 (Thread 0x7ffff7ad0680 (LWP 17073) "bcachefs_write_"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00005555557863d7 in futex (uaddr=0x555555c63eec, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x000055555578687b in schedule () at linux/sched.c:41
#3  0x000055555578ec92 in worker_thread (arg=0x555555c6b500) at linux/workqueue.c:252
#4  0x000055555577f076 in kthread_start_fn (data=0x555555c63ed0) at linux/kthread.c:25
#5  0x00007ffff7c3555a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007ffff7cb2a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 20 (Thread 0x7ffff7ad9680 (LWP 17072) "bcachefs_io"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00005555557863d7 in futex (uaddr=0x555555c63c2c, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x000055555578687b in schedule () at linux/sched.c:41
#3  0x000055555578ec92 in worker_thread (arg=0x555555c6b480) at linux/workqueue.c:252
#4  0x000055555577f076 in kthread_start_fn (data=0x555555c63c10) at linux/kthread.c:25
#5  0x00007ffff7c3555a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007ffff7cb2a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 19 (Thread 0x7ffff7ae2680 (LWP 17071) "bcachefs_copygc"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00005555557863d7 in futex (uaddr=0x555555c6396c, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x000055555578687b in schedule () at linux/sched.c:41
#3  0x000055555578ec92 in worker_thread (arg=0x555555c6b400) at linux/workqueue.c:252
#4  0x000055555577f076 in kthread_start_fn (data=0x555555c63950) at linux/kthread.c:25
#5  0x00007ffff7c3555a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007ffff7cb2a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 18 (Thread 0x7ffff7aeb680 (LWP 17070) "bcachefs_btree_"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00005555557863d7 in futex (uaddr=0x555555c636ac, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x000055555578687b in schedule () at linux/sched.c:41
#3  0x000055555578ec92 in worker_thread (arg=0x555555c62980) at linux/workqueue.c:252
#4  0x000055555577f076 in kthread_start_fn (data=0x555555c63690) at linux/kthread.c:25
#5  0x00007ffff7c3555a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007ffff7cb2a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 17 (Thread 0x7ffff7af4680 (LWP 17069) "bcachefs"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00005555557863d7 in futex (uaddr=0x555555c633ec, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x000055555578687b in schedule () at linux/sched.c:41
#3  0x000055555578ec92 in worker_thread (arg=0x555555c71040) at linux/workqueue.c:252
#4  0x000055555577f076 in kthread_start_fn (data=0x555555c633d0) at linux/kthread.c:25
#5  0x00007ffff7c3555a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007ffff7cb2a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 9 (Thread 0x7ffff7b50680 (LWP 17061) "timers"):
#0  0x00007ffff7c31ebe in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7ffff7b4f930, op=393, expected=0, futex_word=0x555555c5f3ec <timer_cond+44>) at futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x555555c5f3ec <timer_cond+44>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7ffff7b4f930, private=private@entry=0, cancel=cancel@entry=true) at futex-internal.c:87
#2  0x00007ffff7c31f3f in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x555555c5f3ec <timer_cond+44>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7ffff7b4f930, private=private@entry=0) at futex-internal.c:139
#3  0x00007ffff7c34a65 in __pthread_cond_wait_common (abstime=0x7ffff7b4f930, clockid=0, mutex=0x555555c5f380 <timer_lock>, cond=0x555555c5f3c0 <timer_cond>) at pthread_cond_wait.c:503
#4  ___pthread_cond_timedwait64 (cond=0x555555c5f3c0 <timer_cond>, mutex=0x555555c5f380 <timer_lock>, abstime=0x7ffff7b4f930) at pthread_cond_wait.c:643
#5  0x000055555578cb52 in timer_thread (arg=0x0) at linux/timer.c:294
#6  0x000055555577f076 in kthread_start_fn (data=0x555555c61a00) at linux/kthread.c:25
#7  0x00007ffff7c3555a in start_thread (arg=<optimized out>) at pthread_create.c:447
#8  0x00007ffff7cb2a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 8 (Thread 0x7ffff7b59680 (LWP 17060) "shrinkers"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00005555557875d2 in futex (uaddr=0x555555c613bc, op=137, val=1, timeout=0x7ffff7b58940, uaddr2=0x0, val3=-1) at /usr/include/urcu/futex.h:81
#2  0x00005555557879f1 in shrinker_thread (arg=0x0) at linux/shrinker.c:106
#3  0x000055555577f076 in kthread_start_fn (data=0x555555c613a0) at linux/kthread.c:25
#4  0x00007ffff7c3555a in start_thread (arg=<optimized out>) at pthread_create.c:447
#5  0x00007ffff7cb2a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 7 (Thread 0x7ffff7b62680 (LWP 17059) "events_freezabl"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00005555557863d7 in futex (uaddr=0x555555c6116c, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x000055555578687b in schedule () at linux/sched.c:41
#3  0x000055555578ec92 in worker_thread (arg=0x555555c61100) at linux/workqueue.c:252
#4  0x000055555577f076 in kthread_start_fn (data=0x555555c61150) at linux/kthread.c:25
#5  0x00007ffff7c3555a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007ffff7cb2a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 6 (Thread 0x7ffff7b6b680 (LWP 17058) "events_unbound"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00005555557863d7 in futex (uaddr=0x555555c60eac, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x000055555578687b in schedule () at linux/sched.c:41
#3  0x000055555578ec92 in worker_thread (arg=0x555555c60e40) at linux/workqueue.c:252
#4  0x000055555577f076 in kthread_start_fn (data=0x555555c60e90) at linux/kthread.c:25
#5  0x00007ffff7c3555a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007ffff7cb2a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 5 (Thread 0x7ffff7b74680 (LWP 17057) "events_long"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00005555557863d7 in futex (uaddr=0x555555c60bec, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x000055555578687b in schedule () at linux/sched.c:41
#3  0x000055555578ec92 in worker_thread (arg=0x555555c60b80) at linux/workqueue.c:252
#4  0x000055555577f076 in kthread_start_fn (data=0x555555c60bd0) at linux/kthread.c:25
#5  0x00007ffff7c3555a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007ffff7cb2a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 4 (Thread 0x7ffff7b7d680 (LWP 17056) "events_highpri"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00005555557863d7 in futex (uaddr=0x555555c6092c, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x000055555578687b in schedule () at linux/sched.c:41
#3  0x000055555578ec92 in worker_thread (arg=0x555555c608c0) at linux/workqueue.c:252
#4  0x000055555577f076 in kthread_start_fn (data=0x555555c60910) at linux/kthread.c:25
#5  0x00007ffff7c3555a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007ffff7cb2a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 3 (Thread 0x7ffff7b86680 (LWP 17055) "events"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00005555557863d7 in futex (uaddr=0x555555c6066c, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x000055555578687b in schedule () at linux/sched.c:41
#3  0x000055555578ec92 in worker_thread (arg=0x555555c60600) at linux/workqueue.c:252
#4  0x000055555577f076 in kthread_start_fn (data=0x555555c60650) at linux/kthread.c:25
#5  0x00007ffff7c3555a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007ffff7cb2a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 2 (Thread 0x7ffff7b8f680 (LWP 17054) "aio_completion"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007ffff7db2277 in __io_getevents_0_4 () from /usr/lib/libaio.so.1
#2  0x0000555555779538 in aio_completion_thread (arg=0x0) at linux/blkdev.c:276
#3  0x000055555577f076 in kthread_start_fn (data=0x555555c60380) at linux/kthread.c:25
#4  0x00007ffff7c3555a in start_thread (arg=<optimized out>) at pthread_create.c:447
#5  0x00007ffff7cb2a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 1 (Thread 0x7ffff7b90c00 (LWP 17053) "bcachefs"):
#0  __memcpy_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:489
#1  0x0000555555661ddb in __move_gap (array=0x7fff19e01000, element_size=24, nr=71, size=4194304, old_gap=319544, new_gap=0) at libbcachefs/util.h:641
#2  0x0000555555662fa9 in bch2_journal_key_insert_take (c=0x555555cdc000, id=BTREE_ID_alloc, level=0, k=0x555555c82a40) at libbcachefs/btree_journal_iter.c:221
#3  0x000055555566309f in bch2_journal_key_insert (c=0x555555cdc000, id=BTREE_ID_alloc, level=0, k=0x555555c70f00) at libbcachefs/btree_journal_iter.c:248
#4  0x000055555567774f in do_bch2_trans_commit_to_journal_replay (trans=0x555557ffc000) at libbcachefs/btree_trans_commit.c:974
#5  0x0000555555677a8e in __bch2_trans_commit (trans=0x555557ffc000, flags=64) at libbcachefs/btree_trans_commit.c:1033
#6  0x0000555555635d5f in bch2_trans_commit (trans=0x555557ffc000, disk_res=0x0, journal_seq=0x0, flags=64) at libbcachefs/btree_update.h:168
#7  0x000055555563e46e in bch2_gc_alloc_done (c=0x555555cdc000, metadata_only=false) at libbcachefs/btree_gc.c:1475
#8  0x000055555563fe64 in bch2_gc (c=0x555555cdc000, initial=true, metadata_only=false) at libbcachefs/btree_gc.c:1823
#9  0x000055555573c25b in bch2_check_allocations (c=0x555555cdc000) at libbcachefs/recovery.c:515
#10 0x000055555573c8dc in bch2_run_recovery_pass (c=0x555555cdc000, pass=BCH_RECOVERY_PASS_check_allocations) at libbcachefs/recovery.c:685
#11 0x000055555573c97a in bch2_run_recovery_passes (c=0x555555cdc000) at libbcachefs/recovery.c:702
#12 0x000055555573d7b4 in bch2_fs_recovery (c=0x555555cdc000) at libbcachefs/recovery.c:978
#13 0x000055555576bfb2 in bch2_fs_start (c=0x555555cdc000) at libbcachefs/super.c:1024
#14 0x000055555576f9b0 in bch2_fs_open (devices=0x555555c628d0, nr_devices=1, opts=...) at libbcachefs/super.c:2074
#15 0x00005555555e8e6b in cmd_fsck (argc=1, argv=0x555555c62840) at c_src/cmd_fsck.c:256
#16 0x00005555555b0b86 in bcachefs::handle_c_command (args=..., symlink_cmd=...) at src/bcachefs.rs:57
#17 0x00005555555b18a4 in bcachefs::main () at src/bcachefs.rs:120

@Architector4
Copy link
Author

it appears the issue is that in bch2_journal_key_insert_take, when move_gap is used, the keys->gap value for some reason is way higher than keys->nr, meaning the gap starts farther than the last element in the allocation and hence ends after it, which is what causes the segfault.

Adding a small if statement right before the move_gap invocation seems to fix it and let fsck continue, but may just be a horrible hack that worsens everything. At least I have my separate backup copy of the partition in case it does.

if(keys->gap > keys->nr) //bad hack?
        keys->gap = keys->nr;

I'll wait for it finish now...

@Architector4
Copy link
Author

Architector4 commented Feb 21, 2024

...Now it looks like it deadlocked on futexes. I think I experienced the same problem before, but didn't bother with it and just reran fsck with different params until it worked, but that doesn't seem feasible here, as I need to make it finish with --reconstruct_alloc specifically.

Interrupting and rerunning it seems to just give me the same effect.

Here's gdb backtraces:

  Id   Target Id                                           Frame 
* 1    Thread 0x7fc31e8e2c00 (LWP 25727) "bcachefs"        syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  2    Thread 0x7fc31e8e1680 (LWP 25728) "aio_completion"  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  3    Thread 0x7fc31e8d8680 (LWP 25729) "events"          syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  4    Thread 0x7fc31e8cf680 (LWP 25730) "events_highpri"  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  5    Thread 0x7fc31e8c6680 (LWP 25731) "events_long"     syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  6    Thread 0x7fc31e8bd680 (LWP 25732) "events_unbound"  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  7    Thread 0x7fc31e8b4680 (LWP 25733) "events_freezabl" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  8    Thread 0x7fc31e8ab680 (LWP 25734) "shrinkers"       syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  9    Thread 0x7fc31e8a2680 (LWP 25735) "timers"          0x00007fc31e983ebe in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7fc31e8a1930, op=393, expected=0, futex_word=0x601d598973e8 <timer_cond+40>) at futex-internal.c:57
  10   Thread 0x7fc31e846680 (LWP 25743) "bcachefs"        syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  11   Thread 0x7fc31e83d680 (LWP 25744) "bcachefs_btree_" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  12   Thread 0x7fc31e834680 (LWP 25745) "bcachefs_copygc" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  13   Thread 0x7fc31e82b680 (LWP 25746) "bcachefs_io"     syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  14   Thread 0x7fc31e822680 (LWP 25747) "bcachefs_write_" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  15   Thread 0x7fc31e7d7680 (LWP 25748) "bcachefs_journa" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  16   Thread 0x7fc31dead680 (LWP 25749) "btree_update"    syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  17   Thread 0x7fc31e86b680 (LWP 25750) "bch-reclaim//de" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  18   Thread 0x7fc31d600680 (LWP 25751) "events"          syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  19   Thread 0x7fc314961680 (LWP 25754) "bch-gc//dev/map" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38

Thread 19 (Thread 0x7fc314961680 (LWP 25754) "bch-gc//dev/map"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000601d593be3fc in futex (uaddr=0x601d5a27a17c, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x0000601d593be8a0 in schedule () at linux/sched.c:41
#3  0x0000601d59278f3c in bch2_gc_thread (arg=0x601d5a2b3000) at libbcachefs/btree_gc.c:2018
#4  0x0000601d593b709b in kthread_start_fn (data=0x601d5a27a160) at linux/kthread.c:25
#5  0x00007fc31e98755a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007fc31ea04a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 18 (Thread 0x7fc31d600680 (LWP 25751) "events"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007fc31ed01596 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=-1, op=0, uaddr=0x7fc260000bb0) at ../include/urcu/futex.h:85
#2  futex_noasync (timeout=0x0, uaddr2=0x0, val3=0, val=-1, op=0, uaddr=0x7fc260000bb0) at ../include/urcu/futex.h:90
#3  call_rcu_wait (crdp=0x7fc260000b70) at /usr/src/debug/liburcu/userspace-rcu-0.14.0/src/urcu-call-rcu-impl.h:248
#4  call_rcu_thread (arg=0x7fc260000b70) at /usr/src/debug/liburcu/userspace-rcu-0.14.0/src/urcu-call-rcu-impl.h:400
#5  0x00007fc31e98755a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007fc31ea04a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 17 (Thread 0x7fc31e86b680 (LWP 25750) "bch-reclaim//de"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000601d593be3fc in futex (uaddr=0x601d5a23ca2c, op=128, val=16385, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x0000601d593be8a0 in schedule () at linux/sched.c:41
#3  0x0000601d593bea81 in schedule_timeout (timeout=100) at linux/sched.c:96
#4  0x0000601d59351ec0 in bch2_journal_reclaim_thread (arg=0x601d5a2f28c0) at libbcachefs/journal_reclaim.c:749
#5  0x0000601d593b709b in kthread_start_fn (data=0x601d5a23ca10) at linux/kthread.c:25
#6  0x00007fc31e98755a in start_thread (arg=<optimized out>) at pthread_create.c:447
#7  0x00007fc31ea04a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 16 (Thread 0x7fc31dead680 (LWP 25749) "btree_update"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000601d593be3fc in futex (uaddr=0x601d5a3b703c, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x0000601d593be8a0 in schedule () at linux/sched.c:41
#3  0x0000601d593c6cb7 in worker_thread (arg=0x601d5a23c080) at linux/workqueue.c:252
#4  0x0000601d593b709b in kthread_start_fn (data=0x601d5a3b7020) at linux/kthread.c:25
#5  0x00007fc31e98755a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007fc31ea04a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 15 (Thread 0x7fc31e7d7680 (LWP 25748) "bcachefs_journa"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000601d593be3fc in futex (uaddr=0x601d5a2428ac, op=128, val=2, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x0000601d593be8a0 in schedule () at linux/sched.c:41
#3  0x0000601d593b2aa6 in __closure_sync (cl=0x7fc31e7d6890) at linux/closure.c:135
#4  0x0000601d59343df1 in closure_sync (cl=0x7fc31e7d6890) at include/linux/closure.h:194
#5  0x0000601d5934e947 in bch2_journal_write (ws=0x601d5a2f2bd0) at libbcachefs/journal_io.c:2033
#6  0x0000601d593c6d3f in worker_thread (arg=0x601d5a242840) at linux/workqueue.c:262
#7  0x0000601d593b709b in kthread_start_fn (data=0x601d5a242890) at linux/kthread.c:25
#8  0x00007fc31e98755a in start_thread (arg=<optimized out>) at pthread_create.c:447
#9  0x00007fc31ea04a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 14 (Thread 0x7fc31e822680 (LWP 25747) "bcachefs_write_"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000601d593be3fc in futex (uaddr=0x601d5a2422ac, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x0000601d593be8a0 in schedule () at linux/sched.c:41
#3  0x0000601d593c6cb7 in worker_thread (arg=0x601d5a242500) at linux/workqueue.c:252
#4  0x0000601d593b709b in kthread_start_fn (data=0x601d5a242290) at linux/kthread.c:25
#5  0x00007fc31e98755a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007fc31ea04a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 13 (Thread 0x7fc31e82b680 (LWP 25746) "bcachefs_io"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000601d593be3fc in futex (uaddr=0x601d5a23ac6c, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x0000601d593be8a0 in schedule () at linux/sched.c:41
#3  0x0000601d593c6cb7 in worker_thread (arg=0x601d5a242480) at linux/workqueue.c:252
#4  0x0000601d593b709b in kthread_start_fn (data=0x601d5a23ac50) at linux/kthread.c:25
#5  0x00007fc31e98755a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007fc31ea04a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 12 (Thread 0x7fc31e834680 (LWP 25745) "bcachefs_copygc"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000601d593be3fc in futex (uaddr=0x601d5a23a9ac, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x0000601d593be8a0 in schedule () at linux/sched.c:41
#3  0x0000601d593c6cb7 in worker_thread (arg=0x601d5a242400) at linux/workqueue.c:252
#4  0x0000601d593b709b in kthread_start_fn (data=0x601d5a23a990) at linux/kthread.c:25
#5  0x00007fc31e98755a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007fc31ea04a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 11 (Thread 0x7fc31e83d680 (LWP 25744) "bcachefs_btree_"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000601d593be3fc in futex (uaddr=0x601d5a23a6ec, op=128, val=2, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x0000601d593be8a0 in schedule () at linux/sched.c:41
#3  0x0000601d593b2aa6 in __closure_sync (cl=0x7fc31e83c3f0) at linux/closure.c:135
#4  0x0000601d5933bf9d in closure_sync (cl=0x7fc31e83c3f0) at include/linux/closure.h:194
#5  0x0000601d5933f90e in bch2_journal_res_get_slowpath (j=0x601d5a2f28c0, res=0x7fc16804b0c8, flags=21) at libbcachefs/journal.c:621
#6  0x0000601d592ab4aa in bch2_journal_res_get (j=0x601d5a2f28c0, res=0x7fc16804b0c8, u64s=29, flags=21) at libbcachefs/journal.h:382
#7  0x0000601d592af41b in bch2_trans_journal_res_get (flags=21, trans=0x7fc16804b000) at libbcachefs/btree_trans_commit.c:327
#8  bch2_trans_commit_error (trans=0x7fc16804b000, flags=309, i=0x0, ret=-2237, trace_ip=105679166349460) at libbcachefs/btree_trans_commit.c:913
#9  0x0000601d592b0025 in __bch2_trans_commit (trans=0x7fc16804b000, flags=309) at libbcachefs/btree_trans_commit.c:1109
#10 0x0000601d592ba094 in bch2_trans_commit (trans=0x7fc16804b000, disk_res=0x0, journal_seq=0x0, flags=309) at libbcachefs/btree_update.h:168
#11 0x0000601d592c43d8 in __bch2_btree_node_update_key (trans=0x7fc16804b000, iter=0x7fc31e83c810, b=0x601d5b91b900, new_hash=0x0, new_key=0x7fc16004a720, commit_flags=309, skip_triggers=true) at libbcachefs/btree_update_interior.c:2213
#12 0x0000601d592c4772 in bch2_btree_node_update_key (trans=0x7fc16804b000, iter=0x7fc31e83c810, b=0x601d5b91b900, new_key=0x7fc16004a720, commit_flags=309, skip_triggers=true) at libbcachefs/btree_update_interior.c:2277
#13 0x0000601d592c4bb3 in bch2_btree_node_update_key_get_iter (trans=0x7fc16804b000, b=0x601d5b91b900, new_key=0x7fc16004a720, commit_flags=309, skip_triggers=true) at libbcachefs/btree_update_interior.c:2321
#14 0x0000601d59285fbd in btree_node_write_work (work=0x7fc16004a700) at libbcachefs/btree_io.c:1852
#15 0x0000601d593c6d3f in worker_thread (arg=0x601d5a2398c0) at linux/workqueue.c:262
#16 0x0000601d593b709b in kthread_start_fn (data=0x601d5a23a6d0) at linux/kthread.c:25
#17 0x00007fc31e98755a in start_thread (arg=<optimized out>) at pthread_create.c:447
#18 0x00007fc31ea04a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 10 (Thread 0x7fc31e846680 (LWP 25743) "bcachefs"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000601d593be3fc in futex (uaddr=0x601d5a23a42c, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x0000601d593be8a0 in schedule () at linux/sched.c:41
#3  0x0000601d593c6cb7 in worker_thread (arg=0x601d5a248040) at linux/workqueue.c:252
#4  0x0000601d593b709b in kthread_start_fn (data=0x601d5a23a410) at linux/kthread.c:25
#5  0x00007fc31e98755a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007fc31ea04a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 9 (Thread 0x7fc31e8a2680 (LWP 25735) "timers"):
#0  0x00007fc31e983ebe in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7fc31e8a1930, op=393, expected=0, futex_word=0x601d598973e8 <timer_cond+40>) at futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x601d598973e8 <timer_cond+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7fc31e8a1930, private=private@entry=0, cancel=cancel@entry=true) at futex-internal.c:87
#2  0x00007fc31e983f3f in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x601d598973e8 <timer_cond+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7fc31e8a1930, private=private@entry=0) at futex-internal.c:139
#3  0x00007fc31e986a65 in __pthread_cond_wait_common (abstime=0x7fc31e8a1930, clockid=0, mutex=0x601d59897380 <timer_lock>, cond=0x601d598973c0 <timer_cond>) at pthread_cond_wait.c:503
#4  ___pthread_cond_timedwait64 (cond=0x601d598973c0 <timer_cond>, mutex=0x601d59897380 <timer_lock>, abstime=0x7fc31e8a1930) at pthread_cond_wait.c:643
#5  0x0000601d593c4b77 in timer_thread (arg=0x0) at linux/timer.c:294
#6  0x0000601d593b709b in kthread_start_fn (data=0x601d5a238a00) at linux/kthread.c:25
#7  0x00007fc31e98755a in start_thread (arg=<optimized out>) at pthread_create.c:447
#8  0x00007fc31ea04a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 8 (Thread 0x7fc31e8ab680 (LWP 25734) "shrinkers"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000601d593bf5f7 in futex (uaddr=0x601d5a2383bc, op=137, val=1, timeout=0x7fc31e8aa940, uaddr2=0x0, val3=-1) at /usr/include/urcu/futex.h:81
#2  0x0000601d593bfa16 in shrinker_thread (arg=0x0) at linux/shrinker.c:106
#3  0x0000601d593b709b in kthread_start_fn (data=0x601d5a2383a0) at linux/kthread.c:25
#4  0x00007fc31e98755a in start_thread (arg=<optimized out>) at pthread_create.c:447
#5  0x00007fc31ea04a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 7 (Thread 0x7fc31e8b4680 (LWP 25733) "events_freezabl"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000601d593be3fc in futex (uaddr=0x601d5a23816c, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x0000601d593be8a0 in schedule () at linux/sched.c:41
#3  0x0000601d593c6cb7 in worker_thread (arg=0x601d5a238100) at linux/workqueue.c:252
#4  0x0000601d593b709b in kthread_start_fn (data=0x601d5a238150) at linux/kthread.c:25
#5  0x00007fc31e98755a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007fc31ea04a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 6 (Thread 0x7fc31e8bd680 (LWP 25732) "events_unbound"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000601d593be3fc in futex (uaddr=0x601d5a237eac, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x0000601d593be8a0 in schedule () at linux/sched.c:41
#3  0x0000601d593c6cb7 in worker_thread (arg=0x601d5a237e40) at linux/workqueue.c:252
#4  0x0000601d593b709b in kthread_start_fn (data=0x601d5a237e90) at linux/kthread.c:25
#5  0x00007fc31e98755a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007fc31ea04a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 5 (Thread 0x7fc31e8c6680 (LWP 25731) "events_long"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000601d593be3fc in futex (uaddr=0x601d5a237bec, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x0000601d593be8a0 in schedule () at linux/sched.c:41
#3  0x0000601d593c6cb7 in worker_thread (arg=0x601d5a237b80) at linux/workqueue.c:252
#4  0x0000601d593b709b in kthread_start_fn (data=0x601d5a237bd0) at linux/kthread.c:25
#5  0x00007fc31e98755a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007fc31ea04a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 4 (Thread 0x7fc31e8cf680 (LWP 25730) "events_highpri"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000601d593be3fc in futex (uaddr=0x601d5a23792c, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x0000601d593be8a0 in schedule () at linux/sched.c:41
#3  0x0000601d593c6cb7 in worker_thread (arg=0x601d5a2378c0) at linux/workqueue.c:252
#4  0x0000601d593b709b in kthread_start_fn (data=0x601d5a237910) at linux/kthread.c:25
#5  0x00007fc31e98755a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007fc31ea04a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 3 (Thread 0x7fc31e8d8680 (LWP 25729) "events"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000601d593be3fc in futex (uaddr=0x601d5a23766c, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x0000601d593be8a0 in schedule () at linux/sched.c:41
#3  0x0000601d593c6cb7 in worker_thread (arg=0x601d5a237600) at linux/workqueue.c:252
#4  0x0000601d593b709b in kthread_start_fn (data=0x601d5a237650) at linux/kthread.c:25
#5  0x00007fc31e98755a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007fc31ea04a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 2 (Thread 0x7fc31e8e1680 (LWP 25728) "aio_completion"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007fc31eb04277 in __io_getevents_0_4 () from /usr/lib/libaio.so.1
#2  0x0000601d593b155d in aio_completion_thread (arg=0x0) at linux/blkdev.c:276
#3  0x0000601d593b709b in kthread_start_fn (data=0x601d5a237380) at linux/kthread.c:25
#4  0x00007fc31e98755a in start_thread (arg=<optimized out>) at pthread_create.c:447
#5  0x00007fc31ea04a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 1 (Thread 0x7fc31e8e2c00 (LWP 25727) "bcachefs"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000601d593be3fc in futex (uaddr=0x601d5a2372bc, op=128, val=2, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x0000601d593be8a0 in schedule () at linux/sched.c:41
#3  0x0000601d593b2aa6 in __closure_sync (cl=0x7ffe7e088880) at linux/closure.c:135
#4  0x0000601d5933bf9d in closure_sync (cl=0x7ffe7e088880) at include/linux/closure.h:194
#5  0x0000601d5933f90e in bch2_journal_res_get_slowpath (j=0x601d5a2f28c0, res=0x601d5a27c0c8, flags=16) at libbcachefs/journal.c:621
#6  0x0000601d592ab4aa in bch2_journal_res_get (j=0x601d5a2f28c0, res=0x601d5a27c0c8, u64s=17, flags=16) at libbcachefs/journal.h:382
#7  0x0000601d592af41b in bch2_trans_journal_res_get (flags=16, trans=0x601d5a27c000) at libbcachefs/btree_trans_commit.c:327
#8  bch2_trans_commit_error (trans=0x601d5a27c000, flags=80, i=0x0, ret=-2237, trace_ip=105679166993048) at libbcachefs/btree_trans_commit.c:913
#9  0x0000601d592b0025 in __bch2_trans_commit (trans=0x601d5a27c000, flags=80) at libbcachefs/btree_trans_commit.c:1109
#10 0x0000601d59357298 in bch2_trans_commit (trans=0x601d5a27c000, disk_res=0x0, journal_seq=0x0, flags=80) at libbcachefs/btree_update.h:168
#11 0x0000601d59357fae in bch2_check_lrus (c=0x601d5a2b3000) at libbcachefs/lru.c:151
#12 0x0000601d59374901 in bch2_run_recovery_pass (c=0x601d5a2b3000, pass=BCH_RECOVERY_PASS_check_lrus) at libbcachefs/recovery.c:685
#13 0x0000601d5937499f in bch2_run_recovery_passes (c=0x601d5a2b3000) at libbcachefs/recovery.c:702
#14 0x0000601d593757d9 in bch2_fs_recovery (c=0x601d5a2b3000) at libbcachefs/recovery.c:978
#15 0x0000601d593a3fd7 in bch2_fs_start (c=0x601d5a2b3000) at libbcachefs/super.c:1024
#16 0x0000601d593a79d5 in bch2_fs_open (devices=0x601d5a239890, nr_devices=1, opts=...) at libbcachefs/super.c:2074
#17 0x0000601d59220e6b in cmd_fsck (argc=1, argv=0x601d5a238c68) at c_src/cmd_fsck.c:256
#18 0x0000601d591e8b86 in bcachefs::handle_c_command (args=..., symlink_cmd=...) at src/bcachefs.rs:57
#19 0x0000601d591e98a4 in bcachefs::main () at src/bcachefs.rs:120
Detaching from program: /mnt/home/architector4/bcachefs-tools/bcachefs-tools/target/debug/bcachefs, process 25727
[Inferior 1 (process 25727) detached]

May be related to #118 ?

edit: it appears reset-counters might have helped, as now it printed 3 more lines of output (below) and is slowly blipping some CPU usage from the bch_reclaim//de and timers threads... I hope I'm getting anywhere with this

check_lrus... done
check_btree_backpointers... done
check_backpointers_to_extents... done

I suppose it was deadlocking while checking lrus and is now very slowly munching through checking extents to backpointers lol

@Architector4
Copy link
Author

Architector4 commented Feb 22, 2024

It's another deadlock. bch_reclaim//de is spinning in a function bch2_journal_reclaim_thread but otherwise everything is locked in futexes and no progress was happening in the last 14 hours, as far as I can tell. It's a 1TB filesystem on an NVMe SSD and with no subvolumes, so it shouldn't be taking that long I assume lol

the gdb:

  Id   Target Id                                           Frame 
* 1    Thread 0x7aa04e58ec00 (LWP 25888) "bcachefs"        syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  2    Thread 0x7aa04e58d680 (LWP 25889) "aio_completion"  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  3    Thread 0x7aa04e584680 (LWP 25890) "events"          syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  4    Thread 0x7aa04e57b680 (LWP 25891) "events_highpri"  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  5    Thread 0x7aa04e572680 (LWP 25892) "events_long"     syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  6    Thread 0x7aa04e569680 (LWP 25893) "events_unbound"  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  7    Thread 0x7aa04e560680 (LWP 25894) "events_freezabl" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  8    Thread 0x7aa04e557680 (LWP 25895) "shrinkers"       syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  9    Thread 0x7aa04e54e680 (LWP 25896) "timers"          0x00007aa04e62febe in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5f9aaeaf23e8 <timer_cond+40>) at futex-internal.c:57
  10   Thread 0x7aa04e4f2680 (LWP 25904) "bcachefs"        syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  11   Thread 0x7aa04e4e9680 (LWP 25905) "bcachefs_btree_" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  12   Thread 0x7aa04e4e0680 (LWP 25906) "bcachefs_copygc" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  13   Thread 0x7aa04e4d7680 (LWP 25907) "bcachefs_io"     syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  14   Thread 0x7aa04e4ce680 (LWP 25908) "bcachefs_write_" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  15   Thread 0x7aa04e483680 (LWP 25909) "bcachefs_journa" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  16   Thread 0x7aa04db59680 (LWP 25910) "btree_update"    syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  17   Thread 0x7aa04e517680 (LWP 25911) "bch-reclaim//de" bch2_journal_reclaim_thread (arg=0x5f9aaf4738c0) at libbcachefs/journal_reclaim.c:719
  18   Thread 0x7aa04d400680 (LWP 25916) "events"          syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  19   Thread 0x7aa04460d680 (LWP 25917) "bch-gc//dev/map" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38

Thread 19 (Thread 0x7aa04460d680 (LWP 25917) "bch-gc//dev/map"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00005f9aae6193fc in futex (uaddr=0x5f9abd561d2c, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x00005f9aae6198a0 in schedule () at linux/sched.c:41
#3  0x00005f9aae4d3f3c in bch2_gc_thread (arg=0x5f9aaf434000) at libbcachefs/btree_gc.c:2018
#4  0x00005f9aae61209b in kthread_start_fn (data=0x5f9abd561d10) at linux/kthread.c:25
#5  0x00007aa04e63355a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007aa04e6b0a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 18 (Thread 0x7aa04d400680 (LWP 25916) "events"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007aa04e9ad596 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=-1, op=0, uaddr=0x7a9f90000bb0) at ../include/urcu/futex.h:85
#2  futex_noasync (timeout=0x0, uaddr2=0x0, val3=0, val=-1, op=0, uaddr=0x7a9f90000bb0) at ../include/urcu/futex.h:90
#3  call_rcu_wait (crdp=0x7a9f90000b70) at /usr/src/debug/liburcu/userspace-rcu-0.14.0/src/urcu-call-rcu-impl.h:248
#4  call_rcu_thread (arg=0x7a9f90000b70) at /usr/src/debug/liburcu/userspace-rcu-0.14.0/src/urcu-call-rcu-impl.h:400
#5  0x00007aa04e63355a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007aa04e6b0a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 17 (Thread 0x7aa04e517680 (LWP 25911) "bch-reclaim//de"):
#0  bch2_journal_reclaim_thread (arg=0x5f9aaf4738c0) at libbcachefs/journal_reclaim.c:719
#1  0x00005f9aae61209b in kthread_start_fn (data=0x5f9aaf3bda10) at linux/kthread.c:25
#2  0x00007aa04e63355a in start_thread (arg=<optimized out>) at pthread_create.c:447
#3  0x00007aa04e6b0a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 16 (Thread 0x7aa04db59680 (LWP 25910) "btree_update"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00005f9aae6193fc in futex (uaddr=0x5f9aaf53803c, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x00005f9aae6198a0 in schedule () at linux/sched.c:41
#3  0x00005f9aae621cb7 in worker_thread (arg=0x5f9aaf3bd080) at linux/workqueue.c:252
#4  0x00005f9aae61209b in kthread_start_fn (data=0x5f9aaf538020) at linux/kthread.c:25
#5  0x00007aa04e63355a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007aa04e6b0a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 15 (Thread 0x7aa04e483680 (LWP 25909) "bcachefs_journa"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00005f9aae6193fc in futex (uaddr=0x5f9aaf3c38ac, op=128, val=2, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x00005f9aae6198a0 in schedule () at linux/sched.c:41
#3  0x00005f9aae60daa6 in __closure_sync (cl=0x7aa04e482890) at linux/closure.c:135
#4  0x00005f9aae59edf1 in closure_sync (cl=0x7aa04e482890) at include/linux/closure.h:194
#5  0x00005f9aae5a9947 in bch2_journal_write (ws=0x5f9aaf473960) at libbcachefs/journal_io.c:2033
#6  0x00005f9aae621d3f in worker_thread (arg=0x5f9aaf3c3840) at linux/workqueue.c:262
#7  0x00005f9aae61209b in kthread_start_fn (data=0x5f9aaf3c3890) at linux/kthread.c:25
#8  0x00007aa04e63355a in start_thread (arg=<optimized out>) at pthread_create.c:447
#9  0x00007aa04e6b0a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 14 (Thread 0x7aa04e4ce680 (LWP 25908) "bcachefs_write_"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00005f9aae6193fc in futex (uaddr=0x5f9aaf3c32ac, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x00005f9aae6198a0 in schedule () at linux/sched.c:41
#3  0x00005f9aae621cb7 in worker_thread (arg=0x5f9aaf3c3500) at linux/workqueue.c:252
#4  0x00005f9aae61209b in kthread_start_fn (data=0x5f9aaf3c3290) at linux/kthread.c:25
#5  0x00007aa04e63355a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007aa04e6b0a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 13 (Thread 0x7aa04e4d7680 (LWP 25907) "bcachefs_io"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00005f9aae6193fc in futex (uaddr=0x5f9aaf3bbc6c, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x00005f9aae6198a0 in schedule () at linux/sched.c:41
#3  0x00005f9aae621cb7 in worker_thread (arg=0x5f9aaf3c3480) at linux/workqueue.c:252
#4  0x00005f9aae61209b in kthread_start_fn (data=0x5f9aaf3bbc50) at linux/kthread.c:25
#5  0x00007aa04e63355a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007aa04e6b0a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 12 (Thread 0x7aa04e4e0680 (LWP 25906) "bcachefs_copygc"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00005f9aae6193fc in futex (uaddr=0x5f9aaf3bb9ac, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x00005f9aae6198a0 in schedule () at linux/sched.c:41
#3  0x00005f9aae621cb7 in worker_thread (arg=0x5f9aaf3c3400) at linux/workqueue.c:252
#4  0x00005f9aae61209b in kthread_start_fn (data=0x5f9aaf3bb990) at linux/kthread.c:25
#5  0x00007aa04e63355a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007aa04e6b0a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 11 (Thread 0x7aa04e4e9680 (LWP 25905) "bcachefs_btree_"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00005f9aae6193fc in futex (uaddr=0x5f9aaf3bb6ec, op=128, val=2, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x00005f9aae6198a0 in schedule () at linux/sched.c:41
#3  0x00005f9aae60daa6 in __closure_sync (cl=0x7aa04e4e83f0) at linux/closure.c:135
#4  0x00005f9aae596f9d in closure_sync (cl=0x7aa04e4e83f0) at include/linux/closure.h:194
#5  0x00005f9aae59a90e in bch2_journal_res_get_slowpath (j=0x5f9aaf4738c0, res=0x7a9e980010c8, flags=21) at libbcachefs/journal.c:621
#6  0x00005f9aae5064aa in bch2_journal_res_get (j=0x5f9aaf4738c0, res=0x7a9e980010c8, u64s=17, flags=21) at libbcachefs/journal.h:382
#7  0x00005f9aae50a41b in bch2_trans_journal_res_get (flags=21, trans=0x7a9e98001000) at libbcachefs/btree_trans_commit.c:327
#8  bch2_trans_commit_error (trans=0x7a9e98001000, flags=309, i=0x0, ret=-2237, trace_ip=105117954166932) at libbcachefs/btree_trans_commit.c:913
#9  0x00005f9aae50b025 in __bch2_trans_commit (trans=0x7a9e98001000, flags=309) at libbcachefs/btree_trans_commit.c:1109
#10 0x00005f9aae515094 in bch2_trans_commit (trans=0x7a9e98001000, disk_res=0x0, journal_seq=0x0, flags=309) at libbcachefs/btree_update.h:168
#11 0x00005f9aae51f3d8 in __bch2_btree_node_update_key (trans=0x7a9e98001000, iter=0x7aa04e4e8810, b=0x5f9ab01a4700, new_hash=0x0, new_key=0x7a9ea20ab120, commit_flags=309, skip_triggers=true) at libbcachefs/btree_update_interior.c:2213
#12 0x00005f9aae51f772 in bch2_btree_node_update_key (trans=0x7a9e98001000, iter=0x7aa04e4e8810, b=0x5f9ab01a4700, new_key=0x7a9ea20ab120, commit_flags=309, skip_triggers=true) at libbcachefs/btree_update_interior.c:2277
#13 0x00005f9aae51fbb3 in bch2_btree_node_update_key_get_iter (trans=0x7a9e98001000, b=0x5f9ab01a4700, new_key=0x7a9ea20ab120, commit_flags=309, skip_triggers=true) at libbcachefs/btree_update_interior.c:2321
#14 0x00005f9aae4e0fbd in btree_node_write_work (work=0x7a9ea20ab100) at libbcachefs/btree_io.c:1852
#15 0x00005f9aae621d3f in worker_thread (arg=0x5f9aaf3ba8c0) at linux/workqueue.c:262
#16 0x00005f9aae61209b in kthread_start_fn (data=0x5f9aaf3bb6d0) at linux/kthread.c:25
#17 0x00007aa04e63355a in start_thread (arg=<optimized out>) at pthread_create.c:447
#18 0x00007aa04e6b0a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 10 (Thread 0x7aa04e4f2680 (LWP 25904) "bcachefs"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00005f9aae6193fc in futex (uaddr=0x5f9aaf3bb42c, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x00005f9aae6198a0 in schedule () at linux/sched.c:41
#3  0x00005f9aae621cb7 in worker_thread (arg=0x5f9aaf3c9040) at linux/workqueue.c:252
#4  0x00005f9aae61209b in kthread_start_fn (data=0x5f9aaf3bb410) at linux/kthread.c:25
#5  0x00007aa04e63355a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007aa04e6b0a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 9 (Thread 0x7aa04e54e680 (LWP 25896) "timers"):
#0  0x00007aa04e62febe in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5f9aaeaf23e8 <timer_cond+40>) at futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x5f9aaeaf23e8 <timer_cond+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at futex-internal.c:87
#2  0x00007aa04e62ff3f in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5f9aaeaf23e8 <timer_cond+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at futex-internal.c:139
#3  0x00007aa04e632750 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5f9aaeaf2380 <timer_lock>, cond=0x5f9aaeaf23c0 <timer_cond>) at pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x5f9aaeaf23c0 <timer_cond>, mutex=0x5f9aaeaf2380 <timer_lock>) at pthread_cond_wait.c:618
#5  0x00005f9aae61f6b3 in timer_thread (arg=0x0) at linux/timer.c:265
#6  0x00005f9aae61209b in kthread_start_fn (data=0x5f9aaf3b9a00) at linux/kthread.c:25
#7  0x00007aa04e63355a in start_thread (arg=<optimized out>) at pthread_create.c:447
#8  0x00007aa04e6b0a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 8 (Thread 0x7aa04e557680 (LWP 25895) "shrinkers"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00005f9aae61a5f7 in futex (uaddr=0x5f9aaf3b93bc, op=137, val=1, timeout=0x7aa04e556940, uaddr2=0x0, val3=-1) at /usr/include/urcu/futex.h:81
#2  0x00005f9aae61aa16 in shrinker_thread (arg=0x0) at linux/shrinker.c:106
#3  0x00005f9aae61209b in kthread_start_fn (data=0x5f9aaf3b93a0) at linux/kthread.c:25
#4  0x00007aa04e63355a in start_thread (arg=<optimized out>) at pthread_create.c:447
#5  0x00007aa04e6b0a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 7 (Thread 0x7aa04e560680 (LWP 25894) "events_freezabl"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00005f9aae6193fc in futex (uaddr=0x5f9aaf3b916c, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x00005f9aae6198a0 in schedule () at linux/sched.c:41
#3  0x00005f9aae621cb7 in worker_thread (arg=0x5f9aaf3b9100) at linux/workqueue.c:252
#4  0x00005f9aae61209b in kthread_start_fn (data=0x5f9aaf3b9150) at linux/kthread.c:25
#5  0x00007aa04e63355a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007aa04e6b0a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 6 (Thread 0x7aa04e569680 (LWP 25893) "events_unbound"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00005f9aae6193fc in futex (uaddr=0x5f9aaf3b8eac, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x00005f9aae6198a0 in schedule () at linux/sched.c:41
#3  0x00005f9aae621cb7 in worker_thread (arg=0x5f9aaf3b8e40) at linux/workqueue.c:252
#4  0x00005f9aae61209b in kthread_start_fn (data=0x5f9aaf3b8e90) at linux/kthread.c:25
#5  0x00007aa04e63355a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007aa04e6b0a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 5 (Thread 0x7aa04e572680 (LWP 25892) "events_long"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00005f9aae6193fc in futex (uaddr=0x5f9aaf3b8bec, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x00005f9aae6198a0 in schedule () at linux/sched.c:41
#3  0x00005f9aae621cb7 in worker_thread (arg=0x5f9aaf3b8b80) at linux/workqueue.c:252
#4  0x00005f9aae61209b in kthread_start_fn (data=0x5f9aaf3b8bd0) at linux/kthread.c:25
#5  0x00007aa04e63355a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007aa04e6b0a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 4 (Thread 0x7aa04e57b680 (LWP 25891) "events_highpri"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00005f9aae6193fc in futex (uaddr=0x5f9aaf3b892c, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x00005f9aae6198a0 in schedule () at linux/sched.c:41
#3  0x00005f9aae621cb7 in worker_thread (arg=0x5f9aaf3b88c0) at linux/workqueue.c:252
#4  0x00005f9aae61209b in kthread_start_fn (data=0x5f9aaf3b8910) at linux/kthread.c:25
#5  0x00007aa04e63355a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007aa04e6b0a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 3 (Thread 0x7aa04e584680 (LWP 25890) "events"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00005f9aae6193fc in futex (uaddr=0x5f9aaf3b866c, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x00005f9aae6198a0 in schedule () at linux/sched.c:41
#3  0x00005f9aae621cb7 in worker_thread (arg=0x5f9aaf3b8600) at linux/workqueue.c:252
#4  0x00005f9aae61209b in kthread_start_fn (data=0x5f9aaf3b8650) at linux/kthread.c:25
#5  0x00007aa04e63355a in start_thread (arg=<optimized out>) at pthread_create.c:447
#6  0x00007aa04e6b0a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 2 (Thread 0x7aa04e58d680 (LWP 25889) "aio_completion"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007aa04e7b0277 in __io_getevents_0_4 () from /usr/lib/libaio.so.1
#2  0x00005f9aae60c55d in aio_completion_thread (arg=0x0) at linux/blkdev.c:276
#3  0x00005f9aae61209b in kthread_start_fn (data=0x5f9aaf3b8380) at linux/kthread.c:25
#4  0x00007aa04e63355a in start_thread (arg=<optimized out>) at pthread_create.c:447
#5  0x00007aa04e6b0a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 1 (Thread 0x7aa04e58ec00 (LWP 25888) "bcachefs"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00005f9aae6193fc in futex (uaddr=0x5f9aaf3b82bc, op=128, val=2, timeout=0x0, uaddr2=0x0, val3=0) at /usr/include/urcu/futex.h:81
#2  0x00005f9aae6198a0 in schedule () at linux/sched.c:41
#3  0x00005f9aae60daa6 in __closure_sync (cl=0x7fffdec88110) at linux/closure.c:135
#4  0x00005f9aae596f9d in closure_sync (cl=0x7fffdec88110) at include/linux/closure.h:194
#5  0x00005f9aae59a90e in bch2_journal_res_get_slowpath (j=0x5f9aaf4738c0, res=0x5f9aaf3fd0c8, flags=16) at libbcachefs/journal.c:621
#6  0x00005f9aae5064aa in bch2_journal_res_get (j=0x5f9aaf4738c0, res=0x5f9aaf3fd0c8, u64s=15, flags=16) at libbcachefs/journal.h:382
#7  0x00005f9aae50a41b in bch2_trans_journal_res_get (flags=16, trans=0x5f9aaf3fd000) at libbcachefs/btree_trans_commit.c:327
#8  bch2_trans_commit_error (trans=0x5f9aaf3fd000, flags=16, i=0x0, ret=-2237, trace_ip=105117953712868) at libbcachefs/btree_trans_commit.c:913
#9  0x00005f9aae50b025 in __bch2_trans_commit (trans=0x5f9aaf3fd000, flags=16) at libbcachefs/btree_trans_commit.c:1109
#10 0x00005f9aae4a62e4 in bch2_trans_commit (trans=0x5f9aaf3fd000, disk_res=0x0, journal_seq=0x0, flags=16) at libbcachefs/btree_update.h:168
#11 0x00005f9aae4aa3c4 in bch2_check_extents_to_backpointers_pass (trans=0x5f9aaf3fd000, s=0x7fffdec885e0) at libbcachefs/backpointers.c:643
#12 0x00005f9aae4aab30 in bch2_check_extents_to_backpointers (c=0x5f9aaf434000) at libbcachefs/backpointers.c:705
#13 0x00005f9aae5cf901 in bch2_run_recovery_pass (c=0x5f9aaf434000, pass=BCH_RECOVERY_PASS_check_extents_to_backpointers) at libbcachefs/recovery.c:685
#14 0x00005f9aae5cf99f in bch2_run_recovery_passes (c=0x5f9aaf434000) at libbcachefs/recovery.c:702
#15 0x00005f9aae5d07d9 in bch2_fs_recovery (c=0x5f9aaf434000) at libbcachefs/recovery.c:978
#16 0x00005f9aae5fefd7 in bch2_fs_start (c=0x5f9aaf434000) at libbcachefs/super.c:1024
#17 0x00005f9aae6029d5 in bch2_fs_open (devices=0x5f9aaf3ba890, nr_devices=1, opts=...) at libbcachefs/super.c:2074
#18 0x00005f9aae47be6b in cmd_fsck (argc=1, argv=0x5f9aaf3b9c68) at c_src/cmd_fsck.c:256
#19 0x00005f9aae443b86 in bcachefs::handle_c_command (args=..., symlink_cmd=...) at src/bcachefs.rs:57
#20 0x00005f9aae4448a4 in bcachefs::main () at src/bcachefs.rs:120
Detaching from program: /mnt/home/architector4/bcachefs-tools/bcachefs-tools/target/debug/bcachefs, process 25888
[Inferior 1 (process 25888) detached]

edit: i wrote the backup (made before reset-counters) onto the raw nvme partition (removing the LUKS layer), and it now prints the next 3 lines (as in the comment above) too, but still deadlocks.

edit: i copied the recompiled tool to a fat32 partition, did a reboot, and without mounting any bcachefs, got the tool from that partition and tries fsck --reconstruct_alloc -fy again, and this time i didn't get the next 3 lines. Those lines may just be random luck i guess lol

@Architector4
Copy link
Author

IT MOUNTED! after trying fsck with and without reconstruct_alloc a few times, and running into the deadlock every single time, i decided to try and just mount it, and it worked.

...god, now to clean up all the mess i've done lol

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant