ZFS send hangs sometimes #16731

rkojedzinszky · 2024-11-07T07:15:21Z

System information

Type	Version/Name
Distribution Name	TrueNAS-CORE/FreeBSD
Distribution Version	13.3
Kernel Version	13.3-RELEASE-p7
Architecture	amd64
OpenZFS Version	2.2.6 + truenas patches

Describe the problem you're observing

TrueNAS is using zettarepl to replicate zfs datasets to remote sites. During a cycle, sometimes, rarely, zfs send hangs. The symptom is that zfs send hangs, not sending anything to its output, is in idle state. I've applied a workaround, a simpe pipe command which reads output from zfs send and passes data through, and this command is reporting that no output is received from zfs send for minutes. Then it kills zfs send. Also, it is reporting that usually only a few thousand bytes are sent by zfs send, not more. Then, simply killing zfs send solves the problem, upon next cycle it will usually send the snapshots completely, without errors.

~~Must note here that zfs used by TrueNAS contains this PR. I suspect this may be the source of my issue.~~

I suspect that 6bdc725 may be the source of my issue.

Usually, I receive send errors once in a week or two, cannot reproduce, but I will now give a try without this patch, and see the difference.

Describe how to reproduce the problem

Unfortunately, cannot reproduce.

Include any warning/errors/backtraces from the system logs

The text was updated successfully, but these errors were encountered:

openzfs/zfs#16731

vedadkajtaz · 2024-11-12T20:02:04Z

I'm facing a somehow similar issue.

It occurs only on one out of 7 servers that all run the same OS (FreeBSD 14.1, OpenZFS 2.2.4), and use the same (home-made) zfs replication software.

It is basically a series of zfs send | ssh 'zfs receive'.

The zfs send seldom hangs (roughly once a week), on different datasets being sent.

Killing the piped ssh process does nothing either (which is expected, zfs send doesn't get an EPIPE since it doesn't send anything). Killing the zfs send works, and further iterations are okay, usually for a few days only.

There are no related dmesg messages, the pool is healthy (frequently scrub'd).

I'd be glad to help investigating the issue, but don't know where to look.

rkojedzinszky · 2024-11-12T21:01:18Z

@vedadkajtaz thanks!

I have to correct myself, zfs-2.2.4 also contains the suspected patch. If you would be able to test openzfs without commit 6bdc725, that would help. I am running 3 boxes now with that reverted, only for a few days, without hanging zfs send. But, I definitely would need more time to declare this as a possible cause.

rkojedzinszky · 2024-11-14T11:19:39Z

@vedadkajtaz did you have a chance to build zfs userspace with 6bdc725 reverted? Then, that will need some time, but according to your experiments, in a week you'll be able to report some results.

vedadkajtaz · 2024-11-15T11:03:42Z

@vedadkajtaz did you have a chance to build zfs userspace with 6bdc725 reverted? Then, that will need some time, but according to your experiments, in a week you'll be able to report some results.

Hi, I haven't done anything yet regarding this, possibly/likely next week, sorry.

rkojedzinszky · 2024-11-17T12:25:38Z

@nabijaczleweli I can report that reverting the mentioned commit caused no zfs send issues on 3 FreeBSD based NAS servers for more than a week now. Can you have a look at the commit?

nabijaczleweli · 2024-11-17T14:05:43Z

It looked sound back then so it looks sound now. No-one seems to have posted a strace (or backtrace) that would indicate where these hang, that commit basically doesn't touch the actually-sending-stuff thread at all, and all the setup is deterministic AFAICT. This bug hasn't left "oh i see this sometimes". I can't evaluate data you're withholding.

vedadkajtaz · 2024-11-17T15:15:35Z

I have a hung process (with stock binary, FreeBSD 14.1, OpenZFS 2.2.4) right now.

There is no strace on FreeBSD. truss was not helpful, no system calls whatsoever.
Here's the backtrace from gdb:

(gdb) bt
#0  0x000009ad512dae2c in ?? () from /lib/libthr.so.3
#1  0x000009ad512dfa2e in ?? () from /lib/libthr.so.3
#2  0x000009ad4dbcec83 in ?? () from /lib/libzfs.so.4
#3  0x000009ad4dbd0fbe in ?? () from /lib/libzfs.so.4
#4  0x000009ad4dbbf22a in zfs_iter_snapshots_sorted_v2 () from /lib/libzfs.so.4
#5  0x000009ad4dbd0876 in ?? () from /lib/libzfs.so.4
#6  0x000009ad4dbcc684 in ?? () from /lib/libzfs.so.4
#7  0x000009ad4dbcc1b6 in zfs_send () from /lib/libzfs.so.4
#8  0x000009a527a181f0 in ?? ()
#9  0x000009a527a12010 in ?? ()
#10 0x000009ad4f7b0a6a in __libc_start1 () from /lib/libc.so.7
#11 0x000009a527a1108d in ?? ()

Not super helpful without debugging symbols, but it's obviously stuck in libthr, which seems to indicate @rkojedzinszky is likely right.

nabijaczleweli · 2024-11-17T15:28:44Z

Would it be a terrible bother to take a backtrace, with symbols, of all the threads, so we don't have to guess what's happening? Attaching the strace-equivalent should in general be easier and tell you in which syscall each thread is stuck, but I don't really know if FreeBSD possesses this ability.

vedadkajtaz · 2024-11-17T15:36:18Z

I'll rebuild (stock, ie. releng/14.1) binaries with debugging symbols over the next few days, and wait for the next hung.

rkojedzinszky · 2024-11-17T19:00:46Z

@nabijaczleweli unfortunately, I can only add that since I am running my servers with the mentioned patch reverted, I am not facing with hung zfs processes.

rkojedzinszky · 2024-11-29T17:59:50Z

@nabijaczleweli I can report now that since I've reverted the mentioned patch, I am not experiencing the hanging issue. @vedadkajtaz what about your setup?

openzfs/zfs#16731

vedadkajtaz · 2024-12-02T09:40:30Z

It hung this morning, for the first time in a week, this is commit ff3df1211c4a08c36c6ced95eb141a9bb8151e12.

Thread 2 (LWP 252784 of process 16061):
#0  0x000000082a8e26aa in _sigsuspend () from /lib/libc.so.7
#1  0x00000008289a1fb6 in ?? () from /lib/libthr.so.3
#2  0x000000082a858a65 in pause () from /lib/libc.so.7
#3  0x0000000822ead7c9 in send_progress_thread (arg=0x820d26f20) at lib/libzfs/libzfs_sendrecv.c:1010
#4  0x0000000828998b05 in ?? () from /lib/libthr.so.3
#5  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x82ec22000

Thread 1 (LWP 100935 of process 16061):
#0  0x0000000828995e2c in ?? () from /lib/libthr.so.3
#1  0x000000082899aa2e in ?? () from /lib/libthr.so.3
#2  0x0000000822eada3f in send_progress_thread_exit (hdl=0xb6dbd03c000, ptid=0xb6dbd012700, oldmask=oldmask@entry=0x820d26e10) at lib/libzfs/libzfs_sendrecv.c:1066
#3  0x0000000822eb0357 in dump_snapshot (zhp=0xb6dbd05a500, arg=arg@entry=0x820d28b28) at lib/libzfs/libzfs_sendrecv.c:1283
#4  0x0000000822ea014a in zfs_iter_snapshots_sorted_v2 (zhp=zhp@entry=0xb6dbd05a000, flags=flags@entry=0, callback=0x822eaff20 <dump_snapshot>, data=data@entry=0x820d28b28, min_txg=8867784, max_txg=8869382) at lib/libzfs/libzfs_iter.c:352
#5  0x0000000822eafd24 in dump_filesystem (zhp=0xb6dbd05a000, sdd=0x820d28b28) at lib/libzfs/libzfs_sendrecv.c:1356
#6  0x0000000822eaec89 in dump_filesystems (rzhp=0xb6dbd012700, rzhp@entry=0xb6dbd05a000, sdd=0x2, sdd@entry=0x820d28b28) at lib/libzfs/libzfs_sendrecv.c:1424
#7  0x0000000822eae403 in zfs_send_cb_impl (zhp=0xb6dbd05a000, fromsnap=<optimized out>, tosnap=0xb6dbd01040b "zpair-02749186397145403432-kenny-674d4ac4-18805166", flags=0x820d29350, outfd=1, filter_func=0x0, cb_arg=0x820d29330, debugnvp=0x0) at lib/libzfs/libzfs_sendrecv.c:2498
#8  0x0000000822eac26e in zfs_send_cb (outfd=0, arg=<optimized out>) at lib/libzfs/libzfs_sendrecv.c:2551
#9  0x0000000822eac237 in zfs_send (zhp=zhp@entry=0xb6dbd05a000, fromsnap=fromsnap@entry=0xb6dbd0190c0 "zpair-02749186397145403432-kenny-674d2ea4-74205ab9", tosnap=tosnap@entry=0xb6dbd01040b "zpair-02749186397145403432-kenny-674d4ac4-18805166", flags=0x0, flags@entry=0x820d29350, outfd=0, outfd@entry=1, filter_func=0x0, cb_arg=0x820d29330, debugnvp=0x0) at lib/libzfs/libzfs_sendrecv.c:2569
#10 0x0000000000215754 in zfs_do_send (argc=<optimized out>, argv=<optimized out>) at cmd/zfs/zfs_main.c:5099
#11 0x000000000020fd18 in main (argc=6, argv=0x820d294b0) at cmd/zfs/zfs_main.c:9249

rkojedzinszky · 2024-12-02T12:12:47Z

Seems like pause() is not catching pthread cancel request. I am not really familiar with pthread internals. Howewer, even if it may be a bug in freebsd, it is now a regression in ZFS. Can we switch to pipes instead of using signals? A signal handler could send a byte to a pipe, and thread loop just receives from that pipe. Cancelling the thread could be indicated by closing the pipe's sender side.

nabijaczleweli · 2024-12-02T15:01:38Z

Hm, I don't think this is a valid race?

The only way I could see this happening is:

send_progress_thread() is entering pause() (this is a cancellation point), but isn't registered yet as sleeping in pause()
pthread_cancel() runs, and queues a cancellation request for the next cancellation point
in the mean-time send_progress_thread() has entered pause()
so when pthread_cancel() returns, there's a cancellation request queued and send_progress_thread() is already sleeping in pause()

...but then the timer should fire and the next syscall should pop the queue. So maybe step 3 also executes the thread cleanups but still fails to actually kill the thread because of the specific race state it's in?

Either way, I can't construe a valid way through our code that can lead to this if the thread implementation is also valid.

This can probably be trivially fixed with pthread_setcanceltype(PTHREAD_CANCEL_ASYNCHRONOUS, NULL);, since we have no critical sections (or any userland code not in service of a write(2)) in the thread anyway.

Can you try this diff?

From 6c6faeaa81acfc5038ed440a1f89fc42f818c097 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=D0=BD=D0=B0=D0=B1?= <[email protected]>
Date: Mon, 2 Dec 2024 15:58:53 +0100
Subject: [PATCH] FreeBSD: libzfs: send_progress_thread: use asynchronous
 cancellation type asynchronous to work around libthr bug
X-Mutt-PGP: OS

See https://github.com/openzfs/zfs/issues/16731
---
 lib/libzfs/libzfs_sendrecv.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/lib/libzfs/libzfs_sendrecv.c b/lib/libzfs/libzfs_sendrecv.c
index b9780720e..60e5047e9 100644
--- a/lib/libzfs/libzfs_sendrecv.c
+++ b/lib/libzfs/libzfs_sendrecv.c
@@ -975,6 +975,14 @@ send_progress_thread(void *arg)
 	struct tm tm;
 	int err;
 
+	/*
+	 * XXX: the FreeBSD pthread implementation can get stuck
+	 *    in send_progress_thread_exit()'s pthread_join()
+	 *   and send_progress_thread()'s' pause()
+	 * in the default deferred cancel mode.
+	 * See https://github.com/openzfs/zfs/issues/16731. */
+	pthread_setcanceltype(PTHREAD_CANCEL_ASYNCHRONOUS, NULL);
+
 	const struct sigaction signal_action =
 	    {.sa_sigaction = send_progress_thread_act, .sa_flags = SA_SIGINFO};
 	struct sigevent timer_cfg =
-- 
2.39.5

nabijaczleweli · 2024-12-02T15:05:09Z

Oh, you're probably running without a timer (zfs send without -v). In that case there's no timer to bail you out, so if you've entered the deadlock the best you can do is SIGUSR1/SIGINFO.

vedadkajtaz · 2024-12-02T15:19:29Z

It's running without the -v switch indeed. I'll rebuild with the patch, but I'll need a couple of weeks to report that it's no longer deadlocking (as stated previously, it hangs roughly once per week).

rkojedzinszky · 2024-12-02T15:52:04Z

Hm, I don't think this is a valid race?

The only way I could see this happening is:

send_progress_thread() is entering pause() (this is a cancellation point), but isn't registered yet as sleeping in pause()

pthread_cancel() runs, and queues a cancellation request for the next cancellation point

in the mean-time send_progress_thread() has entered pause()

so when pthread_cancel() returns, there's a cancellation request queued and send_progress_thread() is already sleeping in pause()

...but then the timer should fire and the next syscall should pop the queue. So maybe step 3 also executes the thread cleanups but still fails to actually kill the thread because of the specific race state it's in?

Either way, I can't construe a valid way through our code that can lead to this if the thread implementation is also valid.

This can probably be trivially fixed with pthread_setcanceltype(PTHREAD_CANCEL_ASYNCHRONOUS, NULL);, since we have no critical sections (or any userland code not in service of a write(2)) in the thread anyway.

Can you try this diff?
From 6c6faeaa81acfc5038ed440a1f89fc42f818c097 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=D0=BD=D0=B0=D0=B1?= <[email protected]>
Date: Mon, 2 Dec 2024 15:58:53 +0100
Subject: [PATCH] FreeBSD: libzfs: send_progress_thread: use asynchronous
 cancellation type asynchronous to work around libthr bug
X-Mutt-PGP: OS

See https://github.com/openzfs/zfs/issues/16731
---
 lib/libzfs/libzfs_sendrecv.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/lib/libzfs/libzfs_sendrecv.c b/lib/libzfs/libzfs_sendrecv.c
index b9780720e..60e5047e9 100644
--- a/lib/libzfs/libzfs_sendrecv.c
+++ b/lib/libzfs/libzfs_sendrecv.c
@@ -975,6 +975,14 @@ send_progress_thread(void *arg)
 	struct tm tm;
 	int err;
 
+	/*
+	 * XXX: the FreeBSD pthread implementation can get stuck
+	 *    in send_progress_thread_exit()'s pthread_join()
+	 *   and send_progress_thread()'s' pause()
+	 * in the default deferred cancel mode.
+	 * See https://github.com/openzfs/zfs/issues/16731. */
+	pthread_setcanceltype(PTHREAD_CANCEL_ASYNCHRONOUS, NULL);
+
 	const struct sigaction signal_action =
 	    {.sa_sigaction = send_progress_thread_act, .sa_flags = SA_SIGINFO};
 	struct sigevent timer_cfg =
-- 
2.39.5

So there is a case when order of calls are:

progress thread	parent thread
pause()
	pthread_cancel()

In this case (which is the most frequent, I assume), pause() is woken up by a cancel request, and that being handled.

The second case:

progress thread	parent thread
	pthread_cancel()
pause()

Here, a cancellation request is queued and pending, and pause() should handle that.

Am I right with this?

As the symptom is very rare, it would be nice to trigger this somehow, focusing on threads/cancellation. Do you have an idea with a small code to try this?

rkojedzinszky · 2024-12-02T15:52:55Z

Howewer, I still can add that without this change, I have no hanging zfs send processes.

nabijaczleweli · 2024-12-02T16:39:57Z

AFAICT, pthread_cancel() doesn't wake the thread up from pause() at all; it queues an event that, when the thread enters a cancellation point (like pause() or write()), runs destructors and destroys it. The thread should never return from pause().
(In asynchronous mode it should Just run destructors and destroy it.)
The rarity of this happening indicates to me there's Some race that makes it so when the thread starts entering pause and pthread_cancel() is called, then the event is queued but by that time the thread has already entered pause(2) and will never wake up.

(idk how pthreads on FreeBSD are implemented, maybe the thread is killed with a SIGKILL or whatever, but this should be invisible to the user-code pause())

A reduced version of this is

sigaction(SIGUSR1, &signal_action, NULL);
for(;;) {
  pause();
  // time(2) + write(2), whatever
}

and

pthread_create(&S, NULL, 		    send_progress_thread, NULL);

sigset_t new;
sigemptyset(&new);
sigaddset(&new, SIGUSR1);
pthread_sigmask(SIG_BLOCK, &new, old);

pthread_cancel(S);
pthread_join(S, &...);

Maybe add a pthread_cleanup_push destructor.

rkojedzinszky · 2024-12-02T21:13:28Z

AFAICT, pthread_cancel() doesn't wake the thread up from pause() at all; it queues an event that, when the thread enters a cancellation point (like pause() or write())

I dont really understand this. According to your assumption, pause() is not woken up by a pthread_cancel. Then, almost all the time we should encounter this hanging issue, as the progress thread usually gets to pause() earlier than the stream dump finishes. And then the design of the current implementation is wrong. Howewer, I think that pause() is indeed woken up, this small code works:

#include <pthread.h>
#include <stdio.h>
#include <unistd.h>

void *thread1(void *arg1) {
  fprintf(stderr, "in thread, pause\n");
  pause();
  fprintf(stderr, "thread: after pause\n");

  return NULL;
}

int main() {
  pthread_t tid;

  pthread_create(&tid, NULL, thread1, NULL);
  sleep(1);
  pthread_cancel(tid);
  pthread_join(tid, NULL);
}

This works the same way on Linux and FreeBSD, the thread gets cancelled after 1 second.

nabijaczleweli · 2024-12-02T22:16:47Z

There's a distinct difference between pause() waking up and the thread being destroyed while sleeping in pause(). Your demo clearly shows that the thread does not wake up from pause, it dies while in pause():

pthread_create(140539834910400, nil, 0x563f8e588190, nil)                                    = 0
[pid 655200] sleep(1 <unfinished ...>
[pid 655208] fprintf(0x7fd1fb0d3680, "in thread, pause\n")                                                = 17
[pid 655208] pause(0x7fd1faef9c80, 0x7fd1faef9da0, 0, 0 <unfinished ...>
[pid 655200] <... sleep resumed> )                                                                        = 0
[pid 655200] pthread_cancel(140539834910400 <no return ...>
[pid 655208] --- UNKNOWN_SIGNAL (Unknown signal 32) ---
[pid 655200] <... pthread_cancel resumed> )                                                               = 0
[pid 655200] pthread_join(140539834910400 <no return ...>
[pid 655208] +++ exited (status 0) +++
[pid 655200] <... pthread_join resumed> , nil)                                                            = 0
[pid 655200] +++ exited (status 0) +++

rkojedzinszky · 2024-12-03T05:54:44Z

@nabijaczleweli sorry, I think we simply misunderstood each other. With pause() waking up a thread I meant to handle the cancellation request, and not continuing the thread. Sorry for the confusion. So, your and my example code works as expected, I am happy with this. But then, why does the same stuck in zfs userspace? Are you sure that the async handling of the cancellation will solve this? According to documentations, the code should work as expected. May this be a FreeBSD pthread implementation bug?

vedadkajtaz · 2024-12-03T11:02:08Z

This was quick, new deadlock with the above patch, basically the same backtrace:

Thread 2 (LWP 259843 of process 95307):
#0  0x000000082872c6aa in _sigsuspend () from /lib/libc.so.7
#1  0x0000000825f47fb6 in ?? () from /lib/libthr.so.3
#2  0x00000008286a2a65 in pause () from /lib/libc.so.7
#3  0x0000000820e3d7d5 in send_progress_thread (arg=0x8208cf2c0) at lib/libzfs/libzfs_sendrecv.c:1018
#4  0x0000000825f3eb05 in ?? () from /lib/libthr.so.3
#5  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x82ce7a000

Thread 1 (LWP 100403 of process 95307):
#0  0x0000000825f3be2c in ?? () from /lib/libthr.so.3
#1  0x0000000825f40a2e in ?? () from /lib/libthr.so.3
#2  0x0000000820e3da4f in send_progress_thread_exit (hdl=0x349d6143c000, ptid=0x349d61412700, oldmask=oldmask@entry=0x8208cf1b0) at lib/libzfs/libzfs_sendrecv.c:1074
#3  0x0000000820e40367 in dump_snapshot (zhp=0x349d6145a500, arg=arg@entry=0x8208d0ec8) at lib/libzfs/libzfs_sendrecv.c:1291
#4  0x0000000820e3014a in zfs_iter_snapshots_sorted_v2 (zhp=zhp@entry=0x349d6145a000, flags=flags@entry=0, callback=0x820e3ff30 <dump_snapshot>, data=data@entry=0x8208d0ec8, min_txg=8889024, max_txg=8890857) at lib/libzfs/libzfs_iter.c:352
#5  0x0000000820e3fd34 in dump_filesystem (zhp=0x349d6145a000, sdd=0x8208d0ec8) at lib/libzfs/libzfs_sendrecv.c:1364
#6  0x0000000820e3ec99 in dump_filesystems (rzhp=0x349d61412700, rzhp@entry=0x349d6145a000, sdd=0x2, sdd@entry=0x8208d0ec8) at lib/libzfs/libzfs_sendrecv.c:1432
#7  0x0000000820e3e413 in zfs_send_cb_impl (zhp=0x349d6145a000, fromsnap=<optimized out>, tosnap=0x349d61419106 "zpair-02749186397145403432-cartman-674eaa54-537c2abe", flags=0x8208d16f0, outfd=1, filter_func=0x0, cb_arg=0x8208d16d0, debugnvp=0x0) at lib/libzfs/libzfs_sendrecv.c:2506
#8  0x0000000820e3c26e in zfs_send_cb (outfd=0, arg=<optimized out>) at lib/libzfs/libzfs_sendrecv.c:2559
#9  0x0000000820e3c237 in zfs_send (zhp=zhp@entry=0x349d6145a000, fromsnap=fromsnap@entry=0x349d614190c0 "zpair-02749186397145403432-cartman-674e8e34-16208f9f", tosnap=tosnap@entry=0x349d61419106 "zpair-02749186397145403432-cartman-674eaa54-537c2abe", flags=0x0, flags@entry=0x8208d16f0, outfd=0, outfd@entry=1, filter_func=0x0, cb_arg=0x8208d16d0, debugnvp=0x0) at lib/libzfs/libzfs_sendrecv.c:2577
#10 0x0000000000215754 in zfs_do_send (argc=<optimized out>, argv=<optimized out>) at cmd/zfs/zfs_main.c:5099
#11 0x000000000020fd18 in main (argc=6, argv=0x8208d1850) at cmd/zfs/zfs_main.c:9249

kill -s USR1 95307 unblocked it.

nabijaczleweli · 2024-12-03T16:36:21Z

This can't not be a FreeBSD pthread bug, if only by the virtue of pthread_cancel()... not cancelling the thread.
I've forwarded it here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=283101

@vedadkajtaz thanks for noting that sending USR1 unsticks the process. Here's a new patch that, on FreeBSD, unconditionally starts the timer to do the unsticking:

From 6e9a44b27c82318501556c1febc130e5a7437a85 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=D0=BD=D0=B0=D0=B1?= <[email protected]>
Date: Mon, 2 Dec 2024 15:58:53 +0100
Subject: [PATCH] FreeBSD: libzfs: send_progress_thread: always start timer to
 work around libthr bug
X-Mutt-PGP: OS

See https://github.com/openzfs/zfs/issues/16731#issuecomment-2511777775
---
 lib/libzfs/libzfs_sendrecv.c | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/lib/libzfs/libzfs_sendrecv.c b/lib/libzfs/libzfs_sendrecv.c
index b9780720e..71cd1b6cd 100644
--- a/lib/libzfs/libzfs_sendrecv.c
+++ b/lib/libzfs/libzfs_sendrecv.c
@@ -988,7 +988,23 @@ send_progress_thread(void *arg)
 	sigaction(SIGINFO, &signal_action, NULL);
 #endif
 
-	if ((timer.desired = pa->pa_progress || pa->pa_astitle)) {
+	/*
+	 * XXX: the FreeBSD pthread implementation can get stuck
+	 *    in send_progress_thread_exit()'s pthread_join()
+	 *   and send_progress_thread()'s' pause()
+	 * See https://github.com/openzfs/zfs/issues/16731#issuecomment-2511777775
+	 * and https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=283101.
+	 *
+	 * We work around this by forcibly starting a timer
+	 * to occasionally wake the thread up,
+	 * so it services the unmaskable cancellation request 🙄
+	 */
+#ifdef __FreeBSD__
+	timer.desired = B_TRUE;
+#else
+	if ((timer.desired = pa->pa_progress || pa->pa_astitle))
+#endif
+	{
 		if (timer_create(CLOCK_MONOTONIC, &timer_cfg, &timer.timer))
 			return ((void *)(uintptr_t)errno);
 		(void) timer_settime(timer.timer, 0, &timer_time, NULL);
-- 
2.39.5

There should be no functional change. (Well, if it deadlocks then it maybe sleeps for up to a second instead of exiting instantly, but that beats sleeping forever.)

Suspected patch reverted. See openzfs/zfs#16731

vedadkajtaz · 2024-12-16T21:19:09Z

I've been able to reproduce the pthread_cancel() issue on bare metal as well, it does seem to be a FreeBSD pthread implementation bug. I've followed up on your bug report. Thanks!

nabijaczleweli · 2024-12-25T19:56:28Z

looks like fixed in https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=283101

Remove workaround applied in openzfs/zfs#16731

rkojedzinszky · 2024-12-30T08:44:48Z

looks like fixed in https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=283101

I am going to give it a try, will report back in a few days.

…nightly tests on FreeBSD-14.2 and FreeBSD-13.4) This is a workaround for spurious hangs during zfs send/receive in ~30% of Github Action jobs on QEMU, probably caused by https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=283101 via openzfs/zfs#16731 (comment) Signed-off-by: Wolfgang Hoschek <[email protected]>

rkojedzinszky · 2025-01-21T08:35:31Z

@nabijaczleweli it seems that FreeBSD bugfix did help, I've not encountered a hanging zfs send for 3 weeks now, on 3 TrueNAS boxes.

nabijaczleweli · 2025-01-21T15:57:34Z

Seems fixed to me then.

rkojedzinszky added the Type: Defect Incorrect behavior (e.g. crash, hang) label Nov 7, 2024

rkojedzinszky added a commit to dravanet/truenas-middleware that referenced this issue Nov 7, 2024

feat(openzfs): update openzfs port

8b06b33

openzfs/zfs#16731

rkojedzinszky added a commit to dravanet/truenas-middleware that referenced this issue Nov 30, 2024

feat(openzfs): update openzfs port

db05b56

openzfs/zfs#16731

rkojedzinszky added a commit to dravanet/truenas-middleware that referenced this issue Dec 16, 2024

feat(zettarepl): drop zfssend workaround

60487e9

Suspected patch reverted. See openzfs/zfs#16731

rkojedzinszky added a commit to dravanet/truenas-middleware that referenced this issue Dec 16, 2024

feat(zettarepl): drop zfssend workaround

8ef1fcf

Suspected patch reverted. See openzfs/zfs#16731

rkojedzinszky added a commit to dravanet/truenas-middleware that referenced this issue Dec 30, 2024

feat(openzfs): update openzfs port

5f556a5

Remove workaround applied in openzfs/zfs#16731

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ZFS send hangs sometimes #16731

ZFS send hangs sometimes #16731

rkojedzinszky commented Nov 7, 2024 •

edited

Loading

vedadkajtaz commented Nov 12, 2024

rkojedzinszky commented Nov 12, 2024

rkojedzinszky commented Nov 14, 2024

vedadkajtaz commented Nov 15, 2024

rkojedzinszky commented Nov 17, 2024

nabijaczleweli commented Nov 17, 2024

vedadkajtaz commented Nov 17, 2024

nabijaczleweli commented Nov 17, 2024 •

edited

Loading

vedadkajtaz commented Nov 17, 2024

rkojedzinszky commented Nov 17, 2024

rkojedzinszky commented Nov 29, 2024

vedadkajtaz commented Dec 2, 2024

rkojedzinszky commented Dec 2, 2024

nabijaczleweli commented Dec 2, 2024

nabijaczleweli commented Dec 2, 2024 •

edited

Loading

vedadkajtaz commented Dec 2, 2024

rkojedzinszky commented Dec 2, 2024

rkojedzinszky commented Dec 2, 2024

nabijaczleweli commented Dec 2, 2024

rkojedzinszky commented Dec 2, 2024

nabijaczleweli commented Dec 2, 2024

rkojedzinszky commented Dec 3, 2024

vedadkajtaz commented Dec 3, 2024

nabijaczleweli commented Dec 3, 2024

vedadkajtaz commented Dec 16, 2024

nabijaczleweli commented Dec 25, 2024

rkojedzinszky commented Dec 30, 2024

rkojedzinszky commented Jan 21, 2025

nabijaczleweli commented Jan 21, 2025

ZFS send hangs sometimes #16731

ZFS send hangs sometimes #16731

Comments

rkojedzinszky commented Nov 7, 2024 • edited Loading

System information

Describe the problem you're observing

Describe how to reproduce the problem

Include any warning/errors/backtraces from the system logs

vedadkajtaz commented Nov 12, 2024

rkojedzinszky commented Nov 12, 2024

rkojedzinszky commented Nov 14, 2024

vedadkajtaz commented Nov 15, 2024

rkojedzinszky commented Nov 17, 2024

nabijaczleweli commented Nov 17, 2024

vedadkajtaz commented Nov 17, 2024

nabijaczleweli commented Nov 17, 2024 • edited Loading

vedadkajtaz commented Nov 17, 2024

rkojedzinszky commented Nov 17, 2024

rkojedzinszky commented Nov 29, 2024

vedadkajtaz commented Dec 2, 2024

rkojedzinszky commented Dec 2, 2024

nabijaczleweli commented Dec 2, 2024

nabijaczleweli commented Dec 2, 2024 • edited Loading

vedadkajtaz commented Dec 2, 2024

rkojedzinszky commented Dec 2, 2024

rkojedzinszky commented Dec 2, 2024

nabijaczleweli commented Dec 2, 2024

rkojedzinszky commented Dec 2, 2024

nabijaczleweli commented Dec 2, 2024

rkojedzinszky commented Dec 3, 2024

vedadkajtaz commented Dec 3, 2024

nabijaczleweli commented Dec 3, 2024

vedadkajtaz commented Dec 16, 2024

nabijaczleweli commented Dec 25, 2024

rkojedzinszky commented Dec 30, 2024

rkojedzinszky commented Jan 21, 2025

nabijaczleweli commented Jan 21, 2025

rkojedzinszky commented Nov 7, 2024 •

edited

Loading

nabijaczleweli commented Nov 17, 2024 •

edited

Loading

nabijaczleweli commented Dec 2, 2024 •

edited

Loading