worker available is lost after a while

[jstark1]

we are using mod-gearman-worker-go with OMD 2.90 (also with 3.00).
With the command check_gearman -H localhost:4730 -q worker_<hostname> -x we check the worker queues. Some workers lose the connection to their worker queue after some time.
#6 did not help in this case.
When we restart the gearman_worker, it immediately reconnects.
The normal services queues are working as expected

[sni]

Does it occur with the currently nightly as well?

[jstark1]

I tried version mod_gearman_worker - version 1.1.1 (Build: 3.01.20190412-labs-edition-v1.1.1) with the same result.

[sni]

anything in the worker log?

[jstark1]

the worker logs look normal to me. i can provide you a log per PM if you want. As far as I understand the worker creates his worker_<hostname> queue at daemon start. Is there a maintenance process for the worker-queue if it gets removed for the gearmand in case of an daemon restart or an network outage? Or does the gearmand remove queues without any workers?

i think the main problem is that under some circumstances the "Worker Available" in the worker_<hostname> changed from 1 to 0 and comes only with an Worker restart back.

From the mod-gearman docs https://github.com/sni/mod_gearman#how-to i learned to use ./check_gearman -H <job server hostname> -q worker_<worker hostname> -t 10 -s check.
With my check form above i do not use the -s check so i only check the Available worker in the queue.
I will give this command a try so the worker_queue has some work to do.

[jstark1]

i can now reproduce the Problem.
The worker starts and makes some connections to the gearmand server. I assume these are the connection to the different queues. One of the connection idles and get terminated after 1 hour trough our firewall. After another 1-2 hours the gearmand sets "Worker Available" in the worker_ queue to zero.

If i use the ./check_gearman -H <job server hostname> -q worker_<worker hostname> -t 10 -s check command, the idle connection counter is reset when the command has been run (5 minute check interval).

[sni]

sounds reasonable. Thanks for the heads up. So is there anything from the worker side we could do? As far as i remember, the "old" worker renews the status worker from time to time which probably prevents this issue from happening.
I guess that would be ok for the go worker as well.

[dirtyren]

What is happening to me on CentOS8 is that mod_gearman go will do a denial of service on gearmand, with connections building up till it reaches 1000 open TCP connections and gearmand will not accept connections anymore.
I changed it back to the C version and this is not happening with it.
Strangely enough, on another installation I can not reproduce this problem with the same versions installed.

C Version
Every 2.0s: netstat -anp | grep mod_gearman | wc -l opmoncloud8: Mon Jun 21 09:29:22 2021
6

Go Version - keeps going UP
Every 2.0s: netstat -anp | grep mod_gearman | wc -l opmoncloud8: Mon Jun 21 09:41:41 2021
236
Every 2.0s: netstat -anp | grep mod_gearman | wc -l opmoncloud8: Mon Jun 21 09:43:27 2021
272
Every 2.0s: netstat -anp | grep mod_gearman | wc -l opmoncloud8: Mon Jun 21 10:22:52 2021
969

Tks.

[sni]

i guess that's something else. Do the worker have a reasonable limit of connections/threads (max-worker)? You could also send a SIGUSR1 to the worker process to create a thread dump.

[dirtyren]

No problem, will do it now, the conf is like this:

job_timeout=60
min-worker=5
max-worker=50
idle-timeout=30
max-jobs=1000
spawn-rate=1
fork_on_exec=no

`[2021-06-21 15:40:19.095][Error][mod_gearman_worker_linux.go:29] requested thread dump via signal user defined signal 1
[2021-06-21 15:40:19.096][Error][mod_gearman_worker.go:315] threaddump:
goroutine 9 [running]:
github.com/ConSol/mod-gearman-worker-go.logThreaddump()
/root/go/src/github.com/ConSol/mod-gearman-worker-go/mod_gearman_worker.go:311 +0x6f
github.com/ConSol/mod-gearman-worker-go.mainSignalHandler(0x9f2520, 0xc99230, 0x3)
/root/go/src/github.com/ConSol/mod-gearman-worker-go/mod_gearman_worker_linux.go:30 +0x2d5
github.com/ConSol/mod-gearman-worker-go.Worker.func1(0xc000064f60)
/root/go/src/github.com/ConSol/mod-gearman-worker-go/mod_gearman_worker.go:89 +0x77
created by github.com/ConSol/mod-gearman-worker-go.Worker
/root/go/src/github.com/ConSol/mod-gearman-worker-go/mod_gearman_worker.go:85 +0x1e7

goroutine 1 [select]:
github.com/ConSol/mod-gearman-worker-go.mainLoop(0xc000136900, 0xc0000652c0, 0xc0001e9ee0, 0x0, 0x9811b8, 0x0, 0x0)
/root/go/src/github.com/ConSol/mod-gearman-worker-go/mod_gearman_worker.go:155 +0x6a5
github.com/ConSol/mod-gearman-worker-go.Worker(0x0, 0x0)
/root/go/src/github.com/ConSol/mod-gearman-worker-go/mod_gearman_worker.go:105 +0x279
main.main()
/root/go/src/github.com/ConSol/mod-gearman-worker-go/cmd/mod_gearman_worker/main.go:12 +0x39

goroutine 19 [chan receive]:
github.com/appscode/g2/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0xcd0dc0)
/root/go/src/github.com/appscode/g2/vendor/github.com/golang/glog/glog.go:879 +0x8b
created by github.com/appscode/g2/vendor/github.com/golang/glog.init.0
/root/go/src/github.com/appscode/g2/vendor/github.com/golang/glog/glog.go:410 +0x274

goroutine 8 [syscall]:
os/signal.signal_recv(0x9f2520)
/usr/local/go/src/runtime/sigqueue.go:147 +0x9d
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:23 +0x25
created by os/signal.Notify.func1.1
/usr/local/go/src/os/signal/signal.go:150 +0x45

goroutine 36 [sleep]:
time.Sleep(0xb2d05e00)
/usr/local/go/src/runtime/time.go:188 +0xbf
github.com/ConSol/mod-gearman-worker-go.mainLoop.func2(0xc0000223f0)
/root/go/src/github.com/ConSol/mod-gearman-worker-go/mod_gearman_worker.go:147 +0x53
created by github.com/ConSol/mod-gearman-worker-go.mainLoop
/root/go/src/github.com/ConSol/mod-gearman-worker-go/mod_gearman_worker.go:141 +0x599

goroutine 20 [IO wait]:
internal/poll.runtime_pollWait(0x7ff360a3ada0, 0x72, 0x9eea40)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc0001b4898, 0x72, 0x9eea00, 0xc81698, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0001b4880, 0xc0000d8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:159 +0x1a5
net.(*netFD).Read(0xc0001b4880, 0xc0000d8000, 0x1000, 0x1000, 0x1000, 0x7ff38b0347d0, 0xc0000d8000)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc0000b6030, 0xc0000d8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:182 +0x8e
bufio.(*Reader).Read(0xc0000b81e0, 0xc0000d8000, 0x1000, 0x1000, 0xc, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:213 +0x142
github.com/appscode/g2/worker.(*agent).read(0xc00006ee60, 0xc000216f00, 0x40, 0xc0000b8360, 0xc, 0x0)
/root/go/src/github.com/appscode/g2/worker/agent.go:182 +0xa5
github.com/appscode/g2/worker.(*agent).work(0xc00006ee60)
/root/go/src/github.com/appscode/g2/worker/agent.go:61 +0xc7
created by github.com/appscode/g2/worker.(*agent).Connect
/root/go/src/github.com/appscode/g2/worker/agent.go:44 +0x227

goroutine 21 [chan receive]:
github.com/appscode/g2/worker.(*Worker).Work(0xc0001b4780)
/root/go/src/github.com/appscode/g2/worker/worker.go:220 +0xc6
github.com/ConSol/mod-gearman-worker-go.newWorker.func2(0xc0001b4780)
/root/go/src/github.com/ConSol/mod-gearman-worker-go/worker.go:72 +0x4d
created by github.com/ConSol/mod-gearman-worker-go.newWorker
/root/go/src/github.com/ConSol/mod-gearman-worker-go/worker.go:70 +0x5d2

goroutine 50 [IO wait]:
internal/poll.runtime_pollWait(0x7ff360a3abd0, 0x72, 0x9eea40)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc0000d2118, 0x72, 0x9eea00, 0xc81698, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0000d2100, 0xc000244000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:159 +0x1a5
net.(*netFD).Read(0xc0000d2100, 0xc000244000, 0x1000, 0x1000, 0x1000, 0x7ff38b034e98, 0xc000244000)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc00021e000, 0xc000244000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:182 +0x8e
bufio.(*Reader).Read(0xc00020e0c0, 0xc000244000, 0x1000, 0x1000, 0xc, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:213 +0x142
github.com/appscode/g2/worker.(*agent).read(0xc0000c40f0, 0xc000036f00, 0x40, 0xc00020e480, 0xc, 0x0)
/root/go/src/github.com/appscode/g2/worker/agent.go:182 +0xa5
github.com/appscode/g2/worker.(*agent).work(0xc0000c40f0)
/root/go/src/github.com/appscode/g2/worker/agent.go:61 +0xc7
created by github.com/appscode/g2/worker.(*agent).Connect
/root/go/src/github.com/appscode/g2/worker/agent.go:44 +0x227

goroutine 51 [chan receive]:
github.com/appscode/g2/worker.(*Worker).Work(0xc0000d2000)
/root/go/src/github.com/appscode/g2/worker/worker.go:220 +0xc6
github.com/ConSol/mod-gearman-worker-go.newWorker.func2(0xc0000d2000)
/root/go/src/github.com/ConSol/mod-gearman-worker-go/worker.go:72 +0x4d
created by github.com/ConSol/mod-gearman-worker-go.newWorker
/root/go/src/github.com/ConSol/mod-gearman-worker-go/worker.go:70 +0x5d2

goroutine 52 [IO wait]:
internal/poll.runtime_pollWait(0x7ff360a3aae8, 0x72, 0x9eea40)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000236118, 0x72, 0x9eea00, 0xc81698, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000236100, 0xc0000da000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:159 +0x1a5
net.(*netFD).Read(0xc000236100, 0xc0000da000, 0x1000, 0x1000, 0x1000, 0x7ff38b0347d0, 0xc0000da000)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc00021e010, 0xc0000da000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:182 +0x8e
bufio.(*Reader).Read(0xc00020e1e0, 0xc0000da000, 0x1000, 0x1000, 0xc, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:213 +0x142
github.com/appscode/g2/worker.(*agent).read(0xc000234050, 0xc000212f00, 0x40, 0xc0000b8420, 0xc, 0x0)
/root/go/src/github.com/appscode/g2/worker/agent.go:182 +0xa5
github.com/appscode/g2/worker.(*agent).work(0xc000234050)
/root/go/src/github.com/appscode/g2/worker/agent.go:61 +0xc7
created by github.com/appscode/g2/worker.(*agent).Connect
/root/go/src/github.com/appscode/g2/worker/agent.go:44 +0x227

goroutine 53 [chan receive]:
github.com/appscode/g2/worker.(*Worker).Work(0xc000236000)
/root/go/src/github.com/appscode/g2/worker/worker.go:220 +0xc6
github.com/ConSol/mod-gearman-worker-go.newWorker.func2(0xc000236000)
/root/go/src/github.com/ConSol/mod-gearman-worker-go/worker.go:72 +0x4d
created by github.com/ConSol/mod-gearman-worker-go.newWorker
/root/go/src/github.com/ConSol/mod-gearman-worker-go/worker.go:70 +0x5d2

goroutine 54 [IO wait]:
internal/poll.runtime_pollWait(0x7ff360a3aa00, 0x72, 0x9eea40)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000236298, 0x72, 0x9eea00, 0xc81698, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000236280, 0xc000241000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:159 +0x1a5
net.(*netFD).Read(0xc000236280, 0xc000241000, 0x1000, 0x1000, 0x1000, 0x7ff38b034e98, 0xc000241000)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc00021e020, 0xc000241000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:182 +0x8e
bufio.(*Reader).Read(0xc00020e300, 0xc000241000, 0x1000, 0x1000, 0xc, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:213 +0x142
github.com/appscode/g2/worker.(*agent).read(0xc0002340f0, 0xc000217f00, 0x40, 0xc00020e420, 0xc, 0x0)
/root/go/src/github.com/appscode/g2/worker/agent.go:182 +0xa5
github.com/appscode/g2/worker.(*agent).work(0xc0002340f0)
/root/go/src/github.com/appscode/g2/worker/agent.go:61 +0xc7
created by github.com/appscode/g2/worker.(*agent).Connect
/root/go/src/github.com/appscode/g2/worker/agent.go:44 +0x227

goroutine 55 [chan receive]:
github.com/appscode/g2/worker.(*Worker).Work(0xc000236180)
/root/go/src/github.com/appscode/g2/worker/worker.go:220 +0xc6
github.com/ConSol/mod-gearman-worker-go.newWorker.func2(0xc000236180)
/root/go/src/github.com/ConSol/mod-gearman-worker-go/worker.go:72 +0x4d
created by github.com/ConSol/mod-gearman-worker-go.newWorker
/root/go/src/github.com/ConSol/mod-gearman-worker-go/worker.go:70 +0x5d2

goroutine 66 [IO wait]:
internal/poll.runtime_pollWait(0x7ff360a3a918, 0x72, 0x9eea40)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000236418, 0x72, 0x9eea00, 0xc81698, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000236400, 0xc0002a4000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:159 +0x1a5
net.(*netFD).Read(0xc000236400, 0xc0002a4000, 0x1000, 0x1000, 0x1000, 0x7ff38b035560, 0xc0002a4000)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc00028a000, 0xc0002a4000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:182 +0x8e
bufio.(*Reader).Read(0xc0002840c0, 0xc0002a4000, 0x1000, 0x1000, 0xc, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:213 +0x142
github.com/appscode/g2/worker.(*agent).read(0xc000234190, 0xc000214f00, 0x40, 0xc000284480, 0xc, 0x0)
/root/go/src/github.com/appscode/g2/worker/agent.go:182 +0xa5
github.com/appscode/g2/worker.(*agent).work(0xc000234190)
/root/go/src/github.com/appscode/g2/worker/agent.go:61 +0xc7
created by github.com/appscode/g2/worker.(*agent).Connect
/root/go/src/github.com/appscode/g2/worker/agent.go:44 +0x227

goroutine 67 [chan receive]:
github.com/appscode/g2/worker.(*Worker).Work(0xc000236300)
/root/go/src/github.com/appscode/g2/worker/worker.go:220 +0xc6
github.com/ConSol/mod-gearman-worker-go.newWorker.func2(0xc000236300)
/root/go/src/github.com/ConSol/mod-gearman-worker-go/worker.go:72 +0x4d
created by github.com/ConSol/mod-gearman-worker-go.newWorker
/root/go/src/github.com/ConSol/mod-gearman-worker-go/worker.go:70 +0x5d2

goroutine 68 [IO wait]:
internal/poll.runtime_pollWait(0x7ff360a3a830, 0x72, 0x9eea40)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc00029a118, 0x72, 0x9eea00, 0xc81698, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00029a100, 0xc0002a5000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:159 +0x1a5
net.(*netFD).Read(0xc00029a100, 0xc0002a5000, 0x1000, 0x1000, 0x1000, 0x7ff38b035560, 0xc0002a5000)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc00028a010, 0xc0002a5000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:182 +0x8e
bufio.(*Reader).Read(0xc000284240, 0xc0002a5000, 0x1000, 0x1000, 0xc, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:213 +0x142
github.com/appscode/g2/worker.(*agent).read(0xc000296050, 0xc000228700, 0x40, 0xc0002844e0, 0xc, 0x0)
/root/go/src/github.com/appscode/g2/worker/agent.go:182 +0xa5
github.com/appscode/g2/worker.(*agent).work(0xc000296050)
/root/go/src/github.com/appscode/g2/worker/agent.go:61 +0xc7
created by github.com/appscode/g2/worker.(*agent).Connect
/root/go/src/github.com/appscode/g2/worker/agent.go:44 +0x227

goroutine 69 [chan receive]:
github.com/appscode/g2/worker.(*Worker).Work(0xc00029a000)
/root/go/src/github.com/appscode/g2/worker/worker.go:220 +0xc6
github.com/ConSol/mod-gearman-worker-go.newWorker.func2(0xc00029a000)
/root/go/src/github.com/ConSol/mod-gearman-worker-go/worker.go:72 +0x4d
created by github.com/ConSol/mod-gearman-worker-go.newWorker
/root/go/src/github.com/ConSol/mod-gearman-worker-go/worker.go:70 +0x5d2
`