Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove subscription lock at boot #287

Open
ericlyoung opened this issue Jul 25, 2024 · 5 comments
Open

Remove subscription lock at boot #287

ericlyoung opened this issue Jul 25, 2024 · 5 comments

Comments

@ericlyoung
Copy link

ericlyoung commented Jul 25, 2024

How would I disable the subscription advisory lock at startup of my app.
It is preventing me from being able to smoothly release a new version of my backend.
The lock prevents the new app from being able to subscribe until the lock times out, 60 sec.
The old lock hangs around, while that container is already shut down.

I would like to remove the lock at boot or disable the lock in the first place.

There is no way someone is using this in production and has not run into this issue.

@ericlyoung ericlyoung changed the title Disable subscription lock Remove subscription lock at boot Jul 25, 2024
@ericlyoung
Copy link
Author

ericlyoung commented Jul 25, 2024

What is the easiest way to disable the advisory lock for now?

@ericlyoung
Copy link
Author

been on this for many hours.
I've discovered that docker swarm is too aggressively killing the container with SIGKILL, when running docker stack deploy onto an already running stack (for auto-update), which prevents the lock release
I've discovered that docker service rm ... does correctly send SIGTERM, which allows the lock release to run

So, currently as a work-around I'm doing
docker service rm MYAPPSERVICE && docker stack deploy --with-registry-auth --compose-file stack.yml MYAPPSTACK
which gives me the lock cleanup and app update

I did notice you have a concurrency_limit option for the subscription, which gave me hope that I wouldn't have to mess with any of this but I set that to 99 and it didn't work.

more updates to come as I seek the best solution to this

@ericlyoung
Copy link
Author

I'll be honest, the code is so all-over-the-place that I can't figure out how to disable the subscription advisory locks. Please help me disable it.

@jdewar
Copy link
Contributor

jdewar commented Aug 8, 2024

This is something I will likely see for myself in the next few weeks. If I have any insight, I will post here.

As an aside, if the issue is the KILL stop signal, and a softer signal would suffice, it looks like there are a few options: moby/moby#25696

@ericlyoung
Copy link
Author

This is something I will likely see for myself in the next few weeks. If I have any insight, I will post here.

As an aside, if the issue is the KILL stop signal, and a softer signal would suffice, it looks like there are a few options: moby/moby#25696

After a huge amount of research I discovered that swarm is somewhat abandoned, and one of the things that's broken is the fact that you have no control how containers are stopped. SIGKILL is just plain sent. I tried every single mechanism that is supposed to work, 3-4 different ones. Maybe they worked 4 years ago, but all are ignored now.

Since my needs are very simple, I ended up switching to docker compose on the server, which does send the correct signal to stop containers, and so I do not hit this wall anymore.

All that being said, this project should have an option to force remove subscription lock at boot OR to not use it at all.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants