Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PM2 cluster error "timeout: no answer from master in time" if the master is not already available when the client is created #242

Open
ighunter opened this issue Dec 12, 2023 · 2 comments
Labels
bug Something isn't working

Comments

@ighunter
Copy link

ighunter commented Dec 12, 2023

Following the example at https://github.com/animir/node-rate-limiter-flexible/wiki/PM2-cluster, we have one single-instance pm2 service for the RateLimiterClusterMasterPM2, and then we have another clustered service that uses new RateLimiterCluster() to initialize the rate-limiting, then we call .consume() in the client later.

If the order and timing of the service startup is as follows:

  1. new RateLimiterClusterMasterPM2(pm2) happens in the master service,
  2. new RateLimiterCluster() happens in the client services,

then all is good, and it works as expected.

However, if the order and timing of the service startup is as follows:

  1. new RateLimiterCluster() happens in a client service,
  2. then new RateLimiterClusterMasterPM2(pm2) happens in the master service,

then later when we call .consume() in the client we get this error:

Error: RateLimiterCluster timeout: no answer from master in time
     at Timeout._onTimeout (node_modules/rate-limiter-flexible/lib/RateLimiterCluster.js:188:14)
     at listOnTimeout (node:internal/timers:569:17)
     at process.processTimers (node:internal/timers:512:7)

Setting an arbitrarily large timeoutMs in the options passed to new RateLimiterCluster() such that the master should definitely have started within that many ms appears to make no difference.

This sequence of events also results in the same exception:

  1. new RateLimiterClusterMasterPM2(pm2) happens in the master service,
  2. new RateLimiterCluster() happens in the client services,
  3. One or more .consume()s happen successfully,
  4. The master service restarts while the client services continue to run uninterrupted,

then any further .consume() calls in the clients fail with RateLimiterCluster timeout: no answer from master in time

Is there something we should be doing differently, or is this something that can be fixed?

Node version: 18.17.1
rate-limiter-flexible version: 2.4.1
OS: Ubuntu 22.04.3 LTS

@animir
Copy link
Owner

animir commented Dec 16, 2023

@ighunter Hi, thank you for the detailed description.
This looks like a bug.
Can you start the master process and only after it starts workers?
Can you restart workers, when master restarts?
I know this isn't the best solution, but you can try until we understand the problem.

@animir animir added the bug Something isn't working label Dec 16, 2023
@ighunter
Copy link
Author

Thanks @animir - yes, we have workarounds in place for now 👌

@animir animir changed the title Error "RateLimiterCluster timeout: no answer from master in time" if the master is not already available when the client is created PM2 cluster error "timeout: no answer from master in time" if the master is not already available when the client is created Jan 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants