What's the precise role of desired_concurrency?

[EngineerKhan]

While using Crawlee, I have noticed that it updates its concurrency parameters every minute. I don't understand how desired_concurrency is calculated at each update and why there is such a mismatch between desired and actual concurrency.

Here are some of the outputs for reference

[crawlee._autoscaling.autoscaled_pool] INFO  current_concurrency = 13 ; desired_concurrency = 82
[crawlee._autoscaling.autoscaled_pool] INFO  current_concurrency = 11; desired_concurrency = 114
[crawlee._autoscaling.autoscaled_pool] INFO  current_concurrency = 11; desired_concurrency = 155
[crawlee._autoscaling.autoscaled_pool] INFO  current_concurrency = 11; desired_concurrency = 200
[crawlee._autoscaling.autoscaled_pool] INFO  current_concurrency = 11; desired_concurrency = 200
[crawlee._autoscaling.autoscaled_pool] INFO  current_concurrency = 10; desired_concurrency = 200
[crawlee._autoscaling.autoscaled_pool] INFO  current_concurrency = 9; desired_concurrency = 200
[crawlee._autoscaling.autoscaled_pool] INFO  current_concurrency = 9; desired_concurrency = 200
[crawlee._autoscaling.autoscaled_pool] INFO  current_concurrency = 8; desired_concurrency = 200
[crawlee._autoscaling.autoscaled_pool] INFO  current_concurrency = 5; desired_concurrency = 200
[crawlee._autoscaling.autoscaled_pool] INFO  current_concurrency = 2; desired_concurrency = 200

This pattern is something I have experienced in running other similar tasks too: current_concurrency increasing and then decreasing gradually, while desired_concurrency keeps rising until the cap of 200 (isn't it a bit too high?).

Thanks for your answer.

[B4nan]

current_concurrency is the exact number at the time of logging, desired_concurrency is the target. the desired value is dependent on available system resources.

its a bit weird that your current concurrency is so low all the time, its normal to see low values there, you could even hit a zero if all the tasks just finished before the logging happens. if you can provide a complete reproduction, we can take a look how it behaves on our end.

[EngineerKhan]

Exactly. I am finding it weird that it is targeting a concurrency of 200 while having it around 10. Is it something set by the OS or by Crawlee's own assessment? It continues to amaze me as I encountered even something like this:

[crawlee._autoscaling.autoscaled_pool] INFO current_concurrency = 18; desired_concurrency = 17 (by the way, is there any need for cpu and mem in this line as they always return 0 whenever I run any code?)

Unfortunately, the code is a bit big and messed up, so hard to share it. Can you please guide me to the specific files in the code handling this concurrency stuff, so I can look more into it? Thanks!

[B4nan]

[crawlee._autoscaling.autoscaled_pool] INFO current_concurrency = 18; desired_concurrency = 17 (by the way, is there any need for cpu and mem in this line as they always return 0 whenever I run any code?)

I don't see anything weird about that line, you can have 18 tasks running, and we can see the resources are lacking, so we downscale. Downscaling means we decrease the desired concurrency, it doesn't mean some tasks are cancelled - what was started will continue to run, scaling is about the new tasks, not about those in progress.

[EngineerKhan]

I still find it anomalous that on one hand, it is desiring 200 but running only a dozen tasks, while on another occasion, it's preempting lesser tasks than being actually run. I am quite interested to see the code calculating this target as my intuition says it can improve.

[B4nan]

well, yes, but without you providing a reproduction, we cant really help with that. i am asking for a reproduction, not for your app.