Prevent Jobs from Starving in the Active Queue #144
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
sven-rosenzweig
force-pushed
the
fix/stuck_in_active_queue
branch
2 times, most recently
from
0678f85 to
5b5ec6d
Compare
mutax
previously approved these changes
Feb 11, 2025
mutax
reviewed
Feb 11, 2025
The active queue uses a heap-based priority queue, sorting jobs by priority. Tasks received via RPC from Neutron have the highest priority. Jobs from the sync loop have lower priority. Jobs moved from the passive to the active queue retain their original priority. Every interval x, the agent triggers a background sync, placing objects that need synchronization into the passive queue. These jobs may be moved to the active queue if capacity allows. However, when the agent is fully occupied (handling 40 tasks simultaneously), and more high-priority jobs are coming in, lower-priority jobs in the active queue may become stuck for some time, as higher-priority jobs are continuously processed first. The problem with starvation is amplified by the fact, that items from the passive queue with lower priority will prevent new jobs being added with higher priority -- see the __eq__ method of Runnable in line 78 and the comment. With our approach sort stability is not given. Adding new jobs with HIGHEST priority does not guarantee they are returned in the order the have been added. Details can be found in the heapq documentation (https://docs.python.org/3/library/heapq.html#priority-queue-implementation-notes). To overcome this potential drawbacks the active queue becomes a FiFo Queue. Jobs on the active queue are treated with same equality and are processed with insertion order.
sven-rosenzweig
force-pushed
the
fix/stuck_in_active_queue
branch
from
5b5ec6d to
dfc3b63
Compare
mutax
approved these changes
Feb 12, 2025
|
code cherry picked into PR #145 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The active queue uses a heap-based priority queue, sorting jobs by priority.
Tasks received via RPC from Neutron have the highest priority.
Jobs from the sync loop have lower priority.
Jobs moved from the passive to the active queue retain their original priority.
Every interval x, the agent triggers a background sync, placing objects that
need synchronization into the passive queue.
These jobs may be moved to the active queue if capacity allows.
However, when the agent is fully occupied (handling 40 tasks simultaneously),
and more high-priority jobs are coming in, lower-priority jobs in the active
queue may become stuck for some time, as higher-priority
jobs are continuously processed first.
With our approach sort stability is not given.
Adding new jobs with HIGHEST priority does not guarantee they are
returned in the order the have been added. Details can be found in
the heapq documentation (https://docs.python.org/3/library/heapq.html#priority-queue-implementation-notes).
To overcome this potential drawbacks the active queue becomes a FiFo
Queue. Jobs on the active queue are treated with same equality and are
processed with insertion order.