WIP: Concurrent tasks #101
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
robinheghan
requested changes
May 12, 2025
andrewMacmurray
force-pushed
the
concurrent-tasks
branch
9 times, most recently
from
5a0a78d to
dbf4acb
Compare
robinheghan
reviewed
May 13, 2025
andrewMacmurray
force-pushed
the
concurrent-tasks
branch
2 times, most recently
from
45a945c to
880f5f9
Compare
The scheduler queue starts to become very slow when a large number of tasks are in flight (Array.shift is O(n) as it needs to reindex the array every time an element is removed). This modifies `_Scheduler_enqueue` to loop through the active procs and resetting the queue when done. Larger arrays (1M+) are now handled more efficiently.
andrewMacmurray
force-pushed
the
concurrent-tasks
branch
from
880f5f9 to
2d3dccd
Compare
robinheghan
reviewed
May 13, 2025
| const handled = A2(_Scheduler_onError, onError, success); | ||
| return handled; | ||
| }); | ||
| procs = batch.map(_Scheduler_rawSpawn); |
There was a problem hiding this comment.
nitpick: maybe perform the rawSpawn inside of tasks.map to save one iteration through the array of tasks?
There was a problem hiding this comment.
There was an edge case I found with this in the experiments repo I can't quite remember (something where the behaviour with an immediate failure was different) - will dig it up and write a test for it.
This runs an array of tasks concurrently. If any of them fail the rest of the running tasks are cleaned up.
Because `map2` is now concurrent `Task.sequence` needs to be implemented using `andThen`.
map4+ become less useful as these are just chains of `andMap`. `andMap` runs concurrently as it's defined using `map2`.
andrewMacmurray
force-pushed
the
concurrent-tasks
branch
from
2d3dccd to
a538c8c
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds concurrent tasks to gren. The existing
Taskapi has been modified to:Task.concurrent, which runs an array of tasks concurrently.map2run concurrently.map4,map5in favour ofandMap.The existing
Schedulerkernel implementation is largely unchanged but with the addition of a_Scheduler_concurrenthelper that spawns multiple processes and handles collecting results and signalling errors.