Faster meta producers #1
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When compiling pipelines into something runnable, instead of getting producers ( remote and literal ) into a function that's immediately callable, use a function that returns a channel that will spawn some producers before closing. With this channel we can feed literal producers as normal, while allowing us to get remote producer descriptors during runtime, without any timeout. Refactor RunPipeline to collect producers from the spawner channel in chunks, running them in the order received, until the spawner closes ( which we test for by looking at whether or not the next producer is nil ) Refactor the pipeline testcase struct to specify how many producers to run in parallel so that we can test for that, testing for non-wholly divisible and overflowing parallel counts
This reverts commit 471d1d6.
Make the struct field name consistent across pipeline descriptors and compiled pipelines Fix an error where I just forgot to prop it up with other values LOL
gastrodon
force-pushed
the
faster-meta-producers
branch
from
71ad7f7 to
952f446
Compare
Welcome to Codecov 🎉Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests. Thanks for integrating Codecov - We've got you covered ☂️ |
Open
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.