querying 'kind' of ABT_sched

[shanedsnyder]

For some background, in one of our projects, we have defined our own custom ABT scheduler (abt-snoozer) that we want to enable on all xstreams that call a specific initialization function in our API. This init routine can be called multiple times for a given xstream.

We ran into a problem, though, whenever we try to set a scheduler on an xstream that already has scheduled work units (e.g., on an xstream that has already called our init routine once and has work to perform). Looking over the ABT code, this appears to be expected behavior currently (TODO in src/stream.c @ line 783). Anytime an xstream calls our init routine multiple times, they will hit this error in ABT_xstream_set_main_sched(). We can work around this particular problem by calling ABT_sched_get_size() before setting the main scheduler to ensure there are no scheduled work units yet.

However, the issue we currently face is that we want to be able to guarantee whether or not our custom scheduler has been enabled for the xstream calling our init routine. This is trivial if the xstream's scheduler doesn't already have scheduled work units -- just attempt to set the scheduler and fail out if any errors occur. If a scheduler has already been set and has work units, though, we can't tell the user whether it is our own custom scheduler, some basic scheduler in ABT, or something else. It would be pretty helpful if ABT exposed some sort of method for determining the scheduler type for a given xstream.

Looking at the ABTI_sched type, there is a param called 'kind' that appears to store a unique ID (in the form of a pointer to the underlying scheduler definition type) that we could use to compare with the abt-snoozer scheduler definition type pointer in our code. Is this something that could be exposed to users somehow? Alternatively, would it be easier to allow ABT_xstream_set_main_sched() to change schedulers that already have work units (my guess is that is much harder)? Or do you see any other potential workarounds for this type of problem?

[halimamer]

Would stacking your custom scheduler on top of the main scheduler instead of replacing it would work for your case? You would achieve this by pushing your custom scheduler to the desired pools with ABT_pool_add_sched(). The pending work units on the target pool would be executed by the active (or main) scheduler until the custom the scheduler is reached. When the custom scheduler is reached, it will push the old (or main) scheduler to the stack and become the active one. Any further work unit from this point onward would be scheduled with your custom scheduler.

If this is not sufficient, then can you attach a minimal piece of code that reproduces this problem (something of the size of the basic examples at https://github.com/pmodels/argobots/tree/master/test/basic would do)?

[carns]

Hi @halimamer, we actually need to replace both the pool and scheduler for the main xstream. Can we do that with the ABT_pool_add_sched() function, or will that just replace the scheduler and keep the original pool in place?

For our use case they need to go in tandem because our scheduler sleeps when idle, and then is awoken by the pool when a new work unit is inserted, so they are kind of hooked together.

[halimamer]

@carns , the custom scheduler can pull from whichever pool you want, including the main scheduler's default pool. What you should know is that if the custom scheduler manages the same pool as the one it got push into, then that pool would be managed by that scheduler until the scheduler goes out of scope and the main scheduler would now be the one that manages it.