How to provision new tasks on the fly

[MxmUrw]

In our dpsa-project we are implementing federated machine learning, which means that in each learning round we need to create a new aggregation task wrt which the clients submit their gradients of this round.

Appearently the DAP draft does not describe how new tasks can be provisioned.

Still, it seems that such functionality might be relatively easy implementable. Is it correct that a good starting point would be the provision_tasks function in aggregator/bin/janus_cli?

[tgeoghegan]

Everything you say is correct. Let me provide some more context on various points:

1. Yes, DAP deliberately leaves task negotiation unspecified. Shan Wang, Chris Patton and some others have been working on an extension to DAP for dynamic task provisioning called taskprov. We don't currently have plans to implement it in Janus, but I bring it up because it sounds like you have a pretty compelling use case for something like taskprov, and I'm sure the authors would be interested in talking to you about it.
2. For Divvi Up's purposes, we plan to build a task provisioning/control plane service that is responsible for allowing our subscribers to configure tasks and perform other administrative functions with aggregations. We're tracking that work in Provisioning of tasks into the database #44.
3. In the meantime, yes, we use janus_cli to manually provision tasks into the database of a Janus deployment. We also use janus_cli in a one-time bootstrapping step to provision the database schema and create a "datastore key", which is used to encrypt certain database rows.

[MxmUrw]

Ok, thanks for the info. I will try to adapt the code to work for us, when that is successful I will close this issue. Though to properly test this I guess I will need #718 :)

[inahga]

Related to #760.

[tgeoghegan]

The aggregator API has long since landed, so I'm closing this. @MxmUrw please feel free to re-open if it what's there doesn't meet your needs and we can discuss what else to put in there.