[Proposed Feature] Support for workflows which would automatically update data

[cboettig]

Many data products that would be suited to Source Coop are regularly updated as new observations become available, etc. In such cases, data providers / curators may want to implement automated workflows that process new observations and upload them to source cooperative.

To do this, we need a mechanism for the workflow to authenticate with source cooperative and obtain an active access key, secret key, and session token with write permissions. Y'all probably have a much better idea how this would work than I, but I think the user/workflow machine would have to provide source cooperative with a valid AWS user ARN, and the source coop repo (i.e. web interface) would have to issue that ARN a valid role ARN, and then the machine could make the necessary IAM request to receive it's tokens?

What value is this feature adding to Source Cooperative?

This would support automated workflows and help keep data on source-cooperative up-to-date! (My current use case is updating https://beta.source.coop/eco4cast repos, (one repo gets new data daily, the other uses that data to compute scores for the dashboard, so also updates daily), but I think would apply to many other use cases as well.

[kbgg]

Thanks for this, Carl. Our plan is to open up our API which would enable workflows, like described, to generate credentials for a role which allows data manipulation on the bucket. It would produce the same credentials that you can currently generate on the UI which you could then programmatically feed into boto3 and refresh the credentials as needed. Does that sound like it meets your requirements?

[cboettig]

Nice, that sounds perfect! Looking forward to it 😺