[ENH] Alchemiscale task provenance

[IAlibay]

Something that has been brought up in the past (OpenFreeEnergy/gufe#168) is: what kind of provenance should be reported on protocol execution in the Alchemiscale package?

Especially as clients have a limited lifetime, it might be useful for users to get extra information about how the tasks executed (or failed).

I wanted to start this thread for the purposes of getting more information on this.

[IAlibay]

On my end the following would be very useful:

1. Hardware and GPU driver the task was executed on.
2. How long the task ran (even if it died, knowing if it used up the entire length of a client would prevent me from restarting randomly).
3. How many times the task has been restarted

[dotsdl]

@IAlibay thanks for this; I'm having trouble understanding what you mean by "knowing if it used up the entire length of a client". Can you clarify?

[IAlibay]

Here I mean - did it run and take the entire 6h allowed for the runner. I.e. even if you run it again it'll take too long and get interrupted again.

[dotsdl]

Thanks for this @IAlibay; I'll cast these suggestions into an issue on alchemiscale. As for how many times a Task has been restarted, the Task itself doesn't keep track of the states it has been in, but its failures are retained. You could use AlchemiscaleClient.get_task_failures(...), and the number of those failed ProtocolDAGResults would tell you how many times it's failed.

You could inspect the tracebacks for the ProtocolUnitFailures in each of those ProtocolDAGResults if you want to characterize the error modes the Task experienced.