Skip to content

Commit d0600bd

Browse files
sphubersphuber
committed
Blog post: On provenance
1 parent 6c2b997 commit d0600bd

File tree

6 files changed

+203
-0
lines changed

6 files changed

+203
-0
lines changed

‎docs/news/pics/2023-11-20-on-provenance/image_01.png‎

6.01 KB
Loading

‎docs/news/pics/2023-11-20-on-provenance/image_02.png‎

21.3 KB
Loading

‎docs/news/pics/2023-11-20-on-provenance/image_03.png‎

42.5 KB
Loading

‎docs/news/pics/2023-11-20-on-provenance/image_04.png‎

61.6 KB
Loading

‎docs/news/pics/2023-11-20-on-provenance/image_05.png‎

93.1 KB
Loading

‎docs/news/posts/2023-11-20-on-provenance.md‎

Lines changed: 203 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,203 @@
1+
---
2+
blogpost: true
3+
category: Blog
4+
tags: provenance
5+
author: Sebastiaan Huber
6+
date: 2023-11-01
7+
---
8+
9+
# On provenance
10+
11+
One of the main defining characteristics of AiiDA is its focus on provenance.
12+
It aims to provide its users with the necessary tools to preserve the provenance of data that is produced by automated workflows.
13+
As a tool, AiiDA cannot enforce nor guarantee "perfect" provenance, but AiiDA simply encourages and enables its users to keep provenance as complete as possible and as detailed as necessary.
14+
What this means exactly is not so much a technical question as it is a philosophical question, and is always going to be use-case specific.
15+
In this blog post, I will discuss ideas concerning provenance that are hopefully useful to users of AiiDA when they are designing their workflows.
16+
In the following, I assume that the reader is already familiar with the basic concept of provenance as it is implemented in AiiDA and explained [in the documentation](https://aiida.readthedocs.io/projects/aiida-core/en/latest/topics/provenance/concepts.html).
17+
18+
As mentioned in the introduction, the provenance that is kept by AiiDA is not "perfect".
19+
That is to say, given a provenance graph created by AiiDA, it is not yet possible to fully reproduce the results in an automated way.
20+
Although all the inputs to the various processes are captured, the computing environments in which those processes took place, are not fully captured.
21+
AiiDA has the concepts of `Code`s and `Computer`s, which represent the code that processed the input and the compute environment in which it took place, but these are mere *symbolic* references.
22+
The recently added support for container technologies ([added in v2.1](https://github.com/aiidateam/aiida-core/blob/main/CHANGELOG.md#support-for-running-code-in-containers)) has made a great step in the direction of making perfect provenance possible, but for the time being, we will have to be content with a limited version.
23+
24+
It is important to note though, that not having perfect provenance is not the end of the world.
25+
Having any kind of provenance is often better than having not provenance at all.
26+
We should be wary not to fall victim to a myopic provenance puritism and remind ourselves that tracking provenance is not a goal in and of itself.
27+
Rather it is a solution to a particular problem: making computational results reproducible.
28+
29+
Imagine that we have some data: for the purpose of this example, we will take a simple dictionary.
30+
This dictionary can be stored in AiiDA's provenance graph by wrapping it in a `Dict` node:
31+
```python
32+
In [1]: from aiida import orm
33+
...: dict1 = orm.Dict({'key': 'value'}).store()
34+
...: dict2 = orm.Dict(dict1.get_dict())
35+
...: dict2['key'] = 'other_value'
36+
...: dict2.store()
37+
...: dict2.get_dict()
38+
Out[1]: {'key': 'other_value'}
39+
```
40+
We can now generate the provenance of `dict2` using the following `verdi` command:
41+
```console
42+
verdi node graph generate <PK>
43+
```
44+
which generates the following image:
45+
46+
![image_01](../pics/2023-11-20-on-provenance/image_01.png)
47+
48+
The updated dictionary appears isolated in the provenance graph: the fact that it was created by modifying another dictionary was not captured.
49+
The simplest way to capture the modifications of data is by warpping it in a [`calcfunction`](https://aiida.readthedocs.io/projects/aiida-core/en/latest/topics/calculations/concepts.html#calculation-functions):
50+
```python
51+
In [2]: from aiida import engine, orm
52+
...:
53+
...: @engine.calcfunction
54+
...: def update_dict(dictionary):
55+
...: updated = dictionary.get_dict()
56+
...: updated['key'] = 'other_value'
57+
...: return orm.Dict(updated)
58+
...:
59+
...: dict1 = orm.Dict({'key': 'value'})
60+
...: dict2 = update_dict(dict1)
61+
...: dict2.get_dict()
62+
Out[2]: {'key': 'other_value'}
63+
```
64+
If we recreate the provenance graph for `dict2` in this example, we get something like the following:
65+
66+
![image_02](../pics/2023-11-20-on-provenance/image_02.png)
67+
68+
The modification of the original `Dict` into another `Dict` has now been captured and is represented by the `update_dict` node in the provenance graph.
69+
Given that the source code of the `update_dict` function is stored, together with the original input, the produced output `Dict` can now be reproduced.
70+
71+
Of course output nodes can, and very often will in real scenarios, be modified themselves and in turn become inputs to calculations:
72+
73+
```python
74+
In [3]: from aiida import engine, orm
75+
...:
76+
...: @engine.calcfunction
77+
...: def add_random_key(dictionary):
78+
...: import secrets
79+
...: token = secrets.token_hex(2)
80+
...: updated = dictionary.get_dict()
81+
...: updated[token] = token
82+
...: return orm.Dict(updated)
83+
...:
84+
...: dict1 = orm.Dict()
85+
...: dict2 = add_random_key(dict1)
86+
...: dict3 = add_random_key(dict2)
87+
...: dict3.get_dict()
88+
Out[3]: {'14b4': '14b4', '38f8': '38f8'}
89+
```
90+
91+
![image_03](../pics/2023-11-20-on-provenance/image_03.png)
92+
93+
From this follows that, as a general rule of thumb, tracking the provenance of _inputs_ is just as important as that of outputs.
94+
However, in practice, there are pragmatic justifications for making an exception to this rule.
95+
To demonstrate this, we need to consider a more complex example that more closely resembles real-world use cases.
96+
For the following example, we imagine a workflow, implemented by a `WorkChain`, that wraps a subprocess.
97+
The exact nature of the subprocess is irrelevant, so we take a very straightforward `calcfunction` that simply returns the same content of its input dictionary as a stand-in.
98+
The workflow exposes the inputs of the subprocess, but _also_ adds a particular parameter as an explicit input.
99+
This is typically done to make the workflow easier to use for users as in this way they don't have to know exactly where in the sub process' input namespace the parameter is supposed to go.
100+
101+
102+
```{note}
103+
For a concrete example, see the [`only_initialization` input](https://github.com/aiidateam/aiida-quantumespresso/blob/74bbaa22b383b3323fcc3d41ad5b82fa89895c92/src/aiida_quantumespresso/workflows/ph/base.py#L35) of the `PhBaseWorkChain` of the `aiida-quantumespresso` plugin.
104+
```
105+
106+
```python
107+
In [4]: from aiida import engine, orm
108+
...:
109+
...: @engine.calcfunction
110+
...: def some_subprocess(parameters: orm.Dict):
111+
...: """Example subprocess that simply returns a dict with the same content as the ``parameters`` input."""
112+
...: return orm.Dict(parameters.get_dict())
113+
...:
114+
...:
115+
...: class SomeWorkChain(engine.WorkChain):
116+
...:
117+
...: @classmethod
118+
...: def define(cls, spec):
119+
...: super().define(spec)
120+
...: spec.expose_inputs(some_subprocess, namespace='sub')
121+
...: spec.input('some_parameter', valid_type=orm.Str, serializer=orm.to_aiida_type)
122+
...: spec.outline(cls.run_subprocess)
123+
...: spec.output('parameters')
124+
...:
125+
...: def run_subprocess(self):
126+
...: inputs = self.exposed_inputs(some_subprocess, 'sub')
127+
...: parameters = inputs.parameters.get_dict()
128+
...: parameters['some_parameter'] = self.inputs.some_parameter.value
129+
...: inputs['parameters'] = parameters
130+
...: result = some_subprocess(**inputs)
131+
...: self.out('parameters', result)
132+
...:
133+
...:
134+
...: inputs = {
135+
...: 'sub': {
136+
...: 'parameters': {
137+
...: 'some_parameter': 'value',
138+
...: }
139+
...: },
140+
...: 'some_parameter': 'other_value'
141+
...: }
142+
...: results, node = engine.run.get_node(SomeWorkChain, **inputs)
143+
...: results['parameters'].get_dict()
144+
Out[4]: {'some_parameter': 'other_value'}
145+
```
146+
147+
![image_04](../pics/2023-11-20-on-provenance/image_04.png)
148+
149+
150+
151+
```python
152+
from aiida import engine, orm
153+
154+
@engine.calcfunction
155+
def some_subprocess(parameters: orm.Dict):
156+
"""Example subprocess that returns a dict with the same content as `parameters` input."""
157+
return orm.Dict(parameters.get_dict())
158+
159+
160+
class SomeWorkChain(engine.WorkChain):
161+
162+
@classmethod
163+
def define(cls, spec):
164+
super().define(spec)
165+
spec.expose_inputs(some_subprocess, namespace='sub')
166+
spec.input('some_parameter', valid_type=orm.Str, serializer=orm.to_aiida_type)
167+
spec.outline(cls.run_subprocess)
168+
spec.output('parameters')
169+
170+
@staticmethod
171+
@engine.calcfunction
172+
def prepare_parameters(parameters, some_parameter):
173+
parameters = parameters.get_dict()
174+
parameters['some_parameter'] = some_parameter.value
175+
return orm.Dict(parameters)
176+
177+
def run_subprocess(self):
178+
inputs = self.exposed_inputs(some_subprocess, 'sub')
179+
inputs['parameters'] = self.prepare_parameters(
180+
inputs.pop('parameters'),
181+
self.inputs.some_parameter
182+
)
183+
result = some_subprocess(**inputs)
184+
self.out('parameters', result)
185+
186+
187+
inputs = {
188+
'sub': {
189+
'parameters': {
190+
'some_parameter': 'value',
191+
}
192+
},
193+
'some_parameter': 'other_value'
194+
}
195+
results, node = engine.run.get_node(SomeWorkChain, **inputs)
196+
results['parameters'].get_dict()
197+
```
198+
199+
```{note}
200+
Support for adding [process functions as class member functions](https://aiida.readthedocs.io/projects/aiida-core/en/latest/topics/processes/functions.html#as-class-member-methods) was added in AiiDA v2.3.
201+
```
202+
203+
![image_05](../pics/2023-11-20-on-provenance/image_05.png)

0 commit comments

Comments
 (0)