Improving the approval workflow

[Mohammer5]

I've gotten a small overview over how approval works by @larshelge when introducing the concept of the new data approval app.
After having worked with typo3 versioning & workspaces, maybe I can contribute some ideas how to improve the workflows of the approval process. The versioning system that typo3 uses has no relevance to use as the data that's being managed has a fundamental different purpose. Handling the versioning and publishing process with typo3 has some really nice concepts though, that we could copy. This article describes "the user interface that sits on top of the versioning concept", if you want to read more about it.

Versioning of data

This step is quite important as it help quite significantly with the review process after having requested changes.
The reason why this is so beneficial is because it allows to show the differences between version:

• These help the user who is managing the data to see what has not been addressed yet after having received a change-request
• These help the user who is reviewing the data to see what has changed since the last review

For us, I can see two different ways to create a new version:

• Either when data is ready for approval (when the last data has been added), there must be a person who knows about that. That person can mark a time period for being ready to be reviewed/approved (essentially locking the data until the review process has responded or the the request for approval is taken back because something needs to be changed)
• Or data sets for a time period can be approved at any time. When disapproving data sets for a specific time period, a new version is created that gets locked (like adding a tag in git that points to the last time of review)

In both cases, adding data for a specific time period should be locked while the approval process is pending. If data needs to be added/changed, then the approval process should be aborted and restarted after the data has been altered.

I really like the idea of datomic, but I'm not sure if there's a way to apply their concept with our current database, so this is just a side note, typo3 solves this by duplicating rows in the DB and attaching a version to them). Datomic has a concept of "immutable facts over time" with a very strong data model:

Both Neo4j and Datomic can model arbitrary relationships. They both use, effectively, an EAV (entity-attribute-value) schema so they both can model many of the same problem domains except Datomic's EAV schema also embeds a time dimension (i.e. EAVT) which makes it very powerful if you want to perform efficient queries against your database at arbitrary points in time. This is something that non-immutable data stores (Neo4j included) could simply not do. (source)

User interface for managing versions

Similarly to what we already have when defining a workflow, the workflow must consist of different approval stages. In typo3 a higher stage can approve data without having to wait for a lower/previous stage, I don't know if we want to add that to our model, just want to mention it for reference, as it can be quite useful from time to time.

When reviewing data for approval, the user should be able to see a diff. With typo3 that's very complicated as it's about a page on a website, but with the data approval app, which is only displaying tabular data, this should be a lot easier. New rows/columns can be green, removed ones red and changed ones contain both the old and new data (very similar to a colorized git diff). Ideally all changes can be gone through: Initially the diff between the current version and the current version - 1 is shown, when navigating to the previous diff, it's: current version - 1 and current version - 2.

When rejecting data, a message can (should) be attached to the rejection, which sends the data back to the previous workflow stage/group, these should also be visible when comparing versions during the review process, which allows the reviewing user to know what changes they've requested for a specific time period.

---

I assume this requires quite a bit of work (if adopted at all), but would improve the UX quite significantly.
CC: @larshelge @amcgee @cooper-joe @varl

[jimgrace]

I like these ideas. I wonder if we could make use of the data value history that we already maintain (at least when data is entered through the data entry app), and keep a better approval/unapproval history so we can compare the dates. Might we want to a system option to add to the data value history for data imports (and maybe only when/if the value changes)? We would want to be sure it still performs reasonably well.

I've heard users request something related: When a higher-level user wants to request that lower-level data be corrected and resubmitted, they want to be able to selectively unlock the data that they are requesting to be re-reviewed, for example unlocking only one data element, and only at one facility. Currently, for example, an above-district reviewer would have to unlock all data in the workflow for an entire district. But if we do something like this, it would be good to then see the data changes afterwards as described here.

I have also heard from users the desire to attach a message to a rejection, instead of having to rely on out-of-band communication.

I don't believe either of these two requests has been entered into Jira.

The following open Jira tickets for data approval should also be considered when weighing priorities:

https://jira.dhis2.org/browse/DHIS2-120 Approval for tracker/event data

https://jira.dhis2.org/browse/DHIS2-161 Data approval exchange

https://jira.dhis2.org/browse/DHIS2-179 Alerts for data approval

https://jira.dhis2.org/browse/DHIS2-3102 Data approval should be able to approve total value rather than catcombos
(I've thought that one way to address this might be to optionally assign a report to a workflow. If one is assigned, then that report is used to review the workflow data. If one is not assigned, then we display data set reports as we are doing in the new Data Approval App being designed for 2.37.)

https://jira.dhis2.org/browse/DHIS2-11518 Add Category Combo to Data Approval Workflow

Other notes:

• When it reads "approve data sets" above, I understand that to mean "approve data approval workflows". (If an installation wants to approve data sets independently, they can assign each one to its own workflow.)
• I believe workflows for a given time period can already be approved at any time. We're just making the assumption that most users will want to approve data for periods that have ended, so that's a good starting default for selecting a period.
• We've heard clearly from users that the normal process they want is for each level to review and approve the data. A lower-level data manager might see something that looks wrong even if it looks OK to a higher-level user. We also have an authority for a higher-level user to approve lower-level data. The intent is that this be used in exceptional cases when the lower-level user has not acted in time, but not for routine approval. On the other hand, PEPFAR has written their own script to force approval at all levels if data has not been approved by a certain time. This may be an instance of allowing out-of-order approvals as suggested above.