Skip to content

Commit

Permalink
Set up workflow to compare direct and Strapi-based output (RPB-59)
Browse files Browse the repository at this point in the history
See details and usage in README
  • Loading branch information
fsteeg committed Mar 12, 2024
1 parent 3213689 commit 5dae350
Show file tree
Hide file tree
Showing 2 changed files with 53 additions and 0 deletions.
16 changes: 16 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,22 @@ sbt "runMain rpb.ETL conf/test-export-strapi-to-lobid.flux"

This writes individual `.json` files for Strapi records to `output/`.

### Compare export data

```bash
sbt "runMain rpb.ETL conf/test-export-compare-strapi.flux PICK=all_equal('f36_','u') PATH=articles"
sbt "runMain rpb.ETL conf/test-export-compare-strapi.flux PICK=all_equal('f36_','s') PATH=independent-works"
sbt "runMain rpb.ETL conf/test-export-compare-strapi.flux PICK=all_equal('f36_','sbd') PATH=independent-works"
sbt "runMain rpb.ETL conf/test-export-compare-strapi.flux PICK=all_equal('f36t','MultiVolumeBook') PATH=independent-works"
```

This selects parts of the test data and write two files:

1) for each test record, get the data from the Strapi HTTP API, convert the result to the lobid format, write to `test-lobid-output-from-strapi.json`
2) convert each record directly to lobid, write to `test-lobid-output-from-strapi.json`

We can then compare the two files (e.g. in VSC: Select for Compare, Format Document) to see differences. Since fields that are not defined in the Strapi content types are omitted upon import, missing data here points to missing fields in the Strapi content types.

### Validate output

Prerequisites: `npm install -g ajv-cli ajv-formats`
Expand Down
37 changes: 37 additions & 0 deletions conf/test-export-compare-strapi.flux
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
// Get test data for the specified type; for each record,
// fetch the entry from Strapi, convert that to lobid, write.
FLUX_DIR + "output/test-output-strapi.json"
| open-file
| as-lines
| decode-json
| fix("
unless " + PICK + "
reject()
end
prepend(f00_, 'http://rpb-cms-test.lobid.org/api/" + PATH + "?populate=*&filters[f00_][$eq]=')
retain(f00_)
")
| literal-to-object
| log-object("Strapi URL: ")
| open-http
| as-records
| decode-json(recordPath="data.[*].attributes")
| fix(FLUX_DIR + "rpb-titel-to-lobid.fix")
| encode-json
| write(FLUX_DIR + "output/test-lobid-output-from-strapi.json")
;

// To compare, convert test data directly to lobid, write.
FLUX_DIR + "output/test-output-strapi.json"
| open-file
| as-lines
| decode-json
| fix("
unless " + PICK + "
reject()
end
")
| fix(FLUX_DIR + "rpb-titel-to-lobid.fix")
| encode-json
| write(FLUX_DIR + "output/test-lobid-output-from-file.json")
;

0 comments on commit 5dae350

Please sign in to comment.