[267] Tidy & slim schema transformations #281

jaklinger · 2020-06-03T16:17:34Z

Related to #267, which isn't closed until #280 is addressed (which is the base of this branch).

Prune deprecated schema transformations
Only one ontology per dataset (no project specific ontologies)
Tidy directory naming, and refactor references accordingly
Add entity_type to new dataset ontologies
Remove unused ontologies from tier_1.json (renamed ontology.json)
Add tests to ensure ES mappings are both self-consistent and consistent with the slimmed schema transformations
Rollback any inadvertently changed json files, unrelated to this PR
Confirm all new mappings and old mappings are equivalent, then prune old mappings
Spawn PR [267b] Iterate on development pipelines with new ES config setup #282 to iterate on dev pipeline validation

jaklinger · 2020-06-05T14:50:32Z

@mindrones
For the review, could you please consider any file under nesta/core/schemas/tier_1/. All other changes are in the python code to reflect corresponding simplifications in the directory and config structure.

mindrones

Just noted down a couple of changes I'd like to see, but not really sure what implications these would have.

nesta/core/schemas/tier_1/datasets/arxiv.json

mindrones · 2020-06-05T15:19:20Z

nesta/core/schemas/tier_1/mappings/defaults/defaults.json

@@ -1,24 +1,24 @@
 {
-    "mappings":{
-	"_doc":{


(just a comment, in Svizzle I'm actually going back to tabs as it's editor configurable so it makes everyone happy :)

oops, actually I have a new unit test that this would have failed (all json in the repo must be clean to pass) - so this will fail anyway :)

mindrones

Good to merge for my part 👍

* changed schema_transformor to use new simpler mapping * removed to/from keys * new null syntax mapping implemented * adding temporary eurito-dev index to avoid conflating es7 compatibility issues * adding temporary eurito-dev index to avoid conflating es7 compatibility issues * testing es7 on cordis only * testing es7 on cordis only * testing es7 on cordis only * changes to make cordis es7 run * eurito-dev iteration * compatibility issues between arxlive and eurito arxiv * sorted json * pycountry change no longer assumes not null country * needed to split pathstub args * removed redundant es mappings * old new index paradigm fix Co-authored-by: Joel Klinger <[email protected]>

* changed branch name * mappings build * updated docs * updated docs * updated docs * added docstrings * added dynamic strict to settings * removed index.json in favour of a single defaults file * using soft alias until a future PR to minimise changes * cleaned and sorted json * [267] Tidy & slim schema transformations (#281) * pruned deprecated schema transformations * updated fos fieldname on arxlive * unified data set schema transformations * restructured directory * refactored references to schema_transformation * refactored references to schema_transformation * slimmed down transformations, and included entity_type * pruned ontology * tidied schemas * consistency tests * reverted unrelated json file * harmonised name fieldsofstudy across arxiv * added novelty back in * sorted json * sorted json * sorted json Co-authored-by: Joel Klinger <[email protected]> Co-authored-by: Joel Klinger <[email protected]>

* make sure conf dir is empty * simplified es config * added orm es config reader * modified setup_es to pick up new es config * swapped es_mode for boolean * aliases now consistent with config * aliases now automatically located * added endpoint field to estasks * added endpoint field to sql2estasks * [267] Pool ES mappings across datasets (#280) * changed branch name * mappings build * updated docs * updated docs * updated docs * added docstrings * added dynamic strict to settings * removed index.json in favour of a single defaults file * using soft alias until a future PR to minimise changes * cleaned and sorted json * [267] Tidy & slim schema transformations (#281) * pruned deprecated schema transformations * updated fos fieldname on arxlive * unified data set schema transformations * restructured directory * refactored references to schema_transformation * refactored references to schema_transformation * slimmed down transformations, and included entity_type * pruned ontology * tidied schemas * consistency tests * reverted unrelated json file * harmonised name fieldsofstudy across arxiv * added novelty back in * sorted json * sorted json * sorted json Co-authored-by: Joel Klinger <[email protected]> Co-authored-by: Joel Klinger <[email protected]> * patched out es config setup from tests * removed redundant tests * fixed json formatting * none included for testing * picked up bug in test Co-authored-by: Joel Klinger <[email protected]>

* make sure conf dir is empty * simplified es config * added orm es config reader * modified setup_es to pick up new es config * swapped es_mode for boolean * aliases now consistent with config * aliases now automatically located * added endpoint field to estasks * added endpoint field to sql2estasks * changed branch name * mappings build * updated docs * updated docs * updated docs * added docstrings * pruned deprecated schema transformations * updated fos fieldname on arxlive * unified data set schema transformations * restructured directory * refactored references to schema_transformation * refactored references to schema_transformation * slimmed down transformations, and included entity_type * pruned ontology * tidied schemas * consistency tests * reverted unrelated json file * added dynamic strict to settings * removed index.json in favour of a single defaults file * harmonised name fieldsofstudy across arxiv * using soft alias until a future PR to minimise changes * added novelty back in * sorted json * sorted json * sorted json * changed schema_transformor to use new simpler mapping * removed to/from keys * new null syntax mapping implemented * cleaned and sorted json * adding temporary eurito-dev index to avoid conflating es7 compatibility issues * adding temporary eurito-dev index to avoid conflating es7 compatibility issues * testing es7 on cordis only * testing es7 on cordis only * testing es7 on cordis only * changes to make cordis es7 run * eurito-dev iteration * compatibility issues between arxlive and eurito arxiv * sorted json * pycountry change no longer assumes not null country * needed to split pathstub args * removed redundant es mappings * empty gtr transformation * [267] Pool ES mappings across datasets (#280) * changed branch name * mappings build * updated docs * updated docs * updated docs * added docstrings * added dynamic strict to settings * removed index.json in favour of a single defaults file * using soft alias until a future PR to minimise changes * cleaned and sorted json * [267] Tidy & slim schema transformations (#281) * pruned deprecated schema transformations * updated fos fieldname on arxlive * unified data set schema transformations * restructured directory * refactored references to schema_transformation * refactored references to schema_transformation * slimmed down transformations, and included entity_type * pruned ontology * tidied schemas * consistency tests * reverted unrelated json file * harmonised name fieldsofstudy across arxiv * added novelty back in * sorted json * sorted json * sorted json Co-authored-by: Joel Klinger <[email protected]> Co-authored-by: Joel Klinger <[email protected]> * patched out es config setup from tests * removed redundant tests * fixed json formatting * fixed bad table name (NB table was empty anyway) * fixed bad table name (NB table was empty anyway) * gtr ontology * none included for testing * added schema transformation * picked up bug in test * gtr ontology is self consistent * added gtr mapping * added gtr to config * fixed merge conflicts * fixed merge conflicts * changed json field names * instiutes are now analyzed and text * sorted and cleaned json * added geopoint * fixed bad json * fixed bad json Co-authored-by: Joel Klinger <[email protected]>

* make sure conf dir is empty * simplified es config * added orm es config reader * modified setup_es to pick up new es config * swapped es_mode for boolean * aliases now consistent with config * aliases now automatically located * added endpoint field to estasks * added endpoint field to sql2estasks * changed branch name * mappings build * updated docs * updated docs * updated docs * added docstrings * pruned deprecated schema transformations * updated fos fieldname on arxlive * unified data set schema transformations * restructured directory * refactored references to schema_transformation * refactored references to schema_transformation * slimmed down transformations, and included entity_type * pruned ontology * tidied schemas * consistency tests * reverted unrelated json file * added dynamic strict to settings * removed index.json in favour of a single defaults file * harmonised name fieldsofstudy across arxiv * using soft alias until a future PR to minimise changes * added novelty back in * sorted json * sorted json * sorted json * changed schema_transformor to use new simpler mapping * removed to/from keys * new null syntax mapping implemented * cleaned and sorted json * adding temporary eurito-dev index to avoid conflating es7 compatibility issues * adding temporary eurito-dev index to avoid conflating es7 compatibility issues * testing es7 on cordis only * testing es7 on cordis only * testing es7 on cordis only * changes to make cordis es7 run * eurito-dev iteration * compatibility issues between arxlive and eurito arxiv * sorted json * pycountry change no longer assumes not null country * needed to split pathstub args * removed redundant es mappings * empty gtr transformation * [267] Pool ES mappings across datasets (#280) * changed branch name * mappings build * updated docs * updated docs * updated docs * added docstrings * added dynamic strict to settings * removed index.json in favour of a single defaults file * using soft alias until a future PR to minimise changes * cleaned and sorted json * [267] Tidy & slim schema transformations (#281) * pruned deprecated schema transformations * updated fos fieldname on arxlive * unified data set schema transformations * restructured directory * refactored references to schema_transformation * refactored references to schema_transformation * slimmed down transformations, and included entity_type * pruned ontology * tidied schemas * consistency tests * reverted unrelated json file * harmonised name fieldsofstudy across arxiv * added novelty back in * sorted json * sorted json * sorted json Co-authored-by: Joel Klinger <[email protected]> Co-authored-by: Joel Klinger <[email protected]> * patched out es config setup from tests * removed redundant tests * fixed json formatting * fixed bad table name (NB table was empty anyway) * fixed bad table name (NB table was empty anyway) * gtr ontology * none included for testing * added schema transformation * picked up bug in test * gtr ontology is self consistent * added gtr mapping * added gtr to config * fixed merge conflicts * fixed merge conflicts * changed json field names * instiutes are now analyzed and text * sorted and cleaned json * added gtr batchable * empty test commit * couple of tests * tidied json * added schema module to reqs, finished tests * set up root task * moved to es7 image * removed standard token filter, as it is deprecated in es6.5 then removed in es7 * removed start/end dates since theyre empty * misalignment between batchable keys and field names * fixed mapping and removed outcomes due to mapping explosion * removed seconds from fund date fields * tidied json * added none value edgecase to str truncation * Update elasticsearchplus.py Co-authored-by: Joel Klinger <[email protected]>

* make sure conf dir is empty * simplified es config * added orm es config reader * modified setup_es to pick up new es config * swapped es_mode for boolean * aliases now consistent with config * aliases now automatically located * added endpoint field to estasks * added endpoint field to sql2estasks * [267] Pool ES mappings across datasets (#280) * changed branch name * mappings build * updated docs * updated docs * updated docs * added docstrings * added dynamic strict to settings * removed index.json in favour of a single defaults file * using soft alias until a future PR to minimise changes * cleaned and sorted json * [267] Tidy & slim schema transformations (#281) * pruned deprecated schema transformations * updated fos fieldname on arxlive * unified data set schema transformations * restructured directory * refactored references to schema_transformation * refactored references to schema_transformation * slimmed down transformations, and included entity_type * pruned ontology * tidied schemas * consistency tests * reverted unrelated json file * harmonised name fieldsofstudy across arxiv * added novelty back in * sorted json * sorted json * sorted json Co-authored-by: Joel Klinger <[email protected]> Co-authored-by: Joel Klinger <[email protected]> * patched out es config setup from tests * removed redundant tests * fixed json formatting * none included for testing * picked up bug in test Co-authored-by: Joel Klinger <[email protected]>

* make sure conf dir is empty * simplified es config * added orm es config reader * modified setup_es to pick up new es config * swapped es_mode for boolean * aliases now consistent with config * aliases now automatically located * added endpoint field to estasks * added endpoint field to sql2estasks * changed branch name * mappings build * updated docs * updated docs * updated docs * added docstrings * pruned deprecated schema transformations * updated fos fieldname on arxlive * unified data set schema transformations * restructured directory * refactored references to schema_transformation * refactored references to schema_transformation * slimmed down transformations, and included entity_type * pruned ontology * tidied schemas * consistency tests * reverted unrelated json file * added dynamic strict to settings * removed index.json in favour of a single defaults file * harmonised name fieldsofstudy across arxiv * using soft alias until a future PR to minimise changes * added novelty back in * sorted json * sorted json * sorted json * changed schema_transformor to use new simpler mapping * removed to/from keys * new null syntax mapping implemented * cleaned and sorted json * adding temporary eurito-dev index to avoid conflating es7 compatibility issues * adding temporary eurito-dev index to avoid conflating es7 compatibility issues * testing es7 on cordis only * testing es7 on cordis only * testing es7 on cordis only * changes to make cordis es7 run * eurito-dev iteration * compatibility issues between arxlive and eurito arxiv * sorted json * pycountry change no longer assumes not null country * needed to split pathstub args * removed redundant es mappings * empty gtr transformation * [267] Pool ES mappings across datasets (#280) * changed branch name * mappings build * updated docs * updated docs * updated docs * added docstrings * added dynamic strict to settings * removed index.json in favour of a single defaults file * using soft alias until a future PR to minimise changes * cleaned and sorted json * [267] Tidy & slim schema transformations (#281) * pruned deprecated schema transformations * updated fos fieldname on arxlive * unified data set schema transformations * restructured directory * refactored references to schema_transformation * refactored references to schema_transformation * slimmed down transformations, and included entity_type * pruned ontology * tidied schemas * consistency tests * reverted unrelated json file * harmonised name fieldsofstudy across arxiv * added novelty back in * sorted json * sorted json * sorted json Co-authored-by: Joel Klinger <[email protected]> Co-authored-by: Joel Klinger <[email protected]> * patched out es config setup from tests * removed redundant tests * fixed json formatting * fixed bad table name (NB table was empty anyway) * fixed bad table name (NB table was empty anyway) * gtr ontology * none included for testing * added schema transformation * picked up bug in test * gtr ontology is self consistent * added gtr mapping * added gtr to config * fixed merge conflicts * fixed merge conflicts * changed json field names * instiutes are now analyzed and text * sorted and cleaned json * added geopoint * fixed bad json * fixed bad json Co-authored-by: Joel Klinger <[email protected]>

* make sure conf dir is empty * simplified es config * added orm es config reader * modified setup_es to pick up new es config * swapped es_mode for boolean * aliases now consistent with config * aliases now automatically located * added endpoint field to estasks * added endpoint field to sql2estasks * changed branch name * mappings build * updated docs * updated docs * updated docs * added docstrings * pruned deprecated schema transformations * updated fos fieldname on arxlive * unified data set schema transformations * restructured directory * refactored references to schema_transformation * refactored references to schema_transformation * slimmed down transformations, and included entity_type * pruned ontology * tidied schemas * consistency tests * reverted unrelated json file * added dynamic strict to settings * removed index.json in favour of a single defaults file * harmonised name fieldsofstudy across arxiv * using soft alias until a future PR to minimise changes * added novelty back in * sorted json * sorted json * sorted json * changed schema_transformor to use new simpler mapping * removed to/from keys * new null syntax mapping implemented * cleaned and sorted json * adding temporary eurito-dev index to avoid conflating es7 compatibility issues * adding temporary eurito-dev index to avoid conflating es7 compatibility issues * testing es7 on cordis only * testing es7 on cordis only * testing es7 on cordis only * changes to make cordis es7 run * eurito-dev iteration * compatibility issues between arxlive and eurito arxiv * sorted json * pycountry change no longer assumes not null country * needed to split pathstub args * removed redundant es mappings * empty gtr transformation * [267] Pool ES mappings across datasets (#280) * changed branch name * mappings build * updated docs * updated docs * updated docs * added docstrings * added dynamic strict to settings * removed index.json in favour of a single defaults file * using soft alias until a future PR to minimise changes * cleaned and sorted json * [267] Tidy & slim schema transformations (#281) * pruned deprecated schema transformations * updated fos fieldname on arxlive * unified data set schema transformations * restructured directory * refactored references to schema_transformation * refactored references to schema_transformation * slimmed down transformations, and included entity_type * pruned ontology * tidied schemas * consistency tests * reverted unrelated json file * harmonised name fieldsofstudy across arxiv * added novelty back in * sorted json * sorted json * sorted json Co-authored-by: Joel Klinger <[email protected]> Co-authored-by: Joel Klinger <[email protected]> * patched out es config setup from tests * removed redundant tests * fixed json formatting * fixed bad table name (NB table was empty anyway) * fixed bad table name (NB table was empty anyway) * gtr ontology * none included for testing * added schema transformation * picked up bug in test * gtr ontology is self consistent * added gtr mapping * added gtr to config * fixed merge conflicts * fixed merge conflicts * changed json field names * instiutes are now analyzed and text * sorted and cleaned json * added gtr batchable * empty test commit * couple of tests * tidied json * added schema module to reqs, finished tests * set up root task * moved to es7 image * removed standard token filter, as it is deprecated in es6.5 then removed in es7 * removed start/end dates since theyre empty * misalignment between batchable keys and field names * fixed mapping and removed outcomes due to mapping explosion * removed seconds from fund date fields * tidied json * added none value edgecase to str truncation * Update elasticsearchplus.py Co-authored-by: Joel Klinger <[email protected]>

pruned deprecated schema transformations

2cb692b

jaklinger mentioned this pull request Jun 3, 2020

[267] Pool ES mappings across datasets #280

Merged

5 tasks

jaklinger changed the title ~~pruned deprecated schema transformations~~ [267] Tidy a slim schema transformations Jun 3, 2020

jaklinger self-assigned this Jun 3, 2020

jaklinger added dataset: Crunchbase crunchbase.com dataset: Meetup meetup.com dataset: arxiv arxiv dataset:CORDIS dataset: gtr GtR dataset: patstat tech: ES labels Jun 3, 2020

jaklinger added this to the Create integrated pipeline for data collections to ES milestone Jun 3, 2020

jaklinger mentioned this pull request Jun 3, 2020

[266] Refactor and simplify ES configuration #275

Merged

Joel Klinger added 5 commits June 4, 2020 14:48

updated fos fieldname on arxlive

ed10697

unified data set schema transformations

7184032

restructured directory

7267b3d

refactored references to schema_transformation

29389bd

refactored references to schema_transformation

bdb92bf

jaklinger changed the title ~~[267] Tidy a slim schema transformations~~ [267] Tidy & slim schema transformations Jun 5, 2020

Joel Klinger added 10 commits June 5, 2020 08:58

slimmed down transformations, and included entity_type

984291a

pruned ontology

309f823

tidied schemas

b933014

Merge branch '267_es_mappings' into 267a_schematrans

832c77f

consistency tests

2c2482b

reverted unrelated json file

9bab34e

Merge branch '267_es_mappings' into 267a_schematrans

0f190d5

rmd old files

08fe6da

harmonised name fieldsofstudy across arxiv

cec4dd0

Merge branch '267_es_mappings' into 267a_schematrans

c38af84

Joel Klinger added 4 commits June 5, 2020 13:51

added novelty back in

b5617fc

sorted json

908cdec

sorted json

cde6765

sorted json

697a7b5

jaklinger mentioned this pull request Jun 5, 2020

[267b] Iterate on development pipelines with new ES config setup #282

Merged

5 tasks

jaklinger requested a review from mindrones June 5, 2020 14:50

jaklinger marked this pull request as ready for review June 5, 2020 14:50

mindrones reviewed Jun 5, 2020

View reviewed changes

Merge branch '267_es_mappings' into 267a_schematrans

3797ab7

mindrones approved these changes Jun 5, 2020

View reviewed changes

jaklinger merged commit 844b2a2 into 267_es_mappings Jun 9, 2020

jaklinger deleted the 267a_schematrans branch June 9, 2020 11:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[267] Tidy & slim schema transformations #281

[267] Tidy & slim schema transformations #281

jaklinger commented Jun 3, 2020 •

edited

Loading

jaklinger commented Jun 5, 2020

mindrones left a comment

mindrones Jun 5, 2020

jaklinger Jun 5, 2020

mindrones left a comment

[267] Tidy & slim schema transformations #281

[267] Tidy & slim schema transformations #281

Conversation

jaklinger commented Jun 3, 2020 • edited Loading

jaklinger commented Jun 5, 2020

mindrones left a comment

Choose a reason for hiding this comment

mindrones Jun 5, 2020

Choose a reason for hiding this comment

jaklinger Jun 5, 2020

Choose a reason for hiding this comment

mindrones left a comment

Choose a reason for hiding this comment

jaklinger commented Jun 3, 2020 •

edited

Loading