III. A searchable registry of parsers

[ml-evs]

A searchable registry of parsers

• Any parser code that implements the above functionality can be added (automatically or otherwise) to a registry of parsers that can be filtered against the metadata schema from the I. This could allow for automated support of a new file type (i.e., filter for parsers that support such a file type, and that provide a container, automatically download and deploy the container and use the expected endpoints to parse the data file).
• Such a registry would provide discoverability and automated validation of resources, accelerating the proliferation of shared or overlapping data models, schemas and semantics. The registry can then be used to chain parsers together as composable blocks, for example, a custom semantic layer could be added to a data model as a converter from a JSON file with well-defined schema, into a JSON-LD file.

[markus1978]

Ideally, the registry would find a parser based on a given file directly. Imagine these search parameters:

• file-name (e.g. ending)
• mime-type
• first n KBq of content

[ml-evs]

Registry should also be flexible to "meta-formats" that are somehow distributed across multiple files/folders, like TOPSPIN's awkward project folders, or even VASP's format of requiring different files for e.g., POSCAR, INCAR, etc. Assuming people upload zip files this would make the "first n bytes" detection trickier (though presumably zip has some decodable metadata in these first bytes anyway?)

[ml-evs]

After the first office hours today we made repos for each of main topics. I would hope that this discussion thread can continue for general stuff, but if we want to comment on specific ideas/code then we can use PRs over at https://github.com/marda-alliance/metadata_extractors_registry (you will see a similar comment on each other thread with the appropriate link).