II. A common API specification for executing parser code

[ml-evs]

A common API specification for executing parser code, and associated tooling.

• Parsers could be run natively (in the language they were written in) on local files via the creation of a language-specific harness to which the parsers contribute plugins. This harness will execute the parser in such a way to maintain the link with its reported schema, as above.
• Parsers could be packaged into containers (Docker or otherwise) that provide reproducible environments for their execution. Another harness for executing the containers from various languages, or via an HTTP API, will be investigated. Such containers can then be deployed as part of larger infrastructure, or used in a serverless fashion. This approach is well-suited for asynchronous message queues and streaming of data.

[markus1978]

Could also imagine a web-service directly on top of a parser registry. Convenient drag-and-drop no-install required. Obviously limited in file-size and privacy.

[ml-evs]

Yep! I was hoping to entice the FuncX/Globus people on this (https://funcx.org/), though no bites so far... perhaps another large consortium with some spare compute would be interested in hosting a docker registry + some small compute for us? 😅

I think having a WASM backend available for some parsers would be great too (and we could provide this awkward packaging step for compliant parsers --- e.g. take compatible Python/CPP/JS package and do the build magic in our registry CI), to alleviate the security concerns of running something like this. There could also be some auditing process for the containers that this service executes to ensure they have no network access themselves to exfiltrate the data.

[markus1978]

This could be layered: web-service(docker(cli(python))). I would also like to include the lowest layer (if we can agree on a language, e.g. Python). Not sure, but I guess (reference) implementation would also be in the scope of this WG?

[ml-evs]

This is somewhat what I was imagining, most likely with a Python reference implementation that can call native parsers, and perhaps transparently operate on docker containers from within Python (assuming there are nicer ways of doing this than system calls to docker run...) for non-native ones?

[PeterKraus]

@ml-evs I think the question from the Kick-off meeting on whether to start with WP1 or WP2 has not been really addressed. I'd like to know your opinion, and of anybody that would like to contribute code and will have time to do so before January. I suspect whichever point we start with, we'd like to reach "convergence" (one iteration) on or soon after January anyway, but I'd like to decide what end to focus on first (as in, next week on the Jitsi session)

Maybe we can do it this way:

• 👍 for draft schema (WP1) first
• 👎 for draft API (WP2) first

[ml-evs]

I have helpfully reacted with both 👍 and 👎!

I guess my real answer is 👍, but in the process of testing 👍 we will all be somehow independently writing our own idea of what 👎 would look like (which is maybe actually a helpful way to proceed). While the whole multilingual execution engine aspect of WP2 is a real stretch (within any timeframe!), having a defined mechanism for reporting "I have a function that if you call with a file as input it will give you something with this schema" seems unavoidable? Somehow then WP1 is defining an actual API schema at the same time.

NB. It might be worth us coming up with better terminology to describe WP2 as API conflates so many things (web APIs, registry API, etc.). "Common entrypoint standard" perhaps? I'm sure some of these things will come out in the wash as we start hacking at them.

[ml-evs]

Tagging @kroenlein as this is related to his comments in the meeting, though feel free to ignore it Ken! 🙃

[kroenlein]

I think we should take a hard look at OpenAPI as prior art here for a fairly standard scheme for defining what goes into and comes out of a service. Obviously, this glosses over the definition component of what the terms in I/O actually mean.

[ml-evs]

Here's an example OpenAPI schema that we've generated using FastAPI/pydantic for OPTIMADE: https://schemas.optimade.org/openapi/v1.1/optimade.json. You can also paste the URL into https://validator.swagger.io/ to get a nicer UI for exploring (scroll to the bottom see the actual core data models).

[ml-evs]

After the first office hours today we made repos for each of main topics. I would hope that this discussion thread can continue for general stuff, but if we want to comment on specific ideas/code then we can use PRs over at https://github.com/marda-alliance/metadata_extractors_api (you will see a similar comment on each other thread with the appropriate link).