Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add transducer/reducer APIs for eager RDF processing #104

Open
5 tasks
RickMoynihan opened this issue Jul 13, 2017 · 0 comments
Open
5 tasks

Add transducer/reducer APIs for eager RDF processing #104

RickMoynihan opened this issue Jul 13, 2017 · 0 comments

Comments

@RickMoynihan
Copy link
Member

RickMoynihan commented Jul 13, 2017

This is a bit of a mega ticket which we can break down into separate tickets as and when we need them.

Generally we want to add eager processing capabilities in to grafter that retain some of the composition and streaming benefits of lazy-seqs, but without the GC pressure, and resource life cycle issues.

We already have some undocumented and limited support for transducers/CollReduce, though we may need to revise or add more reducing-functions...

In particular we'd like to:

  • To support CollReduce when reading RDF sources. In particular:
    • When reading triples/quads via statements. I suspect there's some mileage in improving things by making RDFParser extend CollReduce, that we can expose the intention to parse a specific thing as a reified first class entity, and trigger the consumption elsewhere. Also before conducting this work it might be worth considering upgrading to RDF4j Update to RDF4j #95.
    • On SPARQL queries, allowing you to build a query and consume the results with things like (into [] xform query) and have all resources etc cleaned up.
  • To support reducing functions for adding/writing data to RDF destinations
  • Support (into sparql-repo xform quads) (IIRC this might already be supported, but we should check it works in all the contexts we require.
@RickMoynihan RickMoynihan changed the title Add transducer/reducer APIs for eager processing Add transducer/reducer APIs for eager RDF processing Jul 13, 2017
@RickMoynihan RickMoynihan added this to the Improved use of RDF4j milestone Oct 27, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant