[daanjo3]

BREAKING CHANGE: This PR changes the current string substitution for the ?dataset keyword. After merging this sub-SELECT blocks will no longer have automatic substitution of the ?dataset keyword.

One of the issues raised in development of the LDE-based LODA pipeline is the need to provide constant bindings from either the dataset description or just purely static.

Changes

This PR aims to resolve this by making two changes:

1. adding an additional injectValuesOuter method that adds another VALUES expression to bind the following predefined dataset bindings: ?dataset, ?datasetPublisher, ?datasetPublisherName, ?datasetLicense, ?datasetLanguage.
2. providing a DatasetBindingsProvider that can be passed as a StageOption to provide user-defined bindings based on the dataset and distribution. It is merged in the same VALUES expression as the dataset bindings.

Consideration / open questions

1. Originally I combined the new "custom" bindings in the main value bindings produced by the selector. However these end up in the innermost subquery. This would not be ideal however as most of the user-defined bindings are likely to end up in construct statements, and would then need to be passed outside the subquery first, making for a very verbose query. ✔️
2. The new dataset bindings logic is in pipeline/src/sparql/datasetBindings.ts, I'm not sure this is entirely the right place. ✔️
3. Currently only a subset of the dataset properties are added as default bindings, perhaps for completeness sake it is best to just add all the properties. => we'll stick to these for now, add more if needed.

A note on additional row generation (cross-products)

Having multiple used bindings, especially of the same type (multiple dataset languages) may result in additional rows being generated (cross-product). This implementation aims to reduce the impact of this as much as possible. Any unused binding will not end up in the query, and where possible will UNIONs be used. Only if bindings are using in WHERE expressions we may generate additional rows.

Testing

This PR has been tested with the LODA pipeline, where hardcoded bindings could successfully be replaced with custom bindings provided via the DatasetBindingsProvider.

fixes #884

[ddeboer]

Thanks, Daan. The AST-based injection and the DatasetBindingsProvider hook look good. You already flagged the cross product in open question 4, and that’s the one thing that has to change before this can merge: the dataset bindings are injected into every query, so the cross product reaches queries that never use them.

The cross product

You’re right that deduplicate cleans up the output. It doesn’t prevent the duplicates, though, and they turn out to cost more than the output suggests. I ran the generated queries against QLever (commit-6fd6171141) on 50000 items, comparing a plain CONSTRUCT { ?s a ?class } WHERE { ?s a ?class } with the same query plus the VALUES block this PR injects:

Languages × publisher names	Triples	Distinct	Size
no bindings (main)	50000	50000	5.6 MB
2 × 2	200000	50000	22 MB
5 × 5	1250000	50000	139 MB

That query mentions no dataset variable, yet every result is joined with each VALUES row and comes back L × N times. deduplicate can’t fix three consequences of that:

• The endpoint still does all the work. It computes, serialises and sends every duplicate; deduplication only happens in the client afterwards.
• LIMIT truncates. With SELECT ?s … LIMIT 10000, 2 × 2 returns 2500 distinct items and 5 × 5 returns 400. Selectors and VoID queries such as class-partition.rq have an outer LIMIT, so they silently lose results.
• VoID counts inflate. The VoID readers don’t set deduplicate, and partitionMerge sums measure quads with the same subject and predicate, so a 2 × 2 dataset reports 4× the real counts.

Suggested fix

1. Inject only the variables a query references. The AST shows which variables occur in the WHERE clause and the template. A query that uses none of them then runs unchanged, so VoID, the sampler and existing user queries keep their current behaviour.

2. Combine independent multi-valued bindings with UNION instead of a cross product. Single-valued bindings stay joined, so they are visible in every branch:

   VALUES (?dataset ?datasetPublisher ?datasetLicense) { (<…> <…> <…>) }
   { VALUES ?datasetLanguage { … } }
   UNION
   { VALUES ?datasetPublisherName { … } }

   That gives L + N rows instead of L × N, and nothing is lost, because template triples with an unbound variable are skipped. On a template that uses both variables, at 5 × 5: 3750000 triples (322 MB, 2.6 s) with the cross product, 1000000 (92 MB, 0.7 s) with UNION. A caller who really needs language–name pairs in one solution can return them from a DatasetBindingsProvider.

3. Compute the merged rows once in Stage.run and pass them to the selector and the readers under one option name.

Even with UNION, a query that joins the dataset bindings with its item pattern repeats each item L + N times. That’s inherent, but worth a line in the docs: put dataset-level triples in their own UNION branch, away from the item pattern.

Breaking change

?dataset used to be replaced as text everywhere in the query, which the docs explicitly promised. As an outer VALUES, it is no longer visible inside a sub-SELECT, so { SELECT ?s { ?s dcterms:isPartOf ?dataset } } now matches subjects of every dataset. The same shift changes the meaning of the excludeResources example in pipeline-shacl-sampler, where ?dataset is now bound to the registry IRI. That needs a ! in the PR title (feat(pipeline)!:), otherwise it ships as a patch.

Smaller things

See the line comments: a port clash, dropped literal datatypes, ?datasetLanguage as a literal instead of an IRI, provider-row edge cases in values.ts, and assertSafeIri on publisher and license.

Your other open questions

1. Outer placement for dataset bindings is right: they’re constant per dataset, and a sub-SELECT can’t see outer bindings anyway.
2. pipeline/src/sparql/datasetBindings.ts is fine.
3. I wouldn’t add all dataset properties yet. With referenced-only injection an extra binding costs nothing, so we can add them as callers need them.

[ddeboer]

Line comments for the review above.

[ddeboer]

This injects the dataset bindings into every query, including the ones that use none of the variables. Suggest injecting only the variables the query references.

It also changes ?dataset: the old string substitution reached into sub-SELECTs, an outer VALUES doesn’t, so { SELECT ?s { ?s dcterms:isPartOf ?dataset } } now matches every dataset’s subjects.

[daanjo3]

The code has been updated to only bind the variables that are in fact used. Also the documentation now describes the fact that sub-SELECT blocks don't automatically replace ?dataset and the PR is marked as a breaking change.

[ddeboer]

With this check (and the one for the license), one odd publisher or license IRI makes every stage of the dataset throw, even stages that never use the variable. The values go through the AST generator, not string substitution, so the guard isn’t needed here.

[daanjo3]

I've moved this check to the termToAstTerm method, which should make sure that we only throw for values that are being used.

[ddeboer]

A provider row containing dataset silently overwrites the dataset IRI. Should a clash with a predefined variable throw instead?

[daanjo3]

I've added documentation for this behavior to the pipeline reference docs. I think it is a matter of preference as arguments can be made for either approach. In this case my reasoning is that with this current method we don't reserve keywords and provide flexibility to the implementer. Through informing them of this behavior they should be able to make it work however they see fit. But I can also definitely see the value in reserving a keyword and throwing on clashes.

[daanjo3]

@ddeboer I've made the improvements as suggested and the result is much better. My tests show that with this current implementation we no longer face an explosion of triples being generated.