Support both object inlining and referencing

[Silvanoc]

According to the documentation, you can currently compose objects (also known as aggregates) from other objects by either referencing the constituent objects or embedding (inlining) them.

Why is this design so exclusive? I want the flexibility to specify that constituents can be either inlined or referenced, allowing for conversion between the two formats.

Here's an example to illustrate my goal:

• An IoT Gateway within an isolated network can discover and model connected devices, gathering information about modules, manufacturers, etc.
• The IoT Gateway lacks the resources and connectivity to separate an aggregate into its constituent parts, so it provides all information with constituents inlined.
• These models are then sent to a cloud service, which can and should split the aggregate into its constituents. This is useful for deduplication and canonicalization.
• Depending on the needs of the consumer, the models can be provided as inlined aggregates or as aggregates with references to their constituents.

In the end, if the constituent has the right range, what is the difference between inlining and referencing?

[Silvanoc]

I know I already asked this question somewhere, but I could not find it anymore. I suspect that it is a comment hidden in some issue or pull-request. IMO this is a design decision that is worth a discussion by itself. I hope it is fine for you.

[cmungall]

Why is this design so exclusive

Simply because by making the structure more predictable, we make it easier for consumers of the data. This may not make much sense from a relational model or graph database perspective, but for tree-based formalisms (JSON, YAML, Pydantic objects, ...) it's much harder for me as a user of data to defensively code against having the data be in two different possible shapes. The current design also makes the mapping to json-schema more predictable (see existing discussions about supporting Referential Integrity in json-schema)

Some of this could be obviated by using some alternative to Pydantic that takes care of automated de-referencing. In fact this is possible using a LinkML ObjectIndex. See

• Should automatic dereferencing of non-inlined objects be supported? #2176

for discussion. But this doesn't help if a consumer of the JSON/YAML I produce isn't using that.

I want the flexibility to specify that constituents can be either inlined or referenced, allowing for conversion between the two formats

This is totally reasonable. LinkML is designed for a variety of use cases. Perhaps in some scenarios the flexibility is a major win, and there are fewer concerns with making the shape of the data less predictable - it sounds like this is true for you.

In slack you provide two solutions:

• linkml-map transformation both ways
• support for "referenced OR inlined" objects, since now it is an "referenced XOR inlined" objects (this would be my favorite)

On reflection, I agree and linkml-map would be overkill for this (but there may be other scenarios that complicate the picture - e.g. having the id be optional).

Remaining questions:

• should the representation be coerced to one form, or should reads/writes be structure-preserving?
• whether to bundle the "XOR" support with general support for different object serialization forms (see Proposed metamodel enhancement: use enum to determine the different collection serialization types #1251)

[Silvanoc]

I think I understand your point better now. And I think that if I scope my use-case better, the realization might become easier:

1. My main requirement is having the possibility of validating the "inlined objects" and converting them to "referenced objects".
2. I do not need support for both inlined and referenced objects as equivalent. Inlined objects would be like an accepted exception on input where referenced objects are expected.
3. I could accept providing dummy IDs or placeholders in the inlined objects, that get replaced when converting them to referenced objects. But it would be even better not needing an ID, which gets created and assigned on conversion.

I could think of following approach:

1. New flag on linkml-validate to handle inlined where referenced was expected (something like --accept-inlined-instead-of-referenced, but shorter).
2. New flag on linkml-convert to convert inlined into referenced wherever needed (something like --disassemble-aggregate), without failing.
3. Optionally we could create a new attribute on SlotDefinition to specify that the range of the defined slot might exceptionally appear inlined and should get referenced (something like accept_inlined: boolean, defaulting to false). This attribute could help making linkml-validate and linkml-convert even more precise, accepting the exception only on marked slots.

I would not expect equivalence between both inlined and referenced in the models (which is what makes the whole thing difficult) throughout the whole life of an object, only an exceptional way to provide an object meant to be converted (into using referenced objects, wherever specified) for further handling.

Getting JSON-Schemas that can cope with it for the validation is pretty easy using anyOf.

I do not know Pydantic good enough, but I would expect that a try-error process could handle it. First try the validation without any exception, wherever Pydantic fails expecting a referenced and finding an inlined, replace with inlined and re-validate. I do not know if it would be possible to make Pydantic coerce inlined to referenced to avoid the whole try-error.

[cmungall]

I'm not sure the Pydantic automatic handling would be so easy. You might be able to auto-coerce to an ID but if you need to make sure that the referenced object is preserved elsewhere it becomes more difficult..