Bi-directional references

[novoj]

In certain situations we want to specify relations from the perspective of one entity, but allow them to be used (filtered / ordered by) from the other entity. Example situation:

ParameterValue relates to Parameter entity with cardinality EXACTLY_ONE.

But this relationship can also be seen from the opposite side as

Parameter refers to ParameterValue with cardinality ONE_OR_MORE

There is a 'predecessor' ordering attribute on the relation, which can be set when the ParameterValue entity is inserted, but we want to use it when we fetch related ParameterValues when querying the Parameter entity. So the value of the reference attribute must be synchronised with the value in the ParameterValue reference.

Currently evitaDB only supports unidirectional references and the data and its synchronisation has to be managed by the client application, which is quite problematic and requires additional work. The relational databases have these bidirectional relations "for free".

This would allow certain references to be marked as bi-directional and linked to references specified in different entities. This way the reference will be synchronised by the evitaDB engine on every update. The reference attributes do not have to be shared, but can be. The rules for attribute sharing will be as follows:

If attribute A is defined on bi-directional reference in entity X, attribute A: may not exist in reference of entity X.

• must not exist in reference to entity Y
• if it exists in reference of entity Y, it must share it's type and other settings.

The attribute definition from the other side should be similar to UseGlobalAttributeSchemaMutation.

The attribute modifications can be made from both sides of the reference and will automatically propagate to indexes in the other entity collection. evitaDB will ensure that the relation is consistent and in-sync from both points of view.

This should noticeably help the client logic to manage the relations consistently.

[lukashornych]

A few points:

• from which point of view would the user specify the bi-di. reference? Would it be possible to create the reference from any side, and the other would be created/updated automatically? What if user creates manually both sides?
• I don't quite understand the attribute sharing rules, does it even make sense to share the attributes, when the reference from each side has slightly different meaning?

As far as the evitaLab is concerned, if all needed data are on both sides of the bi-di. reference schema, the evitaLab would just render a reference for both entities. The only think that comes to mind is to render flag bi-directional for such references and theoretically it could somehow maybe make them linkable, so that the user could check the bi-di. reference from the other side. It could be possible quite easily through finding the other reference schema in catalog schema.

[novoj]

Reaction to those points:

• the process will be as follows:
  • ParameterValue will define reference parameterType with attribute order (order of the parameter value inside parameter type) - code example:

@Reference(
        name = REFERENCE_PARAMETER,
        faceted = true,
        indexed = true
)
@Nonnull
PublishedParameterValueToParameter getParameter() throws ContextMissingException;

• ParameterType will define reference parameterValues and marks it as bi-directional reference for ParameterValue#parameterType - code example:

@BiDirectionalReference(
        name = REFERENCE_PARAMETER_VALUE,
        targetRelationName = REFERENCE_PARAMETER
)
@Nonnull
List<PublishedParameterToParameterValue> getParameterValues() throws ContextMissingException;

• this is a good point - in this particular case, the order attribute always represents the order of the parameter value within the parameter type list - seen from both sides with the same meaning (it's the same as an attribute on a relational table between two entity tables in a relational database).

[novoj]

We will also have to introduce something like ReferencePredecessor that would be used as the inversed form of Predecessor used on reference. The Predecessor always contains primary key of the previous entity in relation to target entity. But we have to also define an inversed form of predecessor where it represents an order of referenced entity ids within the reference of the entity.

Hence there is difference between:

1. order of the products within category (one product follows another one) - seen and defined from the product
2. vs. category owning list of products (we still need to have predecessor defined for products) - seen and defined from category

The relation can be defined from both places, but it needs to always represent a precedessor order using product ids to declare the chain. In current state of indexes this is not possible because the indexes of specific entity always index primary keys of this
very same entity type and never the referenced ids.

[lukashornych]

After recent discussion with FE team using evitaDB, there may be use case were we want to fetch entity that has existing referenced entities of some type. On top of that however, we want to fetch count of these referenced entities without actually fetching the entities.
Something like that:

{
  listGroup(
    filterBy: {
      attributeCodeEquals: "news-group"
      referenceProductsHaving: {}
    }
  ) {
    productsCount # generated from reference `products`
  }
}

I think it could be solved also with facet summary, probably like this:

{
  queryGroup(
    filterBy: {
      attributeCodeEquals: "news-group"
      referenceProductsHaving: {}
    }
  ) {
    extraResults {
      facetSummary {
        products {
          count
        }
      }
    }
  }
}

But that's quite cumbersome to use on FE. @novoj do you think it would be valid to support the first approach at the GraphQL API level? We could also reuse the filterBy clause from the reference fields.
On the backend it could be translated to basic referenceContent.

[novoj]

Interesting idea - the problem is not only in the query language but also in the output DTOs. We would have to have a new information in the Entity class that would represent the count of non-fetched references. Now we maintain only a Map index with them.

Note: I also found out that the io.evitadb.store.entity.model.entity.ReferencesStoragePart is stored ineffectively and can be compressed in terms of related ids / attributes (which share same structure for all references of the entity). This should be updated in this issue as well.

[novoj]

And we'd need to also support price constraints nested inside referenceProductsHaving - we should have explicit functional test for this situation.

[novoj]

I'm reconsidering the original proposal and want to simplify it a bit. A new model for ReflectedReferenceSchema will be created. This schema will allow configuration of all the properties the standard ReferenceSchema has, but for each of them it will always allow the NULL option, which means that this property will be inherited from the reference schema it reflects. ReflectedReferenceSchema is kept as a separate map in `EntitySchema', although it still needs a unique name among all references (no matter if reflected or default).

By default, ReflectedReferenceSchema inherits all reference attributes, but allows you to disable some or all of them - these settings will be unique to ReflectedReferenceSchema. ReflectedReferenceSchema will still provide means to define it's own unique reference attributes, visible only from it's direction. If an attempt is made to create an attribute in the original reference that would conflict with an existing reference attribute in one of the reflected reference schemas, a conflict will occur. The same behavior is applied if the reflected schema attempts to create an attribute with the same name as an existing reference attribute in the main reference schema. Even if the attribute is not inherited, the reflected reference schema is not allowed to create an attribute with the same name as the source reference, to avoid confusion.

The ReflectedReferenceSchema can be created before the target reference/entity is created. When the target reference/entity finally appears, all validations will take place at that moment. EntitySchema will maintain "calculated" ReferenceSchemas for each of its ReflectedReferenceSchemas once their target is present in the schema collection. These calculated reference schemas will be recalculated accordingly whenever their target reference schema is updated (i.e., only those properties that are set to inherit the original properties of the target reference schema).

Definition from the Java prospective will look like:

@Reference(
        name = "parameter",
        faceted = true,
        indexed = true
)
@Nonnull
PublishedParameterValueToParameter getParameter() throws ContextMissingException;

Parameter type will define reference parameterValues and marks it as bi-directional reference for ParameterValue#parameter - code example:

@Reference(
       name = "parameterValues",
       entity="parameterValue",
       faceted = false
)
@ReflectsReference(
       ofName = "parameter",
       inheritsAttributes=true,
       inheritsAttributesExcept=["a","b"]
)
@Nonnull
List<PublishedParameterToParameterValue> getParameterValues() throws ContextMissingException;

This setup will allow you to define exactly what to inherit and what not to inherit. But the simplest form of:

@Reference(name = "parameterValues", entity="parameterValue",)
@ReflectsReference(ofName = "parameter")
@Nonnull
List<PublishedParameterToParameterValue> getParameterValues() throws ContextMissingException;

will simply inherit all the properties of the original (reflected) reference except its name. (which the developer will probably always want to redefine.)

The reference can be updated (upserted/removed) from either location (even as a reflected reference), but the data is always updated in the primary data source, i.e., the reflected primary reference. If the reflected reference doesn't use/inherit the mandatory reference attribute, and the new reference is upserted from the perspective of the reflected reference, the upsert will fail unless the default value for the attribute is defined in the primary reference schema.

[novoj]

@lukashornych #260 (comment) comment was moved to separate issue #650

[novoj]

Basic functionality with test implemented. To be done:

• documentation
• reversed predecessors