🎨 Confusing name: Curate.add_validated_from

[falexwolf]

The problem with the below is that values in a public ontology are not validated in the LaminDB definition. Hence, the name of the method is highly confusing.

How shall this method be called?

[falexwolf]

This here is still a quite basic thing that we should get resolved, shouldn't we? I think you wanted to work on it at some point @Zethson?

[Zethson]

I'll think about it next week but if we change the default of Curator to also validate against records without a Source record even if a Source is passed, the comment above would certainly be false and the name fine.

[falexwolf]

We need a big clear simple box that defines what “validated” means. This is urgent as it’s so basic.

In my mind “validated” == “exists in registry or an underlying ontology”.

If this is what “validated” is, curator.add_validated_from should not be user-facing but records that are not yet in the ontology source should be added to registry upon the first call of .validate() that touches them.

What’s important about this “auto-import-from-the-ontology-source” is that the definition gets easier in practice: "validated" == "record exists in registry". This means the whole complexity of the ontology source is no longer important as soon as the record is imported.

It’s important that we have an easy practical definition of “validated” that’s free of “validating against source” and “validating against another instance” both of which are complicated things.

[Zethson]

In my mind “validated” == “contained in registry or an underlying ontology source”.

Yes that's also where @sunnyosun and me ended up. However, I think the current behavior when sources is passed to Curate would break this assumption because it would only validate against the specific ontology source and NOT the remainder of the registry (basically values that have no source). This needs to be changed for consistency.

+1 the rest of what you wrote.

[falexwolf]

Can we make this a priority then so that it's in a LaminDB release ASAP? Asking both of you @Zethson @sunnyosun

[Zethson]

While I could figure it out, I think @sunnyosun knows this part better and is probably substantially faster? @sunnyosun I'm happy to look into it if you want me to. Just let me know.

[sunnyosun]

What about we run add_validated_from("all") under the hood of .validate()? Essentially all validated records will be saved before running validation. This way we make add_validated_from no longer user facing.

[Zethson]

That's how I understood @falexwolf proposal, yes.

[falexwolf]

What about we run add_validated_from("all") under the hood of .validate()? Essentially all validated records will be saved before running validation. This way we make add_validated_from no longer user facing.

Yes, that's what I meant. I'd also make it private then (with deprecation time).

[sunnyosun]

What about we run add_validated_from("all") under the hood of .validate()? Essentially all validated records will be saved before running validation. This way we make add_validated_from no longer user facing.

Yes, that's what I meant. I'd also make it private then (with deprecation time).

I think we can just remove it, because users will not be asked call this method anymore.

[falexwolf]

Great! That's going to simplify things!

[falexwolf]

I'm looking forward to this getting simpler:

[falexwolf]

And this: