speed up field indexer by archetype ecs

[linonetwo]

Entity component system with archetype Can filter entity with very fast speed.
And those components are similar to tiddler fields, so maybe we can apply this (only to the indexer is sufficient) . This may also improve filters like tag filter.

I learned about this time ago. Just happen to think about this after dinner, may add more details later.

[pmario]

I did have a look, when the FieldIndexer actually is used. As far as I can see, it is only used once by the filters/field.js with wiki.eachXX.byField() function.

IMO we should start to use it instead of making it faster.

So basically if you hardcode eg: [code-body[yes]] it is used. But that's not a real world usecase, which needs to be fast.

IMO real world would be eg: [has<field-name>get<field-name>], where field-name=code-body. There it's not used at all.

[linonetwo]

Oh, I thought many filter operators was using indexer, so I come up with this.

So maybe let them using the indexer is another low-hanging fruit.

[pmario]

... So maybe let them using the indexer is another low-hanging fruit.

Yes.
As I wrote below. IMO has<field>,<value> may be a candidate.
It would be interesting to implement it and make some performance measurements, if it would be worth it.

[pmario]

eg: My tocP does this to "find all tiddlers, where the parent-field has the value of the __tag__-variable"

<$list filter="""[has<__field__>!has[draft.of]] -[<__tag__>] -[subfilter<__exclude__>] :filter[get<__field__>match<__tag__>] +[subfilter<__sort__>]""">

IMO it should be something similar to

<$list filter="""[has<__field__>,<__tag__>!has[draft.of]] -[<__tag__>] -[subfilter<__exclude__>] +[subfilter<__sort__>]""">

Which imo would significantly speed up the whole thing, because the initial filter-run [has<__field__>,<__tag__>!has[draft.of]] will be the result already.

@Jermolene ... Would it make sense to extend the has-operator with a second parameter, which would be the value. So IMO has could make use of the FieldIndexers. eg: [has<fieldName>,<value>]

[pmario]

There may be a other Filter Operators that could be improved. We would have to have a look, which one of them also deals with fields as their primary parameter.

I've long lost the overview, due to the amount (166) of operators we have atm, that do similar things. :/

[linonetwo]

I read the code of https://github.com/hmans/miniplex (less than 1000 loc)

I find ECS is like:

1. filter expressions: Query in ECS
2. add a new filter expressions: ECS will add a new bucket as cache, storing result of query. So multiple same query don't need to re-execute multiple times
3. run filter expression: instead of execute some JS on querying, it directly read data from cache.
4. change tiddler: in ECS, it will update each query bucket. Like update every filter expression.

So TW's filter expression is already simillar to ECS. Only difference is ECS will cache result of each query. This works when there are many same query. But we probably don't have many same filter expressions, so caching is useless.

So I'm moving my attention to:

#8618 (reply in thread)