Auto-expand transformation #293

HeikoTheissen · 2023-11-10T12:00:41Z

uhlmannm · 2023-11-10T17:59:48Z

vocabularies/Analytics.json

+            "@Core.OptionalParameter": { "DefaultValue": "false" }
+          }
+        ],
+        "$ReturnType": { "$Collection": true, "$Type": "Edm.EntityType" }


I guess the EntityType is not enough. We will need some of the information that we also get from a hierarchical query. Most importantly, the LimitedDescendantCount but also the DrillState and the DistanceFromRoot.

So we need an instance annotation whose type contains a subset of the properties of Hierarchy.RecursiveHierarchyType. Which subset exactly?

@ThomasChadzelek I had found the LimitedDescendantCount, the DrillState and the DistanceFromRoot. Is that it? Or already too much?

Is it sufficient if these instance annotations are returned by the auto-expand function? If one node is expanded, they must be computed by the OData model itself.

I think yes. In the other cases, the model is creating requests to manually expand the visual groups. The backend would not even be aware that we are creating a visual grouping on the screen.

@ThomasChadzelek I had found the LimitedDescendantCount, the DrillState and the DistanceFromRoot. Is that it? Or already too much?

What about LimitedRank? Our new favorite ;-)

DistanceFromRoot might be redundant because the hierarchy is leveled, not recursive?

vocabularies/Analytics.json

It is needed if `AggregatedValuesLeafFilter` is specified or properties from `Aggregation` occur in `SiblingOrder`. Especially the latter is typical.

vocabularies/Hierarchy.xml

ThomasChadzelek · 2024-02-02T11:02:50Z

vocabularies/Analytics.xml

+          <Annotation Term="Core.Description" String="Sort specification to apply to all direct descendants of a given entry in the resulting leveled hierarchy" />
+        </Parameter>
+        <Parameter Name="BeforeAggregationFilter" Type="Edm.String" Nullable="false">
+          <Annotation Term="Core.Description" String="Expression valid for `filter` transformation to restrict the input set before any further procressing" />


Can't you achieve the same thing with a filter() transformation beforehand, in the same $apply? For TopLevels, we do likewise with orderby() or search() or filter().

Yes, but then the backend would have to take all these transformations together and process them as one "auto-expand" transformation. (At least with the Analytical Engine, the filter is part of the auto-expand operation.)

If the backend agrees, we can change this.

If the backend disagrees, do we need support for search as well? Or is that too fuzzy for analytics?

vocabularies/Analytics.xml

HeikoTheissen · 2024-04-26T13:17:30Z

Open issue: Whether and how to instruct the transformation to create one row for an expanded group header above its descendants and another row for the subtotals below its descendants.

uhlmannm · 2024-04-26T15:52:40Z

vocabularies/Analytics.json

+        "$IsBound": true,
+        "@Common.Experimental": true,
+        "@Core.Description": "`$apply` transformation that expands an unnamed leveled hierarchy with custom aggregation of certain properties",
+        "@Core.LongDescription": "Example transformation sequence:\n```\n$apply=filter(Industry eq 'IT')\n/groupby((Country,CountryName,Region,RegionName),\n         filter($these/aggregate(Amount) gt 0 and\n                $these/aggregate(Currency) ne null))\n/concat(\n  Analytics.AutoExpand(\n    Levels=[{\"P\":[\"Country\",\"CountryName\"]},\n            {\"P\":[\"Region\",\"RegionName\"]}],\n    LeafLevel=[\"Segment\",\"Industry\"],\n    Aggregation=[\"Amount\",\"Currency\"],\n    SiblingOrder=[{\"Property\":\"Amount\",\n                   \"Order\":\"desc\"}],\n    ExpandLevels=[{\"Entry\":[\"US\",\"USA\"],\"Levels\":0},\n                  {\"Entry\":[\"DE\",\"Germany\",\"BW\",\"Baden-Württemberg\"],\"Levels\":2}])\n    /concat(aggregate($count) as ResultEntriesCount,\n            skip(20)/top(10)),\n  aggregate(Amount,Currency))\n```\n",


In UI5 we start with the leaves count that is missing in the example. It would be a term groupby((Country,CountryName,Region,RegionName))/aggregate($count as LeavesCount) in the outer concat followed by the grand total term aggregate(Amount,Currency) that is currently last in the example. The third term in the outer concat would be Analytics.AutoExpand(...)/concat(aggregate($count) as ResultEntriesCount,skip(20)/top(10)).

uhlmannm · 2024-04-26T15:57:45Z

vocabularies/Analytics.json

+          {
+            "$Name": "LeafLevel",
+            "$Collection": true,
+            "@Core.Description": "A possibly empty set of property names that constitute, together with the property names from `Levels`, the leaf level of the leveled hierarchy",


A possibly empty set of property names

I wonder how that would work. If LeafLevel is empty then the last level in Levels is not expanded, cannot be expanded and could as well be written into LeafLevel?

Isn't LeafLevel simply the last entry in Levels, then?

There can be several properties used for grouping on the leaf level that are not used for visual grouping / group headers.
The interesting part is that we have three things:

properties used for visual grouping

properties used additionally for grouping on the leaf level

the number of visual groups initially expanded.

I think we had decided somewhen in the past that 3. is not necessary for the AutoExpand function as one can simply as collapsed visual groups to 2. But I wonder whether that decision is still good when introducing ExpandLevels. Should the backend then not know which visual groups are introduced, how grouping is done on leaf level and how deep to expand initially?

The leaf level is just the deepest grouping level, therefore we only need the parameter Levels, not LeafLevel.

In the current definition, nodes not mentioned in ExpandLevels are completely expanded. If you want the default to be "expand N levels", we need an additional parameter for that N.

Yes, we can do it like that. But the question is how to denote the leaf level. Is it always the last of the Levels? And hence this last level is not and cannot be expanded?

Should it be stated explicitly in the Levels description?

If you look at the FE examples, then on the leaf level there are individual (unaggregated) records. If that is the target state, one could also argue that the last level can still be expanded and below this last level are the unaggregated records.

Added

odata-vocabularies/vocabularies/Analytics.xml

Line 224 in 8c7b80b

entries on the finest-grained level cannot be expanded further.

vocabularies/Analytics.xml

auto-expand transformation

4750db8

HeikoTheissen requested a review from GeraldKrause November 10, 2023 12:00

Avoid EnumType

53368e3

uhlmannm reviewed Nov 10, 2023

View reviewed changes

vocabularies/Analytics.json Outdated Show resolved Hide resolved

uhlmannm requested a review from ThomasChadzelek November 13, 2023 16:57

HeikoTheissen added 4 commits November 14, 2023 08:03

Instance annotations for auto-expanded result sets

64e5ae7

Reprased definitions

b5a6f34

SortOrder -> SortOrderType

9238702

Avoid <dt>

34b5a64

uhlmannm reviewed Nov 14, 2023

View reviewed changes

vocabularies/Analytics.json Show resolved Hide resolved

HeikoTheissen added 3 commits November 15, 2023 09:59

After alignment with @uhlmannm

443f99c

rephrasing

2fb8ff2

Make LeafLevel mandatory

f5dea9f

It is needed if `AggregatedValuesLeafFilter` is specified or properties from `Aggregation` occur in `SiblingOrder`. Especially the latter is typical.

HeikoTheissen marked this pull request as ready for review January 19, 2024 15:33

HeikoTheissen requested a review from uhlmannm February 2, 2024 10:03

Don't call out instance annotation

41b289a

ThomasChadzelek reviewed Feb 2, 2024

View reviewed changes

vocabularies/Hierarchy.xml Show resolved Hide resolved

ThomasChadzelek reviewed Feb 2, 2024

View reviewed changes

vocabularies/Analytics.xml Outdated Show resolved Hide resolved

HeikoTheissen added 5 commits April 4, 2024 17:54

Merge remote-tracking branch 'origin/main' into auto-expand

2df14e4

Allowed values for DrillState

defafdf

Merge remote-tracking branch 'origin/main' into auto-expand

8584dfe

auto-refreshed

f27abe8

@uhlmannm's suggestion

3ca799d

HeikoTheissen commented Apr 26, 2024

View reviewed changes

vocabularies/Analytics.xml Show resolved Hide resolved

HeikoTheissen added 3 commits April 26, 2024 13:38

Simplifications agreed in V4 table meeting

92f1abd

Example request

bb78f7d

Rephrased

4181573

Typos

b227e7f

uhlmannm reviewed Apr 26, 2024

View reviewed changes

HeikoTheissen added 9 commits April 29, 2024 08:06

@uhlmannm's comments

e390b0c

explained example

b9ecf4f

expand N levels

b587137

renamed type

ae774a6

typo

3cb5082

more links

c9bbadf

corrected LeavesCount, ResultEntriesCount

3e6379a

example formatting

f98c999

save lines

64f96eb

HeikoTheissen commented Apr 29, 2024

View reviewed changes

vocabularies/Analytics.xml Outdated Show resolved Hide resolved

HeikoTheissen added 4 commits April 29, 2024 15:44

round parentheses for in

bc35559

More elegance

9f24039

reword: deeper -> finer-grained

8c7b80b

Sort order like Common.SortOrderType

4dfd1c6

HeikoTheissen commented May 24, 2024

View reviewed changes

vocabularies/Analytics.xml Outdated Show resolved Hide resolved

HeikoTheissen added 3 commits May 24, 2024 11:51

As agreed in meeting on 2024-05-24

5036eba

Explain D vs. A

192cc17

Merge remote-tracking branch 'origin/main' into auto-expand

fbe94b9

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Auto-expand transformation #293

Auto-expand transformation #293

HeikoTheissen commented Nov 10, 2023 •

edited

Loading

uhlmannm Nov 10, 2023 •

edited

Loading

HeikoTheissen Nov 13, 2023

uhlmannm Nov 13, 2023 •

edited

Loading

HeikoTheissen Nov 14, 2023

uhlmannm Nov 14, 2023

ThomasChadzelek Feb 2, 2024

ThomasChadzelek Feb 2, 2024

HeikoTheissen Feb 2, 2024

ThomasChadzelek Feb 2, 2024

HeikoTheissen commented Apr 26, 2024 •

edited

Loading

uhlmannm Apr 26, 2024

uhlmannm Apr 26, 2024

HeikoTheissen Apr 29, 2024

uhlmannm Apr 29, 2024 •

edited

Loading

HeikoTheissen Apr 29, 2024

uhlmannm Apr 29, 2024

HeikoTheissen Apr 29, 2024

uhlmannm Apr 29, 2024

uhlmannm Apr 29, 2024

HeikoTheissen Apr 29, 2024 •

edited

Loading

Auto-expand transformation #293

Are you sure you want to change the base?

Auto-expand transformation #293

Conversation

HeikoTheissen commented Nov 10, 2023 • edited Loading

uhlmannm Nov 10, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

uhlmannm Nov 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HeikoTheissen commented Apr 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

uhlmannm Apr 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HeikoTheissen Apr 29, 2024 • edited Loading

Choose a reason for hiding this comment

HeikoTheissen commented Nov 10, 2023 •

edited

Loading

uhlmannm Nov 10, 2023 •

edited

Loading

uhlmannm Nov 13, 2023 •

edited

Loading

HeikoTheissen commented Apr 26, 2024 •

edited

Loading

uhlmannm Apr 29, 2024 •

edited

Loading

HeikoTheissen Apr 29, 2024 •

edited

Loading