stats1 with two dimensional tabular output

[nikbucher]

Hi

I try to get an two dimensional tabular output from stats1 with the fields in the x-axis and the accumulators in the y-axis. I think this should be very likely possible, but probably I missed something in the docs or in the samples.

What do I have to put at "..." to get the following output?

mlr --icsv --opprint stats1 -f usedMemory,committedMemory,maxMemory -a min,mean,max,median,p90 ... some/data.csv

        usedMemory  committedMemory maxMemory
min     184850984   607125504       2863661056
mean    577374666   2855677277      3095064524
max     966575984   3100639232      3791650816
median  583792440   3083862016      3083862016
p90     902036688   3099066368      3099066368

Thank you in advance - Niklaus

[aborruso]

@nikbucher starting from

a,b,c
1,1.2,3
2,3,4

and running mlr --icsv --opprint stats1 -f a,b -a min,mean,max,median,p90 input.csv, the standard output has "wide" shape.
And there is not another kind of output.

+-------+----------+-------+----------+-------+----------+----------+----------+----------+-------+
| a_min | a_mean   | a_max | a_median | a_p90 | b_min    | b_mean   | b_max    | b_median | b_p90 |
+-------+----------+-------+----------+-------+----------+----------+----------+----------+-------+
| 1     | 1.500000 | 2     | 2        | 2     | 1.200000 | 2.100000 | 3.000000 | 3        | 3     |
+-------+----------+-------+----------+-------+----------+----------+----------+----------+-------+

You could use other miller verbs to reshape it.

Running

mlr --icsv --opprint  stats1 -f a,b -a min,mean,max,median,p90 then reshape -r "_" -o k,v input.csv

you get

+----------+----------+
| k        | v        |
+----------+----------+
| a_min    | 1        |
| a_mean   | 1.500000 |
| a_max    | 2        |
| a_median | 2        |
| a_p90    | 2        |
| b_min    | 1.200000 |
| b_mean   | 2.100000 |
| b_max    | 3.000000 |
| b_median | 3        |
| b_p90    | 3        |
+----------+----------+

Then you can explode k

mlr --icsv --opprint  stats1 -f a,b -a min,mean,max,median,p90 then reshape -r "_" -o k,v then nest --explode --values --across-fields -f k --nested-fs "_" input.csv

to get

+-----+--------+----------+
| k_1 | k_2    | v        |
+-----+--------+----------+
| a   | min    | 1        |
| a   | mean   | 1.500000 |
| a   | max    | 2        |
| a   | median | 2        |
| a   | p90    | 2        |
| b   | min    | 1.200000 |
| b   | mean   | 2.100000 |
| b   | max    | 3.000000 |
| b   | median | 3        |
| b   | p90    | 3        |
+-----+--------+----------+

And then reshape it againg long to wide

mlr --icsv --opprint --barred stats1 -f a,b -a min,mean,max,median,p90 then reshape -r "_" -o k,v then nest --explode --values --across-fields -f k --nested-fs "_" then reshape -s k_1,v then label param input.csv

to get

+--------+----------+----------+
| param  | a        | b        |
+--------+----------+----------+
| min    | 1        | 1.200000 |
| mean   | 1.500000 | 2.100000 |
| max    | 2        | 3.000000 |
| median | 2        | 3        |
| p90    | 2        | 3        |
+--------+----------+----------+

[nikbucher]

Wow - that's awesome! Thanks a lot @aborruso
I see, I've to dive deeper in mlr :-)

[aborruso]

I see, I've to dive deeper in mlr :-)

It's really a great tool, you will like it a lot

[nikbucher]

yeah - maybe I can "reuse" some experience from jq.

[johnkerl]

@nikbucher maybe ... jq and mlr are similar in spirit, but wildly different in execution/syntax ...

[nikbucher]

Yes, I agree with that.
There was just the verb 'reshape' that reminded me a bit of 'to_entries, from_entries, with_entries' in jq. In that sense, by "reuse" I meant more the way of thinking of how to transform data. Sorry for my inaccurate diction and my mediocre English. 🙈