What about to add JSON Lines format?

[aborruso]

Dear @johnkerl ,
using mlr is already possible to work with this kind of file format. It is more and more used, because it is much more comfortable in terms of processing.

If I have this CSV

id,v
1,a
2,b

and run

mlr --c2j  cat input.csv

I have

{
  "id": 1,
  "v": "a"
}
{
  "id": 2,
  "v": "b"
}

This standard JSON output is a wrong JSON, whether you consider it as a JSON or as a JSON Lines.

In my opinion it would be better that when you set simply --c2j, mlr will have also "inside" it the --jlistwrap flag.

And you could add JSON Lines format, in which mlr will have "inside" it the --no-jvstack flag to have:

{"id": 1, "v": "a"}
{"id": 2, "v": "b"}

It occurred to me, because I had to work with standard mlr JSON output, and my client couldn't read it, because it wasn't a real JSON Lines file.

Thank you

[johnkerl]

Hi @aborruso & sorry for the delay! Is this related to #346 ?

[aborruso]

Hi @aborruso & sorry for the delay! Is this related to #346 ?

Partially. I think it's a more general issue.

This is the default JSON output. Running echo "a=1,b=a\na=2,b=c" | mlr --ojson cat, you have

{
  "a": 1,
  "b": "a"
}
{
  "a": 2,
  "b": "c"
}

Is this a valid JSON output? If no, why to have as default an invalid JSON file?
Shouldn't that be so?

[
   {
      "a":1,
      "b":"a"
   },
   {
      "a":2,
      "b":"c"
   }
]

The default output looks like a JSON lines file. But it's not properly formatted because each Line is not a valid JSON value. It should be

{"a":1,"b":"a"}
{"a":2,"b":"c"}

Then my message has a dual purpose:

• to check the default JSON output (is it valid?). To add by default --jlistwrap?
• to add JSON lines output (--ojsonl, adding --no-jvstack), because it's widely used in the data world and is very convenient to process and manage.

Thank you

[johnkerl]

Hi @aborruso --

There are several things going on here

• Miller has always accepted JSON objects either with or without the enclosing square brackets
  • jq does this
  • It's streaming-friendly -- records can be processed in or out without waiting for the final ] (the Miller 5-and-below JSON reader wasn't streaming as I wanted it to be; now in Miller 6 it is -- as long as there are no enclosing [...])
  • This is better for memory usage -- see https://miller.readthedocs.io/en/latest/streaming-and-memory/
  • It's punctuation-friendly -- no special-casing that all but the last object need to end with },
  • If anyone really needs the outermost [...] then: Miller accepts them on input without any flags, and produces them on output on request with --jlistwrap
• Separately from that: whether records are single-line by default or not. In Miller 5-and-below they were single-line by default (and --jvstack to change the default); in Miller 6 they're multi-line by default (and --no-jvstack to change the default)
• Separately from that: JSON lines format is something we haven't supported yet (but should!), and today's JSON handling resembles it only by accident.

In sum I had Miller's JSON output omit the enclosing [...] by default for general stream-friendliness, believing it was easily customizable by --jlistwrap and/or piping through jq -- and each record is pure JSON. That was years ago; it reflected (and still reflects) my preference. But perhaps it's time to update the default behavior. There's an argument to be made though (which you've made above) that including outermost [...] by default would make the entire stream pure JSON, making the stream-friendliness opt-in rather than opt-out.

The questions reduce to:

• Should we prioritize stream-friendliness or pure-JSON for Miller's JSON output?
• Supporting JSON lines: we can and will support this per se, just not yet.

[aborruso]

First of all, thank you very much for your interesting message.

• Should we prioritize stream-friendliness or pure-JSON for Miller's JSON output?

I'm with you and I think we should prioritize stream-friendliness. However, I would put a sentence at the beginning of the text on the JSON format documentation, to warn the user about it: he/she might try to read a json output and fail to read it with his/her client, without understanding why.

[johnkerl]

@aborruso I'm not convinced of my position. Non-wrapping in [...] is important for stream-friendliness on input and Miller already handles input formatted either with or without them. The --jlistwrap flag is for output and only becomes a stream-friendliness issue for data piped into another Miller run.

Maybe the best thing is to list-wrap on output by default; we still get stream-friendliness on input by default if the input data allows it; and the doc note can let people know they have a --no-jlistwrap option if they want it ...

[aborruso]

Hi @johnkerl, in my guide in Italian under construction, following questions that friends and colleagues have asked me, I have inserted at the top a warning: something like "Miller's default output JSON is not really a JSON".

And then I have added a note about the Miller 5 default output (it's the great JSON Lines format) and a note about Miller 6 default output.

It is a bit long, but it seems necessary to me.
I wanted to inform you.

Thank you

[johnkerl]

Thanks! But do know that Miller 5 JSON is not JSON lines -- it's one line per record, but there is more to the JSON lines spec than is implemented in either Miller 5 or 6.

The difference between Miller 5 & 6 here is only that for Miller 5, --no-jvstack is the default and for Miller 6, --jvstack is the default. For both of them, --no-jlistwrap is the default -- which is what I think you mean by 'not really a JSON'.

[aborruso]

@johnkerl thank you. I don't want to write wrong things, I love Miller, so if I'm wrong, please correct me.

Thanks! But do know that Miller 5 JSON is not JSON lines -- it's one line per record, but there is more to the JSON lines spec than is implemented in either Miller 5 or 6.

Is the Miller 5 default output, not a JSON lines?

I know, this is a too easy example, but this is not a JSON lines?

echo -e "a=lorem,b=ipsum\na=hello,b=world" | mlr --ojson cat
{ "a": "lorem", "b": "ipsum" }
{ "a": "hello", "b": "world" }

Using the default Miller 6 I have the below result, than by default I do not have that each Line is a Valid JSON Value. Am I wrong?

{
  "a": "lorem",
  "b": "ipsum"
}
{
  "a": "hello",
  "b": "world"
}

The difference between Miller 5 & 6 here is only that for Miller 5, --no-jvstack is the default and for Miller 6, --jvstack is the default. For both of them, --no-jlistwrap is the default -- which is what I think you mean by 'not really a JSON'.

John without --no-jlistwrap you do not have a valid JSON, you cannot pass it to a standard client, without setting some import parameters. Am I wrong?

If I have not understood anything, I apologize. Please correct me and I will correct my guide draft right away. I care a lot about Miller and my Italian guide.

Thank you

[johnkerl]

Miller 5 and Miller 6 both have --no-jlistwrap as default output. So Miller 5 and Miller 6 both don't produce valid JSON by default. They only differ in whether records are all on one line or not.

I thought JSON lines (https://jsonlines.org/) had the requirement of letting "header" be in the first record and "data" in subsequents -- as in CSV -- but that's not part of the JSON lines spec. What is part of the JSON lines spec is utf-8 encoding and I'm not definitely doing that in Miller 5 or 6.

• Miller 5 and 6 both accept -- on input -- outermost [...] or not. They both accept pure JSON input.
• Both Miller 5 and Miller 6 produce -- on output -- fully compliant JSON as long as --jlistwrap is used.
• Miller 5, with default flags, produces -- on output -- something a little closer to JSON Lines but not purely compliant JSON Lines. And I'm hesitant (I may be wrong) to change the default output from --no-jlistwrap to--jlistwrap for Miller 6, for backward-compatibility concerns. I'm hoping ~/.mlrrc will be enough to accommodate everyone's needs.
• Another option, maybe Miller 6 should have --no-jvstack by default like Miller 5 does.

[aborruso]

Miller 5 and Miller 6 both have --no-jlistwrap as default. So Miller 5 and Miller 6 both don't produce valid JSON. They only differ in whether records are all on one line or not.

Then my warning is correct, "Miller's default output JSON is not really a JSON". I think it's important so write as the first thing.

[johnkerl]

What is part of the JSON lines spec is utf-8 encoding and I'm not definitely doing that in Miller 5 or 6.

I am mistaken:

$ echo 'x=παράδειγμα' | mlr --d2j cat
{
  "x": "παράδειγμα"
}

Maybe we should have --json and --json-lines:

• --json will default to --jlistwrap and --jvstack for output
• --json-lines will default to --no-jlistwrap and --no-jvstack for output
• This may break some Miller 5 scripts people have, since it changes the default behavior for --json

[aborruso]

Maybe we should have --json and --json-lines:

WOW, this was my request somehow 👍

[johnkerl]

@aborruso Another way I've been thinking about backward-compatibility issues is they affect only people who were using Miller before Miller 6 -- they don't affect new users. Adhering overmuch to backward-compatible awkward behavior might turn off new users from using the tool.

So it's perhaps worth making some well-justified and well-documented non-backward-compatible changes (which is what 'major' versions like 5.x.x. -> 6.x.x are for) -- documenting them clearly so existing users know what to change in their scripts; making the default behavior less awkward so hopefully more future users will use and enjoy the tool.

I think I'll do as proposed above. There will be a goodly amount of doc mods now that there will be two formats -- JSON and JSON Lines -- which will touch many doc pages.

[johnkerl]

@aborruso #844 and https://miller.readthedocs.io/en/latest/new-in-miller-6/#json-and-json-lines-formatting

[aborruso]

@aborruso #844 and https://miller.readthedocs.io/en/latest/new-in-miller-6/#json-and-json-lines-formatting

I'm very proud of it!

Thank you🙏