BTW, can you also point out the Java code in this pr?

Other LGTM but I think it worths issue a disscussion...

I have sent a discussion thread to the dev ML. It would be good if you can take a look. Thanks! @emkornfield @pitrou @gszadovszky @rdblue @etseidl @clairemcginty

@etseidl I think we are on the same page that current rule 3 has an issue that it does not forbid the inner-most layer to be repeated. However, I feel that it is not a good idea to add would otherwise be a valid 3-level structure as outlined above to rule 3. The reason is that a normal 3-level structure can simply ignore its synthetic middle layer but the middle layer produces a OneTuple in the rule 3, which may also confuse readers. In addition, it is not that straight-forward. So I agree with @rdblue that it would be good to insert a new rule for the new case. IMO the new rule should be placed above the current rule 3 so we are pretty sure that the inner layer cannot be repeated in the remaining cases (including the rule 3). The only caveat is that the new rule 5 seems to be unreachable.

Ok, so the new Rule 3 is in effect saying that if the child of the repeated middle group is repeated, then because 1-level and LIST/MAP annotated schemas cannot mix, the middle group cannot be the type so the child is the type. The side consequence of this is that the middle group has to have a LIST or MAP annotation to allow the 3rd level to be repeated.

So is it true then that the repeated keyword may only be used for the child of a LIST or MAP annotated field when those annotations are in use? And if that's true, then a 2-level structure cannot be repeated as stated above?

In contrast to 3-level structure, the repetition of 2-level structure can be `optional`, `required`, or `repeated`.

Ok, so the new Rule 3 is in effect saying that if the child of the repeated middle group is repeated, then because 1-level and LIST/MAP annotated schemas cannot mix, the middle group cannot be the type so the child is the type. The side consequence of this is that the middle group has to have a LIST or MAP annotation to allow the 3rd level to be repeated

No, the middle level can never be a LIST-annotated or MAP-annotated 3-level structure, because they cannot be repeated. As you have said, the middle level cannot be an unannotated group with a single repeated field due to mixing LIST-annotation and repeated field without annotation. So the only valid case for rule 3 is that the middle level is a LIST-annotated 2-level structure.

So is it true then that the repeated keyword may only be used for the child of a LIST or MAP annotated field when those annotations are in use? And if that's true, then a 2-level structure cannot be repeated as stated above?

You're right. A LIST-annotated 2-level structure can only be repeated when it is a direct child of another LIST-annotated 2-level structure. I had missed the statement that disallows annotated and unannotated types to be mixed and modified these rules back and forth. Now this is pretty clear. Let me reflect these discussions in the PR.

No, the middle level can never be a LIST-annotated or MAP-annotated 3-level structure, because they cannot be repeated.

Ah, ok. Now I understand. That's why the 2-level list needs the ability to be repeated, but only as part of a nested list. Thanks again for the clarification @wgtmac. I think all my questions are answered now.

@gszadovszky @etseidl @rdblue @pitrou @emkornfield Do you have any concern about this PR? I think we need to move forward now.

The rules part is looking good, but I think that spending time documenting what people did incorrectly years ago makes the doc more confusing and increases chances that people will write invalid lists. I'd prefer to revert most of the changes that explain what people did incorrectly.

I agree. But I think those can be posted on the pull-request description

@rdblue Thanks for your review! I have removed all unnecessary changes. Please take a look again.

@rdblue @wgtmac I'm not sure I agree with removing the examples. They make it less likely that an implementation would interpret the written rules wrongly, IMO. Maybe we can instead strengthen the wording that these examples show existing legacy schemas, not anything to be written by modern writers.

Concretely, if I have to review a PR implementing said compatibility rules, I'd rather have the spec spell out the examples because it makes it easier to check the PR's implementation (and unit tests!).

Perhaps add them like this? @pitrou

WARNING: WRITERS SHOULD NOT PRODUCE LIST TYPES LIKE THESE EXAMPLES!
THEY ARE FOR THE PURPOSE OF READING EXISTING DATA ONLY.

// List<Integer> (non-null list, non-null elements)
repeated int32 num;

// List<Tuple<Integer, String>> (non-null list, non-null elements)
repeated group my_list {
  required int32 num;
  optional binary str (STRING);
}

Ok, please don't use UPPERCASE :)