Clarify compatibility and use cases for test data extensions

[hairez]

There are several supported file extensions: .in, .ans, .out, .in.statement, .ans.statement, .interaction, as well as .download, and maybe even .yaml.

We should make it clearer which test cases each of these extensions can be used for, and what they are compatible with.

There is already a table under the section for Test data, but I would like to extend it to include all supported file extensions.

Here are the file extensions that I have found and would like to be included:

Extension	Described In	Summary
.in	Input	Input piped to standard input
.ans	Output Validator	Answer file given to the Output Validator
.out	TODO	TODO
.in.statement	TODO	TODO
.ans.statement	TODO	TODO
.interaction	TODO	TODO
*.download	TODO	TODO
.files	Input	Input available via file I/O
.yaml	Test Case Configuration	Additional configuration of the test case
.png, .jpg, .jpeg, .svg	Illustrations	Illustration of the test case

I also want to add another column called Incompatible with or something similar, since for example .interaction is not compatible with .in.statement or .ans.statement.

Another column should maybe be Allowed in, that clarifies where each file type could be used. For example, .in.statement should probably not be allowed in Invalid Input or Invalid Output. This is especially useful, since it is not obvious if .statement should be allowed in Secret if feedback for all test cases is allowed for a problem (mentioned to me by @niemela and @Matistjati).

[hairez]

I will start working on a PR adding this.

Are there any other extensions that I have missed that should be added to the table?

[hairez]

Started three related issues that would be nice to resolve before completing this one:

• Remove the functionality to add .download to any kind of file and directory #506
• Restrict output validator checks to .out files only #507
• Allow .statement files for test cases in Secret #508
• Allow .download in Secret/ #515

(Edit: Added Issue #515)

[RagnarGrootKoerkamp]

• .in: Stdin, or input to interactive for interactive problems
• .ans: input to (default) output validator
• .out: example team output
• .in.statement: input shown to team in case the .in has wrong format or secret information for the interactor. (In case the public interactor in /attachments is different than the private one.)
• .ans.statement: answer shown to team in case the .ans has the wrong format for the public output validator/interactor. Or simply to have public answer 3.14 instead of .ans containing 3.14159265 (but this could also be done with .out). Or: to abbreviate a very long answer into blah ..... blah.
• .interaction: to render interactive samples. Not validated.
• .download to override the file that will be available for download. How does this interact with .in, .in.statement, and .in.download? Could be some situation where .in is private, .in.statement is the abbreviated public version with ... in it, and .in.download is the full public version.

[RagnarGrootKoerkamp]

If .in and .in.statement are available, which of the two will be the one available for download?

• In case .in is private and .in.statement is public, it should be .in.statement
• In case .in is public but long, and .in.statement is abbreviated, it shoudl be .in.

One can always add a .in.download of course, but not sure which is the better default. .in.statement is safer at least.

[mzuenni]

.statement is never available for download, this is just to change how stuff is displayed. If you want to modify what can be downloaded, you must provide a .download.

[niemela]

I also want to add another column called Incompatible with or something similar, since for example .interaction is not compatible with .in.statement or .ans.statement.

I don't want that column. It does not make it easier to understand, in fact I would argue it makes it seem more complicated. When you explain it where we describe these, it's clear why you can't have both. (On a bit of a side note, this is similar to the discussion in #508 about allowing both .out and .ans.statement. In both cases these are disallowed because it does nothing, so that might imply that an mistake has been made (why would you add a file that does nothing). And in both cases the counterargument is that the spec is simpler if we have fewer rules and exceptions. Whatever we do, we should be consistent. There is also a third choice in not disallowing it but adding a note about it warn about it in tools.)

Another column should maybe be Allowed in, that clarifies where each file type could be used.

I would prefer a "Comment" or "Descriptions" column that adds that kind of information. That is what we do elsewhere.

• .ans: input to (default) output validator

It's the input to all output validators.

• .in.statement:

Also not validated.

• .ans.statement:

Also not validated.

• In case .in is private and .in.statement is public, it should be .in.statement

I don't understand this comment. We have not why to mark individual files as private of public. We can mark test cases as full_feedback (which is default for sample), and that makes it public in some sense. But that's the test case as a whole, not individual files.

.statement is never available for download, this is just to change how stuff is displayed. If you want to modify what can be downloaded, you must provide a .download.

Correct.

[RagnarGrootKoerkamp]

In case .in is private and .in.statement is public, it should be .in.statement

For interactive problems, the .in to the interactor (the private one) might be different to the .in to the public testing tool. In that case you'd want .in.download and .interaction, but no .in.statement (since the interaction is shown).

In case all cases are generated on the fly (technically interactive), but look like a standard non-interactive problem for teams, you'd have .in for the interactor=generator, .in.statement and .in.download for the teams, and those last two files should be the same, which is somewhat redundant. But well, this is a niche case; it's OK.

[niemela]

For interactive problems, the .in to the interactor (the private one) might be different to the .in to the public testing tool. In that case you'd want .in.download and .interaction, but no .in.statement (since the interaction is shown).

Right.

In case all cases are generated on the fly (technically interactive), but look like a standard non-interactive problem for teams, you'd have .in for the interactor=generator, .in.statement and .in.download for the teams, and those last two files should be the same, which is somewhat redundant. But well, this is a niche case; it's OK.

Right, and agreed.

[mzuenni]

.statement is never available for download, this is just to change how stuff is displayed. If you want to modify what can be downloaded, you must provide a .download.

Correct.

(Actually i was wrong with this one, sorry. The priority is indeed .in.download, .in.statement, .in)

[niemela]

(Actually i was wrong with this one, sorry. The priority is indeed .in.download, .in.statement, .in)

Ah, indeed. I was also wrong.