directory vs extension for samples

[mzuenni]

While implementing the spec for the new sample directories (data/sample/statement/ and data/sample/download/) for BAPCtools I noticed a few quirks and potential pitfalls.
For example the existence of data/sample/download/ shadows all testcases (not only those for which a overwrite file is provided).
Additionally the directory structure makes it more likely the directories get out of sync, i.e. a sample is only added/removed from one of the samples.
Also somehow the point of data/sample/statement/ is missed? IMO this is only useful to present non validated input files (else it could just be a normal testcase?) so i explicitly want it to be not validated? (see the attached PR)

I would propose to instead introduce the extensions .statement and .download for this.
For example data/sample/x.in would than be replaced by data/sample/x.in.statement for the problem statement.

This approach has a few benefits:

• all testcases related files are in the same directory
• files get shadowed on a per file basis not in generell
• all files belonging to the testcase data/sample/<x> match the glob data/sample/<x>.*

[RagnarGrootKoerkamp]

I'm neutral on this. But if playing around with it seems better, yeah I get that and sounds reasonable.

[evouga]

For example the existence of data/sample/download/ shadows all testcases (not only those for which a overwrite file is provided).

This is intentional as otherwise there is no way to remove test cases (such as those in /data/sample/statement) that should not be included.

IMO this is only useful to present non validated input files (else it could just be a normal testcase?)

Often you want to show different sample output than what's in the .ans file (for example, because the .ans file contains way more precision than is needed by the problem statement; or the .ans file contains some certificate used for output validation that is unrelated to the output expected from the contestant submission). In this case you want to use the samples folder and you want the data to be validated (with the .ans and .out files treated as described in the spec).

[mzuenni]

For example the existence of data/sample/download/ shadows all testcases (not only those for which a overwrite file is provided).

This is intentional as otherwise there is no way to remove test cases (such as those in /data/sample/statement) that should not be included.

Having a statement test case without a .in would already signal this?

IMO this is only useful to present non validated input files (else it could just be a normal testcase?)

Often you want to show different sample output than what's in the .ans file (for example, because the .ans file contains way more precision than is needed by the problem statement; or the .ans file contains some certificate used for output validation that is unrelated to the output expected from the contestant submission). In this case you want to use the samples folder and you want the data to be validated (with the .ans and .out files treated as described in the spec).

ok, so that is another use case. I still think it is more valuable to support both scenarios, which is only possible if statement files are not validated at all. Note that you can still validate the statement ans file by adding/symlinking it as valid_output.

What is the general opinion about extensions instead of directories?