Write test report artifacts directly

[cbandy]

Describe the feature:

During local development, I like to print test results to the terminal. During CI, I like to see similar, human-readable test results in the log and a machine-readable report that can be tracked and visualized by other CI tooling.

To accomplish this today, I must run Goss twice with different --format flags and shell redirection. It would be great if Goss could produce both forms of feedback in a single execution.

Describe the solution you'd like

I want to specify one or more additional files that are filled with test results formatted differently than the terminal output.

For example,

• Gingko produces one file for each report flag it is passed:

  ginkgo --json-report=report.json --junit-report=report.xml
  

• RSpec produces one file for each format and out flag pair it is passed:

  rspec --format progress --format documentation --out rspec.txt
  

• Syft produces one file for each output flag it is passed:

  syft scan --output cyclonedx-xml=report.xml --output syft-json=report.json
  

[aelsabbahy]

I like this as a suggestion. Marked as approved, anyone can submit a PR for this.

To be completely honest, I probably won't get around to it myself for a while.

[tknerr]

I was about to ask for the same feature, that would be super helpful indeed.

Alternatively to passing --format twice, we could also just pass a comma separated list of formatters, e.g. goss validate --format documentation,junit. I have seen that approach in doctest recently, but it's probably just a matter of taste (or ease in commandline parsing).

[aelsabbahy]

Alternatively to passing --format twice, we could also just pass a comma separated list of formatters, e.g. goss validate --format documentation,junit. I have seen that approach in doctest recently, but it's probably just a matter of taste (or ease in commandline parsing).

Would that print both out to stdout or write to two different files?

[cbandy]

🤔 I suspect it may be best to leave the --format flag alone. It can control the (singular) terminal output as it always has.

Of the three examples in the request, I like Syft one most. As I understand it,

• the --output flag can be specified any number of times
• each occurrence indicates a file to be written
• each occurrence takes a single argument that contains a =;
  the file format is left of the =, and the file path is to the right
• the same file format can be used in two --output arguments, but a file path should occur only once

🤔 I'm not sure what to do with formatter options in this arrangement.
Maybe a comma? --output json,pretty=results.json
Maybe --format-options affects everything, terminal and files?

$ goss --format documentation
User sees "documentation" on the terminal

$ goss --output junit=report.xml
User sees "rspecish" on the terminal and a JUnit file "report.xml"

$ goss --output junit=one.xml --output junit=two.xml
User sees "rspecish" on the terminal and two JUnit files "one.xml" and "two.xml"

$ goss --format tap --output junit=junit.xml --output json=results.json
User sees TAP on the terminal, a JUnit file "junit.xml" and a JSON file "results.json"

[tknerr]

Would that print both out to stdout or write to two different files?

@aelsabbahy you are right, it's missing the specification of the output file. In doctest, if I remember correctly, it would output both formats to stdout, one after the other, which is a bit ugly because you then have to split stdout at the right place...

🤔 I'm not sure what to do with formatter options in this arrangement. Maybe a comma? --output json,pretty=results.json Maybe --format-options affects everything, terminal and files?

Hmmm.... comma would be good enough imho, or maybe square brackets? (would imho read a bit better, e.g. --output json[pretty]=results.json).

Regarding formatter options, I think it would be easier / less edge cases to deal with when defining the formatter options for each output individually rather than having --format-options apply globally to everything.

Would it also make sense to explicitly specify stdout instead of file output, so that using --format junit would then just be a "shortcut" for the more versatile --output junit=stdout?

[mattlorimor]

Because goss doesn't have this, YaleUniversity/packer-plugin-goss is attempting to work around the "problem" on their end: YaleUniversity/packer-plugin-goss#100.

As you likely already know, this is an important ability in CI/CD workloads. Specifically, for exporting/attaching build artifacts to builds - of which the goss validate test results would be a part.

(not attempting to be pushy; just providing more context)