Adding JUnit support to chart verifier report

[dcurran90]

Adds JUnit output support. Example output:

<?xml version="1.0" encoding="UTF-8"?>
 <testsuites name="chart-verifier test run" tests="12" failures="5" skipped="1" timestamp="2024-04-15T11:00:44-05:00">
   <properties>
     <property name="config">
       <ApiVersion>v1</ApiVersion>
       <Kind>verify-report</Kind>
       <reportOptions></reportOptions>
       <Metadata>
         <ToolMetadata>
           <Version>1.13.3</Version>
           <Profile>
             <VendorType>partner</VendorType>
             <Version>v1.2</Version>
           </Profile>
           <ReportDigest>uint64:6719283521726907025</ReportDigest>
           <ChartUri>../helm-tester/charts/hello-world</ChartUri>
           <Digests>
             <Chart>sha256:472ca4bd30d0ac4af65d4107a2e73bc4593ea8271476e110d4b71834348c5478</Chart>
             <Package></Package>
             <PublicKey></PublicKey>
           </Digests>
           <LastCertifiedTimestamp>2024-04-15T11:00:44.838016-05:00</LastCertifiedTimestamp>
           <CertifiedOpenShiftVersions></CertifiedOpenShiftVersions>
           <TestedOpenShiftVersion>N/A</TestedOpenShiftVersion>
           <SupportedOpenShiftVersions>N/A</SupportedOpenShiftVersions>
           <ProviderDelivery>false</ProviderDelivery>
           <WebCatalogOnly>false</WebCatalogOnly>
         </ToolMetadata>
         <Overrides></Overrides>
         <ChartData>
           <Name>hello-world</Name>
           <Home></Home>
           <Version>0.1.0</Version>
           <Description>A Helm chart for Kubernetes</Description>
           <Icon></Icon>
           <APIVersion>v2</APIVersion>
           <Condition></Condition>
           <Tags></Tags>
           <AppVersion>1.16.0</AppVersion>
           <Deprecated>false</Deprecated>
           <KubeVersion></KubeVersion>
           <Type>application</Type>
           <Sources></Sources>
           <Keywords></Keywords>
           <Annotations></Annotations>
           <Maintainers></Maintainers>
           <Dependencies></Dependencies>
         </ChartData>
       </Metadata>
     </property>
   </properties>
   <testsuite name="chart-verifier test run" tests="12" failures="5" skipped="1" timestamp="2024-04-15T11:00:44-05:00">
     <testcase name="v1.1/has-kubeversion" classname="Kubernetes version is not specified" assertions="1">
       <system-out>Kubernetes version is not specified</system-out>
     </testcase>
     <testcase name="v1.0/has-readme" classname="Chart has a README" assertions="1">
       <system-out>Chart has a README</system-out>
     </testcase>
     <testcase name="v1.0/not-contains-crds" classname="Chart does not contain CRDs" assertions="1">
       <system-out>Chart does not contain CRDs</system-out>
     </testcase>
     <testcase name="v1.0/is-helm-v3" classname="API version is V2, used in Helm 3" assertions="1">
       <system-out>API version is V2, used in Helm 3</system-out>
     </testcase>
     <testcase name="v1.0/contains-test" classname="Chart test files do not exist" assertions="1">
       <system-out>Chart test files do not exist</system-out>
     </testcase>
     <testcase name="v1.0/contains-values-schema" classname="Values schema file does not exist" assertions="1">
       <system-out>Values schema file does not exist</system-out>
     </testcase>
     <testcase name="v1.0/contains-values" classname="Values file exist" assertions="1">
       <system-out>Values file exist</system-out>
     </testcase>
     <testcase name="v1.0/required-annotations-present" classname="Missing required annotations: [charts.openshift.io/name]" assertions="1">
       <system-out>Missing required annotations: [charts.openshift.io/name]</system-out>
     </testcase>
     <testcase name="v1.0/chart-testing" classname="chart Install failure: Kubernetes cluster unreachable: the server has asked for the client to provide credentials" assertions="1">
       <system-out>chart Install failure: Kubernetes cluster unreachable: the server has asked for the client to provide credentials</system-out>
     </testcase>
     <testcase name="v1.0/helm-lint" classname="Helm lint successful" assertions="1">
       <system-out>Helm lint successful</system-out>
     </testcase>
     <testcase name="v1.0/signature-is-valid" classname="Chart is not signed : Signature verification not required" assertions="1">
       <system-out>Chart is not signed : Signature verification not required</system-out>
     </testcase>
     <testcase name="v1.0/not-contain-csi-objects" classname="CSI objects do not exist" assertions="1">
       <system-out>CSI objects do not exist</system-out>
     </testcase>
   </testsuite>
 </testsuites>

Closes #232

[github-advanced-security]

gosec found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

[komish]

Thanks for your work on this @dcurran90!

There are a couple of things I'd like to see changed here.

---

(1)

The JUnitXML output format should be a supplemental output format instead of being a valid value to the --output flag. The reason for this is because we will not accept junitxml reports as a part of Certification.

What I would prefer to see here is something like this (the flag's name is not prescriptive, it's just an example):

chart-verifier verify --write-junit-to /this/path/report-junit.xml <params> <chart>

This tells chart-verifier that it needs to write a junit xml representation of the test case outcomes also, in addition to writing the normal format to whatever output was specified.

Alternatively, you can choose the file for the user, and write it to the chartverifier/ directory. I kind of prefer the former, but either will work.

The reason we have to do it this way is because chart-verifier, by default, writes to STDOUT as mentioned here.

---

(2)

I don't necessarily believe the JUnitXML needs to include all of the metadata from the report, especially if it's supplemental. Similar to preflight, I would just focus on writing the test case definitions, and skipping properties and other metadata that aren't immediately relevant.

---

(3)

I think we can simplify the calls we make here from the xml library by using structs and tags. You may even be able to borrow and re-work some of those linked.

---

(4)

If it's possible (and I believe it should be, at least, at first glance), let's make this code internal to chart-verifier.

[dcurran90]

thanks for the feedback @komish
Looking into those changes now

[komish]

@dcurran90

• I've rebase this for you and resolved the file conflict in the proecss.
• You probably need to remove your report test.
• Anything else you might need to do before this comes out of "draft" status

[komish]

Closing this in favor of #471