Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add "merge --input-file-list NAME ..." as a way to exceed CLI limits #328

Open
wants to merge 13 commits into
base: main
Choose a base branch
from

Conversation

jimklimov
Copy link
Contributor

@jimklimov jimklimov commented Jul 17, 2023

...or to handle filenames in paths which have spaces.

Not tested yet, and C# is not my native language, so please bear with me :)
UPDATE: Now tested, as detailed in comments below.

So far, posting to make sure CI passes on all platforms with this trivial change. Not sure how to test this properly (with an additional file to list SBOM files) - suggestions or maintainer commits are welcome :)

@sonatype-lift
Copy link

sonatype-lift bot commented Jul 17, 2023

Sonatype Lift is retiring

Sonatype Lift will be retiring on Sep 12, 2023, with its analysis stopping on Aug 12, 2023. We understand that this news may come as a disappointment, and Sonatype is committed to helping you transition off it seamlessly. If you’d like to retain your data, please export your issues from the web console.
We are extremely grateful and thank you for your support over the years.

📖 Read about the impacts and timeline

@jimklimov
Copy link
Contributor Author

jimklimov commented Jul 19, 2023

Now actually tested locally, at least. Outputs posted below (made a bit noisier to see what's happening with parameter handling).

  • With a CRLF-separated list file:
:; cyclonedx.exe merge --input-files top*json --input-files-list top*txt \
    --output-file myProject_deploystd~da72200.json
Got 1 individual input file name(s): ['topSBOM.myProject_deploystd~da72200.CollectedComponents.json']
Got 1 file(s) with actual input file names: ['topSBOM.myProject_deploystd~da72200.CollectedComponents.list.txt']
Adding to input file list from topSBOM.myProject_deploystd~da72200.CollectedComponents.list.txt
Got 301 entries from topSBOM.myProject_deploystd~da72200.CollectedComponents.list.txt
Processing input file topSBOM.myProject_deploystd~da72200.CollectedComponents.json
    Contains 18 components
Processing input file common-service-discovery-14.0.0-cyclonedx.json
    Contains 12 components
Processing input file db-12.0.0-cyclonedx.json
    Contains 119 components
...
  • Similar for a NUL-separated file (as would be provided by find ... -print0):
:; tr -d '\r' < top*txt | tr '\n' '\0' > top*txt0
:; cyclonedx.exe merge --input-files top*json --input-files-nul-list top*txt0 \
    --output-file myProject_deploystd~da72200.json
Got 1 individual input file name(s): ['topSBOM.myProject_deploystd~da72200.CollectedComponents.json']
Got 1 file(s) with NUL-separated actual input file names: ['topSBOM.myProject_deploystd~da72200.CollectedComponents.list.txt0']
Adding to input file list from topSBOM.myProject_deploystd~da72200.CollectedComponents.list.txt0
Got 301 entries from topSBOM.myProject_deploystd~da72200.CollectedComponents.list.txt0
Processing input file topSBOM.myProject_deploystd~da72200.CollectedComponents.json
    Contains 18 components
Processing input file common-service-discovery-14.0.0-cyclonedx.json
    Contains 12 components
Processing input file db-12.0.0-cyclonedx.json
    Contains 119 components
...
  • UPDATE: option renamed to --input-files-nul-list to avoid conflicts in C# Options parser

…ork around stupid Options parser)

Signed-off-by: Jim Klimov <[email protected]>
In particular, avoid starting lines with "Got" for different concepts -
improve readability.

Signed-off-by: Jim Klimov <[email protected]>
@jimklimov
Copy link
Contributor Author

Updated report style:

:; cyclonedx.exe merge --input-files top*json --input-files-list top*txt \
    --output-file myProject_deploystd~da72200.json
Got 1 individual input file name(s): ['topSBOM.myProject_deploystd~da72200.CollectedComponents.json']
Processing 1 file(s) with list of actual input file names: ['topSBOM.myProject_deploystd~da72200.CollectedComponents.list.txt']
Adding to input file list from topSBOM.myProject_deploystd~da72200.CollectedComponents.list.txt
Got 295 new entries from topSBOM.myProject_deploystd~da72200.CollectedComponents.list.txt
Determined 296 input files to merge
Processing input file topSBOM.myProject_deploystd~da72200.CollectedComponents.json
    Contains 18 components
...
Writing output file...
    Total 26726 components

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants