feat(fs): add --max-file-size flag to skip the files greater than a particular size #7304

zesiar0 · 2024-08-04T10:01:13Z

Description

Related issues

Close feat: add an option to skip the files greater than a particular size #7190

…cular size

CLAassistant · 2024-08-04T10:01:18Z

All committers have signed the CLA.

CLAassistant · 2024-08-04T10:01:18Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

nikpivkin · 2024-08-05T05:03:38Z

pkg/fanal/analyzer/analyzer.go

@@ -74,13 +76,19 @@ type analyzer interface {
 	Version() int
 	Analyze(ctx context.Context, input AnalysisInput) (*AnalysisResult, error)
 	Required(filePath string, info os.FileInfo) bool
+	Description() string


This method is not used anywhere. I think the point is that the description should return a human-readable description of the analyser and be used in generating documentation.

/cc @knqyf263

Yes, but we can do it in another PR.

I agree with @knqyf263
Let's move adding of this function to another PR.

nikpivkin · 2024-08-05T05:03:56Z

pkg/fanal/analyzer/analyzer.go

 }

 type PostAnalyzer interface {
 	Type() Type
 	Version() int
 	PostAnalyze(ctx context.Context, input PostAnalysisInput) (*AnalysisResult, error)
 	Required(filePath string, info os.FileInfo) bool
+	Description() string


Same as above.

nikpivkin · 2024-08-05T05:22:00Z

pkg/fanal/analyzer/analyzer.go

+	if _, ok := ag.fileSizes[analyzerType]; !ok {
+		return true
+	}
+	if fileInfo.Size() > ag.fileSizes[analyzerType] {
+		return false
+	}
+	return true


Suggested change

if _, ok := ag.fileSizes[analyzerType]; !ok {

return true

}

if fileInfo.Size() > ag.fileSizes[analyzerType] {

return false

}

return true

maxSize, ok := ag.fileSizes[analyzerType]

if !ok {

return true

}

return fileInfo.Size() <= maxSize

nikpivkin · 2024-08-05T05:24:26Z

Tests should be added for the new flag, similar to the file-patterns flag.

nikpivkin · 2024-08-05T05:31:24Z

pkg/fanal/analyzer/analyzer.go

@@ -411,6 +435,10 @@ func (ag AnalyzerGroup) AnalyzeFile(ctx context.Context, wg *sync.WaitGroup, lim
 			continue
 		}

+		if ag.fileSizeMatch(a.Type(), info) {


Should the condition be inverted?

nikpivkin · 2024-08-05T05:49:57Z

A flag needs to be added when the group is initialised. https://github.com/aquasecurity/trivy/blob/main/pkg/flag/scan_flags.go#L147

DmitriyLewen · 2024-08-19T06:14:33Z

docs/docs/analyzers/analyzers.md

@@ -0,0 +1,68 @@
+
+## Analyzers
+Use the following types of analyzers for the [--max-file-size] flags


Suggested change

Use the following types of analyzers for the [--max-file-size] flags

Use the following types of analyzers for the [--max-file-size] and [--file-pattern] flags

DmitriyLewen · 2024-08-19T06:21:49Z

pkg/fanal/analyzer/analyzer.go

+	for _, f := range opts.MaxFileSize {
+		// e.g. "jar:5mb secret:5kb"
+		s := strings.SplitN(f, separator, 2)
+		if len(s) != 2 {
+			return group, xerrors.Errorf("invalid max file size (%s) expected format: \"analyzerType:maxFileSize\" e.g. \"jar:5mb secret:5kb\"", f)
+		}
+
+		analyzerType := Type(s[0])
+		maxFileSize, err := units.FromHumanSize(s[1])
+		if err != nil {
+			return group, xerrors.Errorf("invalid file size (%s): %w", s[1], err)
+		}
+		group.fileSizes[analyzerType] = maxFileSize
+	}


What if we move this logic into the flag package:

trivy/pkg/flag/scan_flags.go

Lines 183 to 212 in 1aa5158

func (f *ScanFlagGroup) ToOptions(args []string) (ScanOptions, error) {

if err := parseFlags(f); err != nil {

return ScanOptions{}, err

}

var target string

if len(args) == 1 {

target = args[0]

}

parallel := f.Parallel.Value()

if f.Parallel != nil && parallel == 0 {

log.Info("Set '--parallel' to the number of CPUs", log.Int("parallel", runtime.NumCPU()))

parallel = runtime.NumCPU()

}

return ScanOptions{

Target: target,

SkipDirs: f.SkipDirs.Value(),

SkipFiles: f.SkipFiles.Value(),

OfflineScan: f.OfflineScan.Value(),

Scanners: xstrings.ToTSlice[types.Scanner](f.Scanners.Value()),

FilePatterns: f.FilePatterns.Value(),

MaxFileSize: f.MaxFileSize.Value(),

Parallel: parallel,

SBOMSources: f.SBOMSources.Value(),

RekorURL: f.RekorURL.Value(),

DetectionPriority: ftypes.DetectionPriority(f.DetectionPriority.Value()),

}, nil

}

PS. opts.MaxFileSize will be map[Type]int64.

Then users will get an error message if the wrong template is used immediately when launching Trivy.
@knqyf263 wdyt?

Sounds good

DmitriyLewen · 2024-08-26T07:51:04Z

pkg/flag/scan_flags.go

@@ -196,14 +203,30 @@ func (f *ScanFlagGroup) ToOptions(args []string) (ScanOptions, error) {
 		parallel = runtime.NumCPU()
 	}

+	maxFileSize := make(map[analyzer.Type]int64)
+	for _, f := range f.MaxFileSize.Value() {


You should not overwrite a variable that is in use:

Suggested change

for _, f := range f.MaxFileSize.Value() {

for _, fileSize := range f.MaxFileSize.Value() {

DmitriyLewen · 2024-08-26T08:04:57Z

docs/docs/analyzers/analyzers.md

You need to add this page to docs.
Update mkdocs.yml for that.

But i am not sure about place for this page.
I think we can add this page to References.
@knqyf263 wdyt?

Yes, sounds good

DmitriyLewen · 2024-08-26T08:08:19Z

pkg/fanal/analyzer/analyzer.go

@@ -74,13 +76,19 @@ type analyzer interface {
 	Version() int
 	Analyze(ctx context.Context, input AnalysisInput) (*AnalysisResult, error)
 	Required(filePath string, info os.FileInfo) bool
+	Description() string


I agree with @knqyf263
Let's move adding of this function to another PR.

DmitriyLewen · 2024-08-26T08:11:36Z

pkg/fanal/analyzer/analyzer_test.go

+			if tt.args.maxFileSize != nil {
+				maxFileSize = tt.args.maxFileSize
+			} else {
+				maxFileSize = map[analyzer.Type]int64{}
+			}


Suggested change

if tt.args.maxFileSize != nil {

maxFileSize = tt.args.maxFileSize

} else {

maxFileSize = map[analyzer.Type]int64{}

}

maxFileSize := map[analyzer.Type]int64{}

if tt.args.maxFileSize != nil {

maxFileSize = tt.args.maxFileSize

}

DmitriyLewen · 2024-08-26T08:12:28Z

pkg/flag/scan_flags.go

+			return ScanOptions{}, xerrors.Errorf("invalid max file size (%s) expected format: \"analyzerType:maxFileSize\" e.g. \"jar:5mb secret:5kb\"", f)
+		}
+
+		analyzerType := analyzer.Type(s[0])


You added AllAnalyzerTypes.
We can currently check analyzerType

Suggested change

analyzerType := analyzer.Type(s[0])

analyzerType := analyzer.Type(s[0])

if !slices.Contains(analyzer.AllAnalyzerTypes(), analyzerType) {

return ScanOptions{}, xerrors.Errorf("invalid analyzer type %q. See analyzer list in %s", analyzerType, doc.URL("/docs/references/analyzers", ""))

}

feat: add --max-file-size flag to skip the files greater than a parti…

1aa5158

…cular size

zesiar0 requested review from simar7, nikpivkin, knqyf263 and DmitriyLewen as code owners August 4, 2024 10:01

nikpivkin reviewed Aug 5, 2024

View reviewed changes

DmitriyLewen reviewed Aug 19, 2024

View reviewed changes

update logic

e713e01

DmitriyLewen reviewed Aug 26, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(fs): add --max-file-size flag to skip the files greater than a particular size #7304

feat(fs): add --max-file-size flag to skip the files greater than a particular size #7304

zesiar0 commented Aug 4, 2024

CLAassistant commented Aug 4, 2024 •

edited

Loading

CLAassistant commented Aug 4, 2024

nikpivkin Aug 5, 2024 •

edited

Loading

knqyf263 Aug 5, 2024

DmitriyLewen Aug 26, 2024

nikpivkin Aug 5, 2024

nikpivkin Aug 5, 2024

nikpivkin commented Aug 5, 2024

nikpivkin Aug 5, 2024

nikpivkin commented Aug 5, 2024

DmitriyLewen Aug 19, 2024

DmitriyLewen Aug 19, 2024

knqyf263 Aug 21, 2024

DmitriyLewen Aug 26, 2024

DmitriyLewen Aug 26, 2024

knqyf263 Aug 29, 2024

DmitriyLewen Aug 26, 2024

DmitriyLewen Aug 26, 2024

DmitriyLewen Aug 26, 2024

		@@ -0,0 +1,68 @@

		## Analyzers
		Use the following types of analyzers for the [--max-file-size] flags

	func (f *ScanFlagGroup) ToOptions(args []string) (ScanOptions, error) {
	if err := parseFlags(f); err != nil {
	return ScanOptions{}, err
	}

	var target string
	if len(args) == 1 {
	target = args[0]
	}

	parallel := f.Parallel.Value()
	if f.Parallel != nil && parallel == 0 {
	log.Info("Set '--parallel' to the number of CPUs", log.Int("parallel", runtime.NumCPU()))
	parallel = runtime.NumCPU()
	}

	return ScanOptions{
	Target: target,
	SkipDirs: f.SkipDirs.Value(),
	SkipFiles: f.SkipFiles.Value(),
	OfflineScan: f.OfflineScan.Value(),
	Scanners: xstrings.ToTSlice[types.Scanner](f.Scanners.Value()),
	FilePatterns: f.FilePatterns.Value(),
	MaxFileSize: f.MaxFileSize.Value(),
	Parallel: parallel,
	SBOMSources: f.SBOMSources.Value(),
	RekorURL: f.RekorURL.Value(),
	DetectionPriority: ftypes.DetectionPriority(f.DetectionPriority.Value()),
	}, nil
	}

	for _, f := range f.MaxFileSize.Value() {
	for _, fileSize := range f.MaxFileSize.Value() {

feat(fs): add --max-file-size flag to skip the files greater than a particular size #7304

Are you sure you want to change the base?

feat(fs): add --max-file-size flag to skip the files greater than a particular size #7304

Conversation

zesiar0 commented Aug 4, 2024

Description

Related issues

CLAassistant commented Aug 4, 2024 • edited Loading

CLAassistant commented Aug 4, 2024

nikpivkin Aug 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nikpivkin commented Aug 5, 2024

Choose a reason for hiding this comment

nikpivkin commented Aug 5, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CLAassistant commented Aug 4, 2024 •

edited

Loading

nikpivkin Aug 5, 2024 •

edited

Loading