Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: enable repo finder to support more languages via Open Source Insights #388

Merged
merged 31 commits into from
Sep 21, 2023
Merged

feat: enable repo finder to support more languages via Open Source Insights #388

merged 31 commits into from
Sep 21, 2023

Conversation

benmss
Copy link
Member

@benmss benmss commented Jul 28, 2023

The repo finder is refactored to:

  • be usable from anywhere within Macaron
  • accept a PURL string as input
  • support more languages via Open Source Insights (deps.dev)

Added languages include: Python, NodeJS, .Net, Rust.
Note that these additions won't do anything until dependency analysers for these languages are added.

Use of Open Source Insights can be disabled via a configuration value.

@oracle-contributor-agreement oracle-contributor-agreement bot added the OCA Verified All contributors have signed the Oracle Contributor Agreement. label Jul 28, 2023
@benmss benmss changed the base branch from main to staging July 28, 2023 02:08
@benmss benmss changed the title Repo finder refactor feat: enable repo finder to support more languages via Open Source Insights Jul 28, 2023
@benmss benmss marked this pull request as ready for review July 28, 2023 02:58
@benmss benmss requested review from behnazh-w and tromai as code owners July 28, 2023 02:58
@behnazh-w behnazh-w requested a review from nicallen July 30, 2023 22:14
behnazh-w
behnazh-w reviewed Aug 4, 2023
View reviewed changes
src/macaron/repo_finder/repo_finder_dd.py Outdated Show resolved Hide resolved
src/macaron/repo_finder/repo_finder_dd.py Outdated Show resolved Hide resolved
tromai
tromai reviewed Aug 8, 2023
View reviewed changes
src/macaron/__main__.py Outdated Show resolved Hide resolved
tromai
tromai reviewed Aug 8, 2023
View reviewed changes
src/macaron/__main__.py Outdated Show resolved Hide resolved
@benmss benmss linked an issue Aug 23, 2023 that may be closed by this pull request
Repo Finder Refactoring #428
Closed
@benmss benmss added the repository_finder The issues related to the repository finder label Aug 23, 2023
@benmss benmss force-pushed the repo-finder-refactor branch 3 times, most recently from f7fadaa to 57d13c9 Compare August 31, 2023 02:07
@benmss benmss requested a review from nathanwn August 31, 2023 06:36
behnazh-w
behnazh-w reviewed Aug 31, 2023
View reviewed changes
src/macaron/dependency_analyzer/cyclonedx.py Outdated Show resolved Hide resolved
src/macaron/dependency_analyzer/dependency_resolver.py Outdated
@@ -31,9 +29,13 @@ class DependencyTools(str, Enum):
class DependencyInfo(TypedDict):
"""The information of a resolved dependency."""

version: str
group: str
scheme: str
Copy link
Member

@behnazh-w behnazh-w Aug 31, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of duplicating the PURL spec here, wouldn't it be better to store the PURL string directly, which will be parsed using the packageurl module later before analyzing the dependency?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we store the PURL string as well as the old information (in their fields), we have duplicates. If we don't keep the old information, we have to create the PURL from the string every time we want to extract any information from it. Perhaps storing the actual PURL would make the most sense?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with the option of storing the actual PURL object.

Copy link
Member

@behnazh-w behnazh-w Sep 5, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Storing a PackageURL object sounds good to me.

benmss added 22 commits September 21, 2023 14:37
@benmss
chore: minor fix
d274dd0 
Signed-off-by: Ben Selwyn-Smith <[email protected]>
@benmss
chore: extended docstring of repo finder
0a7096f 
Signed-off-by: Ben Selwyn-Smith <[email protected]>
@benmss
chore: Addressed PR feedback.
0a8b665 
- Moved dependency resolution functions to dependency_resolver.py
- Moved find_valid_url to repo_finder to avoid circular dependencies
- Added URL for deps.dev API documentation as comment

Signed-off-by: Ben Selwyn-Smith <[email protected]>
@benmss
chore: moved repo finder integration test to new file
7745019 
Signed-off-by: Ben Selwyn-Smith <[email protected]>
@benmss
chore: try to derive the SBOM component type
60a5cc7 
Signed-off-by: Ben Selwyn-Smith <[email protected]>
@benmss
chore: Add PURL to DependencyInfo; Try to retrieve PURL from SBOM for…
64a7472 
… use with Repo Finder.

Signed-off-by: Ben Selwyn-Smith <[email protected]>
@benmss
chore: renaming of deps.dev files
556514e 
Signed-off-by: Ben Selwyn-Smith <[email protected]>
@benmss
chore: added integration tests for more languages
3ad9c4a 
Signed-off-by: Ben Selwyn-Smith <[email protected]>
@benmss
chore: restored removed test
abcd9a0 
Signed-off-by: Ben Selwyn-Smith <[email protected]>
@benmss
chore: repo finder interface refactoring
011da8b 
Signed-off-by: Ben Selwyn-Smith <[email protected]>
@benmss
chore: updated repo finder return values
3ab5664 
Signed-off-by: Ben Selwyn-Smith <[email protected]>
@benmss
chore: correctly added repo finder integration tests; fixed duplicate…
83be471 
… string replacement in java repo finder

Signed-off-by: Ben Selwyn-Smith <[email protected]>
@benmss
chore: removed repo finder test in docker integration tests
8eae174 
Signed-off-by: Ben Selwyn-Smith <[email protected]>
@benmss
chore: moved URL validation to within Repo Finders
8bce886 
Signed-off-by: Ben Selwyn-Smith <[email protected]>
@benmss
chore: moved url validator to repo finder
4c8cba6 
Signed-off-by: Ben Selwyn-Smith <[email protected]>
@benmss
chore: added repo validator
f0ee636 
Signed-off-by: Ben Selwyn-Smith <[email protected]>
@benmss
chore: updated docs
50137f7 
Signed-off-by: Ben Selwyn-Smith <[email protected]>
@benmss
chore: rebase and integrate with config purl change
9d4f657 
Signed-off-by: Ben Selwyn-Smith <[email protected]>
@benmss
chore: addressed review feedback
b369fe6 
Signed-off-by: Ben Selwyn-Smith <[email protected]>
@benmss
chore: addressed review feedback
11ab3aa 
Signed-off-by: Ben Selwyn-Smith <[email protected]>
@benmss
chore: rebase and make use of updated send_get_http_raw
5b600bf 
Signed-off-by: Ben Selwyn-Smith <[email protected]>
@benmss
chore: enable repo finder for sboms
bca62d5 
Signed-off-by: Ben Selwyn-Smith <[email protected]>
@benmss benmss force-pushed the repo-finder-refactor branch from a8ccd34 to bca62d5 Compare September 21, 2023 04:38
@benmss
chore: updated docs
056b3f3 
Signed-off-by: Ben Selwyn-Smith <[email protected]>
@benmss benmss merged commit bf118b3 into staging Sep 21, 2023
9 checks passed
@tromai tromai deleted the repo-finder-refactor branch September 21, 2023 23:45
tromai added a commit that referenced this pull request Sep 23, 2023
@tromai
chore: revert the changes in DependencyAnalyzer.resolve_dependencies
ebc9696 
These changes were to separate the resolving dependencies using SBOM generator and resolving dependencies
from SBOM files. These changes were necessary for this PR back when I first work on it (before the merging
of #388). However, after #388 is merged and further discussion, this change is no longer necessary.

Signed-off-by: Trong Nhan Mai <[email protected]>
art1f1c3R pushed a commit that referenced this pull request Nov 29, 2024
@benmss
feat: enable repo finder to support more languages via Open Source In…
cb52a9d 
…sights (#388)

This feature modifies the Repo Finder, so that it can: be usable from anywhere within Macaron; accept PURL strings as input; and, support more languages via Google's Open Source Insights (deps.dev)

This enables Macaron to accept artifact PURLs as input, whereby the Repo Finder will be used to attempt to retrieve the related repository. 

Additional languages include those supported by deps.dev: Python, NodeJS, .Net, and Rust. Note that currently these will only work when specifying an artifact PURL as input, or providing an SBOM. Full support for these extra languages will require the addition of new dependency analyzers.

A new config option is also provided to disable API calls to Google's Open Source Insights, if desired.

Signed-off-by: Ben Selwyn-Smith <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nathanwn nathanwn nathanwn approved these changes

@tromai tromai tromai approved these changes

@behnazh-w behnazh-w behnazh-w approved these changes

@nicallen nicallen nicallen approved these changes

Assignees

@benmss benmss

Labels
OCA Verified All contributors have signed the Oracle Contributor Agreement. repository_finder The issues related to the repository finder
Projects
None yet
Milestone
Extend the source repository finder
Development

Successfully merging this pull request may close these issues.

Repo Finder Refactoring
5 participants
@benmss @nicallen @behnazh-w @tromai @nathanwn