Skip to content

Migrate Alpine importer to advisory V2#2111

Merged
ziadhany merged 8 commits intoaboutcode-org:mainfrom
ziadhany:alpine-migrate
Feb 10, 2026
Merged

Migrate Alpine importer to advisory V2#2111
ziadhany merged 8 commits intoaboutcode-org:mainfrom
ziadhany:alpine-migrate

Conversation

@ziadhany
Copy link
Collaborator

@ziadhany ziadhany commented Jan 8, 2026 •
edited
Loading

@ziadhany
Copy link
Collaborator Author

ziadhany commented Jan 8, 2026 •
edited
Loading 

Importing data using alpine_linux_importer_v2
INFO 2026-02-10 18:01:50.339169 UTC Pipeline [AlpineLinuxImporterPipeline] starting
INFO 2026-02-10 18:01:50.339308 UTC Step [clone] starting
INFO 2026-02-10 18:01:50.339358 UTC Cloning `git+https://github.com/aboutcode-org/aboutcode-mirror-alpine-secdb/`
INFO 2026-02-10 18:01:52.503890 UTC Step [clone] completed in 2 seconds
INFO 2026-02-10 18:01:52.504022 UTC Step [collect_and_store_advisories] starting
INFO 2026-02-10 19:00:33.772204 UTC Successfully collected 104,566 advisories
INFO 2026-02-10 19:00:33.772337 UTC Step [collect_and_store_advisories] completed in 3521 seconds (58.7 minutes)
INFO 2026-02-10 19:00:33.772390 UTC Pipeline completed in 3523 seconds (58.7 minutes)

Process finished with exit code 0

from vulnerabilities.models import AdvisoryV2
from django.db.models import Count
duplicates = (
    AdvisoryV2.objects
    .values('avid')
    .annotate(count=Count('id'))
    .filter(count__gt=1)
)
len(duplicates)
Out[2]: 0
AdvisoryV2.objects.count()
Out[3]: 104531

@ziadhany
Copy link
Collaborator Author

ziadhany commented Jan 15, 2026 •
edited
Loading

@TG1999 @pombredanne I have a question about Alpine migration. We are fetching one URL and processing the data without grouping by CVE.

The problem is that each URL reports a package version along with its fixed CVEs. How can we obtain a unique identifier for this importer? Is it a good idea to restructure the data and create a large mapping, using the CVE as the unique identifier?

Proposed structure:
CVE: [purl_1, purl_2, ...]

Example:
Package: aom

Sources:
https://secdb.alpinelinux.org/v3.22/main.json -> CVEs: "CVE-2021-30473", "CVE-2021-30474", "CVE-2021-30475"
https://secdb.alpinelinux.org/v3.21/main.json -> CVEs: "CVE-2021-30473", "CVE-2021-30474", "CVE-2021-30475"

ziadhany
ziadhany commented Jan 26, 2026
View reviewed changes
vulnerabilities/pipelines/v2_importers/alpine_linux_importer.py Outdated
)

for cve in aliases:
advisory_id = f"{pkg_infos['name']}/{qualifiers['distroversion']}/{cve}"
Copy link
Collaborator Author

@ziadhany ziadhany Jan 26, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ex:

alpine_linux_importer_v2/aardvark-dns/edge/1.12.2-r0/CVE-2024-8418

@ziadhany
Copy link
Collaborator Author

ziadhany commented Jan 28, 2026 •
edited
Loading

The logs in debug mode:

alpine.zip

@ziadhany ziadhany requested a review from keshav-space January 28, 2026 13:50
keshav-space
keshav-space requested changes Jan 29, 2026
View reviewed changes
Copy link
Member

@keshav-space keshav-space left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ziadhany, see comments below.

@ziadhany ziadhany requested a review from keshav-space February 9, 2026 12:55
keshav-space
keshav-space requested changes Feb 10, 2026
View reviewed changes
Copy link
Member

@keshav-space keshav-space left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ziadhany, see few comments below. And let’s get rid of the optional logger in helper functions. There is no point in having logger as optional when we are already passing self.log, this only makes the code less readable.

@ziadhany
Migrate Alpine importer to advisory V2
547d742 
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
@ziadhany
Fix typo Alpine Linux importer pipeline inherits from VulnerableCodeB…
bba1080 
…aseImporterPipelineV2

Signed-off-by: ziad hany <ziadhany2016@gmail.com>
@ziadhany
Update Alpine to avoid fetching the same URL links multiple times
86a1c65 
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
@ziadhany
Update alpine linux so for every vulnerability id AdvisoryData
fdfef36 
Fix duplication on advisory_id

Signed-off-by: ziad hany <ziadhany2016@gmail.com>
@ziadhany
Update Alpine pipeline to use aboutcode-mirror-alpine-secdb
8da445f 
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
@ziadhany
Update advisories_count function
eff2f9b 
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
@ziadhany
Update Alpine Linux to use the new AdvisoryDataV2
d61a505 
Update the logger to prevent None from being used as a default value
Move the tests to a new file expected-advisories-v3.3.json

Signed-off-by: ziad hany <ziadhany2016@gmail.com>
keshav-space
keshav-space approved these changes Feb 10, 2026
View reviewed changes
Copy link
Member

@keshav-space keshav-space left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ziadhany, LGTM!

@ziadhany ziadhany merged commit 376a4d5 into aboutcode-org:main Feb 10, 2026
4 of 5 checks passed
@ziadhany ziadhany deleted the alpine-migrate branch February 10, 2026 19:26
@ziadhany ziadhany mentioned this pull request Feb 10, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@keshav-space keshav-space keshav-space approved these changes

@TG1999 TG1999 Awaiting requested review from TG1999

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ziadhany @keshav-space