nixpkgs-clarity: Add on-demand indexing for Nixpkgs in PurlDB

[pombredanne]

Working code for on-demand indexing and crawling for all Nixpkgs or a select subset of these. Determine and index the corresponding source code repositories as needed using ecosystem conventions.

[chinyeungli]

should we display source download archives for Nix packages?

Currently, I get only the binary NAR archives (the Nix store outputs) as shown below (http://127.0.0.1:8001/api/collect/?purl=pkg:nix/nixpkgs/Axel)

[
    {
        "url": "http://127.0.0.1:8001/api/packages/2cd5621d-e951-47b9-8f6a-68405c5efe03/",
        "uuid": "2cd5621d-e951-47b9-8f6a-68405c5efe03",
        "filename": "0i3yfkkdfl7kjrcpa9r7p6w3agqmkb18qb2za2wmpfvbq1fzzdxh.nar.zst",
        "package_sets": [
            {
                "uuid": "cd80b38a-b189-4e33-9267-590d6764c76b",
                "packages": [
                    "http://127.0.0.1:8001/api/packages/f83f18d6-b1d0-4558-b868-195275c52ff4/",
                    "http://127.0.0.1:8001/api/packages/2cd5621d-e951-47b9-8f6a-68405c5efe03/",
                    "http://127.0.0.1:8001/api/packages/60cedddd-6aa8-44cb-a9ac-8d6f6b4abd34/"
                ]
            }
        ],
        "package_content": "binary",
        "purl": "pkg:nix/nixpkgs/Axel@2.17.14?commit=3d46470bb3030020f7e1361f33514854f5bfa86d&output=out&system=aarch64-darwin",
        "type": "nix",
        "namespace": "nixpkgs",
        "name": "Axel",
        "version": "2.17.14",
        "qualifiers": "commit=3d46470bb3030020f7e1361f33514854f5bfa86d&output=out&system=aarch64-darwin",
        "subpath": "",
        "primary_language": null,
        "description": "Console downloading program with some features for parallel connections for faster downloading",
        "release_date": "2026-06-27T07:37:20Z",
        "parties": [],
        "keywords": [],
        "homepage_url": "https://github.com/axel-download-accelerator/axel",
        "download_url": "https://cache.nixos.org/nar/0i3yfkkdfl7kjrcpa9r7p6w3agqmkb18qb2za2wmpfvbq1fzzdxh.nar.zst",
        "bug_tracking_url": null,
        "code_view_url": null,
        "vcs_url": null,
        "repository_homepage_url": null,
        "repository_download_url": null,
        "api_data_url": null,
        "size": null,
        "md5": null,
        "sha1": null,
        "sha256": null,
        "sha512": null,
        "copyright": null,
        "holder": null,
        "declared_license_expression": "gpl-2.0-plus",
        "declared_license_expression_spdx": "GPL-2.0-or-later",
        "license_detections": [
            {
                "matches": [
                    {
                        "score": 50.0,
                        "matcher": "1-hash",
                        "end_line": 1,
                        "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/spdx_license_id_gpl-2.0-or-later_for_gpl-2.0-plus.RULE",
                        "from_file": null,
                        "start_line": 1,
                        "matched_text": "GPL-2.0-or-later",
                        "match_coverage": 100.0,
                        "matched_length": 5,
                        "rule_relevance": 50,
                        "rule_identifier": "spdx_license_id_gpl-2.0-or-later_for_gpl-2.0-plus.RULE",
                        "license_expression": "gpl-2.0-plus",
                        "license_expression_spdx": "GPL-2.0-or-later"
                    }
                ],
                "identifier": "gpl_2_0_plus-5b1e08b4-d11a-8dda-a3ae-2a8e74495478",
                "license_expression": "gpl-2.0-plus",
                "license_expression_spdx": "GPL-2.0-or-later"
            }
        ],
        "other_license_expression": null,
        "other_license_expression_spdx": null,
        "other_license_detections": [],
        "extracted_license_statement": "- GPL-2.0-or-later\n",
        "notice_text": null,
        "source_packages": [],
        "extra_data": {},
        "package_uid": "pkg:nix/nixpkgs/Axel@2.17.14?commit=3d46470bb3030020f7e1361f33514854f5bfa86d&output=out&system=aarch64-darwin&uuid=2cd5621d-e951-47b9-8f6a-68405c5efe03",
        "datasource_id": null,
        "file_references": [],
        "dependencies": [],
        "resources": "http://127.0.0.1:8001/api/packages/2cd5621d-e951-47b9-8f6a-68405c5efe03/resources/",
        "history": "http://127.0.0.1:8001/api/packages/2cd5621d-e951-47b9-8f6a-68405c5efe03/history/"
    },
    {
        "url": "http://127.0.0.1:8001/api/packages/f10ebc00-5a25-4026-bdaf-9e385db029e1/",
        "uuid": "f10ebc00-5a25-4026-bdaf-9e385db029e1",
        "filename": "1kfvy3vpv4gvirpknjkdjjz5ppcm653fcaxypyq37kp847c57shm.nar.xz",
        "package_sets": [
            {
                "uuid": "1cd729d1-ce79-4ef6-8bc9-f4de7acacb1a",
                "packages": [
                    "http://127.0.0.1:8001/api/packages/f83f18d6-b1d0-4558-b868-195275c52ff4/",
                    "http://127.0.0.1:8001/api/packages/f10ebc00-5a25-4026-bdaf-9e385db029e1/",
                    "http://127.0.0.1:8001/api/packages/60cedddd-6aa8-44cb-a9ac-8d6f6b4abd34/"
                ]
            }
        ],
        "package_content": "binary",
        "purl": "pkg:nix/nixpkgs/Axel@2.17.14?commit=3d46470bb3030020f7e1361f33514854f5bfa86d&output=out&system=aarch64-linux",
        "type": "nix",
        "namespace": "nixpkgs",
        "name": "Axel",
        "version": "2.17.14",
        "qualifiers": "commit=3d46470bb3030020f7e1361f33514854f5bfa86d&output=out&system=aarch64-linux",
        "subpath": "",
        "primary_language": null,
        "description": "Console downloading program with some features for parallel connections for faster downloading",
        "release_date": "2026-06-27T07:37:20Z",
        "parties": [],
        "keywords": [],
        "homepage_url": "https://github.com/axel-download-accelerator/axel",
        "download_url": "https://cache.nixos.org/nar/1kfvy3vpv4gvirpknjkdjjz5ppcm653fcaxypyq37kp847c57shm.nar.xz",
        "bug_tracking_url": null,
        "code_view_url": null,
        "vcs_url": null,
        "repository_homepage_url": null,
        "repository_download_url": null,
        "api_data_url": null,
        "size": null,
        "md5": null,
        "sha1": null,
        "sha256": null,
        "sha512": null,
        "copyright": null,
        "holder": null,
        "declared_license_expression": "gpl-2.0-plus",
        "declared_license_expression_spdx": "GPL-2.0-or-later",
        "license_detections": [
            {
                "matches": [
                    {
                        "score": 50.0,
                        "matcher": "1-hash",
                        "end_line": 1,
                        "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/spdx_license_id_gpl-2.0-or-later_for_gpl-2.0-plus.RULE",
                        "from_file": null,
                        "start_line": 1,
                        "matched_text": "GPL-2.0-or-later",
                        "match_coverage": 100.0,
                        "matched_length": 5,
                        "rule_relevance": 50,
                        "rule_identifier": "spdx_license_id_gpl-2.0-or-later_for_gpl-2.0-plus.RULE",
                        "license_expression": "gpl-2.0-plus",
                        "license_expression_spdx": "GPL-2.0-or-later"
                    }
                ],
                "identifier": "gpl_2_0_plus-5b1e08b4-d11a-8dda-a3ae-2a8e74495478",
                "license_expression": "gpl-2.0-plus",
                "license_expression_spdx": "GPL-2.0-or-later"
            }
        ],
        "other_license_expression": null,
        "other_license_expression_spdx": null,
        "other_license_detections": [],
        "extracted_license_statement": "- GPL-2.0-or-later\n",
        "notice_text": null,
        "source_packages": [],
        "extra_data": {},
        "package_uid": "pkg:nix/nixpkgs/Axel@2.17.14?commit=3d46470bb3030020f7e1361f33514854f5bfa86d&output=out&system=aarch64-linux&uuid=f10ebc00-5a25-4026-bdaf-9e385db029e1",
        "datasource_id": null,
        "file_references": [],
        "dependencies": [],
        "resources": "http://127.0.0.1:8001/api/packages/f10ebc00-5a25-4026-bdaf-9e385db029e1/resources/",
        "history": "http://127.0.0.1:8001/api/packages/f10ebc00-5a25-4026-bdaf-9e385db029e1/history/"
    },

In addition, I have code to get the source tarball URLs for the package:

https://github.com/axel-download-accelerator/axel/archive/v2.17.14.tar.gz
pkg:nix/nixpkgs/Axel@2.17.14

https://github.com/axel-download-accelerator/axel/archive/v2.17.13.tar.gz
pkg:nix/nixpkgs/Axel@2.17.13

https://github.com/axel-download-accelerator/axel/archive/v2.17.11.tar.gz
pkg:nix/nixpkgs/Axel@2.17.11

https://github.com/axel-download-accelerator/axel/archive/v2.17.10.tar.gz
pkg:nix/nixpkgs/Axel@2.17.10

https://github.com/axel-download-accelerator/axel/archive/v2.17.9.tar.gz
pkg:nix/nixpkgs/Axel@2.17.9

However, when I try to store the source URLs in download_url, I get the following error:

ERROR Internal Server Error: /api/collect/
Traceback (most recent call last):
  File "/home/thomas/vscode/purldb/venv/lib/python3.11/site-packages/django/db/models/query.py", line 948, in get_or_create
    return self.get(**kwargs), False
           ^^^^^^^^^^^^^^^^^^
  File "/home/thomas/vscode/purldb/venv/lib/python3.11/site-packages/django/db/models/query.py", line 649, in get
    raise self.model.DoesNotExist(
packagedb.models.Package.DoesNotExist: Package matching query does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/thomas/vscode/purldb/venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 105, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/vscode/purldb/venv/lib/python3.11/site-packages/psycopg/cursor.py", line 97, in execute
    raise ex.with_traceback(None)
psycopg.errors.UniqueViolation: duplicate key value violates unique constraint "packagedb_package_download_url_f8544ea7_uniq"
DETAIL:  Key (download_url)=(https://github.com/axel-download-accelerator/axel/archive/v2.17.9.tar.gz) already exists.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/thomas/vscode/purldb/venv/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/vscode/purldb/venv/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.pyenv/versions/3.11.8/lib/python3.11/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/vscode/purldb/venv/lib/python3.11/site-packages/django/views/decorators/csrf.py", line 65, in _view_wrapper
    return view_func(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/vscode/purldb/venv/lib/python3.11/site-packages/rest_framework/viewsets.py", line 125, in view
    return self.dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/vscode/purldb/venv/lib/python3.11/site-packages/rest_framework/views.py", line 515, in dispatch
    response = self.handle_exception(exc)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/vscode/purldb/venv/lib/python3.11/site-packages/rest_framework/views.py", line 475, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/home/thomas/vscode/purldb/venv/lib/python3.11/site-packages/rest_framework/views.py", line 486, in raise_uncaught_exception
    raise exc
  File "/home/thomas/vscode/purldb/venv/lib/python3.11/site-packages/rest_framework/views.py", line 512, in dispatch
    response = handler(request, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/vscode/purldb/packagedb/api.py", line 852, in list
    get_source_package_and_add_to_package_set(package)
  File "/home/thomas/vscode/purldb/purl2vcs/src/purl2vcs/find_source_repo.py", line 157, in get_source_package_and_add_to_package_set
    source_package, _created = Package.objects.get_or_create(
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/vscode/purldb/venv/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/vscode/purldb/venv/lib/python3.11/site-packages/django/db/models/query.py", line 955, in get_or_create
    return self.create(**params), True
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/vscode/purldb/venv/lib/python3.11/site-packages/django/db/models/query.py", line 679, in create
    obj.save(force_insert=True, using=self.db)
  File "/home/thomas/vscode/purldb/venv/lib/python3.11/site-packages/django/db/models/base.py", line 892, in save
    self.save_base(
  File "/home/thomas/vscode/purldb/venv/lib/python3.11/site-packages/django/db/models/base.py", line 998, in save_base
    updated = self._save_table(
              ^^^^^^^^^^^^^^^^^
  File "/home/thomas/vscode/purldb/venv/lib/python3.11/site-packages/django/db/models/base.py", line 1161, in _save_table
    results = self._do_insert(
              ^^^^^^^^^^^^^^^^
  File "/home/thomas/vscode/purldb/venv/lib/python3.11/site-packages/django/db/models/base.py", line 1202, in _do_insert
    return manager._insert(
           ^^^^^^^^^^^^^^^^
  File "/home/thomas/vscode/purldb/venv/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/vscode/purldb/venv/lib/python3.11/site-packages/django/db/models/query.py", line 1847, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/vscode/purldb/venv/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1836, in execute_sql
    cursor.execute(sql, params)
  File "/home/thomas/vscode/purldb/venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 79, in execute
    return self._execute_with_wrappers(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/vscode/purldb/venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 92, in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/vscode/purldb/venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 100, in _execute
    with self.db.wrap_database_errors:
  File "/home/thomas/vscode/purldb/venv/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/thomas/vscode/purldb/venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 105, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/vscode/purldb/venv/lib/python3.11/site-packages/psycopg/cursor.py", line 97, in execute
    raise ex.with_traceback(None)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "packagedb_package_download_url_f8544ea7_uniq"
DETAIL:  Key (download_url)=(https://github.com/axel-download-accelerator/axel/archive/v2.17.9.tar.gz) already exists.
ERROR "GET /api/collect/?purl=pkg:nix/nixpkgs/Axel HTTP/1.1" 500 145

I think the error basically means the value of the download_url already exist in the database with other purl, and we cannot have the same download_url with different purl.

So, if this is the case, for nix, we won't display the source entries, correct? i.e. we won't display the source entry along with the source download_url ?

[chinyeungli]

Jono Yang  [6:24 AM]
it would be good to add them if the source packages are 1 to 1 with the binary packages

Chin Yeung Li  [6:52 AM]
The issue is that Nix packages are not 1 to 1 with upstream source packages.

A Nix package almost always fetches its upstream source (GitHub, GitLab, PyPI, crates.io, SourceForge, etc.) and then applies patches/configuration defined in the Nix expression. Nixpkgs itself does not host source archives.

In other words, the “actual” source is pkg:github, pkg:gitlab, pkg:pypi, pkg:crate, etc., depending on where the upstream project lives.

 So I guess adding a “source” entry in the PURL doesn’t make sense in most cases?

Jono Yang  [6:55 AM]
i would focus on the binary packages from nixpkgs for now

[chinyeungli]

PR created: #895

[chinyeungli]

For references, nix packages can be found at https://search.nixos.org/packages