Revert "Disable AOSC for permanent HTTP/500"

[xtexChooser]

This reverts commit 024146e.

The issue should be fixed by AOSC-Dev/packages-site-rs#9.

[AMDmi3]

• Repository data misses upstream download URLs, these are required.
• Repository data is broken for most packages (app-utils/wtf for instance), having empty category and section fields, which does not allow to construct links to package sources, this is fatal.

[xtexChooser]

What is the "upstream download URLs"? Do you mean the upstream source of the package (e.g. "github.com/xxxx/xxx-software") or the URL on packages.aosc.io (e.g. "https://packages.aosc.io/packages/xxxx")?

[AMDmi3]

What is the "upstream download URLs"? Do you mean the upstream source of the package (e.g. "github.com/xxxx/xxx-software")

That. E.g., the contents of SRCS from spec files, after interpolation.

or the URL on packages.aosc.io (e.g. "https://packages.aosc.io/packages/xxxx")?

Repology constructs these links itself using category, section and directory fields from json.

[xtexChooser]

We have some packages with SRCS directly pointing to a git repository (the build system will partially clone those repository and checkout a given version), what should the download URL be like (I think there needs some way to include the tag in the URL)?

[Cyanoxygen]

Hi, the empty section field is because our package site was just refactored using Rust, and some of the metadata are missing as the corresponding package is not updated or touched after the refactoring.

I will try to fill out the metadata as soon as possible.

[Cyanoxygen]

We have some packages with SRCS directly points to a git repository (the build system will partial clone those repository and checkout a given version), what should the download URL be like (I think there needs some way to include the tag in the URL)?

Since our spec file uses a custom format to specify the files to be downloaded or git repositories to be cloned, it requires extra work to extract these URLs (and to make the problem worse, we use various Bash parameter expansions within SRCS fields).

Is there a way Repology can extract these source URLs? Or should we provide the fully-expanded URLs via the package site?

[AMDmi3]

Since our spec file uses a custom format to specify the files to be downloaded or git repositories to be cloned, it requires extra work to extract these URLs (and to make the problem worse, we use various Bash parameter expansions within SRCS fields).

Is there a way Repology can extract these source URLs?

If you mean support for custom schemas like git:: and tbl:: (which is the difference between tbl::<url> and plain url btw?), I think I can handle this on Repologys side, but I do not plan to support something like for each package entry in json, do a HTTP request, or go to git repo and get extra data from the corresponding spec file as I cannot afford maintaining non-trivial parsers (for there are about hundred parsers to maintain) and I will not run any third party code (which is impiled by running specs through bash).

Upstream URLs are required to tell equally named projects from each other, and there are a lot (about a hundred) of these in AOSC.

Or should we provide the fully-expanded URLs via the package site?

I expect these URLs in the json dump.

Just my 2¢, if you generate json by processing each package individually, you could probably just do e.g. (cat $spec; echo 'echo $SRCS') | bash, this seems to produce expected result.

[Cyanoxygen]

Hi,

I cannot afford maintaining non-trivial parsers (for there are about hundred parsers to maintain) and I will not run any third party code (which is impiled by running specs through bash).

Yes, you don't have to do that. It is too complex to parse it yourself, and some of the source specifications have extra options in them. But I think that:

I expect these URLs in the json dump.

we do have links to the source tarball in our API:

 curl https://packages.aosc.io/packages/firefox?type=json | jq ".srcurl"
"https://archive.mozilla.org/pub/firefox/releases/128.0.3/source/firefox-128.0.3.source.tar.xz"

And for Git repositories:

curl https://packages.aosc.io/packages/pipewire?type=json | jq ".srcurl"
"https://gitlab.freedesktop.org/pipewire/pipewire"

[Cyanoxygen]

And we have packages with dummy sources for 1) metapackages, 2) transitional packages and 3) some applications that we do use daily but they are not redistributable so we have to add a dummy package to inform the user about that, and the actual package will be downloaded during postinst - one of these packages is vscode. For those packages, they have DUMMYSRC=1 in their spec file, and srcurls in their JSON dumps are empty.

[AMDmi3]

we do have links to the source tarball in our API:

Repology uses https://packages.aosc.io/list.json and only that, it lacks these urls.

And we have packages with dummy sources for 1) metapackages, 2) transitional packages and 3) some applications that we do use daily but they are not redistributable so we have to add a dummy package to inform the user about that, and the actual package will be downloaded during postinst - one of these packages is vscode. For those packages, they have DUMMYSRC=1 in their spec file, and srcurls in their JSON dumps are empty.

That's ok, but it would also be nice to have metapackages and transitional packages marked in some way in json so repology could exclude these.

[xtexChooser]

I made AOSC-Dev/packages-site-rs#10 to include the source URLs in the list.json. However, because some packages have different source URLs for different architectures, like vscode, they will have an empty source URL in responses.

[AMDmi3]

srcurls have appeared, but empty categories and sections are still there. When these are fixed we're ready to go.

[Cyanoxygen]

srcurls have appeared, but empty categories and sections are still there. When these are fixed we're ready to go.

Hi, I have just committed the database backing our package site, packages with empty category and section are filled. Please check.

[AMDmi3]

There are still 42 with emmpty category/section:

"eg25-manager"
"firmware-xiaomi-wt88047"
"linux-kernel-asahi-5.18.0"
"linux-kernel-lts-6.6.31"
"linux-kernel-msm8916-6.3.0"
"linux+kernel+pinephone"
"linux-kernel-pinephone-5.18.4"
"linux+kernel+rk64"
"linux-kernel-rk64-5.18.0-rc7"
"linux+kernel+rpi64"
"linux-kernel-rpi64-5.18.9"
"linux+kernel+rpi64+lts"
"linux-kernel-rpi64-lts-5.15.50"
"linux+kernel+sunxi64"
"linux-kernel-sunxi64-5.12.13"
"msbuild"
"qmic"
"qrtr"
"rmtfs"
"rpi-firmware-boot"
"rpi-userland"
"rtl8723bt-firmware"
"u-boot-aosc-utils"
"u-boot-rk3328-rock64"
"u-boot-rk3399-nanopc-t4"
"u-boot-rk3399-pinebook-pro"
"u-boot-rk3399-rockpro64"
"u-boot-rk3566-quartz64-a"
"u-boot-sun50i-a64-bananapi-m64"
"u-boot-sun50i-a64-pine64-plus"
"u-boot-sun50i-a64-pinebook"
"u-boot-sun50i-a64-pinephone-1.1"
"u-boot-sun50i-a64-pinephone-1.2"
"u-boot-sun50i-a64-pinetab-early-adopter"
"u-boot-sun50i-a64-sopine-baseboard"
"u-boot-sun50i-a64-teres-i"
"u-boot-sun50i-h5-libretech-all-h3-cc"
"u-boot-sun50i-h5-nanopi-neo2"
"u-boot-sun50i-h5-orangepi-pc2"
"u-boot-sun50i-h5-orangepi-prime"
"u-boot-sun50i-h6-pine-h64"
"u-boot-sun50i-h6-pine-h64-model-b"

[Cyanoxygen]

There are still 42 with emmpty category/section:

"eg25-manager"

"firmware-xiaomi-wt88047"

"linux-kernel-asahi-5.18.0"

"linux-kernel-lts-6.6.31"

"linux-kernel-msm8916-6.3.0"

"linux+kernel+pinephone"

"linux-kernel-pinephone-5.18.4"

"linux+kernel+rk64"

"linux-kernel-rk64-5.18.0-rc7"

"linux+kernel+rpi64"

"linux-kernel-rpi64-5.18.9"

"linux+kernel+rpi64+lts"

"linux-kernel-rpi64-lts-5.15.50"

"linux+kernel+sunxi64"

"linux-kernel-sunxi64-5.12.13"

"msbuild"

"qmic"

"qrtr"

"rmtfs"

"rpi-firmware-boot"

"rpi-userland"

"rtl8723bt-firmware"

"u-boot-aosc-utils"

"u-boot-rk3328-rock64"

"u-boot-rk3399-nanopc-t4"

"u-boot-rk3399-pinebook-pro"

"u-boot-rk3399-rockpro64"

"u-boot-rk3566-quartz64-a"

"u-boot-sun50i-a64-bananapi-m64"

"u-boot-sun50i-a64-pine64-plus"

"u-boot-sun50i-a64-pinebook"

"u-boot-sun50i-a64-pinephone-1.1"

"u-boot-sun50i-a64-pinephone-1.2"

"u-boot-sun50i-a64-pinetab-early-adopter"

"u-boot-sun50i-a64-sopine-baseboard"

"u-boot-sun50i-a64-teres-i"

"u-boot-sun50i-h5-libretech-all-h3-cc"

"u-boot-sun50i-h5-nanopi-neo2"

"u-boot-sun50i-h5-orangepi-pc2"

"u-boot-sun50i-h5-orangepi-prime"

"u-boot-sun50i-h6-pine-h64"

"u-boot-sun50i-h6-pine-h64-model-b"

They are considered "out-of-tree" (not part of AOSC OS ABBS), and some of them are dropped from the tree such as MSBuild (it will get revived). I can clear them out, but I would like to ask for other maintainers. @MingcongBai Can we remove these entries (except MSBuild)?

[MingcongBai]

There are still 42 with emmpty category/section:

"eg25-manager"

"firmware-xiaomi-wt88047"

"linux-kernel-asahi-5.18.0"

"linux-kernel-lts-6.6.31"

"linux-kernel-msm8916-6.3.0"

"linux+kernel+pinephone"

"linux-kernel-pinephone-5.18.4"

"linux+kernel+rk64"

"linux-kernel-rk64-5.18.0-rc7"

"linux+kernel+rpi64"

"linux-kernel-rpi64-5.18.9"

"linux+kernel+rpi64+lts"

"linux-kernel-rpi64-lts-5.15.50"

"linux+kernel+sunxi64"

"linux-kernel-sunxi64-5.12.13"

"msbuild"

"qmic"

"qrtr"

"rmtfs"

"rpi-firmware-boot"

"rpi-userland"

"rtl8723bt-firmware"

"u-boot-aosc-utils"

"u-boot-rk3328-rock64"

"u-boot-rk3399-nanopc-t4"

"u-boot-rk3399-pinebook-pro"

"u-boot-rk3399-rockpro64"

"u-boot-rk3566-quartz64-a"

"u-boot-sun50i-a64-bananapi-m64"

"u-boot-sun50i-a64-pine64-plus"

"u-boot-sun50i-a64-pinebook"

"u-boot-sun50i-a64-pinephone-1.1"

"u-boot-sun50i-a64-pinephone-1.2"

"u-boot-sun50i-a64-pinetab-early-adopter"

"u-boot-sun50i-a64-sopine-baseboard"

"u-boot-sun50i-a64-teres-i"

"u-boot-sun50i-h5-libretech-all-h3-cc"

"u-boot-sun50i-h5-nanopi-neo2"

"u-boot-sun50i-h5-orangepi-pc2"

"u-boot-sun50i-h5-orangepi-prime"

"u-boot-sun50i-h6-pine-h64"

"u-boot-sun50i-h6-pine-h64-model-b"

They are considered "out-of-tree" (not part of AOSC OS ABBS), and some of them are dropped from the tree such as MSBuild (it will get revived). I can clear them out, but I would like to ask for other maintainers. @MingcongBai Can we remove these entries (except MSBuild)?

Fair enough, msbuild is already back, by the way.