maven-heaven: Extend indexing for Apache source and binaries in PurlDB

[pombredanne]

Working code for indexing and crawling for source and binary distributions of Apache projects. Determine and index the corresponding source code repositories from Apache project metadata

[chinyeungli]

@pombredanne what would be the package type for these packages? I assume it should be "maven"?

[chinyeungli]

The official Apache repository is located at https://downloads.apache.org/. However, https://archive.apache.org/dist/ hosts a wider range of packages, including deprecated and older releases. For indexing purposes, I’ll be using the archive repository.

Since the Apache repository doesn’t contain an index file like other repositories such as PyPI or Maven, I won’t use the index‑based approach as in in https://github.com/aboutcode-org/purldb/tree/main/minecode_pipelines/pipelines. Instead, I will use a crawling approach similar to the one in https://github.com/aboutcode-org/purldb/blob/main/minecode/management/commands/maven_crawler.py

The maven_crawler determines whether a page is a package page based on the presence of a maven-metadata.xml file. I tried a similar approach by checking for KEYS files since many package pages include them. However, this approach may not be reliable as not all package pages contain KEYS files, not all directory links within a package page point to actual package versions, and not all archives follow the {package_page}/{version}/ structure.

For example:

• Missing KEYS files on some package pages
  https://archive.apache.org/dist/activemq/activemq-artemis-console/
• Directory links that don’t correspond to package versions
  https://archive.apache.org/dist/activemq/ contains a KEYS file, but also includes the subdirectory activemq-apollo, which is not a version pointer.
• Archives not organized under {package_page}/{version}/ structure
  https://archive.apache.org/dist/perl/

[chinyeungli]

We cannot apply the same logic used in maven_crawler for the following reasons:

• maven_crawler determines the package version from the URL path. This approach is not always valid, as seen in cases like https://archive.apache.org/dist/perl/old/ and https://archive.apache.org/dist/deltacloud/rc1
• maven_crawler assumes that a single package URL corresponds to one package with multiple versions. However, some Apache directories may contain multiple distinct packages under the same URL, such as https://archive.apache.org/dist/devicemap/

[chinyeungli]

I've some workable code that have the following behaviors.
Given a download url to the package archive:

• If the final segment of the parent URL is a version:
  • Treat every segment before that version (and after https://archive.apache.org/dist/) as the namespace, and prefix it with org.apache/.
  • In the filename, treat everything between the version and the file extension as the classifier.
  • A predefined list of common classifiers and artifact extensions is used to validate this.
• If the final segment of the parent URL is not a version:
  • Attempt to extract the version from the filename instead.
  • If no version can be identified, then no version is assigned

Final Segment of the Parent URL is a Version

Download URL	PURL
https://archive.apache.org/dist/activemq/5.15.3/activemq-parent-5.15.3-source-release.zip	pkg:maven/org.apache/activemq/activemq-parent@5.15.3?classifier=source-release&download_url=https://archive.apache.org/dist/activemq/5.15.3/activemq-parent-5.15.3-source-release.zip
https://archive.apache.org/dist/activemq/5.15.3/apache-activemq-5.15.3-bin.tar.gz	pkg:maven/org.apache/activemq/apache-activemq@5.15.3?classifier=bin&download_url=https://archive.apache.org/dist/activemq/5.15.3/apache-activemq-5.15.3-bin.tar.gz
https://archive.apache.org/dist/activemq/5.15.3/apache-activemq-5.15.3-bin.zip	pkg:maven/org.apache/activemq/apache-activemq@5.15.3?classifier=bin&download_url=https://archive.apache.org/dist/activemq/5.15.3/apache-activemq-5.15.3-bin.zip
https://archive.apache.org/dist/activemq/activemq-apollo/1.0-beta1/apache-apollo-1.0-beta1-unix-distro.tar.gz	pkg:maven/org.apache/activemq/activemq-apollo/apache-apollo@1.0-beta1?classifier=unix-distro&download_url=https://archive.apache.org/dist/activemq/activemq-apollo/1.0-beta1/apache-apollo-1.0-beta1-unix-distro.tar.gz
https://archive.apache.org/dist/activemq/activemq-apollo/1.0-beta1/apache-apollo-1.0-beta1-windows-distro.zip	pkg:maven/org.apache/activemq/activemq-apollo/apache-apollo@1.0-beta1?classifier=windows-distro&download_url=https://archive.apache.org/dist/activemq/activemq-apollo/1.0-beta1/apache-apollo-1.0-beta1-windows-distro.zip
https://archive.apache.org/dist/airflow/2.3.2/apache_airflow-2.3.2-py3-none-any.whl	pkg:maven/org.apache/airflow/apache_airflow@2.3.2?classifier=py3-none-any&download_url=https://archive.apache.org/dist/airflow/2.3.2/apache_airflow-2.3.2-py3-none-any.whl
https://archive.apache.org/dist/airflow/2.3.3/apache-airflow-2.3.3-source.tar.gz	pkg:maven/org.apache/airflow/apache-airflow@2.3.3?classifier=source&download_url=https://archive.apache.org/dist/airflow/2.3.3/apache-airflow-2.3.3-source.tar.gz
https://archive.apache.org/dist/airflow/2.3.3/apache-airflow-2.3.3.tar.gz	pkg:maven/org.apache/airflow/apache-airflow@2.3.3?download_url=https://archive.apache.org/dist/airflow/2.3.3/apache-airflow-2.3.3.tar.gz

Final Segment of the Parent URL is Not a Version

Download URL	PURL
https://archive.apache.org/dist/perl/win32-bin/apxs_win32-0.1.tar.gz	pkg:maven/org.apache/perl/win32-bin/apxs_win32@0.1?download_url=https://archive.apache.org/dist/perl/win32-bin/apxs_win32-0.1.tar.gz
https://archive.apache.org/dist/perl/win32-bin/apxs_win32.tar.gz	pkg:maven/org.apache/perl/win32-bin/apxs_win32?download_url=https://archive.apache.org/dist/perl/win32-bin/apxs_win32.tar.gz

Potential Issues

pkg:maven/org.apache/perl/win32-bin/apxs_win32@0.1?download_url=https://archive.apache.org/dist/perl/win32-bin/apxs_win32-0.1.tar.gz , the win32-bin perhaps should not be considered as a namespace.

Should we only consider the "first" level after https://archive.apache.org/dist/ as namepsace and ignoring all the sub-folders?
For instance,
namespace for https://archive.apache.org/dist/activemq/activemq-apollo/1.0-beta1/apache-apollo-1.0-beta1-windows-distro.zip should become org.apache/activemq/ instead of the current org.apache/activemq/activemq-apollo/

and

namespace for https://archive.apache.org/dist/airflow/providers/apache-airflow-providers-1.0.0-source.tar.gz should become org.apache/airflow/ instead of 'org.apache/airflow/providers/` ?

[JonoYang]

@chinyeungli @pombredanne

The spec says that the groupId is the namespace: https://github.com/package-url/purl-spec/blob/main/types-doc/maven-definition.md#namespace-definition

so it would be pkg:maven/org.apache.perl.win32-bin/apxs_win32@0.1?download_url=https://archive.apache.org/dist/perl/win32-bin/apxs_win32-0.1.tar.gz, I think?

[chinyeungli]

@JonoYang How do I construct/find the groupid? is it based on the path with excluding the version (if present)? Is the following correct?

Download URL	PURL
https://archive.apache.org/dist/abdera/1.1.2/apache-abdera-1.1.2-src.tar.gz	pkg:maven/org.apache.abdera/apache-abdera@1.1.2?classifier=src&download_url=https://archive.apache.org/dist/abdera/1.1.2/apache-abdera-1.1.2-src.tar.gz
https://archive.apache.org/dist/datasketches/cpppy/4.1.0/pypi/datasketches-4.1.0-cp310-cp310-macosx_10_9_x86_64.whl	pkg:maven/org.apache.datasketches.cpppy/datasketches@4.1.0?classifier=cp310-cp310-macosx_10_9_x86_64&download_url=https://archive.apache.org/dist/datasketches/cpppy/4.1.0/pypi/datasketches-4.1.0-cp310-cp310-macosx_10_9_x86_64.whl
https://archive.apache.org/dist/incubator/kie/10.1.0/apache-kie-10.1.0-incubating-business-automation-chrome-extension-editors.zip	pkg:maven/org.apache.incubator.kie/apache-kie@10.1.0-incubating?classifier=business-automation-chrome-extension-editors&download_url=https://archive.apache.org/dist/incubator/kie/10.1.0/apache-kie-10.1.0-incubating-business-automation-chrome-extension-editors.zip

It's challenging to decide what to be feteched/mined and what not
Probably no?
https://archive.apache.org/dist/directory/studio/update/2.x/plugins/ because it contains libs not from apache directory?
Some no?
https://archive.apache.org/dist/couchdb/binary/win/1.6.0/ as some of them is log files?
Probably yes?
https://archive.apache.org/dist/geronimo/eclipse/updates/plugins/

If https://archive.apache.org/dist/geronimo/eclipse/updates/features/org.apache.geronimo.feature_0.5.0.jar should be fetched, is the PURL be
pkg:mavne/org.apache.geronimo.eclipse.updates.features/org.apache.geronimo.feature@0.5.0?download_url=https://archive.apache.org/dist/geronimo/eclipse/updates/features/org.apache.geronimo.feature_0.5.0.jar

[chinyeungli]

I conclude name, version and classifier based on the filename.
Please review the list below if they are all correct:

filename	name	version	classifier
griffin-0.5.0-source-release.zip	griffin	0.5.0	source-release
sample.zip	sample	-	-
apxs_win32-0.3.tar.gz	apxs_win32	0.3	-
accumulo-1.4.1-1.amd64.rpm	accumulo	1.4.1-1	amd64
org.apache.felix.webconsole-4.3.0-debug.jar	org.apache.felix.webconsole	4.3.0	debug
accumulo-2.0.0-alpha-2-bin.tar.gz	accumulo	2.0.0-alpha-2	bin
commons-csv-1.0-SNAPSHOT-rev1148315.jar	commons-csv	1.0-SNAPSHOT-rev1148315	-
apache-carbondata-1.6.1-bin-spark2.1.0-hadoop2.7.2.jar	apache-carbondata	1.6.1	bin-spark2.1.0-hadoop2.7.2
apache_beam-v2.57.0-prism-darwin-amd64.zip	apache_beam	2.57.0	prism-darwin-amd64
apache-beam-f6ec9cb0c167815f942cf70a674f92a04819c83b-source-release.zip	apache-beam	f6ec9cb0c167815f942cf70a674f92a04819c83b	source-release
avalon-activation-api-1.0b1.jar	avalon-activation-api	1.0b1	-
cayenne-4.1.B1-src.tar.gz	cayenne	4.1.B1	src
ApacheDirectoryStudio-linux-x86_64-2.0.0.v20120127.tar.gz	ApacheDirectoryStudio-linux	2.0.0.v20120127	x86_64
pyarrow-0.11.1-py35hfc679d8_0.tar.bz2	pyarrow	0.11.1	py35hfc679d8_0
camel-k-client-1.0.0-mac-64bit.tar.gz	camel-k-client	1.0.0	mac-64bit
apache_airflow_providers_cncf_kubernetes-10.4.3-py3-none-any.whl	apache_airflow_providers_cncf_kubernetes	10.4.3	py3-none-any
datasketches-5.0.0-cp310-cp310-macosx_10_14_x86_64.whl	datasketches	5.0.0	cp310-cp310-macosx_10_14_x86_64
apache-flex-sdk-installer-3.3.2-64bit-bin.zip	apache-flex-sdk-installer	3.3.2-64bit	bin
excalibur-javadoc-r508111-15022007.tar.gz	excalibur-javadoc	r508111-15022007	-
apache_1.3.1-m68k-apple-aux3.tar.gz	apache	1.3.1	m68k-apple-aux3

All of the test cases above are passing successfully. The one shown below is the remaining case I’m still working on to get passing.

filename	name	version	classifier
psol-1.14.36.1-apache-incubating-x64.tar.gz	psol	1.14.36.1-apache-incubating	x64

[JonoYang]

@chinyeungli

https://archive.apache.org/dist/abdera/1.1.2/apache-abdera-1.1.2-src.tar.gz would probably map to pkg:maven/org.apache.abdera/abdera@1.1.2?download_url=https://archive.apache.org/dist/abdera/1.1.2/apache-abdera-1.1.2-src.tar.gz

https://archive.apache.org/dist/datasketches/cpppy/4.1.0/pypi/datasketches-4.1.0-cp310-cp310-macosx_10_9_x86_64.whl

I think this would be a PyPI package and not maven

I'm still unsure about how to map these apache archive download urls to maven packages, since there is no equivalent download target on repo1.maven.org, would these be apache packages? Are we going to use the package type of maven to represent all java packages? In the case of pkg:maven/org.apache.abdera/abdera@1.1.2?download_url=https://archive.apache.org/dist/abdera/1.1.2/apache-abdera-1.1.2-src.tar.gz, the corresponding location in maven has only poms (https://repo1.maven.org/maven2/org/apache/abdera/abdera/1.1.2/)

I will need some help from @pombredanne to get clarification on this and determining classifiers from filenames

[chinyeungli]

We already have code in place to fetch from "repo1.maven.org". I think the purpose here is to fetch/mine Apache packages that aren’t available there, so there’s no need to map these Apache packages back to Maven packages.
I think there are several issues we need to clarify:

• package type - should all mined packages from Apache be classified as "maven"?

• namespace - should the namespace be derived from the URL path? For example, given
  https://archive.apache.org/dist/geronimo/eclipse/updates/plugins/org.apache.geronimo.common.deployment.model.edit_1.0.0.jar,
  would the namespace be "org.apache.geronimo.eclipse.updates.plugins"?

• scope of fetching - are we expected to fetch all archive packages from Apache repos? For instance, would we also mine the following even they are not Maven packages?

  • https://archive.apache.org/dist/datasketches/cpppy/4.1.0/pypi/datasketches-4.1.0-cp310-cp310-macosx_10_9_x86_64.whl (Python)
  • https://archive.apache.org/dist/couchdb/binary/win/1.6.0/setup-couchdb-1.6.0_R14B04.exe (Windows)
  • also. how do we decide which packages to fetch and which to skip? I noticed a pattern where most Apache archives include ".asc" files (or corresponding .md5/.sha1 checksums). However, there are exceptions. For example, in https://archive.apache.org/dist/directory/studio/update/2.x/plugins/, not all packages originate from "directory".

• classifier - can we assume a classifier as, typically found between the version and the extension? (See maven-heaven: Extend indexing for Apache source and binaries in PurlDB #631 (comment))

[pombredanne]

https://archive.apache.org/dist/abdera/1.1.2/apache-abdera-1.1.2-src.tar.gz would probably map to pkg:maven/org.apache.abdera/abdera@1.1.2?download_url=https://archive.apache.org/dist/abdera/1.1.2/apache-abdera-1.1.2-src.tar.gz

@JonoYang an Apache project cannot be a maven PURL. This is an apache PURL or some generic or else which is progress: See:

• Define package types for ASF projects (Apache Software Foundation) package-url/purl-spec#305
• Define package types for ASF projects (Apache Software Foundation) package-url/purl-spec#851