how does the p2.index exactly work?

[jcompagner]

i am trying to upgrade from 2.7.5 to 4.0
but now i get weird errors on a 3rd party product p2 site that we use (that works fine under 2.7.5)
they have a few sites, one is a composite thing which has a p2.index file that has:

metadata.repository.factory.order=compositeContent.xml,!
artifact.repository.factory.order=compositeArtifacts.xml,!

but then tycho does ask for the "jar" variant:

compositeContent.jar

which fails because that one is not there (if i try to download it, its the xml file that is there)
why does tycho try to download the jar variant which is not there for that site? or should that be there?

if i look at our sites then

and the p2.index does say for those:

artifact.repository.factory.order=artifacts.xml.xz,artifacts.xml,!
version=1
metadata.repository.factory.order=content.xml.xz,content.xml,!

so that is kind of what tycho now does i guess?
the "xz" files are in the p2 as is and those are downloaded? but an "xml" file is always downloaded as a jar (expected to)?

[mdaloia]

Regarding the p2.index file, look at this section: https://wiki.eclipse.org/Equinox/p2/p2_index#Why_say_.27xml.27_instead_of_.27jar.27_in_factory_name.3F

Please note that the values in this file denotes repository factories and
not files. The factory '.xml' will look for both the .jar
and the .xml file, in that order

Basically, they are factory names, not file names. Those factories will always look for the .jar files first.

In our case, we were hosting a Composite Update Site (CUS) in a non-public AWS S3 bucket but the actual files of the CUS were made public. With this scenario, it seems that S3 replies HTTP 403 if a requested file (the .jar) is not found instead of an HTTP 404 (Ref 1 and Ref 2).
Since Tycho 4.0.0 a validation of the response was added and if it is not HTTP 404 or 410 it throws an exception that makes the build to fail:

tycho/p2-maven-plugin/src/main/java/org/eclipse/tycho/p2maven/transport/Response.java

Lines 41 to 47 in e692ddd

	if (code >= HttpURLConnection.HTTP_BAD_REQUEST) {
	if (code == HttpURLConnection.HTTP_NOT_FOUND || code == HttpURLConnection.HTTP_GONE) {
	throw new FileNotFoundException(getURI().toString());
	} else {
	throw new java.io.IOException("Server returned HTTP code: " + code + " for URL " + getURI().toString());
	}
	}

Also from the first link,

As an aside, jarred versions of p2 content and artifacts files should always be provided for production sites, when the files are of any substantial size.

Our CUS was being generated using the org.eclipse.ant.core.antRunner Eclipse application and using the p2.composite.repository target (reference) using an initial .xml file. Even after setting the compressed attribute to true it ended up generating only the .xml files. So we ended up compressing them after the CUS generation.

Hope this helps!

[laeubi]

Our CUS was being generated using the org.eclipse.ant.core.antRunner Eclipse application and using the p2.composite.repository target (reference) using an initial .xml file. Even after setting the compressed attribute to true it ended up generating only the .xml files. So we ended up compressing them after the CUS generation.

The best would be to use Tycho what usually handles all those cases well, if any feature is missing (e.g. creating composites) we should actually add support for it.

[jcompagner]

if you want S3 to give you a a 404 then you need to adjust the buckets policy:

{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::your.domain.com/*"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::your.domain.com"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "s3:GetObjectAttributes",
      "Resource": "arn:aws:s3:::your.domain.com/*"
    }
  ]
}

if you have that and you hit your s3 bucket for unknow files then it will reply with 404 (not found) instead of 403 (forbidden)