Support referrers responses in the Image Layout

[sudo-bmitch]

This defines how referrers should be implemented with the Image Layout. Presently, they can be stored in the Layout using the fallback tag. That has issues for tooling that doesn't want to pollute the tag namespace. The proposed solution adds a org.opencontainers.image.referrer.subject annotation, which indicates the descriptor references a referrers response for the specified subject digest, as a drop in replacement for the fallback tag.

I've tested these annotations in a registry that is based on a filesystem of OCI Image Layout content.

[shizhMSFT]

oras-go (versions >= v2.0.0) and oras (versions >= 1.0.0) support referrers natively for the image layout without the fallback tag. Basically, oras puts the referrer manifests in the index.json file without specifying a tag so that oras can traverse to that node without scanning all blobs. The benefits of this methods are

• Clean: fallback tag is not required.
• Backward compatible: backfill is not required. Backward compatible with existing image layout and tools.
• Organized: index.json is de facto a manifest list where some of the manifests have a tag.
• Performant: scanning the blobs folder is not required.

Here's a sample output using oras 1.2.0-beta.1.

$ echo foo > foo.txt
$ oras push --oci-layout hello:foo foo.txt
✓ Uploaded  foo.txt                                                                                4/4  B 100.00%    2ms
  └─ sha256:b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c
✓ Uploaded  application/vnd.oci.empty.v1+json                                                      2/2  B 100.00%    2ms
  └─ sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
✓ Uploaded  application/vnd.oci.image.manifest.v1+json                                         585/585  B 100.00%  992µs
  └─ sha256:7f1216714a7ecbe4525a55ec07c4bb1d06d32cc862f6e4c6598395aeb89e1be8
Pushed [oci-layout] hello:foo
Digest: sha256:7f1216714a7ecbe4525a55ec07c4bb1d06d32cc862f6e4c6598395aeb89e1be8
$ echo bar > bar.txt
$ oras attach --oci-layout --artifact-type application/bar hello:foo bar.txt
✓ Exists    application/vnd.oci.empty.v1+json                                                      2/2  B 100.00%     0s
  └─ sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
✓ Uploaded  bar.txt                                                                                4/4  B 100.00%  826µs
  └─ sha256:7d865e959b2466918c9863afca942d0fb89d7c9ac0c99bafc3749504ded97730
✓ Uploaded  application/vnd.oci.image.manifest.v1+json                                         897/897  B 100.00%  444µs
  └─ sha256:0570b97df7a9bf4bcef65e724cc60d663f09b7650d5010d03a3ed4571d385e81
Attached to [oci-layout] hello@sha256:7f1216714a7ecbe4525a55ec07c4bb1d06d32cc862f6e4c6598395aeb89e1be8
Digest: sha256:0570b97df7a9bf4bcef65e724cc60d663f09b7650d5010d03a3ed4571d385e81
$ oras discover --oci-layout hello:foo
Discovered 1 artifact referencing foo
Digest: sha256:7f1216714a7ecbe4525a55ec07c4bb1d06d32cc862f6e4c6598395aeb89e1be8

Artifact Type     Digest
application/bar   sha256:0570b97df7a9bf4bcef65e724cc60d663f09b7650d5010d03a3ed4571d385e81
$ tree hello
hello
├── blobs
│   └── sha256
│       ├── 0570b97df7a9bf4bcef65e724cc60d663f09b7650d5010d03a3ed4571d385e81
│       ├── 44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
│       ├── 7d865e959b2466918c9863afca942d0fb89d7c9ac0c99bafc3749504ded97730
│       ├── 7f1216714a7ecbe4525a55ec07c4bb1d06d32cc862f6e4c6598395aeb89e1be8
│       └── b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c
├── index.json
├── ingest
└── oci-layout

3 directories, 7 files

The resulted hello/index.json after formating is

{
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:7f1216714a7ecbe4525a55ec07c4bb1d06d32cc862f6e4c6598395aeb89e1be8",
      "size": 585,
      "annotations": {
        "org.opencontainers.image.created": "2024-02-21T09:33:26Z",
        "org.opencontainers.image.ref.name": "foo"
      },
      "artifactType": "application/vnd.unknown.artifact.v1"
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:0570b97df7a9bf4bcef65e724cc60d663f09b7650d5010d03a3ed4571d385e81",
      "size": 897,
      "annotations": {
        "org.opencontainers.image.created": "2024-02-21T09:34:11Z"
      },
      "artifactType": "application/bar"
    }
  ]
}

/cc @sajayantony

[sudo-bmitch]

Performant: scanning the blobs folder is not required.

Wouldn't oras discover need to read the manifest of every untagged index entry from the blob store? This proposal would eliminate that.

[shizhMSFT]

Wouldn't oras discover need to read the manifest of every untagged index entry from the blob store?

Yes, oras does traverse all manifests to build a graph in the memory. BTW, reading from OCI image layout usually happens on local disk (with system-level file cache). Currently, we don't observe any performance issues yet.

[tianon]

We discussed this on the OCI call quite a few weeks ago now, and I'm going to do my best to summarize some of the concerns several of us on that call had:

• our gut reaction to the added subject annotation was that it must be for objects which have a subject field (to "pull up" their subject)
  • this was on the assumption that the most common way to bundle "subject-having" objects in a layout would be via something akin to docker save which is then used to move them around
• this PR is concretely about adding the actual "referrers API" response directly as an index object within the OCI layout (and adding these annotations to that)
  • Brandon is doing something more ambitious (a registry serving from oci-layout), and makes the argument that clients need to generate this object either way
  • on the flip side, clients do need to generate this, but always in a "merge with existing object from registry" fashion

I'm not entirely convinced we should add this (but I'm also not feeling super strongly that it shouldn't) -- perhaps officially an even softer NACK than my normal soft NACK.

[sudo-bmitch]

The inspiration for the annotation pointing to the referrers response, rather than each entry in the response, was to consolidate client workflows. It allows the OCI Layout to be treated the same as a distribution-spec 1.0 registry. By having the annotation on every entry, the client would need a separate workflow for handling an OCI Layout.

Some of this gets into how clients are treating the OCI Layout, whether it's a static transport for a batch process (export/import), or if it's treated like a repository. I think there will always be the first use case. The latter use case opens up a lot of efficiencies and security opportunities for things like CI pipelines that want to review and possibly add metadata before finally uploading it. For those latter use cases, the more an OCI Layout can look like a repository to the tooling, the easier the tooling is to write.

[sudo-bmitch]

Revisiting this since there's some value in other tools supporting referrers in an OCI Layout and we don't have guidance in the spec for that. Between oras, olareg, regclient, and buildkit, there are multiple competing implementations:

1. oras saves individual manifests in the index.json, rather than generating the referrers response. Tooling would need to read every manifest listed in the index to discover any referrers to a specified subject.
2. regclient treats the OCI Layout as a v1 registry and pushes the fallback tag (sha256-....) with the referrers response to the index.json.
3. olareg upconverts a regclient style index.json to use a project specific annotation on entries that are a referrers response (similar to this PR but the annotations are olareg specific). This avoids polluting the tag listing.
4. buildkit includes referrers inside of the multi-platform image index with an "unknown" platform, with an option to specify the "subject" inside the nested manifests. I'm not sure how that would support a subject that points to the image index itself, and if that's supported, what would happen if a new referrer is later added to a nested image (changing the digest of the image index).

There are compatibility issues between each of these. In almost every case, an artifact with a subject pushed by one tool won't be discovered by any other tool.

Are there other tools that currently support referrers/subject in the OCI Layout we should consider?

• on the flip side, clients do need to generate this, but always in a "merge with existing object from registry" fashion

Revisiting this comment, I'm not sure that all uses of an OCI Layout involve merging content to/from a registry. And if support was improved and standardized between tooling, workflows to operate on an OCI Layout within a CI pipeline, or during transit across an air-gapped environment, could allow signing, sbom generation, vulnerability scanning, etc, while content remains in the Layout. I could also foresee edge environments that want to operate without a registry.

That's starting to repeat my comment above about whether the OCI Layout is a static transport or a standalone/offline representation of a repository. And I think there's value in supporting both use cases.

[cyphar]

@sudo-bmitch

Are there other tools that currently support referrers/subject in the OCI Layout we should consider?

FWIW, I still haven't figured out how I want to do it for umoci. umoci 0.6 supports the raw blob format for subjects but we don't do any conversions (partially because we also don't yet support distribution-spec -- it's on my TODO list for umoci 0.7). I am interested in what model we end up with here, though I was thinking of something similar to what you've proposed here.

Revisiting this comment, I'm not sure that all uses of an OCI Layout involve merging content to/from a registry.

This is definitely the case for umoci -- we build images without any direct access to registries (for openSUSE images, the artefacts are built in an isolated environment without network access in OBS with KIWI+umoci and then published from the OBS registry but not by umoci). I do want to add support for managing stuff from registries, but I suspect it's not going to be the primary use-case for umoci.

[sudo-bmitch]

I'm switching this to a draft state to keep this from being merged while waiting for community feedback. containerd is also taking the approach of listing individual referrer manifests in the index.json rather than the generated referrers response.

Part of my own reasoning for generating the referrers response was that it aligned nicely with code already written for the managing the fallback tag on v1 registries. Changing the tooling to generate that response at consumption time rather than generation time would make three scenarios to support instead of two (v1.1 registries, v1 registries, and OCI Layouts). If OCI decided to only list the individual referrer manifests in the index.json, in olareg I'd probably extend the index.json with a custom field to cache the referrers response for each digest so I don't need to do generate it dynamically per request. The custom field would then hopefully be dropped by other tools that modify the index.json so I avoid using a dirty cache.

I also think it would be a good idea to remove org.opencontainers.image.referrer.convert from this PR since there are an unknown number of ways tools are doing this today, so I'll push that change shortly.

[tianon]

Reviewed again and had stronger feelings this time so leaving an explicit NACK -- I don't think how things currently exist precludes your desired usage (in fact, I think what we have already supports it in a reasonably clean way, as I've detailed in my line-comment below).

If we wanted to codify something, I'd prefer to see something along the lines of BuildKit's vnd.docker.reference.digest annotation, but I'm not convinced officially codifying that is really all that necessary (and it's orthogonal to what you're trying to achieve anyhow, since you don't want the individual referrers entries in index.json but that's also a really common way to lay this out, as you've noted 😅).

[tianon]

Looking at this with fresh eyes, I'm much more strongly -1 on this specific annotation unless it's for pulling up the subject field of the manifest we're listing.

For the use case you've described (having the "registry view" referrers response object as part of the image layout), that would be more obviously annotated by setting org.opencontainers.image.ref.name to the fallback tag format, which would have the same fidelity as the proposed new field (with the added benefit that it could/would be per-"name" instead of per-digest, because referrers are technically scoped to a namespace just like manifests are, and would make conversions to/from the fallback automatic and transparent and even trivial to support querying in both ways with a consistent view).

[sudo-bmitch]

Making the annotation point to the same content as the fallback tag pointed to was intentional, just as the registry referrers response should look just like a manifest get to the fallback tag. One key advantage of not using the fallback tag is getting rid of the tags in the tag listing.

I can convert my implementations to use an annotation of the individual artifacts, but it's going to be a lot more code to write, without the reuse of the existing fallback tag code. I'd be curious to know how many of the projects looking to store these as individual entries in the top level index.json have also implemented the fallback tag logic in addition to the referrers query logic in their code.

[tianon]

I know I certainly have not, and have no intention to (for my uses, the fallback tag as a solution is completely dead on arrival; if we don't have referrers support in the registry, I just won't use them at all).