Support referrers responses in the Image Layout

[sudo-bmitch]

This defines how referrers should be implemented with the Image Layout. Presently, they can be stored in the Layout using the fallback tag. That has issues for tooling that doesn't want to pollute the tag namespace. The proposed solution uses two annotations. One annotation is on the descriptors of the index.json file, org.opencontainers.image.referrer.subject, which indicates the descriptor references a referrers response for the specified subject digest. The other annotation, org.opencontainers.image.referrer.convert, set on the top level manifest of the index.json file, is used as a flag for tooling to know the file has already had any content converted over to use the org.opencontainers.image.referrer.subject annotation, and further conversions may be skipped.

I've tested these annotations in a registry that is based on a filesystem of OCI Image Layout content.

[shizhMSFT]

oras-go (versions >= v2.0.0) and oras (versions >= 1.0.0) support referrers natively for the image layout without the fallback tag. Basically, oras puts the referrer manifests in the index.json file without specifying a tag so that oras can traverse to that node without scanning all blobs. The benefits of this methods are

• Clean: fallback tag is not required.
• Backward compatible: backfill is not required. Backward compatible with existing image layout and tools.
• Organized: index.json is de facto a manifest list where some of the manifests have a tag.
• Performant: scanning the blobs folder is not required.

Here's a sample output using oras 1.2.0-beta.1.

$ echo foo > foo.txt
$ oras push --oci-layout hello:foo foo.txt
✓ Uploaded  foo.txt                                                                                4/4  B 100.00%    2ms
  └─ sha256:b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c
✓ Uploaded  application/vnd.oci.empty.v1+json                                                      2/2  B 100.00%    2ms
  └─ sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
✓ Uploaded  application/vnd.oci.image.manifest.v1+json                                         585/585  B 100.00%  992µs
  └─ sha256:7f1216714a7ecbe4525a55ec07c4bb1d06d32cc862f6e4c6598395aeb89e1be8
Pushed [oci-layout] hello:foo
Digest: sha256:7f1216714a7ecbe4525a55ec07c4bb1d06d32cc862f6e4c6598395aeb89e1be8
$ echo bar > bar.txt
$ oras attach --oci-layout --artifact-type application/bar hello:foo bar.txt
✓ Exists    application/vnd.oci.empty.v1+json                                                      2/2  B 100.00%     0s
  └─ sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
✓ Uploaded  bar.txt                                                                                4/4  B 100.00%  826µs
  └─ sha256:7d865e959b2466918c9863afca942d0fb89d7c9ac0c99bafc3749504ded97730
✓ Uploaded  application/vnd.oci.image.manifest.v1+json                                         897/897  B 100.00%  444µs
  └─ sha256:0570b97df7a9bf4bcef65e724cc60d663f09b7650d5010d03a3ed4571d385e81
Attached to [oci-layout] hello@sha256:7f1216714a7ecbe4525a55ec07c4bb1d06d32cc862f6e4c6598395aeb89e1be8
Digest: sha256:0570b97df7a9bf4bcef65e724cc60d663f09b7650d5010d03a3ed4571d385e81
$ oras discover --oci-layout hello:foo
Discovered 1 artifact referencing foo
Digest: sha256:7f1216714a7ecbe4525a55ec07c4bb1d06d32cc862f6e4c6598395aeb89e1be8

Artifact Type     Digest
application/bar   sha256:0570b97df7a9bf4bcef65e724cc60d663f09b7650d5010d03a3ed4571d385e81
$ tree hello
hello
├── blobs
│   └── sha256
│       ├── 0570b97df7a9bf4bcef65e724cc60d663f09b7650d5010d03a3ed4571d385e81
│       ├── 44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
│       ├── 7d865e959b2466918c9863afca942d0fb89d7c9ac0c99bafc3749504ded97730
│       ├── 7f1216714a7ecbe4525a55ec07c4bb1d06d32cc862f6e4c6598395aeb89e1be8
│       └── b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c
├── index.json
├── ingest
└── oci-layout

3 directories, 7 files

The resulted hello/index.json after formating is

{
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:7f1216714a7ecbe4525a55ec07c4bb1d06d32cc862f6e4c6598395aeb89e1be8",
      "size": 585,
      "annotations": {
        "org.opencontainers.image.created": "2024-02-21T09:33:26Z",
        "org.opencontainers.image.ref.name": "foo"
      },
      "artifactType": "application/vnd.unknown.artifact.v1"
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:0570b97df7a9bf4bcef65e724cc60d663f09b7650d5010d03a3ed4571d385e81",
      "size": 897,
      "annotations": {
        "org.opencontainers.image.created": "2024-02-21T09:34:11Z"
      },
      "artifactType": "application/bar"
    }
  ]
}

/cc @sajayantony

[sudo-bmitch]

Performant: scanning the blobs folder is not required.

Wouldn't oras discover need to read the manifest of every untagged index entry from the blob store? This proposal would eliminate that.

[shizhMSFT]

Wouldn't oras discover need to read the manifest of every untagged index entry from the blob store?

Yes, oras does traverse all manifests to build a graph in the memory. BTW, reading from OCI image layout usually happens on local disk (with system-level file cache). Currently, we don't observe any performance issues yet.

[tianon]

We discussed this on the OCI call quite a few weeks ago now, and I'm going to do my best to summarize some of the concerns several of us on that call had:

• our gut reaction to the added subject annotation was that it must be for objects which have a subject field (to "pull up" their subject)
  • this was on the assumption that the most common way to bundle "subject-having" objects in a layout would be via something akin to docker save which is then used to move them around
• this PR is concretely about adding the actual "referrers API" response directly as an index object within the OCI layout (and adding these annotations to that)
  • Brandon is doing something more ambitious (a registry serving from oci-layout), and makes the argument that clients need to generate this object either way
  • on the flip side, clients do need to generate this, but always in a "merge with existing object from registry" fashion

I'm not entirely convinced we should add this (but I'm also not feeling super strongly that it shouldn't) -- perhaps officially an even softer NACK than my normal soft NACK.

[sudo-bmitch]

The inspiration for the annotation pointing to the referrers response, rather than each entry in the response, was to consolidate client workflows. It allows the OCI Layout to be treated the same as a distribution-spec 1.0 registry. By having the annotation on every entry, the client would need a separate workflow for handling an OCI Layout.

Some of this gets into how clients are treating the OCI Layout, whether it's a static transport for a batch process (export/import), or if it's treated like a repository. I think there will always be the first use case. The latter use case opens up a lot of efficiencies and security opportunities for things like CI pipelines that want to review and possibly add metadata before finally uploading it. For those latter use cases, the more an OCI Layout can look like a repository to the tooling, the easier the tooling is to write.

[sudo-bmitch]

Revisiting this since there's some value in other tools supporting referrers in an OCI Layout and we don't have guidance in the spec for that. Between oras, olareg, regclient, and buildkit, there are multiple competing implementations:

1. oras saves individual manifests in the index.json, rather than generating the referrers response. Tooling would need to read every manifest listed in the index to discover any referrers to a specified subject.
2. regclient treats the OCI Layout as a v1 registry and pushes the fallback tag (sha256-....) with the referrers response to the index.json.
3. olareg upconverts a regclient style index.json to use a project specific annotation on entries that are a referrers response (similar to this PR but the annotations are olareg specific). This avoids polluting the tag listing.
4. buildkit includes referrers inside of the multi-platform image index with an "unknown" platform, with an option to specify the "subject" inside the nested manifests. I'm not sure how that would support a subject that points to the image index itself, and if that's supported, what would happen if a new referrer is later added to a nested image (changing the digest of the image index).

There are compatibility issues between each of these. In almost every case, an artifact with a subject pushed by one tool won't be discovered by any other tool.

Are there other tools that currently support referrers/subject in the OCI Layout we should consider?

• on the flip side, clients do need to generate this, but always in a "merge with existing object from registry" fashion

Revisiting this comment, I'm not sure that all uses of an OCI Layout involve merging content to/from a registry. And if support was improved and standardized between tooling, workflows to operate on an OCI Layout within a CI pipeline, or during transit across an air-gapped environment, could allow signing, sbom generation, vulnerability scanning, etc, while content remains in the Layout. I could also foresee edge environments that want to operate without a registry.

That's starting to repeat my comment above about whether the OCI Layout is a static transport or a standalone/offline representation of a repository. And I think there's value in supporting both use cases.

[cyphar]

@sudo-bmitch

Are there other tools that currently support referrers/subject in the OCI Layout we should consider?

FWIW, I still haven't figured out how I want to do it for umoci. umoci 0.6 supports the raw blob format for subjects but we don't do any conversions (partially because we also don't yet support distribution-spec -- it's on my TODO list for umoci 0.7). I am interested in what model we end up with here, though I was thinking of something similar to what you've proposed here.

Revisiting this comment, I'm not sure that all uses of an OCI Layout involve merging content to/from a registry.

This is definitely the case for umoci -- we build images without any direct access to registries (for openSUSE images, the artefacts are built in an isolated environment without network access in OBS with KIWI+umoci and then published from the OBS registry but not by umoci). I do want to add support for managing stuff from registries, but I suspect it's not going to be the primary use-case for umoci.