Defining the diffID Generation Order in the Model Spec

[gaius-qi]

Model Spec uses files as layers and image layers have a sequential relationship, it should define a fixed order for generating diffIDs of files within a directory.

If use application/vnd.oci.image.config.v1+json as config media type now, Model Spec should consider how to define the generation logic of diffIDs to prevent containerd from generating different snapshots.

If use application/vnd.cnai.model.config.v1+json as config media type, the definition of diffIDs can be ignored, refer to #19 (comment).

[gaius-qi]

Generating diffIDs in alphabetical order by file name is a simple solution.

[gorkem]

I think the main issue is using the application/vnd.oci.image.config.v1+json as mediaType which makes it an invalid OCI Artifact.

[amisevsk]

I'm not sure I'm following here -- could you perhaps share an example of the problem?

DiffIDs in my understanding refer to something very specific (which only really applies to container images, not artifacts). Are we talking about the order of layers in the manifest?

[caozhuozi]

I think the main issue is using the application/vnd.oci.image.config.v1+json as mediaType which makes it an invalid OCI Artifact.

Agreed.... I haven’t watched the meeting recording yet. Did we conclude that we must use application/vnd.oci.image.config.v1+json?

[caozhuozi]

I'm not sure I'm following here -- could you perhaps share an example of the problem?

DiffIDs in my understanding refer to something very specific (which only really applies to container images, not artifacts). Are we talking about the order of layers in the manifest?

I think he might be referring to the chainID, which conceptually seems more likely to describe the order?

[gaius-qi]

If the media type is application/vnd.oci.image.config.v1+json and containerd is used to download the model, containerd will calculate the chain ID to represent a unique snapshot. Each layer of the image follows a specific order, so the snapshot ID corresponds to a unique image. However, the model is a directory, not composed of layers, and each file is treated as a layer. If the order of these layers is not certain, containerd may calculate a different chain ID for each build.

[caozhuozi]

After thinking for several days, I think I now understand what @gaius-qi meant. The 'order' here seems to refer to the sequence of items in the rootfs.diff_ids array.

The following quote from the application/vnd.oci.image.config.v1+json spec suggests that the items in the array should be in order:

An array of layer content hashes (DiffIDs), in order from first to last.

@gaius-qi wants the implementations (currently modctl) to make the items order in that array deterministic and the same if the content of a directory is identical.

@gaius-qi please correct me if Im wrong.

Holp this helps.