Storing the same layer under different media types causes blobs to be erroneously garbage collected

[amisevsk]

Describe the bug
If the local store for ModelKits contains the same layer under different mediaTypes (e.g. as both a docs and code layer, in two separate ModelKits), automatic garbage collection of blobs will be broken and the local store may encounter errors when removing ModelKits. This occurs because for the purposes of the oras library's garbage collection, descriptors are compared for equality even if they have the same digest (this may be a bug on the library side).

In other words, packing the same content in two different media types may result in ModelKit manifests with layers like

• ModelKit one:

      "layers": [
        {
          "mediaType": "application/vnd.kitops.modelkit.model.v1.tar",
          "digest": "sha256:65c705d810aa0ec82d1ea886c170462c9f17203aa313bfc2cf01d099d4e5ff96",
          "size": 2048
        }
      ],

• ModelKit two:

    "layers": [
      {
        "mediaType": "application/vnd.kitops.modelkit.docs.v1.tar",
        "digest": "sha256:65c705d810aa0ec82d1ea886c170462c9f17203aa313bfc2cf01d099d4e5ff96",
        "size": 2048
      }
    ],

These ModelKits refer to the same file -- stored at $KITOPS_HOME/storage/blobs/sha256/65c705d810aa0ec82d1ea886c170462c9f17203aa313bfc2cf01d099d4e5ff96 -- but when deleting the first ModelKit, the file will be removed since no other descriptor in the store is identical (i.e. including mediaType).

To Reproduce

cd $(mktemp -d)

# Make a temporary Kit context directory to avoid breaking the default one
mkdir .kit-temp
export KITOPS_HOME=$(pwd)/.kit-temp

# Create file to be packed
echo "testfile" > file.txt

# Create ModelKit storing file as a model layer
cat <<EOF > Kitfile
manifestVersion: 1.0.0
model:
  path: file.txt
EOF
kit pack -t issue-reproducer:model -f Kitfile .

# Create ModelKit storing files as a docs layer
cat <<EOF > Kitfile2
manifestVersion: 1.0.0
docs:
  - path: file.txt
EOF
kit pack -t issue-reproducer:docs -f Kitfile2 .

# Try to remove all ModelKits, for example -- this will fail
kit remove -af

Version
latest

Additional context
Add any other context about the problem here.

[srikary12]

Can I start working on this?

[amisevsk]

@srikary12 You're welcome to take a look, but this is a fairly complicated issue -- the underlying behavior is due to garbage collection in the oras library, which we use for managing local storage. From a quick glance, fixing this issue may require re-implementing a significant portion of oras' code.

[srikary12]

Will give it a shot. Need to learn, need to grow, need to give back to the society :).

[srikary12]

@amisevsk in the removeModel function, why are we creating a new local repo?

[amisevsk]

in the removeModel function, why are we creating a new local repo?

It's not creating a new model (i.e. it's not touching disk) -- it's a utility function for getting a local OCI store with the correct configuration (index file, etc.). The new word in there is just echoing oci.New.