Shared folder inotify issue with multiple containers on EKS using gVisor

[NahumLitvin]

Description

I encountered an issue when running a Pod with two containers sharing a folder in a Kubernetes cluster using gVisor. The expected behavior is for one container to detect file changes in the shared folder made by the other container, and send an event when a file is modified. However, this works as expected on GKE, but fails on EKS using the latest version of gVisor.

it is similar to #8089
but i see it works on GKE

---

Additional Information

• Assumption: This issue is likely related to inotify, which chokidar relies on for file change detection.
• Environment Details:
  • gVisor version: 20241217.0
  • EKS Kubernetes version: 1.30 (gvisor is installed with default settings)
  • GKE Kubernetes version: 1.30

---

Debugging Done

1. Verified that changes to the shared folder are visible within both containers using stat and cat.
2. Tested on GKE and confirmed that the issue does not occur there.
3. Attempted enabling polling mode in chokidar, which bypasses inotify, and confirmed that events are then detected correctly, suggesting the issue lies with inotify.

---

Request

Please investigate why inotify-based file change detection in shared folders fails on EKS with gVisor, but works on GKE.

Let me know if further details or logs are required. Thank you!

---

This format should provide enough clarity and context for the maintainers to understand and replicate the issue.

Steps to reproduce

1. Environment Setup:

   • Kubernetes cluster running on EKS with gVisor installed.
   • Pod specification:
     • Two containers.
     • Shared folder mounted as a volume.
   • chokidar library in Node.js is used to listen for file changes. This library relies on inotify.

2. Pod Configuration:

   apiVersion: v1
   kind: Pod
   metadata:
     name: shared-folder-test
   spec:
     runtimeClassName: gvisor
     containers:
     - name: container1
       image: node:14
       command: ["node", "watcher.js"]
       volumeMounts:
       - name: shared-folder
         mountPath: /shared
     - name: container2
       image: busybox
       command: ["/bin/sh", "-c", "while true; do echo 'hello' > /shared/test.txt; sleep 2; done"]
       volumeMounts:
       - name: shared-folder
         mountPath: /shared
     volumes:
     - name: shared-folder
       emptyDir: {}

3. Container 1 Code (watcher.js):

   const chokidar = require('chokidar');
   
   const watcher = chokidar.watch('/shared', {
     persistent: true,
   });
   
   watcher.on('change', (path) => {
     console.log(`${path} changed`);
   });

4. Expected Behavior:

   • Container 1 logs changes to /shared/test.txt every time container 2 modifies the file.

5. Observed Behavior:

   • On EKS with gVisor, no changes are detected by chokidar when container 2 modifies the file.
   • On GKE with gVisor, chokidar detects changes correctly.

runsc version

runsc version release-20241217.0
spec: 1.1.0-rc.1

docker version (if using docker)

uname

Linux ip-10-59-118-142.ec2.internal 5.10.226-214.880.amzn2.x86_64 #1 SMP Tue Oct 8 16:18:15 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

kubectl (if using Kubernetes)

Client Version: v1.31.4
Kustomize Version: v5.4.2
Server Version: v1.30.7-eks-56e63d8

repo state (if built from source)

No response

runsc debug logs (if available)

[milantracy]

would you mind sharing gvisor debugging logs from you EKS cluster?

also, have you tried the workflow w/o gvisor sandbox on EKS, how does that work for you?

[fvoznika]

In Kubernetes, a pod is started one container at a time, with no single view over the entire pod. gVisor mounts container volumes independently from each other and as containers are starting up, it doesn’t know whether a volume will be shared with other containers or not. Because they are treated as independent volumes, changes to shared volumes from one container are viewed as external changes to other containers. While these changes are detected when files are accessed, inofity doesn’t work on them (due to #8089).

To solve this problem, we added a set of “mount hints”, that are passed in the form of annotations, to tell how volumes should be mounted. GKE does that automatically in the control plane, that’s why you see this working in GKE and not in vanilla K8s.

Here’s an overview of the annotations that need to be set:

dev.gvisor.spec.mount.<NAME>.share: container | pod | shared
dev.gvisor.spec.mount.<NAME>.type: tmpfs | bind
dev.gvisor.spec.mount.<NAME>.options: rw | ro

NAME can be anything unique to the volume, normally the volume name from the pod spec, e.g. shared-folder. Then for the individual fields:

• share: indicates how the volume is shared with other containers and external entities:

  • container: The volume is only used by one container, and not shared with the outside. Enable fast access.
  • pod: The volume is used by multiple containers inside the pod, and not shared with the outside. Enable fast access and mount sharing.
  • shared: The volume is shared with the outside. Check for external changes on every access.

• type: indicates how the volume is mounted.

  • tmpfs: The volume uses tmpfs inside the sandbox. If using EmptyDir, always set this to tmpfs to enable further optimizations (see details).
  • bind: The volume is a bind mount from host.

• options: CSV indicating what options to use when mounting the volume. This must be the “least common denominator” from all mounts. Ex: volume is mounted as ro in container A and rw in container B, the pod option must be rw so that container B can function. Individual containers will maintain their respective options.

In your case, you want to set annotations for shared-folder indicating that it’s pod shared and type is tmpfs since you’re using EmptyDir. This may be confusing since shared-folder’s Medium is not Memory. What will happen is that, gVisor will mount the shared-folder as an internal tmpfs volume for faster access, but will make it a file backed tmpfs. IOW, bytes written to it are stored in the file system and not memory.

I haven’t tested, but these annotations should work for you:

  annotations:
    dev.gvisor.spec.mount.shared-folder.share: “pod”
    dev.gvisor.spec.mount.shared-folder.type: “tmp”
    dev.gvisor.spec.mount.shared-folder.options: “rw,rprivate”

Also, apart from inotify working, performance will be better since file access will not require revalidation and the volume is mounted directly inside gVisor. Please, give it a try and let us know if it works. Also, make sure to use runsc shim in the node, as there is some handling of the annotations in there.

The documentation for these annotations are a bit spread out in the code. I’ll consolidate them to a single place, so it’s easier to reference in the future.

[NahumLitvin]

@fvoznika this did not work i have reread your guide and all the links multiple times.

after setting the annotations to be the same as in GKE and or as in your guide and also multiple other configurations.

i do see annotations are recognized in the logs but inotify still doesn't work if the file is edited from the second container.

all works if gvisor is off or using the same container as the file system watcher

../b4b/gvisor.create.log:I0109 10:31:17.076094  116102 mount_hints.go:105] Mount annotation found, name: gvisorinternaldevshm, source: "/run/containerd/io.containerd.grpc.v1.cri/sandboxes/b4b/shm", type: tmpfs, share: pod
../b4b/gvisor.boot.log:I0109 10:31:17.213576       1 mount_hints.go:105] Mount annotation found, name: gvisorinternaldevshm, source: "/run/containerd/io.containerd.grpc.v1.cri/sandboxes/b4b/shm", type: tmpfs, share: pod

is all i see in the logs. when putting gvisor off all still works.
also tried other types/share any advice what else can i do?

is there any chance to get your help in a debug session as part of enterprise support?

[fvoznika]

According to the logs above, the annotations are not making into runsc. There is one more step to follow, which to tell containerd that you allow certain pod annotations to be propagated to the runtime. Please add pod_annotations = [ "dev.gvisor.*" ] to /etc/containerd/config.toml like in the example below:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.gvisor]
  runtime_type = "io.containerd.runsc.v1"
  pod_annotations = [ "dev.gvisor.*" ]

This should do it. If it still doesn't work, please share debug logs and the contents of /proc/mounts from inside the containers.

[NahumLitvin]

@fvoznika thank you!
I have successfully proved it works.
would you accept a PR that will update the user guide based on what you wrote here?

[ayushr2]

would you accept a PR that will update the user guide based on what you wrote here?

Yes we most definitely would!