[QUESTION/HELP] k3d shared volumes

[Jean-Baptiste-Lasselle]

Question / Where do you need Help?

I categorized this issue as help, not bug, because I am not 100% sure yet it is a bug. I am trying to create a cluster with the --volume option, and the shared attribute. I get a fatal error, and the cluster is not created

Scope of your Question

• Is your question related to a specific version of k3d (or k3s)?
  • Please paste the output of k3d version here

bash-3.2$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.15.5
BuildVersion:	19F101
bash-3.2$ k3d --version
k3d version v4.4.1
k3s version v1.20.5-k3s1 (default)
bash-3.2$ docker version
Client: Docker Engine - Community
 Cloud integration: 1.0.9
 Version:           20.10.5
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        55c4c88
 Built:             Tue Mar  2 20:13:00 2021
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.5
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       363e9a8
  Built:            Tue Mar  2 20:15:47 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
bash-3.2$ 

Here is how I create my cluster, on MacOS :

# first, I delete the cluster if it exists 
k3d cluster delete jblCluster
export K3DVOL_GITFLOW_EXPERIMENT='/tmp/k3dvolumeforcsi'
if [ -d ${K3DVOL_GITFLOW_EXPERIMENT} ]; then
  rm -fr ${K3DVOL_GITFLOW_EXPERIMENT}
fi;
mkdir -p ${K3DVOL_GITFLOW_EXPERIMENT}/volume1
mkdir -p ${K3DVOL_GITFLOW_EXPERIMENT}/volume2
mkdir -p ${K3DVOL_GITFLOW_EXPERIMENT}/volume3
mkdir -p ${K3DVOL_GITFLOW_EXPERIMENT}/volume4

export VOLUME_OPT=""
export VOLUME_OPT="${VOLUME_OPT} --volume ${K3DVOL_GITFLOW_EXPERIMENT}/volume1:/tmp/k3dvol/app1:shared "
export VOLUME_OPT="${VOLUME_OPT} --volume ${K3DVOL_GITFLOW_EXPERIMENT}/volume2:/tmp/k3dvol/app2:shared "
export VOLUME_OPT="${VOLUME_OPT} --volume ${K3DVOL_GITFLOW_EXPERIMENT}/volume3:/tmp/k3dvol/app3:shared "
export VOLUME_OPT="${VOLUME_OPT} --volume ${K3DVOL_GITFLOW_EXPERIMENT}/volume4:/tmp/k3dvol/app4:shared "

echo "VOLUME_OPT=[${VOLUME_OPT}]"
k3d cluster create jblCluster ${VOLUME_OPT} --agents 3 --servers 3 --network jbl_network  -p 8080:80@agent[0] -p 8081:80@agent[1] -p 8090:8090@server[0]  -p 8091:8090@server[1] --api-port 0.0.0.0:7888

and here is my stdoutput :

WARN[0000] No node filter specified                     
WARN[0000] No node filter specified                     
WARN[0000] No node filter specified                     
WARN[0000] No node filter specified                     
INFO[0000] Prep: Network                                
INFO[0000] Network with name 'jbl_network' already exists with ID '6f56aff5f5b8890744f5a970a04899f0bd29124e61e07655a41c9123283e675c' 
INFO[0000] Created volume 'k3d-jblCluster-images'       
INFO[0000] Creating initializing server node            
INFO[0000] Creating node 'k3d-jblCluster-server-0'      
INFO[0001] Creating node 'k3d-jblCluster-server-1'      
INFO[0002] Creating node 'k3d-jblCluster-server-2'      
INFO[0002] Creating node 'k3d-jblCluster-agent-0'       
INFO[0002] Creating node 'k3d-jblCluster-agent-1'       
INFO[0002] Creating node 'k3d-jblCluster-agent-2'       
INFO[0002] Creating LoadBalancer 'k3d-jblCluster-serverlb' 
INFO[0002] Starting cluster 'jblCluster'                
INFO[0002] Starting the initializing server...          
INFO[0002] Starting Node 'k3d-jblCluster-server-0'      
ERRO[0002] Failed to start node 'k3d-jblCluster-server-0' 
ERRO[0002] Failed Cluster Start: Failed to start initializing server node: Error response from daemon: path /host_mnt/private/tmp/k3dvolumeforcsi/volume2 is mounted on /host_mnt but it is not a shared mount 
ERRO[0002] Failed to create cluster >>> Rolling Back    
INFO[0002] Deleting cluster 'jblCluster'                
INFO[0002] Deleted k3d-jblCluster-server-0              
INFO[0002] Deleted k3d-jblCluster-server-1              
INFO[0002] Deleted k3d-jblCluster-server-2              
INFO[0002] Deleted k3d-jblCluster-agent-0               
INFO[0002] Deleted k3d-jblCluster-agent-1               
INFO[0002] Deleted k3d-jblCluster-agent-2               
INFO[0002] Deleted k3d-jblCluster-serverlb              
INFO[0002] Deleting image volume 'k3d-jblCluster-images' 
FATA[0002] Cluster creation FAILED, all changes have been rolled back! 
bash-3.2$ k3d cluster delete jblCluster
INFO[0000] No clusters found                            
bash-3.2$ echo  ${VOLUME_OPT}
--volume /tmp/k3dvolumeforcsi/volume1:/tmp/k3dvol/app1:shared --volume /tmp/k3dvolumeforcsi/volume2:/tmp/k3dvol/app2:shared --volume /tmp/k3dvolumeforcsi/volume3:/tmp/k3dvol/app3:shared --volume /tmp/k3dvolumeforcsi/volume4:/tmp/k3dvol/app4:shared
bash-3.2$ k3d cluster delete jblCluster
INFO[0000] No clusters found          

[iwilltry42]

Hey @Jean-Baptiste-Lasselle , thanks for opening this issue (converted to discussion).
I think this is an issue with mount-propagation from rootfs to the Docker VM and then to the containers, where the rootfs -> docker-vm mount is not shared and cannot be made shared (as in e.g. moby/moby#39093).
So I feel like this could not work at all with Docker for Desktop 🤔

[Jean-Baptiste-Lasselle]

Hi @iwilltry42 ,

Thank you so much for your answer :

• As I opened the issue, I just made an experiment, and described the problem I experienced as a result
• Thanks to your answer, I now understand that the issue I experienced is caused by the infrastructure on top of which I run k3d (running on macos, so a macos vm with docker inside) :
  • I experienced other difficulties explained by the infrastructure on top of which I run k3d (running on macos, so a macos vm with docker inside) : onthe networking side especially
  • so basically here, I will change the infrastructure on top of which I am running k3d, to get rid of the complexity brought by that infrastructure. I will go for pure GNU/Linux debian on standard PC and standard armv8

Thank you so much for your response :)