Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When docker overlay2.size is enabled, the actual size of the pvc request is always the overlay2.size setting size #176

Open
wutz opened this issue Mar 30, 2024 · 5 comments

Comments

@wutz
Copy link

wutz commented Mar 30, 2024

Describe the bug: Always docker overlay2.size and not pvc request size.

Expected behaviour: The actual size should be the pvc request size.

Steps to reproduce the bug:

  1. openebs localpv and docker in same xfs with prjquota enabled

  2. enable overlay.size in docker daemon config

cat /etc/docker/daemon.json
{
 "storage-driver": "overlay2",
    "storage-opts": [
        "overlay2.size=10GB"
    ]
}
  1. deploy a pod and pvc with size equal 5 GB

The output of the following commands will help us better understand what's going on:

  • kubectl get pods -n <openebs_namespace> --show-labels

    NAME                                          READY   STATUS    RESTARTS   AGE   LABELS
    localpv-localpv-provisioner-7d59d8f6c-zdt5c   1/1     Running   0          22h   app=localpv-provisioner,chart=localpv-provisioner-4.0.0,component=localpv-provisioner,heritage=Helm,name=openebs-localpv-provisioner,openebs.io/component-name=openebs-localpv-provisioner,openebs.io/version=4.0.0,pod-template-hash=7d59d8f6c,release=localpv
  • kubectl logs <upgrade_job_pod> -n <openebs_namespace>

    I0330 07:08:37.197263       1 controller.go:1366] provision "default/localpv-vol" class "local-ssd": started
    I0330 07:08:37.201411       1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"localpv-vol", UID:"115f2690-dc12-49a5-bd12-adb1b5824f52", APIVersion:"v1", ResourceVersion:"11761250", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/localpv-vol"
    I0330 07:08:37.203873       1 provisioner_hostpath.go:77] Creating volume pvc-115f2690-dc12-49a5-bd12-adb1b5824f52 at node with labels {map[kubernetes.io/hostname:cn002.zw1.local]}, path:/var/openebs/local/pvc-115f2690-dc12-49a5-bd12-adb1b5824f52,ImagePullSecrets:[]
    2024-03-30T07:08:45.254Z	INFO	app/provisioner_hostpath.go:131		{"eventcode": "local.pv.quota.success", "msg": "Successfully applied quota", "rname": "pvc-115f2690-dc12-49a5-bd12-adb1b5824f52", "storagetype": "hostpath"}
    2024-03-30T07:08:45.254Z	INFO	app/provisioner_hostpath.go:215		{"eventcode": "local.pv.provision.success", "msg": "Successfully provisioned Local PV", "rname": "pvc-115f2690-dc12-49a5-bd12-adb1b5824f52", "storagetype": "hostpath"}
    I0330 07:08:45.254731       1 controller.go:1449] provision "default/localpv-vol" class "local-ssd": volume "pvc-115f2690-dc12-49a5-bd12-adb1b5824f52" provisioned
    I0330 07:08:45.254744       1 controller.go:1462] provision "default/localpv-vol" class "local-ssd": succeeded
    I0330 07:08:45.254752       1 volume_store.go:212] Trying to save persistentvolume "pvc-115f2690-dc12-49a5-bd12-adb1b5824f52"
    I0330 07:08:45.257734       1 volume_store.go:219] persistentvolume "pvc-115f2690-dc12-49a5-bd12-adb1b5824f52" saved
    I0330 07:08:45.257856       1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"localpv-vol", UID:"115f2690-dc12-49a5-bd12-adb1b5824f52", APIVersion:"v1", ResourceVersion:"11761250", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-115f2690-dc12-49a5-bd12-adb1b5824f52

Anything else we need to know?:
Add any other context about the problem here.

Environment details:

  • OpenEBS version (use kubectl get po -n openebs --show-labels): 4.0.0
  • Kubernetes version (use kubectl version): v1.28.7+k3s1
  • Cloud provider or hardware configuration: AMD EPYC 7452 32-Core Processor / Intel D3-S4510 Series 480GB TLC SATA 6Gbps
  • OS (e.g: cat /etc/os-release): ubuntu 22.04
  • kernel (e.g: uname -a): 5.15.0-97-generic
  • others:
@wutz wutz changed the title When docker overlay2.size is enabled, the actual size of the pvc application is always the overlay2.size setting size When docker overlay2.size is enabled, the actual size of the pvc request is always the overlay2.size setting size Mar 30, 2024
@wutz
Copy link
Author

wutz commented Mar 30, 2024

I notice that overwriting hostpathClass.name from openebs-hostpath to local-ssd will trigger this problem, as described in helm chart values. You must redeploy openebs without modifying hostpathClass.name to work properly

@wutz
Copy link
Author

wutz commented Mar 30, 2024

I should have followed the tutorial to create a new storageClass instead of renaming openebs-hostpath

@wutz wutz closed this as completed Mar 30, 2024
@wutz
Copy link
Author

wutz commented Mar 30, 2024

The use of quota on the worker node is set correctly, but setting quota on the master node is still equivalent to docker overlay.size

The master and the worker nodes with different, the master node has taint node-role.kubernetes.io/control-plane:NoSchedule and docker overlay.size is 30GB

Create a size 50GB pvc, the localpv logs:

I0330 09:27:25.421180       1 controller.go:1366] provision "default/localpv-vol" class "openebs-hostpath": started
I0330 09:27:25.427411       1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"localpv-vol", UID:"d6b4589f-1c60-449f-9a62-dba6849d09a7", APIVersion:"v1", ResourceVersion:"11873609", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/localpv-vol"
I0330 09:27:25.429837       1 provisioner_hostpath.go:77] Creating volume pvc-d6b4589f-1c60-449f-9a62-dba6849d09a7 at node with labels {map[kubernetes.io/hostname:mn03.zw1.local]}, path:/var/openebs/local/pvc-d6b4589f-1c60-449f-9a62-dba6849d09a7,ImagePullSecrets:[]
2024-03-30T09:27:31.477Z	INFO	app/provisioner_hostpath.go:131		{"eventcode": "local.pv.quota.success", "msg": "Successfully applied quota", "rname": "pvc-d6b4589f-1c60-449f-9a62-dba6849d09a7", "storagetype": "hostpath"}
2024-03-30T09:27:31.478Z	INFO	app/provisioner_hostpath.go:215		{"eventcode": "local.pv.provision.success", "msg": "Successfully provisioned Local PV", "rname": "pvc-d6b4589f-1c60-449f-9a62-dba6849d09a7", "storagetype": "hostpath"}
I0330 09:27:31.478134       1 controller.go:1449] provision "default/localpv-vol" class "openebs-hostpath": volume "pvc-d6b4589f-1c60-449f-9a62-dba6849d09a7" provisioned
I0330 09:27:31.478150       1 controller.go:1462] provision "default/localpv-vol" class "openebs-hostpath": succeeded
I0330 09:27:31.478162       1 volume_store.go:212] Trying to save persistentvolume "pvc-d6b4589f-1c60-449f-9a62-dba6849d09a7"
I0330 09:27:31.481748       1 volume_store.go:219] persistentvolume "pvc-d6b4589f-1c60-449f-9a62-dba6849d09a7" saved
I0330 09:27:31.481862       1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"localpv-vol", UID:"d6b4589f-1c60-449f-9a62-dba6849d09a7", APIVersion:"v1", ResourceVersion:"11873609", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-d6b4589f-1c60-449f-9a62-dba6849d09a7

@wutz wutz reopened this Mar 30, 2024
@wutz
Copy link
Author

wutz commented Mar 31, 2024

" PID=`xfs_quota -x -c 'report -h' /data | tail -2 | awk 'NR==1{print substr ($1,2)}+0'` ;" +
" PID=`expr $PID + 1` ;" +

Generating a project id based on the above code may cause docker to compete for concurrency

@tiagolobocastro
Copy link

This could be somewhat related to #155 ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants