Cannot set readOnlyRootFilesystem for ui container

[zsolooo]

Describe the bug
As a mandatory K8S cluster hardening policy in our organisation it's required to set readOnlyRootFilesystem for each container inside a pod.
The actual issues are with the docker-entrypoint script: https://github.com/opencost/opencost/blob/develop/ui/docker-entrypoint.sh#L21
It tries to write a few files to the root volume (/var/www and /etc/nginx/conf.d).
In case of /var/www it was possible to workaround the issue by attaching a writable emptyDir volume to /var/www, however the same approach was not possible with /etc/nginx/conf.d, because mounting a volume on that path hides files that are already existing on that path (entrypoint script tries to read config template to use for envsubst).

This issue makes the pod unable to start (CrashLoopBackoff).

To Reproduce with the tried workaround
Steps to reproduce the behavior:

1. Deploy opencost via this helmchart: https://github.com/opencost/opencost-helm-chart , chart-version: 1.32.0
   Using the following helm values:

   opencost:
     exporter:
       startupProbe:
         enabled: false
       livenessProbe:
         enabled: false
       readinessProbe:
         enabled: false
     ui:
       securityContext:
         capabilities:
           drop:
           - ALL
         allowPrivilegeEscalation: false
         readOnlyRootFilesystem: true
         runAsNonRoot: true
         runAsUser: 1000

2. Check the ui container logs with kubectl
3. See error:

cp: can't create '/var/www/MaterialIcons-Regular.12b3b105.woff': Read-only file system
cp: can't create '/var/www/MaterialIcons-Regular.333251c4.ttf': Read-only file system
cp: can't create '/var/www/MaterialIcons-Regular.b99eb5ce.woff2': Read-only file system
cp: can't create '/var/www/MaterialIcons-Regular.e9e55c63.eot': Read-only file system
cp: can't create '/var/www/favicon.7eff484d.ico': Read-only file system
cp: can't create '/var/www/index.3406705b.css': Read-only file system
cp: can't create '/var/www/index.3406705b.css.map': Read-only file system
cp: can't create '/var/www/index.4170b679.js': Read-only file system
cp: can't create '/var/www/index.4170b679.js.map': Read-only file system
cp: can't create '/var/www/index.6d70e179.js': Read-only file system
cp: can't create '/var/www/index.6d70e179.js.map': Read-only file system
cp: can't create '/var/www/index.html': Read-only file system
cp: can't create '/var/www/index.runtime.256cff63.js': Read-only file system
cp: can't create '/var/www/index.runtime.256cff63.js.map': Read-only file system
cp: can't create '/var/www/index.runtime.997a1df7.js': Read-only file system
cp: can't create '/var/www/index.runtime.997a1df7.js.map': Read-only file system
cp: can't create '/var/www/logo.b9464e00.png': Read-only file system

To Reproduce with the described workaround (still failing)
Steps to reproduce the behavior:

1. Deploy opencost via this helmchart: https://github.com/opencost/opencost-helm-chart , version: 1.32.0
   Using the following helm values:

   opencost:
     exporter:
       startupProbe:
         enabled: false
       livenessProbe:
         enabled: false
       readinessProbe:
         enabled: false
     ui:
       securityContext:
         capabilities:
           drop:
           - ALL
         allowPrivilegeEscalation: false
         readOnlyRootFilesystem: true
         runAsNonRoot: true
         runAsUser: 1000
       extraVolumeMounts:
       - mountPath: /var/www
         name: ui-www
       - mountPath: /etc/nginx/conf.d
         name: ui-nginx-config
     extraVolumes:
     - name: ui-www
       emptyDir: {}
     - name: ui-nginx-config
       emptyDir: {}

2. Check the ui container logs with kubectl
3. See error:

'/opt/ui/dist/MaterialIcons-Regular.12b3b105.woff' -> '/var/www/MaterialIcons-Regular.12b3b105.woff'
'/opt/ui/dist/MaterialIcons-Regular.333251c4.ttf' -> '/var/www/MaterialIcons-Regular.333251c4.ttf'
'/opt/ui/dist/MaterialIcons-Regular.b99eb5ce.woff2' -> '/var/www/MaterialIcons-Regular.b99eb5ce.woff2'
'/opt/ui/dist/MaterialIcons-Regular.e9e55c63.eot' -> '/var/www/MaterialIcons-Regular.e9e55c63.eot'
'/opt/ui/dist/favicon.7eff484d.ico' -> '/var/www/favicon.7eff484d.ico'
'/opt/ui/dist/index.3406705b.css' -> '/var/www/index.3406705b.css'
'/opt/ui/dist/index.3406705b.css.map' -> '/var/www/index.3406705b.css.map'
'/opt/ui/dist/index.4170b679.js' -> '/var/www/index.4170b679.js'
'/opt/ui/dist/index.4170b679.js.map' -> '/var/www/index.4170b679.js.map'
'/opt/ui/dist/index.6d70e179.js' -> '/var/www/index.6d70e179.js'
'/opt/ui/dist/index.6d70e179.js.map' -> '/var/www/index.6d70e179.js.map'
'/opt/ui/dist/index.html' -> '/var/www/index.html'
'/opt/ui/dist/index.runtime.256cff63.js' -> '/var/www/index.runtime.256cff63.js'
'/opt/ui/dist/index.runtime.256cff63.js.map' -> '/var/www/index.runtime.256cff63.js.map'
'/opt/ui/dist/index.runtime.997a1df7.js' -> '/var/www/index.runtime.997a1df7.js'
'/opt/ui/dist/index.runtime.997a1df7.js.map' -> '/var/www/index.runtime.997a1df7.js.map'
'/opt/ui/dist/logo.b9464e00.png' -> '/var/www/logo.b9464e00.png'
running with BASE_URL=/model
/usr/local/bin/docker-entrypoint.sh: line 21: can't open /etc/nginx/conf.d/default.nginx.conf.template: no such file

Expected behavior
Pod starts up correctly with readonly root filesystem

Screenshots
Not relevant

Which version of OpenCost are you using?
OpenCost: v1.109.0
Helm chart: v1.32.0

Additional context
None

[mkilchhofer]

I also wanted to raise an issue for this. We have the same approach forcing every workload to set the field pod.spec.containers.securityContext.readOnlyRootFilesystem to true.

For /var/www there was a workaround as mentioned until

• Expose and make the API and UI ports configurable  opencost#2366

was implemented. Now we are unable to workaround this.

---

Update:
Found a recent PR which should fix this:

• Allow opencost-ui to run with a read-only root filesystem opencost#2521

But it still seems not possible.
@kaitimmer does it still work for you?

[kaitimmer]

But it still seems not possible. @kaitimmer does it still work for you?

Actually no, because I missed this second part. Currently, we run opencost with an exception to the rule. My hope is that this fixes all of the places where it is needed. But if you can check again, that would be great. 4-eyes more than 2 :)

[jamalsms]

I still get the message in version 1.111.0

/usr/local/bin/docker-entrypoint.sh: line 21: can't open /etc/nginx/conf.d/default.nginx.conf.template: no such file

Has anyone managed to overcome this problem?

[morawat]

has there been any fix or known workaround?

[ervinb]

This worked for me:

# values.yaml
opencost:
  ui:
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: false
      runAsNonRoot: true
      runAsUser: 1001

    extraVolumeMounts:
      - name: empty-var-www
        mountPath: /var/www

extraVolumes:
  - name: empty-var-www
    emptyDir: {}

[kastl-ars]

I also encountered this issue today. I'll try the emptyDir workaround, but a proper fix would be nice...

[kastl-ars]

OK, the error changed:

/usr/local/bin/docker-entrypoint.sh: line 21: can't create /etc/nginx/conf.d/default.nginx.conf: Permission denied

(Please note that I am not using 1001 as runAsUser, as this is not allowed in this openShift cluster...)

Using readOnlyRootFilesystem set to true the error is not Permission denied but read-only filesystem)

[kippsterr]

It's not a fix to the underlying problem but in our case this workaround helped getting a modified default.nginx.conf.template into the game:

values.yaml

opencost:
  ui:
    extraVolumeMounts:
    - name: opencost-ui-nginx-config-volume
      mountPath: /etc/nginx/conf.d/default.nginx.conf.template
      subPath: default.nginx.conf.template

extraVolumes:
- name: opencost-ui-nginx-config-volume
  configMap:
    name: opencost-ui-nginx-config

The configMap itself is provided by kustomize which we use around helm.

[kastl-ars]

It's not a fix to the underlying problem but in our case this workaround helped getting a modified default.nginx.conf.template into the game:

But then the docker-entrypoint.sh would still try to write things to /etc/nginx/conf.d/default.nginx.conf? Or what am I missing?