Cannot set readOnlyRootFilesystem for ui container #4

zsolooo · 2024-03-20T15:47:56Z

Describe the bug
As a mandatory K8S cluster hardening policy in our organisation it's required to set readOnlyRootFilesystem for each container inside a pod.
The actual issues are with the docker-entrypoint script: https://github.com/opencost/opencost/blob/develop/ui/docker-entrypoint.sh#L21
It tries to write a few files to the root volume (/var/www and /etc/nginx/conf.d).
In case of /var/www it was possible to workaround the issue by attaching a writable emptyDir volume to /var/www, however the same approach was not possible with /etc/nginx/conf.d, because mounting a volume on that path hides files that are already existing on that path (entrypoint script tries to read config template to use for envsubst).

This issue makes the pod unable to start (CrashLoopBackoff).

To Reproduce with the tried workaround
Steps to reproduce the behavior:

Deploy opencost via this helmchart: https://github.com/opencost/opencost-helm-chart , chart-version: 1.32.0
Using the following helm values:

opencost:
  exporter:
    startupProbe:
      enabled: false
    livenessProbe:
      enabled: false
    readinessProbe:
      enabled: false
  ui:
    securityContext:
      capabilities:
        drop:
        - ALL
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      runAsNonRoot: true
      runAsUser: 1000

Check the ui container logs with kubectl
See error:

cp: can't create '/var/www/MaterialIcons-Regular.12b3b105.woff': Read-only file system
cp: can't create '/var/www/MaterialIcons-Regular.333251c4.ttf': Read-only file system
cp: can't create '/var/www/MaterialIcons-Regular.b99eb5ce.woff2': Read-only file system
cp: can't create '/var/www/MaterialIcons-Regular.e9e55c63.eot': Read-only file system
cp: can't create '/var/www/favicon.7eff484d.ico': Read-only file system
cp: can't create '/var/www/index.3406705b.css': Read-only file system
cp: can't create '/var/www/index.3406705b.css.map': Read-only file system
cp: can't create '/var/www/index.4170b679.js': Read-only file system
cp: can't create '/var/www/index.4170b679.js.map': Read-only file system
cp: can't create '/var/www/index.6d70e179.js': Read-only file system
cp: can't create '/var/www/index.6d70e179.js.map': Read-only file system
cp: can't create '/var/www/index.html': Read-only file system
cp: can't create '/var/www/index.runtime.256cff63.js': Read-only file system
cp: can't create '/var/www/index.runtime.256cff63.js.map': Read-only file system
cp: can't create '/var/www/index.runtime.997a1df7.js': Read-only file system
cp: can't create '/var/www/index.runtime.997a1df7.js.map': Read-only file system
cp: can't create '/var/www/logo.b9464e00.png': Read-only file system

To Reproduce with the described workaround (still failing)
Steps to reproduce the behavior:

Deploy opencost via this helmchart: https://github.com/opencost/opencost-helm-chart , version: 1.32.0
Using the following helm values:

opencost:
  exporter:
    startupProbe:
      enabled: false
    livenessProbe:
      enabled: false
    readinessProbe:
      enabled: false
  ui:
    securityContext:
      capabilities:
        drop:
        - ALL
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      runAsNonRoot: true
      runAsUser: 1000
    extraVolumeMounts:
    - mountPath: /var/www
      name: ui-www
    - mountPath: /etc/nginx/conf.d
      name: ui-nginx-config
  extraVolumes:
  - name: ui-www
    emptyDir: {}
  - name: ui-nginx-config
    emptyDir: {}

Check the ui container logs with kubectl
See error:

'/opt/ui/dist/MaterialIcons-Regular.12b3b105.woff' -> '/var/www/MaterialIcons-Regular.12b3b105.woff'
'/opt/ui/dist/MaterialIcons-Regular.333251c4.ttf' -> '/var/www/MaterialIcons-Regular.333251c4.ttf'
'/opt/ui/dist/MaterialIcons-Regular.b99eb5ce.woff2' -> '/var/www/MaterialIcons-Regular.b99eb5ce.woff2'
'/opt/ui/dist/MaterialIcons-Regular.e9e55c63.eot' -> '/var/www/MaterialIcons-Regular.e9e55c63.eot'
'/opt/ui/dist/favicon.7eff484d.ico' -> '/var/www/favicon.7eff484d.ico'
'/opt/ui/dist/index.3406705b.css' -> '/var/www/index.3406705b.css'
'/opt/ui/dist/index.3406705b.css.map' -> '/var/www/index.3406705b.css.map'
'/opt/ui/dist/index.4170b679.js' -> '/var/www/index.4170b679.js'
'/opt/ui/dist/index.4170b679.js.map' -> '/var/www/index.4170b679.js.map'
'/opt/ui/dist/index.6d70e179.js' -> '/var/www/index.6d70e179.js'
'/opt/ui/dist/index.6d70e179.js.map' -> '/var/www/index.6d70e179.js.map'
'/opt/ui/dist/index.html' -> '/var/www/index.html'
'/opt/ui/dist/index.runtime.256cff63.js' -> '/var/www/index.runtime.256cff63.js'
'/opt/ui/dist/index.runtime.256cff63.js.map' -> '/var/www/index.runtime.256cff63.js.map'
'/opt/ui/dist/index.runtime.997a1df7.js' -> '/var/www/index.runtime.997a1df7.js'
'/opt/ui/dist/index.runtime.997a1df7.js.map' -> '/var/www/index.runtime.997a1df7.js.map'
'/opt/ui/dist/logo.b9464e00.png' -> '/var/www/logo.b9464e00.png'
running with BASE_URL=/model
/usr/local/bin/docker-entrypoint.sh: line 21: can't open /etc/nginx/conf.d/default.nginx.conf.template: no such file

Expected behavior
Pod starts up correctly with readonly root filesystem

Screenshots
Not relevant

Which version of OpenCost are you using?
OpenCost: v1.109.0
Helm chart: v1.32.0

Additional context
None

The text was updated successfully, but these errors were encountered:

mkilchhofer · 2024-04-03T18:58:42Z

I also wanted to raise an issue for this. We have the same approach forcing every workload to set the field pod.spec.containers.securityContext.readOnlyRootFilesystem to true.

For /var/www there was a workaround as mentioned until

Expose and make the API and UI ports configurable opencost#2366

was implemented. Now we are unable to workaround this.

Update:
Found a recent PR which should fix this:

Allow opencost-ui to run with a read-only root filesystem opencost#2521

But it still seems not possible.
@kaitimmer does it still work for you?

kaitimmer · 2024-04-05T11:58:25Z

But it still seems not possible. @kaitimmer does it still work for you?

Actually no, because I missed this second part. Currently, we run opencost with an exception to the rule. My hope is that this fixes all of the places where it is needed. But if you can check again, that would be great. 4-eyes more than 2 :)

jamalsms · 2024-07-26T07:22:10Z

I still get the message in version 1.111.0

/usr/local/bin/docker-entrypoint.sh: line 21: can't open /etc/nginx/conf.d/default.nginx.conf.template: no such file

Has anyone managed to overcome this problem?

morawat · 2024-08-16T19:53:48Z

has there been any fix or known workaround?

ervinb · 2024-09-27T09:19:08Z

This worked for me:

# values.yaml
opencost:
  ui:
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: false
      runAsNonRoot: true
      runAsUser: 1001

    extraVolumeMounts:
      - name: empty-var-www
        mountPath: /var/www

extraVolumes:
  - name: empty-var-www
    emptyDir: {}

kastl-ars · 2024-12-04T10:13:34Z

I also encountered this issue today. I'll try the emptyDir workaround, but a proper fix would be nice...

kastl-ars · 2024-12-04T10:17:53Z

OK, the error changed:

/usr/local/bin/docker-entrypoint.sh: line 21: can't create /etc/nginx/conf.d/default.nginx.conf: Permission denied

(Please note that I am not using 1001 as runAsUser, as this is not allowed in this openShift cluster...)

Using readOnlyRootFilesystem set to true the error is not Permission denied but read-only filesystem)

kippsterr · 2024-12-04T11:36:58Z

It's not a fix to the underlying problem but in our case this workaround helped getting a modified default.nginx.conf.template into the game:

values.yaml

opencost:
  ui:
    extraVolumeMounts:
    - name: opencost-ui-nginx-config-volume
      mountPath: /etc/nginx/conf.d/default.nginx.conf.template
      subPath: default.nginx.conf.template

extraVolumes:
- name: opencost-ui-nginx-config-volume
  configMap:
    name: opencost-ui-nginx-config

The configMap itself is provided by kustomize which we use around helm.

kastl-ars · 2024-12-04T12:36:33Z

It's not a fix to the underlying problem but in our case this workaround helped getting a modified default.nginx.conf.template into the game:

But then the docker-entrypoint.sh would still try to write things to /etc/nginx/conf.d/default.nginx.conf? Or what am I missing?

kastl-ars · 2024-12-04T13:15:11Z

This seems to work:

      extraVolumeMounts:
        - name: empty-var-www
          mountPath: /var/www
        - name: empty-nginx-confd
          mountPath: /etc/nginx/conf.d/
        - name: opencost-ui-nginx-config-volume
          mountPath: /etc/nginx/conf.d/default.nginx.conf.template
          subPath: default.nginx.conf.template
  extraVolumes:
    - name: empty-var-www
      emptyDir: {}
    - name: empty-nginx-confd
      emptyDir: {}
    - name: opencost-ui-nginx-config-volume
      configMap:
        name: opencost-ui-nginx-config

I created a PR to changed the opencost-ui entrypoint script to only create the default.nginx.conf, if it does not exist. This way people could mount the configuration file from a configMap directly. The emptyDir for /var/www/ would still be needed, however.

kippsterr · 2024-12-04T13:18:44Z

Okay, nice. I can't currently recall where I read it or if it was just figured out by looking at it in the pod, but the volume mount basically overwrote the securityContext configuration leading to the permission denied error you mentioned above. The subPath hack, somehow, does not have this problem.

only create /etc/nginx/conf.d/default.nginx.conf if it does not yet exist (fix #4)

asdfgugus mentioned this issue Apr 22, 2024

Install helm chart with non-root user opencost/opencost-helm-chart#198

Closed

mattray transferred this issue from opencost/opencost May 23, 2024

github-actions bot added the needs-follow-up label Jul 26, 2024

mattray added the bug Something isn't working label Jul 31, 2024

kastl-ars mentioned this issue Dec 4, 2024

only create /etc/nginx/conf.d/default.nginx.conf if it does not yet exist (fix #4) #49

Merged

mittal-ishaan mentioned this issue Dec 7, 2024

Draft: allow creating an OpenShift route opencost/opencost-helm-chart#239

Open

cliffcolvin closed this as completed in c7d3146 Dec 23, 2024

cliffcolvin added a commit that referenced this issue Dec 23, 2024

Merge pull request #49 from kastl-ars/20241204_entrypoint_openshift

aa7fbcb

only create /etc/nginx/conf.d/default.nginx.conf if it does not yet exist (fix #4)

github-actions bot removed the needs-follow-up label Dec 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot set readOnlyRootFilesystem for ui container #4

Cannot set readOnlyRootFilesystem for ui container #4

zsolooo commented Mar 20, 2024

mkilchhofer commented Apr 3, 2024 •

edited

Loading

kaitimmer commented Apr 5, 2024

jamalsms commented Jul 26, 2024

morawat commented Aug 16, 2024

ervinb commented Sep 27, 2024

kastl-ars commented Dec 4, 2024

kastl-ars commented Dec 4, 2024 •

edited

Loading

kippsterr commented Dec 4, 2024

kastl-ars commented Dec 4, 2024 •

edited

Loading

kastl-ars commented Dec 4, 2024

kippsterr commented Dec 4, 2024 •

edited

Loading

Cannot set readOnlyRootFilesystem for ui container #4

Cannot set readOnlyRootFilesystem for ui container #4

Comments

zsolooo commented Mar 20, 2024

mkilchhofer commented Apr 3, 2024 • edited Loading

kaitimmer commented Apr 5, 2024

jamalsms commented Jul 26, 2024

morawat commented Aug 16, 2024

ervinb commented Sep 27, 2024

kastl-ars commented Dec 4, 2024

kastl-ars commented Dec 4, 2024 • edited Loading

kippsterr commented Dec 4, 2024

kastl-ars commented Dec 4, 2024 • edited Loading

kastl-ars commented Dec 4, 2024

kippsterr commented Dec 4, 2024 • edited Loading

mkilchhofer commented Apr 3, 2024 •

edited

Loading

kastl-ars commented Dec 4, 2024 •

edited

Loading

kastl-ars commented Dec 4, 2024 •

edited

Loading

kippsterr commented Dec 4, 2024 •

edited

Loading