MSSQL pod fail on crashloopBackOff

[sebglon]

Steps To Reproduce

deploy minimal helm values:

database:
  enabled: true
  volume:
    backups:
      size: 20Gi
    data:
      size: 20Gi
    log:
      size: 10Gi
general:
  admins: xxxx
  domain: bitwarden.env.fr
  email:
    replyToEmail: [email protected]
    smtpHost: toto.env.fr
    smtpPort: "587"
  volumeAccessMode: ReadWriteOnce
secrets:
  secretName: custom

Expected Result

mssql pod is up and running

Actual Result

mssql log:

SQL Server 2019 will run as non-root by default.
This container is running as user mssql.
To learn more visit https://go.microsoft.com/fwlink/?linkid=2099216.
2023-12-04 16:54:47.94 Server      The licensing PID was successfully processed. The new edition is [Express Edition].
2023-12-04 16:54:48.81 Server      Setup step is copying system data file 'C:\templatedata\master.mdf' to '/var/opt/mssql/data/master.mdf'.
2023-12-04 16:54:48.82 Server      ERROR: Setup FAILED copying system data file 'C:\templatedata\master.mdf' to '/var/opt/mssql/data/master.mdf':  5(Access is denied.)
ERROR: BootstrapSystemDataDirectories() failure (HRESULT 0x80070005)
Stream closed EOF for bitwarden/bitwarden-self-host-mssql-0 (bitwarden-self-host-mssql)

kube event:

    state:
      waiting:
        message: back-off 5m0s restarting failed container=bitwarden-self-host-mssql
          pod=bitwarden-self-host-mssql-0_bitwarden(61aea36e-717f-4732-81ff-ab64ca6bbc90)
        reason: CrashLoopBackOff

Screenshots or Videos

No response

Additional Context

No response

Chart Version

self-host-0.1.7-Beta

Environment Details

kubernetes self-hosted: 1.22.2

Issue Tracking Info

• I understand that work is tracked outside of Github. A PR will be linked to this issue should one be opened to address it, but Bitwarden doesn't use fields like "assigned", "milestone", or "project" to track progress.

[sebglon]

If i replace the default image mcr.microsoft.com/mssql/server:2019-CU17-ubuntu-20.04 by this one bitwarden/mssql:2023.8.3 all work well

I don't know why

[jhbeskow]

I have seen this caused by having DB files out there created from a previous install that is running under a different security context (i.e. a different service account) or issues with the storage class permissions. Which flavor of Kubernetes are you running?

[ebCrypto]

I am experiencing the same problem here. Running on Talos 1.6.1.

apiVersion: v1
kind: Namespace
metadata:
  name: bitwarden
  labels:
    toolkit.fluxcd.io/tenant: dev-team
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/audit: privileged
    pod-security.kubernetes.io/warn: privileged

 #
    # Configure database
    #
    database:
      # deploy the database pod?  if false, a connection string to a SQL Server will need to be provided through one of the configuration providers.
      enabled: true
      # Labels to add to the MSSQL deployment
      labels: {}
      # Image name, tag, and pull policy
      image:
        name: mcr.microsoft.com/mssql/server
        # Tag of the image to use. (Defaults to general.coreVersion)
        tag: 2019-CU17-ubuntu-20.04
      # The container is limited to the resources below.  Adjust for your environment.
      resources:
        requests:
          memory: "2G"
          cpu: "100m"
        limits:
          memory: "2G"
          cpu: "500m"

      # The MSSQL volumes for the PVCs
      volume:
        backups:
          # Storage size
          size: 1Gi
        data:
          # Storage size
          size: 10Gi
        log:
          # Storage size
          size: 10Gi
      # Specifies the strategy used to replace old Pods by new ones. The value can be "OnDelete" or "RollingUpdate". "OnDelete" is the default value.
      # When set to OnDelete, the SQL pod will not be recreated automatically on update.  It must be manually deleted to update.
      updateStrategy: OnDelete
      # This will set the Kubernetes container security context
      securityContext:
      # Run the pod under a service account you create.  This is especially useful for OpenShift deployments
      podServiceAccount:

[Kaylleur]

I think this charts has some missing init containers to set up correct rights on volume before starting main containers. I have the same error on all the pods about permission denied on each volumes

[momelod]

In my case i was attaching a PVC for the data volume and was getting the same permission error. I can get the pod to start, but I know there must be a better way.

helm template bitwarden/self-host --values /tmp/bitwarden-values.yaml > /tmp/bitwarden-rendered.yaml

Then i edit /tmp/bitwarden-rendered.yaml and add spec.template.spec.securityContext.fsGroup: 10001:

      securityContext:
        fsGroup: 10001

Finally kubectl apply -f /tmp/bitwarden-rendered.yaml

And here is just a sample from the file where the change needs to be made:

---
# Source: self-host/templates/mssql.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  ...
spec:
  serviceName: release-name-self-host-mssql
  template:
     spec:
      securityContext:
        fsGroup: 10001
      containers:
        - name: release-name-self-host-mssql
          image: "mcr.microsoft.com/mssql/server:2022-CU11-ubuntu-22.04"

[Niich]

I was also having this issue.

I am using the k8s sigs nfs provisioner and I was able to get past this error by setting runAsUser: 10001 under the database.securityContext available in the values file.

I also removed the old files so the pod could recreate them with the correct UID.

#
# Configure database
#
database:
  # deploy the database pod?  if false, a connection string to a SQL Server will need to be provided through one of the configuration providers.
  enabled: true
  # Labels to add to the MSSQL deployment
  labels: {}
  # Image name, tag, and pull policy
  image:
    name: mcr.microsoft.com/mssql/server
    # Tag of the image to use. (Defaults to general.coreVersion)
    tag: 2022-CU11-ubuntu-22.04


  # This will set the Kubernetes container security context
  securityContext:
    runAsUser: 10001
  # Run the pod under a service account you create.  This is especially useful for OpenShift deployments
  podServiceAccount:

[jorgevisentini]

I was also having this issue.

I am using the k8s sigs nfs provisioner and I was able to get past this error by setting runAsUser: 10001 under the database.securityContext available in the values file.

I also removed the old files so the pod could recreate them with the correct UID.

#
# Configure database
#
database:
  # deploy the database pod?  if false, a connection string to a SQL Server will need to be provided through one of the configuration providers.
  enabled: true
  # Labels to add to the MSSQL deployment
  labels: {}
  # Image name, tag, and pull policy
  image:
    name: mcr.microsoft.com/mssql/server
    # Tag of the image to use. (Defaults to general.coreVersion)
    tag: 2022-CU11-ubuntu-22.04


  # This will set the Kubernetes container security context
  securityContext:
    runAsUser: 10001
  # Run the pod under a service account you create.  This is especially useful for OpenShift deployments
  podServiceAccount:

For me, only as root user. But worked.

securityContext:
runAsUser: 0

[The-Pizza]

I have resolved this with a few changes to the mssql.yaml template. Hopefully this can be corrected in the official helm charts.

First is to change the "startupProbe" command on the container to:
'/opt/mssql-tools18/bin/sqlcmd -S localhost -U sa -P ${SA_PASSWORD} -Q "SELECT 1" -C || exit 1'

This will ignore the cert error the command gets with -C as well as corrects the broken path to "mssql-tools18"

Second is to correct permissions on the folder structure by adding an init container:

  initContainers:
    - name: init-permissions
      image: busybox
      command:
        - '/bin/sh'
        - '-c'
        - 'chown -R 10001:root /var/opt/mssql/data /var/opt/mssql/log /var/opt/mssql/backups'
      resources: {}
      volumeMounts:
        - name: mssql-data
          mountPath: /var/opt/mssql/data
        - name: mssql-log
          mountPath: /var/opt/mssql/log
        - name: mssql-backups
          mountPath: /var/opt/mssql/backups
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      imagePullPolicy: Always
      securityContext:
        runAsUser: 0