v1.115.0 cannot read .immich file even with permissions

[zionun]

The bug

When updating immich server to version v1.115.0, the container does not start. The issue is related to the newly introduced check on mounted volumes.
In my case, the upload folder is a mounted volume from a NAS connected to the LAN. The folder is mounted with what I believe are the right permissions; as a confirmation to this, the .immich file in the "encoded-video" folder gets actually written with a timestamp in it, but soon after that the docker logs show an error opening this file just created and the server refuses to start.

The OS that Immich Server is running on

Ubuntu 22.04

Version of Immich Server

v1.115.0

Version of Immich Mobile App

v1.115.0

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
    healthcheck:
      disable: false

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:e3b17ba9479deec4b7d1eeec1548a253acc5374d68d3b27937fcfe4df8d18c7e
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command: ["postgres", "-c", "shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]
    restart: always

networks:
  default:
    name: caddy_net
    external: true

volumes:
  model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/media/photos
# The location where your database files are stored
DB_DATA_LOCATION=/home/[redacted]/docker/data/postgres

# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
TZ=Europe/Rome

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
# Please use only the characters `A-Za-z0-9`, without special characters or spaces
DB_PASSWORD=[redacted]

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

HOSTNAME=[redacted]

Reproduction steps

1. Download latest release with sudo docker compose pull
2. Execute sudo docker-compose up -d
3. Immich server doesn't start; see log output

Relevant log output

[Nest] 17  - 09/13/2024, 9:41:53 AM     LOG [Api:StorageService] Verifying system mount folder checks
[Nest] 17  - 09/13/2024, 9:41:53 AM     LOG [Api:StorageService] Writing initial mount file for the encoded-video folder
[Nest] 17  - 09/13/2024, 9:41:53 AM   ERROR [Api:StorageService] Failed to write upload/encoded-video/.immich: Error: EACCES: permission denied, open 'upload/encoded-video/.immich'
[Nest] 17  - 09/13/2024, 9:41:53 AM   ERROR [Api:StorageService] The "encoded-video" folder cannot be written to, please make sure the volume is mounted with the correct permission

Additional information

The .immich file actually does get written inside the encoded-video folder.
this is the entry in fstab to mount the drive:

//[redacted local ip/Photos /media/photos cifs credentials=/home/[redacted]/.smbcredentials,uid=1000,gid=1000,x-systemd.automount,x-systemd.requires=network-online.target 0 0

these are the permissions of /media/photos
drwxr-xr-x 2 [redacted user - uid 1000] [redacted group - gid 1000] 0 Sep 12 22:00 encoded-video

[pwyde]

Similar setup and the same issue here too. Difference from OP is that I use TrueNAS SCALE as storage backend to my Kubernetes cluster. NFS shares are used for storage in my cluster.

This setup worked previously with v1.114.0.

Permissions on TrueNAS:

$ ls -lan /mnt/hdd-data/Immich/
total 42
drwxrwx--- 3 5000 3010 3 Sep 13 12:34 .
drwxr-xr-x 9    0    0 9 Sep 13 11:20 ..
drwxr-xr-x 2 5000 3010 2 Sep 13 12:34 library

From within the Immich deployment:

...@immich-machine-learning-58885c5dcc-k8p8s:/usr/src/app$ id
uid=5000 gid=3010 groups=3010

...@immich-machine-learning-58885c5dcc-k8p8s:/usr/src/app$ echo $UPLOAD_LOCATION
/usr/src/app/upload

...@immich-machine-learning-58885c5dcc-k8p8s:/usr/src/app$ ls -la /usr/src/app/upload
total 17
drwxrwx--- 3 5000 3010  3 Sep 13 12:34 .
drwxr-xr-x 1 root root 20 Sep 13 12:36 ..
drwxr-xr-x 2 5000 3010  2 Sep 13 12:34 library

Note that container is running with the UID and GID of 5000/3010 which is the user and group owner of the directory used by Immich.

The directory is shared to Immich via NFS. One strange thing is that I am able to write to the NFS share from the Immich deployment. The immich-machine-learningcontainer gets deployed and from it I can access NFS share:

...@immich-machine-learning-7cb9644f4-zj288:/usr/src/app$ touch /usr/src/app/upload/library/test-file
...@immich-machine-learning-7cb9644f4-zj288:/usr/src/app$ mkdir /usr/src/app/upload/library/test-dir
...@immich-machine-learning-7cb9644f4-zj288:/usr/src/app$ ls -la /usr/src/app/upload/library/

total 26
drwxr-xr-x 3 5000 3010 4 Sep 13 12:40 .
drwxrwx--- 3 5000 3010 3 Sep 13 12:34 ..
drwxr-xr-x 2 5000 3010 2 Sep 13 12:40 test-dir
-rw-r--r-- 1 5000 3010 0 Sep 13 12:40 test-file

...@immich-machine-learning-7cb9644f4-zj288:/usr/src/app$ rm /usr/src/app/upload/library/test-file
...@immich-machine-learning-7cb9644f4-zj288:/usr/src/app$ rm -rf /usr/src/app/upload/library/test-dir/

In contrast to OP, my deployment is never able to create the .immich file, but able to access the directory and write to it according to the test result above.

Log from the main Immich container:

$ kubectl -n self-hosted logs deployments/immich --follow
.
.
.
[Nest] 37  - 09/13/2024, 12:42:54 PM   ERROR [Api:StorageService] Failed to read upload/encoded-video/.immich: Error: ENOENT: no such file or directory, open 'upload/encoded-video/.immich'
[Nest] 37  - 09/13/2024, 12:42:54 PM   ERROR [Api:StorageService] The "encoded-video" folder appears to be offline/missing, please make sure the volume is mounted with the correct permissions
api worker exited with code 1

Edit: Upgrading from v1.114.0 to v1.115.0 actually works. The error above appears during a completely new deployment.

[metri]

I have the same problem, but after upgrading from version 1.14, not after a clean install.

[zackpollard]

Similar setup and the same issue here too. Difference from OP is that I use TrueNAS SCALE as storage backend to my Kubernetes cluster. NFS shares are used for storage in my cluster.

This setup worked previously with v1.114.0.

Permissions on TrueNAS:

$ ls -lan /mnt/hdd-data/Immich/
total 42
drwxrwx--- 3 5000 3010 3 Sep 13 12:34 .
drwxr-xr-x 9    0    0 9 Sep 13 11:20 ..
drwxr-xr-x 2 5000 3010 2 Sep 13 12:34 library

From within the Immich deployment:

...@immich-machine-learning-58885c5dcc-k8p8s:/usr/src/app$ id
uid=5000 gid=3010 groups=3010

...@immich-machine-learning-58885c5dcc-k8p8s:/usr/src/app$ echo $UPLOAD_LOCATION
/usr/src/app/upload

...@immich-machine-learning-58885c5dcc-k8p8s:/usr/src/app$ ls -la /usr/src/app/upload
total 17
drwxrwx--- 3 5000 3010  3 Sep 13 12:34 .
drwxr-xr-x 1 root root 20 Sep 13 12:36 ..
drwxr-xr-x 2 5000 3010  2 Sep 13 12:34 library

Note that container is running with the UID and GID of 5000/3010 which is the user and group owner of the directory used by Immich.

The directory is shared to Immich via NFS. One strange thing is that I am able to write to the NFS share from the Immich deployment. The immich-machine-learningcontainer gets deployed and from it I can access NFS share:

...@immich-machine-learning-7cb9644f4-zj288:/usr/src/app$ touch /usr/src/app/upload/library/test-file
...@immich-machine-learning-7cb9644f4-zj288:/usr/src/app$ mkdir /usr/src/app/upload/library/test-dir
...@immich-machine-learning-7cb9644f4-zj288:/usr/src/app$ ls -la /usr/src/app/upload/library/

total 26
drwxr-xr-x 3 5000 3010 4 Sep 13 12:40 .
drwxrwx--- 3 5000 3010 3 Sep 13 12:34 ..
drwxr-xr-x 2 5000 3010 2 Sep 13 12:40 test-dir
-rw-r--r-- 1 5000 3010 0 Sep 13 12:40 test-file

...@immich-machine-learning-7cb9644f4-zj288:/usr/src/app$ rm /usr/src/app/upload/library/test-file
...@immich-machine-learning-7cb9644f4-zj288:/usr/src/app$ rm -rf /usr/src/app/upload/library/test-dir/

In contrast to OP, my deployment is never able to create the .immich file, but able to access the directory and write to it according to the test result above.

Log from the main Immich container:

$ kubectl -n self-hosted logs deployments/immich --follow
.
.
.
[Nest] 37  - 09/13/2024, 12:42:54 PM   ERROR [Api:StorageService] Failed to read upload/encoded-video/.immich: Error: ENOENT: no such file or directory, open 'upload/encoded-video/.immich'
[Nest] 37  - 09/13/2024, 12:42:54 PM   ERROR [Api:StorageService] The "encoded-video" folder appears to be offline/missing, please make sure the volume is mounted with the correct permissions
api worker exited with code 1

Edit: Upgrading from v1.114.0 to v1.115.0 actually works. The error above appears during a completely new deployment.

Hey, could you please post your NFS mount point config on the immich host machine?

[zackpollard]

I have the same problem, but after upgrading from version 1.14, not after a clean install.

Please confirm if you're using a networked filesystem and if so, please post your config parameters for that mount point.

[metri]

I have the same problem, but after upgrading from version 1.14, not after a clean install.

Please confirm if you're using a networked filesystem and if so, please post your config parameters for that mount point.

Yes, like the topic author I use cifs.

[pwyde]

Hey, could you please post your NFS mount point config on the immich host machine?

I am running a Kubernetes cluster managed by Flux. Hence, I am not mounting the NFS share on the host machine. The NFS share is mounted inside the Immich pod like this:

persistence:
  upload:
    type: nfs
    server: <TrueNAS SCALE IP>
    path: /mnt/hdd-data/Immich
    globalMounts:
      - path: /usr/src/app/upload

[ShunsukeMatsuno]

I encounter the same issue after updating to v1.115. In addition to this, I am running immich along with tailscale, and the tailscale server complains that connection to immich server is refused, and immich does not start. So, I am guessing that there is some permission setting messed up in the new release. Reverting back to v1.114 solves the issue. Here is the relevant part of docker compose:

  services:
  immich-ts:
    image: tailscale/tailscale:latest
    container_name: immich-ts
    hostname: immich-ts
    environment:
      - TS_AUTHKEY=REDACTED
      - "TS_EXTRA_ARGS=--advertise-tags=tag:container --reset"
      - TS_SERVE_CONFIG=/config/immich.json
      - TS_STATE_DIR=/var/lib/tailscale
      - TS_USERSPACE=false
    volumes:
      - ${PWD}/config:/config
      - immich-ts:/var/lib/tailscale
      - /dev/net/tun:/dev/net/tun
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    cap_add:
      - net_admin
      - sys_module
    restart: unless-stopped
    
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    network_mode: "service:immich-ts"
    depends_on:
      - redis
      - database
    restart: always

and .env file:

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/mnt/hdd/immich-library

# The location where your database files are stored
DB_DATA_LOCATION=./postgres

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
#IMMICH_VERSION=release    # This does not work
IMMICH_VERSION=v1.114.0   

# Connection secret for postgres. You should change it to a random password
DB_PASSWORD=postgres

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

My OS is Fedora 40 Server Edition.

Edit:
UPLOAD_LOCATION is set us an external HDD that is plugged into the computer, and it is mounted using the default option, which I understand use async:

/dev/sda1 on /mnt/hdd type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)

[nilsburg]

I am having the same issues after updating from 1.114.0

immich_server            | [Nest] 17  - 09/14/2024, 11:37:16 AM     LOG [Api:StorageService] Verifying system mount folder checks
immich_server            | [Nest] 17  - 09/14/2024, 11:37:16 AM     LOG [Api:StorageService] Writing initial mount file for the encoded-video folder
immich_server            | [Nest] 17  - 09/14/2024, 11:37:16 AM     LOG [Api:StorageService] Writing initial mount file for the library folder
immich_server            | [Nest] 17  - 09/14/2024, 11:37:16 AM     LOG [Api:StorageService] Writing initial mount file for the upload folder
immich_server            | [Nest] 17  - 09/14/2024, 11:37:16 AM     LOG [Api:StorageService] Writing initial mount file for the profile folder
immich_server            | [Nest] 17  - 09/14/2024, 11:37:16 AM   ERROR [Api:StorageService] Failed to write upload/profile/.immich: Error: EACCES: permission denied, mkdir 'upload/profile'
immich_server            | [Nest] 17  - 09/14/2024, 11:37:16 AM   ERROR [Api:StorageService] The "profile" folder cannot be written to, please make sure the volume is mounted with the correct permissions

.env file:

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/mnt/media/library/
# The location where your database files are stored
DB_DATA_LOCATION=./postgres

# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
# TZ=Etc/UTC

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=v1.115.0

# Connection secret for postgres. You should change it to a random password
DB_PASSWORD=postgres

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

/mnt/media/library is a NFS share

My docker-compose.yml

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      - /mnt/media/imagenes/camara/:/media/libraries/camara:ro
      - /mnt/media/camera-roll/:/media/libraries/camera-roll:ro
      - /mnt/media/imagenes/fotos/:/media/libraries/fotos:ro
      - /mnt/media/imagenes/movil/:/media/libraries/movil:ro
      - /mnt/media/videos/:/media/libraries/videos:ro
      - /mnt/media/imagenes/ipod/:/media/libraries/ipod:ro
      - /mnt/media/imagenes/import/:/media/libraries/import:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-openvino
    extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
      file: hwaccel.ml.yml
      service: openvino # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:d6c2911ac51b289db208767581a5d154544f2b2fe4914ea5056443f62dc6e900
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
#      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
       - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command: ["postgres", "-c" ,"shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]
    restart: always

volumes:
  model-cache:
  pgdata:

Version 1.114.0 works without issues I can create any file inside the upload/profile/ folder without issues from inside the docker immich_server container

[IanMajor12]

I'm also seeing this problem, using a CIFS share, with credentials stored in file (more secure) and accessing a folder on TrueNAS core (but not a specific dataset. I've reset all the permissions on TrueNAS core to be the UID/PID and it was working in 1.14 for me just fine.

Can also confirm the file is written just fine on the start of the the docker-compose but fails the check to find the file in

[Nest] 17 - 09/14/2024, 8:00:02 AM LOG [Api:StorageService] Verifying system mount folder checks
[Nest] 17 - 09/14/2024, 8:00:02 AM LOG [Api:StorageService] Writing initial mount file for the encoded-video folder
[Nest] 17 - 09/14/2024, 8:00:02 AM ERROR [Api:StorageService] Failed to write upload/encoded-video/.immich: Error: EACCES: permission denied, open 'upload/encoded-video/.immich'
[Nest] 17 - 09/14/2024, 8:00:02 AM ERROR [Api:StorageService] The "encoded-video" folder cannot be written to, please make sure the volume is mounted with the correct permissions

Seems to be in the path for the check has an extra upload/ folder appended maybe?

[bo0tzz]

Can you all check whether your network mounts are mounted with sync or async writes?

[zionun]

@bo0tzz I'm not sure how to check this. With cat /proc/mounts I see no "sync/async" info on that mount point. I guess it is async since afaik it should be default and I did not set a specific type

[IanMajor12]

I'm using the defaults on Debian 12.5 to TrueNAS Core 13.6U2
According to this, it's async for TrueNAS for SMB and sync for NFS

#https://www.truenas.com/community/threads/why-is-smb-async-and-nfs-sync.107227/

I don't see anything in the underlying config files or TrueNAS config files

[ShunsukeMatsuno]

@bo0tzz I edited my comment above, but I am not using network mount. I mount an external HDD in the xfs file system, and I understand that it is mounted with the default async

[bo0tzz]

Can you try mounting explicitly with sync and see if the issue still happens?

[ShunsukeMatsuno]

In my case mounting the external hdd (xfs) with sync did not resolves the issue

[hoyazot]

I am having this problem in encoded-video, but immich is able to write .immich -- confirmed that by removing the file (immich recreates it), touching an empty file (immich overwrites it and file size increases)

[zackpollard]

Can you confirm what os you're running , what os your network share is on and could you show us the mount options for that share in fstab?

[hoyazot]

Can you confirm what os you're running ,

Windows, Docker Desktop 4.34.2

what os your network share is on

TrueNAS Core 13.0-U6.1

and could you show us the mount options for that share in fstab?

https://i.imgur.com/52bb5D9.png

--

this is how the CIFS mount is configured in the YML:

volumes:
  nas-share:
    driver_opts:
      type: cifs
      o: "username=username,password=password,addr=192.168.25.240,vers=3.0,file_mode=0777,dir_mode=0777"
      device: "//SERVERNAME/ShareName/immich/upload"
  model-cache:

[jrasm91]

Turns out the latest code is incompatible, not with writing, but with overwriting hidden files in Windows. This will be fixed in the next release.

[wouterverduin]

@jrasm91

I am running in the same issue. Also truenas share and docker compose in a proxmox lxc.

This is my error:

[Nest] 17 - 09/20/2024, 8:36:47 AM LOG [Api:StorageService] Verifying system mount folder checks [Nest] 17 - 09/20/2024, 8:36:47 AM LOG [Api:StorageService] Writing initial mount file for the encoded-video folder [Nest] 17 - 09/20/2024, 8:36:47 AM ERROR [Api:StorageService] Failed to write upload/encoded-video/.immich: Error: EACCES: permission denied, open 'upload/encoded-video/.immich' [Nest] 17 - 09/20/2024, 8:36:47 AM ERROR [Api:StorageService] The "encoded-video" folder cannot be written to, please make sure the volume is mounted with the correct permissions

However. Creating the profile folder and doing touch .immich in each folder doenst resolve the error.

The way i mount my share:
//192.168.1.118/Media /MEDIA cifs nobrl,x-systemd.automount,x-systemd.requires=network-online.target,user=XXX,pass=XXX 0 0

Must be noted that i dont have any permission issues with other folders/containers.

Is this also going to be fixed next release although i dont use Windows?

Thanks in advance!

[deddc23efb]

Turns out the latest code is incompatible, not with writing, but with overwriting hidden files in Windows. This will be fixed in the next release.

Can you provide more detail on this? Does this ONLY affect Immich hosted on windows systems accessing a NAS? TrueNAS only? CIFS?

How about Linux Docker containers hosting Immich accessing the same NAS filesystems? Are they safe to upgrade?

[mertalev]

From what we've seen so far, the hidden file issue affects Immich instances hosted on a Windows machine that use a CIFS mount to a share with the hide dot files setting enabled. We haven't seen this issue when Immich is on native Linux (not WSL2). Even on Windows, some people can reproduce the issue and others can't.

There are other cases of permission issues where immich can't create the needed folder, like if the parent folder forbids immich from writing but the child folders don't. For existing instances, this can affect the profile and encoded-videos folders: the former only exists if you set a profile picture and the latter only exists if you ever transcoded a video. If you never did these and either manually created the library, thumbs and upload folders with different permissions than the parent folder or later changed the parent folder's permissions, it can cause this situation.

It's unclear to me if NFS is affected by the hidden file issue and whether the ENOENT error reported on NFS is related.

[deddc23efb]

Thanks for the details. For the record, based on the description I went ahead and upgraded from 114 to 115 Ubuntu Server Host running Immich docker accessing a mount point on the host, CIFS mount on TrueNAS network server. .immich files created no issues, immich is happy.

[sacredd]

The same problem with a clean installation of v1.115.0 When should I expect the update?

[bo0tzz]

Probably next week

[jrasm91]

You can temporarily fix it by removing the hidden attribute from the .immich files

[sacredd]

Sorry, but how I can do it. I use Hdd (ext4) connect to Zyxel Keenetic and shared. I am use raspberry pi4 B, mount hdd from fstab
//192.168.7.1/1TBMedia /mnt/Keenetic cifs _netdev,credentials=/home/pi/.smbclient,uid=pi,gid=pi,x-systemd.automount 0 0
immich create folders and file /mnt/Keenetic/opt/immich-app/encoded-video/.immich
but program doesnt start
Failed to write /opt/immich-app/encoded-video/.immich: Error: EACCES: permission denied, open '/opt/immich-app/encoded-video/.immich'
I must chmod 755 /mnt/Keenetic/opt/immich-app/encoded-video/.immich ?

[mertalev]

No, chmod isn't enough in this case. I think configuring the SMB server with hide dot files = false and deleting the .immich file should also work.

[dgtal1]

Experienced the same issue when using CIFS volumes (essentially doing indexing+transcoding on a PC while storing the files on a NAS). Downgraded to 1.114 and it solved it for me. Must be either some breaking change in 1.115 or some bug in it.

[adamhl8]

I was able to get v1.115.0 to start by recreating the .immich file with the proper permissions in each directory (encoded-video, library, profile, thumbs, upload):

rm .immich && touch .immich && chmod 666 .immich

[metri]

This doesn't work for me either.

[adamhl8]

This doesn't work for me either.

I found that just recreating the file won't work. It needs to be readable/writable by other. Hence the chmod 666

[sacredd]

This only works on Windows share, not on Linux ore ext4, I haven’t fully figured it out yet. On ext4 this file is considered system and it is not possible to change its attributes.

[metri]

Yep. I also use cifs share.

[grannysmith59]

Works for me on Unraid.

Thanks

[zackpollard]

For everyone here, a new release should be coming today (1.116.0) which will resolve the issues with CIFS and windows shares. If you're still experiencing issues after that, it'll either be because of trying and failing to upgrade to 1.115.0, or actual permission problems. There will be some instructions in the release notes if you still experience issues, and if not you can continue posting here or on discord for support 😄

[aaronspruit]

Sadly, I'm still experiencing the error with 1.116. The error has slightly changed though.

immich-server �[32m[Nest] 17 - �[39m09/26/2024, 10:31:06 AM �[32m LOG�[39m �[33m[Api:StorageService]�[39m �[32mVerifying system mount folder checks (enabled=true)�[39m immich-server �[31m[Nest] 17 - �[39m09/26/2024, 10:31:06 AM �[31m ERROR�[39m �[33m[Api:StorageService]�[39m �[31mFailed to read upload/encoded-video/.immich: Error: ENOENT: no such file or directory, open 'upload/encoded-video/.immich'�[39m immich-server �[31m[Nest] 17 - �[39m09/26/2024, 10:31:06 AM �[31m ERROR�[39m �[33m[Api:StorageService]�[39m �[31mThe "encoded-video" folder appears to be offline/missing, please make sure the volume is mounted with the correct permissions�[39m

I'm running this in a k3s cluster and my library is mounted to a remote NFS share via a PVC. If i change the tags to 1.114 everything works fine, so it appears something in the way it's access this share has changed (maybe it's how it's using the user account to access it) in 1.115.0.

[jrasm91]

This means the database says "you should have .immich files" but the filesystem is saying "there are no .immich files". If you deleted them when you rolled-back that would cause this error. You can just create .immich files in the top level folders and that should fix it.

[RK3356]

I have just upgraded from version 115.1 (which was working) to 116.0 and am getting this error [Nest] 17 - 09/26/2024, 8:16:25 PM ERROR [Api:StorageService] Failed to read upload/library/.immich: Error: ENOENT: no such file or directory, open 'upload/library/.immich'
[Nest] 17 - 09/26/2024, 8:16:25 PM ERROR [Api:StorageService] The "library" folder appears to be offline/missing, please make sure the volume is mounted with the correct permissions

How can I resolve this as I am unclear if I should create .immich files, and if so what they should contain, where they should be placed, and what suffix to use for the files.

Thanks

[joshsingley]

Mine is finally working. I had to change my auto mount in /etc/fstab to use a different user.

[Luvbe3]

I just encountered this error too. and solved it by deleting all the .immich files in the folders. then restarting docker immich is ok.

[psla]

I got this error on upload folder. I do not use NFS or network mounted folders. The folder was mounted and working normally on 1.115.

This started happening after upgrade from 1.115 to 1.116.

Honestly, this could have been my fault - I was cleaning upload folder from orphans, and I potentially might have removed the .immich file. Posting the "repair" steps here for others.

[Nest] 17  - 09/28/2024, 2:31:25 AM     LOG [Api:StorageService] Verifying system mount folder checks (enabled=true)
[Nest] 17  - 09/28/2024, 2:31:25 AM   ERROR [Api:StorageService] Failed to read upload/upload/.immich: Error: ENOENT: no such file or directory, open 'upload/upload/.immich'
[Nest] 17  - 09/28/2024, 2:31:25 AM   ERROR [Api:StorageService] The "upload" folder appears to be offline/missing, please make sure the volume is mounted with the correct permissions

(it's interesting to see upload/upload/).

      - ${UPLOAD_LOCATION}:/usr/src/app/upload

UPLOAD_LOCATION=/mnt/nas/srv/immich/pictures

unprivileged-docker@usa:~/immich$ ls -ald /mnt/nas/srv/immich/pictures
drwxr-xr-x 1 unprivileged-docker unprivileged-docker 78 Jul 31  2023 /mnt/nas/srv/immich/pictures
unprivileged-docker@usa:~/immich$ ls -ald /mnt/nas/srv/immich/pictures/upload/
drwxr-xr-x 1 unprivileged-docker unprivileged-docker 144 Sep 15 13:23 /mnt/nas/srv/immich/pictures/upload/
unprivileged-docker@usa:~/immich$ ls -al /mnt/nas/srv/immich/pictures/upload/
total 0
drwxr-xr-x 1 unprivileged-docker unprivileged-docker  144 Sep 15 13:23 .
drwxr-xr-x 1 unprivileged-docker unprivileged-docker   78 Jul 31  2023 ..
drwxr-xr-x 1 unprivileged-docker unprivileged-docker 1024 Sep 15 13:23 d38cadab-260f-4e27-8c8a-a11a7a627894
drwxr-xr-x 1 unprivileged-docker unprivileged-docker 1024 Sep 15 13:23 da2f2967-4e69-4be5-ae15-385e3b4b0ea9

The solution was to

cd /mnt/nas/srv/immich/pictures/upload
touch .immich
chown unprivileged-docker:unprivileged-docker .immich      ## use the username/account that your docker should have access to

[g0fish]

I am using v116.2 from HomeAssistant addon build (alexbelgium). It occurred in v115, so I uninstalled, removed directories, and performed a new version install where the problem continues. I created the .immich manually after the error appeared and the output shows in screenshot

Other files and folders around the system are all with root and do not have any other user I know of.

I am using CIFS; and window share, and it's running on VMware Workstation.

[jrasm91]

It looks like the error is with creating the .immich file. It's not expecting it to exist but it already does

[g0fish]

The original error

[Luvbe3]

i'm solved it by deleting all the .immich files in the library folders. then restarting docker immich is ok.

[tuupola]

Same problem with K3S + Longhorn setup. Fresh v1.114.0 install works fine. With both fresh install or upgrade of v1.116.2 and server startup fails with:

[Nest] 17  - 10/01/2024, 11:53:02 AM     LOG [Api:StorageService] Verifying system mount folder checks (enabled=true)
[Nest] 17  - 10/01/2024, 11:53:02 AM   ERROR [Api:StorageService] Failed to read upload/encoded-video/.immich: Error: ENOENT: no such file or directory, open 'upload/encoded-video/.immich'
[Nest] 17  - 10/01/2024, 11:53:02 AM   ERROR [Api:StorageService] The "encoded-video" folder appears to be offline/missing, please make sure the volume is mounted with the correct permissions
api worker exited with code 1

[Luvbe3]

Same problem with K3S + Longhorn setup. Fresh v1.114.0 install works fine. With both fresh install or upgrade of v1.116.2 and server startup fails with:

[Nest] 17  - 10/01/2024, 11:53:02 AM     LOG [Api:StorageService] Verifying system mount folder checks (enabled=true)
[Nest] 17  - 10/01/2024, 11:53:02 AM   ERROR [Api:StorageService] Failed to read upload/encoded-video/.immich: Error: ENOENT: no such file or directory, open 'upload/encoded-video/.immich'
[Nest] 17  - 10/01/2024, 11:53:02 AM   ERROR [Api:StorageService] The "encoded-video" folder appears to be offline/missing, please make sure the volume is mounted with the correct permissions
api worker exited with code 1

deleting all the .immich files in the library folders. then restarting docker immich is done.

[dery-dery]

After removing .immich files from library folders my setup worked again.

[IanMajor12]

Finally got this to work with v.1.117.0 by logging into the server and creating the .immich files in the 4 folders under the path you have set for UPLOAD_LOCATION in .env. Note I had to create the profile folder too. All done as the same user that the docker runs as (so that permissions align). Now able to start up. The error message is misleading as it says upload/encoded-video which I read as encoded-video should be under the upload folder, but they are separate folders at the root level of the UPLOAD_LOCATION.

[thirtythreeforty]

Agreed, had to remediate this manually. Can confirm it works after creating profile/ and the various .immich files.

[rhardih]

I've been holding off upgrading from 1.114, because of this error, but now we're at 1.120. Has anyone successfully upgraded without manual intervention afterwards?

[alextran1502]

Many AFAIK

[dgtal1]

Creating the files didn't fix it for me even in 0.120, but fortunately you can workaround this by disabling the checks in the config.
The setup works fine for me on my Synology hosting NAS, but on my PC which I use for performance indexing via SMB it doesn't work without the checks being disabled

[scrampker]

I mean the fix is dead simple even if you do experience it. Like 30 seconds of cli effort. I was able to handle it via windows on my cifs share. I definitely wouldn't let it stop you. Make a backup first of it's a real concern.

[IanMajor12]

I upgraded to .116 and it's been smooth ever since. I did have to create the folders and files manually, but all good since that.

[julian-alarcon]

I had the same issue, I just edited one path (Backup) in my TrueNAS 24.10 to another location, I even moved the backup files to the new path, but I was not aware of that .immich file, I just moved it to the new location and that was all that I needed. Service started with no issues.