Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Healthcheck on db fails #10219

Closed
1 of 3 tasks
dronnikovigor opened this issue Jun 12, 2024 · 6 comments · Fixed by #10221
Closed
1 of 3 tasks

Healthcheck on db fails #10219

dronnikovigor opened this issue Jun 12, 2024 · 6 comments · Fixed by #10221

Comments

@dronnikovigor
Copy link

The bug

Healthcheck on db fails. Container is marked as unhealthy.

In logs of db i have:

2024-06-12 14:55:34.901 UTC [1] LOG: starting PostgreSQL 14.10 (Debian 14.10-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
2024-06-12 14:55:34.902 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2024-06-12 14:55:34.902 UTC [1] LOG: listening on IPv6 address "::", port 5432
2024-06-12 14:55:34.915 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2024-06-12 14:55:34.930 UTC [28] LOG: database system was shut down at 2024-06-12 14:55:34 UTC
[2024-06-12T14:55:34Z INFO service::utils::clean] Find directory "pg_vectors/indexes/17338".
[2024-06-12T14:55:34Z INFO service::utils::clean] Find directory "pg_vectors/indexes/17337".
[2024-06-12T14:55:34Z INFO service::utils::clean] Find directory "pg_vectors/indexes/17338/segments/6bf96433-b28b-4757-ba3a-b32f108eaf0e".
2024-06-12 14:55:34.952 UTC [1] LOG: database system is ready to accept connections
[2024-06-12T14:55:35Z INFO service::utils::clean] Find directory "pg_vectors/indexes/17337/segments/1fc08397-86a6-448e-b68f-b3eb84191800".
2024-06-12 14:56:04.960 UTC [71] FATAL: role "root" does not exist
2024-06-12 14:56:35.165 UTC [82] FATAL: role "root" does not exist
2024-06-12 14:57:05.314 UTC [226] FATAL: role "root" does not exist
2024-06-12 14:57:35.442 UTC [237] FATAL: role "root" does not exist
2024-06-12 14:58:05.616 UTC [269] FATAL: role "root" does not exist
2024-06-12 14:58:35.809 UTC [280] FATAL: role "root" does not exist
2024-06-12 14:59:06.010 UTC [290] FATAL: role "root" does not exist
2024-06-12 14:59:36.197 UTC [301] FATAL: role "root" does not exist
2024-06-12 15:00:06.360 UTC [311] FATAL: role "root" does not exist
2024-06-12 15:00:36.498 UTC [322] FATAL: role "root" does not exist
2024-06-12 15:05:36.694 UTC [337] FATAL: role "root" does not exist
2024-06-12 15:10:36.885 UTC [352] FATAL: role "root" does not exist
2024-06-12 15:15:37.078 UTC [367] FATAL: role "root" does not exist
2024-06-12 15:20:37.279 UTC [400] FATAL: role "root" does not exist
2024-06-12 15:25:37.448 UTC [419] FATAL: role "root" does not exist
2024-06-12 15:30:37.648 UTC [436] FATAL: role "root" does not exist
2024-06-12 15:35:37.856 UTC [457] FATAL: role "root" does not exist
2024-06-12 15:40:38.070 UTC [494] FATAL: role "root" does not exist

The OS that Immich Server is running on

docker

Version of Immich Server

v1.106.2

Version of Immich Mobile App

v1.106.2

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      - POSTGRES_PASSWORD=${DB_PASSWORD}
      - POSTGRES_USER=${DB_USERNAME}
      - POSTGRES_DB=${DB_DATABASE_NAME}
      - POSTGRES_INITDB_ARGS=--data-checksums
    volumes:
      - postgres:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT SUM(checksum_failures) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command: ["postgres", "-c" ,"shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]
    restart: always

Your .env content

...

Reproduction steps

...

Relevant log output

No response

Additional information

No response

@dronnikovigor
Copy link
Author

Seems to be duplication of #10134

@mmomjian
Copy link
Contributor

mmomjian commented Jun 12, 2024

I don't think this is a direct duplicate. Can you try something? Change your test: under database to this:

test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT SUM(checksum_failures) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1

@dronnikovigor
Copy link
Author

Hi, yes, with adding username now i don't have errors in log.
Btw, container is still unhealthy. But it looks like checksums are disabled for db.

data_checksums

off

@mmomjian
Copy link
Contributor

If checksums are disabled you can use this version. Please let me know if that works.

test: pg_isready --dbname='${DB_DATABASE_NAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1

@dronnikovigor
Copy link
Author

Yep. Works.
Also I recreated database & restoring backup and initial health check is working now
Thanks

@mmomjian
Copy link
Contributor

Thanks! Glad it worked now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants