OPAHealthCheck Internal Server Error for url: http://opa:8181/health?bundles

[Shadowfear36]

Actions before raising this issue

• I searched the existing issues and did not find anything similar.
• I read/searched the docs

Steps to Reproduce

CVAT Version:
CVAT:DEV
Been running this for awhile now. Only decided to rebuild the container and since then this appears to be happening.

Error:
OPAHealthCheck ... unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles

Command used to build:
sudo -E docker compose -f docker-compose.yml -f docker-compose.external_db.yml -f components/serverless/docker-compose.serverless.yml -f docker-compose.https.yml up --build --force-recreate

cvat_opa

{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp: lookup cvat-server on 127.0.0.11:53: server misbehaving","name":"cvat","plugin":"bundle","time":"2024-09-17T21:15:00Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp: lookup cvat-server on 127.0.0.11:53: server misbehaving","name":"cvat","plugin":"bundle","time":"2024-09-17T21:15:00Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp: lookup cvat-server on 127.0.0.11:53: server misbehaving","name":"cvat","plugin":"bundle","time":"2024-09-17T21:15:00Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp: lookup cvat-server on 127.0.0.11:53: server misbehaving","name":"cvat","plugin":"bundle","time":"2024-09-17T21:15:00Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp: lookup cvat-server on 127.0.0.11:53: server misbehaving","name":"cvat","plugin":"bundle","time":"2024-09-17T21:15:01Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.29.0.14:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2024-09-17T21:15:02Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.29.0.14:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2024-09-17T21:15:03Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.29.0.14:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2024-09-17T21:15:05Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.29.0.14:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2024-09-17T21:15:07Z"}

Health Check

python manage.py health_check
WARNING:HEALTH:getting Cache backend: default health status
WARNING:HEALTH:getting Cache backend: media health status
WARNING:HEALTH:gettin OPA health status
WARNING:HEALTH:DONE getting Cache backend: default health status
ERROR:health-check:unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles
Traceback (most recent call last):
  File "/home/django/cvat/apps/health/backends.py", line 29, in check_status
    response.raise_for_status()
  File "/opt/venv/lib/python3.8/site-packages/requests/models.py", line 953, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/health_check/backends.py", line 30, in run_check
    self.check_status()
  File "/home/django/cvat/apps/health/backends.py", line 31, in check_status
    raise HealthCheckException(str(e))
health_check.exceptions.HealthCheckException: unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles
WARNING:HEALTH:DONE getting Cache backend: media health status
Cache backend: default   ... working
Cache backend: media     ... working
DatabaseBackend          ... working
DiskUsage                ... working
MigrationsHealthCheck    ... working
OPAHealthCheck           ... unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles

OpenPolicyAgent appears to be running.

 docker ps
CONTAINER ID   IMAGE                                       COMMAND                  CREATED          STATUS                    PORTS                                                                                                                                                            NAMES
c8ed8eaafca0   cvat/ui:dev                                 "/docker-entrypoint.…"   13 minutes ago   Up 13 minutes             80/tcp                                                                                                                                                           cvat_ui
db4f33da87eb   cvat/server:dev                             "./backend_entrypoin…"   13 minutes ago   Up 13 minutes             8080/tcp                                                                                                                                                         cvat_worker_quality_reports
4ee86010847f   cvat/server:dev                             "./backend_entrypoin…"   13 minutes ago   Up 13 minutes             8080/tcp                                                                                                                                                         cvat_utils
5656f3bf0a32   cvat/server:dev                             "./backend_entrypoin…"   13 minutes ago   Up 13 minutes             8080/tcp                                                                                                                                                         cvat_worker_analytics_reports
2b335307065c   cvat/server:dev                             "./backend_entrypoin…"   13 minutes ago   Up 13 minutes             8080/tcp                                                                                                                                                         cvat_server
d7dbe243a0bd   cvat/server:dev                             "./backend_entrypoin…"   13 minutes ago   Up 13 minutes             8080/tcp                                                                                                                                                         cvat_worker_import
bbf0b8c3e148   timberio/vector:0.26.0-alpine               "/usr/local/bin/vect…"   13 minutes ago   Up 13 minutes                                                                                                                                                                              cvat_vector
6d15378648f2   cvat/server:dev                             "./backend_entrypoin…"   13 minutes ago   Up 13 minutes             8080/tcp                                                                                                                                                         cvat_worker_webhooks
40faf79a3111   cvat/server:dev                             "./backend_entrypoin…"   13 minutes ago   Up 13 minutes             8080/tcp                                                                                                                                                         cvat_worker_annotation
8f6e68500767   cvat/server:dev                             "./backend_entrypoin…"   13 minutes ago   Up 13 minutes             8080/tcp                                                                                                                                                         cvat_worker_export
c596d96f3c43   quay.io/nuclio/dashboard:1.13.0-amd64       "/docker-entrypoint.…"   13 minutes ago   Up 13 minutes (healthy)   80/tcp, 0.0.0.0:8070->8070/tcp, :::8070->8070/tcp                                                                                                                nuclio
7b8c0bffc8d4   grafana/grafana-oss:10.1.2                  "sh -euc 'mkdir -p /…"   13 minutes ago   Up 13 minutes             3000/tcp                                                                                                                                                         cvat_grafana
026b6421eca6   redis:7.2.3-alpine                          "docker-entrypoint.s…"   13 minutes ago   Up 13 minutes             6379/tcp                                                                                                                                                         cvat_redis_inmem
62db1ec657cb   clickhouse/clickhouse-server:23.11-alpine   "/entrypoint.sh"         13 minutes ago   Up 13 minutes             8123/tcp, 9000/tcp, 9009/tcp                                                                                                                                     cvat_clickhouse
307193fa0112   traefik:v2.9                                "/entrypoint.sh trae…"   13 minutes ago   Up 13 minutes             0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp, 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp, 0.0.0.0:8090->8090/tcp, :::8090->8090/tcp   traefik
939fe31fa958   apache/kvrocks:2.7.0                        "kvrocks -c /var/lib…"   13 minutes ago   Up 13 minutes (healthy)   6666/tcp                                                                                                                                                         cvat_redis_ondisk
5f88ded404cb   openpolicyagent/opa:0.63.0                  "/opa run --server -…"   13 minutes ago   Up 13 minutes                                                                                                                                                                              cvat_opa
b9fb69b2b1f0   gcr.io/iguazio/alpine:3.17                  "/bin/sh -c '/bin/sl…"   23 minutes ago   Up 23 minutes                                                                                                                                                                              nuclio-local-storage-reader

Expected Behavior

I expected the built container to work properly. I'm not using a specific version of CVAT. Just using CVAT:DEV which could ultimately be my problem. But it's what was already established as I already have data both in a volume and on the database I need access to.

Possible Solution

No response

Context

No response

Environment

Docker version 24.0.5, build ced0996
Linux/Ubuntu

[azhavoro]

Please attach full log from cvat_server container docker logs cvat_server