[Bug]: Server constantly restarting after installation using docker-compose

[dawid-czarnecki]

Preflight Checklist

• I agree to follow the Code of Conduct that this project adheres to.
• I have searched the issue tracker for an issue that matches the one I want to file, without success.
• I am not looking for support or already pursued the available support channels without success.

Version

1.15.0

Installation Type

Official Docker Compose

Service Name

Docker(DongTai-Base-Image), Deploy(DongTai-deploy), DongTai-Core

Describe the details of the bug and the steps to reproduce it

DongTai server docker container constantly restarts right after installing DongTai according to this documentation: https://docs.dongtai.io/docs/getting-started/server/deploy-docker-compose

Additional Information

Docker version 24.0.5, build 24.0.5-0ubuntu1~20.04.1
Docker Compose version v2.22.0
Linux dongtai 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Logs

$ docker ps | fgrep server-1
caa9e6207a07   registry.cn-beijing.aliyuncs.com/huoxian_pub/dongtai-server:1.15.0      "/bin/bash /opt/dong…"   14 minutes ago   Restarting (1) 31 seconds ago                                       dongtai-dongtai-server-1

$ netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      20885/docker-proxy  
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      934/systemd-resolve 
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1027/sshd: /usr/sbi 
tcp        0      0 127.0.0.1:38141         0.0.0.0:*               LISTEN      10733/node          
tcp        0      0 127.0.0.1:45287         0.0.0.0:*               LISTEN      991/containerd      
tcp6       0      0 :::80                   :::*                    LISTEN      20891/docker-proxy  
tcp6       0      0 :::22                   :::*                    LISTEN      1027/sshd: /usr/sbi 


$ ./dtctl logs server
"docker logs" requires exactly 1 argument.
See 'docker logs --help'.

Usage:  docker logs [OPTIONS] CONTAINER

Fetch the logs of a container

$ ./dtctl logs web
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /doccereker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
172.16.147.1 - - [22/Sep/2023:18:55:21 +0000] "GET /login HTTP/1.1" 304 0 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36"
172.16.147.1 - - [22/Sep/2023:18:55:24 +0000] "GET /api/v1/captcha/refresh HTTP/1.1" 502 559 "http://dongtai.local/login" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36"
2023/09/22 18:55:24 [error] 28#28: *1 connect() failed (113: No route to host) while connecting to upstream, client: 172.16.147.1, server: 0.0.0.0, request: "GET /upload/assets/img/logo_en.png?v=0.7883261240552661 HTTP/1.1", upstream: "http://172.31.200.3:8000/upload/assets/img/logo_en.png?v=0.7883261240552661", host: "dongtai.local", referrer: "http://dongtai.local/login"
2023/09/22 18:55:24 [error] 29#29: *3 connect() failed (113: No route to host) while connecting to upstream, client: 172.16.147.1, server: 0.0.0.0, request: "GET /api/v1/captcha/refresh HTTP/1.1", upstream: "http://172.31.200.3:8000/api/v1/captcha/refresh", host: "dongtai.local", referrer: "http://dongtai.local/login"
2023/09/22 18:55:24 [error] 28#28: *2 connect() failed (113: No route to host) while connecting to upstream, client: 172.16.147.1, server: 0.0.0.0, request: "GET /upload/assets/img/logo_en.png?v=0.25444061191113265 HTTP/1.1", upstream: "http://172.31.200.3:8000/upload/assets/img/logo_en.png?v=0.25444061191113265", host: "dongtai.local", referrer: "http://dongtai.local/login"
172.16.147.1 - - [22/Sep/2023:18:55:24 +0000] "GET /upload/assets/img/logo_en.png?v=0.7883261240552661 HTTP/1.1" 502 559 "http://dongtai.local/login" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36"
172.16.147.1 - - [22/Sep/2023:18:55:24 +0000] "GET /upload/assets/img/logo_en.png?v=0.25444061191113265 HTTP/1.1" 502 559 "http://dongtai.local/login" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36"
2023/09/22 18:55:27 [error] 28#28: *2 connect() failed (113: No route to host) while connecting to upstream, client: 172.16.147.1, server: 0.0.0.0, request: "GET /upload/assets/img/favicon.ico HTTP/1.1", upstream: "http://172.31.200.3:8000/upload/assets/img/favicon.ico", host: "dongtai.local", referrer: "http://dongtai.local/login"
172.16.147.1 - - [22/Sep/2023:18:55:27 +0000] "GET /upload/assets/img/favicon.ico HTTP/1.1" 502 559 "http://dongtai.local/login" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36"

[Bidaya0]

The 1.15.0 image has been re-released, try reinstalling it.

[dawid-czarnecki]

I tried reinstalling it but the result is the same.
I also tried installing it from the github main, develop, and release-1.15.0 branches and directly from the zip release package.
Result is the same.

[Bidaya0]

I tried reinstalling it but the result is the same. I also tried installing it from the github main, develop, and release-1.15.0 branches and directly from the zip release package. Result is the same.

Have you tried ./dtctl rm -d?
The previously released version has data problems and needs to be cleaned and rebuilt.

[dawid-czarnecki]

Before no. Now I retried it with -d on the same three branches. Result is still the same.

[Bidaya0]

Before no. Now I retried it with -d on the same three branches. Result is still the same.

Could you give me some logs?
Just use this command docker logs dongtai-dongtai-server-1

[dawid-czarnecki]

Sure, below are these logs. This seems like the scheme was not properly removed and installed or there is still some scheme error in the latest version.

启动uwsgi服务                                                                                                          
INFO 2023-10-10 20:16:25,339 [__init__.monkey_patch:68] app: dongtai_common module: dongtai_engine.plugins.method_pool class:  func : method_pool_after_scan is patched by PlugMethodPoolAfterScan                                            
INFO 2023-10-10 20:16:25,707 [report_handler_factory.wrapper:80] Registration report type 130 handler FilePathHandler  
INFO 2023-10-10 20:16:25,709 [report_handler_factory.wrapper:80] Registration report type 129 handler ThirdPartyServiceHandler                                                                                                                
INFO 2023-10-10 20:16:25,713 [report_handler_factory.wrapper:80] Registration report type 97 handler ApiRouteHandler   
INFO 2023-10-10 20:16:25,715 [report_handler_factory.wrapper:80] Registration report type 81 handler ErrorLogHandler   
INFO 2023-10-10 20:16:25,717 [report_handler_factory.wrapper:80] Registration report type 37 handler HardEncodeVulHandler                                                                                                                     
INFO 2023-10-10 20:16:25,721 [report_handler_factory.wrapper:80] Registration report type 1 handler HeartBeatHandler   
INFO 2023-10-10 20:16:25,726 [report_handler_factory.wrapper:80] Registration report type 33 handler NormalVulnHandler 
INFO 2023-10-10 20:16:25,738 [report_handler_factory.wrapper:80] Registration report type 36 handler SaasMethodPoolHandler                                                                                                                    
INFO 2023-10-10 20:16:25,852 [report_handler_factory.wrapper:80] Registration report type 17 handler ScaHandler        
INFO 2023-10-10 20:16:25,852 [report_handler_factory.wrapper:80] Registration report type 18 handler ScaBulkHandler
use local file                                                                                                         
is in gevent patched : False                                                                                           
preheat settings now : True
File “/opt/dongtai/static/i18n/zh/LC_MESSAGES/django.po” is already compiled and up to date.
File “/opt/dongtai/static/i18n/en/LC_MESSAGES/django.po” is already compiled and up to date.

Get the latest vulnerability rules.
INFO 2023-10-10 20:16:29,864 [__init__.monkey_patch:68] app: dongtai_common module: dongtai_engine.plugins.method_pool class:  func : method_pool_after_scan is patched by PlugMethodPoolAfterScan
INFO 2023-10-10 20:16:30,269 [report_handler_factory.wrapper:80] Registration report type 130 handler FilePathHandler
INFO 2023-10-10 20:16:30,273 [report_handler_factory.wrapper:80] Registration report type 129 handler ThirdPartyServiceHandler
INFO 2023-10-10 20:16:30,281 [report_handler_factory.wrapper:80] Registration report type 97 handler ApiRouteHandler
INFO 2023-10-10 20:16:30,283 [report_handler_factory.wrapper:80] Registration report type 81 handler ErrorLogHandler
INFO 2023-10-10 20:16:30,287 [report_handler_factory.wrapper:80] Registration report type 37 handler HardEncodeVulHandler
INFO 2023-10-10 20:16:30,292 [report_handler_factory.wrapper:80] Registration report type 1 handler HeartBeatHandler
INFO 2023-10-10 20:16:30,295 [report_handler_factory.wrapper:80] Registration report type 33 handler NormalVulnHandler
INFO 2023-10-10 20:16:30,304 [report_handler_factory.wrapper:80] Registration report type 36 handler SaasMethodPoolHandler
INFO 2023-10-10 20:16:30,398 [report_handler_factory.wrapper:80] Registration report type 17 handler ScaHandler
INFO 2023-10-10 20:16:30,398 [report_handler_factory.wrapper:80] Registration report type 18 handler ScaBulkHandler
use local file
is in gevent patched : False
preheat settings now : True
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/usr/local/lib/python3.10/site-packages/pymysql/cursors.py", line 148, in execute
    result = self._query(query)
  File "/usr/local/lib/python3.10/site-packages/pymysql/cursors.py", line 310, in _query
    conn.query(q)
  File "/usr/local/lib/python3.10/site-packages/pymysql/connections.py", line 548, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/usr/local/lib/python3.10/site-packages/pymysql/connections.py", line 775, in _read_query_result
    result.read()
  File "/usr/local/lib/python3.10/site-packages/pymysql/connections.py", line 1156, in read
    first_packet = self.connection._read_packet()
  File "/usr/local/lib/python3.10/site-packages/pymysql/connections.py", line 725, in _read_packet
    packet.raise_for_error()
  File "/usr/local/lib/python3.10/site-packages/pymysql/protocol.py", line 221, in raise_for_error
    err.raise_mysql_exception(self._data)
  File "/usr/local/lib/python3.10/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.OperationalError: (1364, "Field 'modified' doesn't have a default value")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/dongtai/manage.py", line 21, in <module>
    main()
  File "/opt/dongtai/manage.py", line 17, in main
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.10/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.10/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv) 
  File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/opt/dongtai/deploy/commands/management/commands/load_hook_strategy.py", line 166, in handle
    HookStrategy.objects.create(hooktype=policy_hook_type, **hook_strategy)
  File "/usr/local/lib/python3.10/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 453, in create
    obj.save(force_insert=True, using=self.db)
  File "/usr/local/lib/python3.10/site-packages/django/db/models/base.py", line 739, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/usr/local/lib/python3.10/site-packages/django/db/models/base.py", line 776, in save_base
    updated = self._save_table(
  File "/usr/local/lib/python3.10/site-packages/django/db/models/base.py", line 881, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/usr/local/lib/python3.10/site-packages/django/db/models/base.py", line 919, in _do_insert
    return manager._insert(
  File "/usr/local/lib/python3.10/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 1270, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/usr/local/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1416, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 79, in _execute
    with self.db.wrap_database_errors:
  File "/usr/local/lib/python3.10/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/usr/local/lib/python3.10/site-packages/pymysql/cursors.py", line 148, in execute
    result = self._query(query)
  File "/usr/local/lib/python3.10/site-packages/pymysql/cursors.py", line 310, in _query
    conn.query(q)
  File "/usr/local/lib/python3.10/site-packages/pymysql/connections.py", line 548, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/usr/local/lib/python3.10/site-packages/pymysql/connections.py", line 775, in _read_query_result
    result.read()
  File "/usr/local/lib/python3.10/site-packages/pymysql/connections.py", line 1156, in read
    first_packet = self.connection._read_packet()
  File "/usr/local/lib/python3.10/site-packages/pymysql/connections.py", line 725, in _read_packet
    packet.raise_for_error()
  File "/usr/local/lib/python3.10/site-packages/pymysql/protocol.py", line 221, in raise_for_error
    err.raise_mysql_exception(self._data)
  File "/usr/local/lib/python3.10/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
    raise errorclass(errno, errval)
django.db.utils.OperationalError: (1364, "Field 'modified' doesn't have a default value")
ERROR: Lost connection to MySQL server !!!

[Bidaya0]

Check whether there is a container with the normal running image registry.cn-beijing.aliyuncs.com/huoxian_pub/dongtai-server:1.15.0
Run the following command

docker exec -it dongtai-dongtai-server-1 bash -c 'python manage.py migrate'

[dawid-czarnecki]

All of these containers are based on the image you mentioned:

root@cerebrates:~/DongTai/deploy/docker-compose# docker ps | fgrep registry.cn-beijing.aliyuncs.com/huoxian_pub/dongtai-server:1.15.0
4ba5533f4bf8   registry.cn-beijing.aliyuncs.com/huoxian_pub/dongtai-server:1.15.0      "/opt/dongtai/deploy…"   2 days ago   Up 2 days                                          dongtai-dongtai-worker-beat-1
e1d83b0a6fd3   registry.cn-beijing.aliyuncs.com/huoxian_pub/dongtai-server:1.15.0      "/opt/dongtai/deploy…"   2 days ago   Up 2 days                                          dongtai-dongtai-worker-es-1
450f3ba79a20   registry.cn-beijing.aliyuncs.com/huoxian_pub/dongtai-server:1.15.0      "/bin/bash /opt/dong…"   2 days ago   Up 6 seconds                                       dongtai-dongtai-server-1
958e98453dc3   registry.cn-beijing.aliyuncs.com/huoxian_pub/dongtai-server:1.15.0      "/opt/dongtai/deploy…"   2 days ago   Up 2 days                                          dongtai-dongtai-worker-other-1
6e6ef618c530   registry.cn-beijing.aliyuncs.com/huoxian_pub/dongtai-server:1.15.0      "/opt/dongtai/deploy…"   2 days ago   Up 2 days                                          dongtai-dongtai-worker-sca-1
888b3b5011de   registry.cn-beijing.aliyuncs.com/huoxian_pub/dongtai-server:1.15.0      "/opt/dongtai/deploy…"   2 days ago   Up 2 days                                          dongtai-dongtai-worker-high-freq-1
015d3a829b71   registry.cn-beijing.aliyuncs.com/huoxian_pub/dongtai-server:1.15.0      "/opt/dongtai/deploy…"   2 days ago   Up 2 days                                          dongtai-dongtai-worker-task-1

Since the server-1 is constantly restarting I cannot execute any migrate command on it:

$ docker exec -it dongtai-dongtai-server-1 bash -c 'python manage.py migrate'
Error response from daemon: Container 450f3ba79a20a464a9a4b9d700d8ab7e7412c3f30b53fdd3f6c0e876b916ef89 is restarting, wait until the container is running

[Bidaya0]

dongtai-dongtai-worker

Try other container like dongtai-dongtai-worker-xxxx.

[dawid-czarnecki]

I got the following:

Running migrations:
  No migrations to apply.
  Your models in app(s): 'dongtai_common' have changes that are not yet reflected in a migration, and so won't be applied.
  Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.

So I run makemigrations and than migrate on all containers, even on server. I was able to catch it in between restarts. Some of the containers created and applied migrations, others didn't. The result is still the same. Server is constantly restarting. Example log of migrations on one container:

$ docker exec -it dongtai-dongtai-server-1 bash -c 'python manage.py makemigrations; python manage.py migrate'
use local file
is in gevent patched : False
INFO 2023-10-17 00:54:18,738 [__init__.monkey_patch:68] app: dongtai_common module: dongtai_engine.plugins.method_pool class:  func : method_pool_after_scan is patched by PlugMethodPoolAfterScan
preheat settings now : True
INFO 2023-10-17 00:54:19,058 [report_handler_factory.wrapper:80] Registration report type 130 handler FilePathHandler
INFO 2023-10-17 00:54:19,059 [report_handler_factory.wrapper:80] Registration report type 129 handler ThirdPartyServiceHandler
INFO 2023-10-17 00:54:19,062 [report_handler_factory.wrapper:80] Registration report type 97 handler ApiRouteHandler
INFO 2023-10-17 00:54:19,064 [report_handler_factory.wrapper:80] Registration report type 81 handler ErrorLogHandler
INFO 2023-10-17 00:54:19,067 [report_handler_factory.wrapper:80] Registration report type 37 handler HardEncodeVulHandler
INFO 2023-10-17 00:54:19,071 [report_handler_factory.wrapper:80] Registration report type 1 handler HeartBeatHandler
INFO 2023-10-17 00:54:19,076 [report_handler_factory.wrapper:80] Registration report type 33 handler NormalVulnHandler
INFO 2023-10-17 00:54:19,087 [report_handler_factory.wrapper:80] Registration report type 36 handler SaasMethodPoolHandler
INFO 2023-10-17 00:54:19,183 [report_handler_factory.wrapper:80] Registration report type 17 handler ScaHandler
INFO 2023-10-17 00:54:19,183 [report_handler_factory.wrapper:80] Registration report type 18 handler ScaBulkHandler
No changes detected
use local file
is in gevent patched : False
INFO 2023-10-17 00:54:21,529 [__init__.monkey_patch:68] app: dongtai_common module: dongtai_engine.plugins.method_pool class:  func : method_pool_after_scan is patched by PlugMethodPoolAfterScan
preheat settings now : True
INFO 2023-10-17 00:54:21,915 [report_handler_factory.wrapper:80] Registration report type 130 handler FilePathHandler
INFO 2023-10-17 00:54:21,916 [report_handler_factory.wrapper:80] Registration report type 129 handler ThirdPartyServiceHandler
INFO 2023-10-17 00:54:21,920 [report_handler_factory.wrapper:80] Registration report type 97 handler ApiRouteHandler
INFO 2023-10-17 00:54:21,921 [report_handler_factory.wrapper:80] Registration report type 81 handler ErrorLogHandler
INFO 2023-10-17 00:54:21,924 [report_handler_factory.wrapper:80] Registration report type 37 handler HardEncodeVulHandler
INFO 2023-10-17 00:54:21,928 [report_handler_factory.wrapper:80] Registration report type 1 handler HeartBeatHandler
INFO 2023-10-17 00:54:21,933 [report_handler_factory.wrapper:80] Registration report type 33 handler NormalVulnHandler
INFO 2023-10-17 00:54:21,944 [report_handler_factory.wrapper:80] Registration report type 36 handler SaasMethodPoolHandler
INFO 2023-10-17 00:54:22,032 [report_handler_factory.wrapper:80] Registration report type 17 handler ScaHandler
INFO 2023-10-17 00:54:22,032 [report_handler_factory.wrapper:80] Registration report type 18 handler ScaBulkHandler
System check identified some issues:

WARNINGS:
dongtai_common.IastAssetVulRelationMetaData.vul_asset_key: (mysql.W003) MySQL may not allow unique CharFields to have a max_length > 255.
        HINT: See: https://docs.djangoproject.com/en/3.2/ref/databases/#mysql-character-fields
Operations to perform:
  Apply all migrations: admin, auth, authtoken, captcha, contenttypes, db, django_celery_beat, dongtai_common, sessions
Running migrations:
  Applying dongtai_common.0026_auto_20231017_0046... OK

[johnniesong]

reinstall latest docker and docker-compose v1 can fix this problem.
https://docs.docker.com/compose/install/standalone/

./dtctl install -v 1.15.0

[dawid-czarnecki]

Is there any simpler way to run it locally? I've been trying to quickly see how it works and potentially use it in future but seems like either I'm unlucky with choosing the OS/environment or the setup process is troublesome.