ProxySQL Error: signal 11 Crash

[MattMencel]

If you are submitting a reproducible bug report, please provide:

• A clear description of the issue
  I run proxysql in a Kubernetes StatefulSet and have been noticing quite a lot of restarts recently. I have the crash error but at the moment I don't have any good idea on the cause. I can't tie it to any specific application behavior yet anyway.

• ProxySQL version

# proxysql --version
2024-11-25 22:38:11 [INFO] Using jemalloc with MALLOC_CONF: config.xmalloc:1, lg_tcache_max:16, opt.prof_accum:1, opt.prof_leak:1, opt.lg_prof_sample:20, opt.lg_prof_interval:30, rc:0
ProxySQL version 2.7.1-16-g2726c27, codename Truls

• OS version

# uname -a
Linux proxysql-0 5.15.0-1073-azure #82-Ubuntu SMP Mon Sep 2 11:36:34 UTC 2024 x86_64 GNU/Linux

• The steps to reproduce the issue

• The full ProxySQL error log (default location: /var/lib/proxysql/proxysql.log)

2024-11-25 20:32:44 MyHGC.cpp:225:get_random_MySrvC(): [ERROR] Hostgroup 2 has no servers ready for GTID '3f3c3decb4b811eeb0b8002248b710e9:1355391570'. Waiting for replication...
Error: signal 11:
proxysql(_Z13crash_handleri+0x2a)[0x55b0ac52a4ea]
/lib/x86_64-linux-gnu/libc.so.6(+0x3c050)[0x7f34a4fd4050]
proxysql(+0xafd4c3)[0x55b0accb44c3]
proxysql(_ZN17MySQL_Data_StreamD1Ev+0x3da)[0x55b0ac58b1fa]
proxysql(_ZN13MySQL_Backend5resetEv+0x8b)[0x55b0ac5cdf0b]
proxysql(_ZN13MySQL_Session18reset_all_backendsEv+0x3f)[0x55b0ac5a78ff]
proxysql(_ZN13MySQL_Session5resetEv+0x150)[0x55b0ac5a7a80]
proxysql(_ZN13MySQL_SessionD1Ev+0xe)[0x55b0ac5b2dae]
proxysql(_ZN12MySQL_Thread20process_all_sessionsEv+0x341)[0x55b0ac5a1ce1]
proxysql(_ZN12MySQL_Thread3runEv+0x3a0)[0x55b0ac5a2290]
proxysql(_Z24mysql_worker_thread_funcPv+0xcc)[0x55b0ac501017]
/lib/x86_64-linux-gnu/libc.so.6(+0x89144)[0x7f34a5021144]
/lib/x86_64-linux-gnu/libc.so.6(+0x1097dc)[0x7f34a50a17dc]
 ---- proxysql(_Z13crash_handleri+0x2a) [0x55b0ac52a4ea] : crash_handler(int)
 ---- proxysql(_ZN17MySQL_Data_StreamD1Ev+0x3da) [0x55b0ac58b1fa] : MySQL_Data_Stream::~MySQL_Data_Stream()
 ---- proxysql(_ZN13MySQL_Backend5resetEv+0x8b) [0x55b0ac5cdf0b] : MySQL_Backend::reset()
 ---- proxysql(_ZN13MySQL_Session18reset_all_backendsEv+0x3f) [0x55b0ac5a78ff] : MySQL_Session::reset_all_backends()
 ---- proxysql(_ZN13MySQL_Session5resetEv+0x150) [0x55b0ac5a7a80] : MySQL_Session::reset()
 ---- proxysql(_ZN13MySQL_SessionD1Ev+0xe) [0x55b0ac5b2dae] : MySQL_Session::~MySQL_Session()
 ---- proxysql(_ZN12MySQL_Thread20process_all_sessionsEv+0x341) [0x55b0ac5a1ce1] : MySQL_Thread::process_all_sessions()
 ---- proxysql(_ZN12MySQL_Thread3runEv+0x3a0) [0x55b0ac5a2290] : MySQL_Thread::run()
 ---- proxysql(_Z24mysql_worker_thread_funcPv+0xcc) [0x55b0ac501017] : mysql_worker_thread_func(void*)

If this is a crashing bug, please also include:

• The package used to install ProxySQL
  ProxySQL is installed via a Helm chart I maintain.

• The compressed proxysql binary

• The compressed core dump (Note: if you're worried it may contain sensitive data, please contact us for information on sharing it securely: https://proxysql.com/contact-us/)
  Is there any guidance available on getting a proxysql core dump from an ephemeral container?

If the above information is not provided, this issue is likely to be closed.

Please use markdown to format any code or SQL: https://guides.github.com/features/mastering-markdown/

Thank you!

[JavierJF]

Hi @MattMencel,

for crashing issues of this type, either a reproduction case or at least coredumps are required to perform an investigation. Just a fraction of the error log with the backtrace (the issue template also specifies the full error log) isn't sufficient, coredumps for ProxySQL are no different to any other applications, and their collection is also like for any other application.

Regards, Javier.

[flmmartins]

Hello we have the same issue on version 2.7.0 and afterwards proxysql process dies. I see that for us to take core dumps we need to do it while process is running. Since this happens randomly and we need to have the coredump running until it does that has the risk of filling up the disk. Is there another way to do coredumps? Or some other command we can do?

[renecannao]

@flmmartins : a core dump is automatically generated by the Linux kernel when a program crashes.
The link you referenced to is to create core dump on demand , not on crash.
Please consult your specific OS documentation on how to enable core dump when a program crashes, if core dump on crashes are not enabled.