large memory usage for some log_proxy instances

[ajw1980]

We run log_proxy running for several hosts and services (1000+ services). I think we have just one host with unusual memory usage for a small number of log_proxy instances:

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        1281  0.1  0.8 578928 256336 ?       Sl   Feb20   3:05 /opt/bin/log_proxy -c 644 -n 0 -s 0 -t 0 -d /opt/logs a.041257.log
root        1305  2.1  9.4 3529988 3004128 ?     Sl   Feb20  36:15 /opt/bin/log_proxy -c 644 -n 0 -s 0 -t 0 -d /opt/logs b.041257.log
root        1407  0.0  0.0 296264   164 ?        Sl   Feb20   0:07 /opt/bin/log_proxy -c 644 -n 0 -s 0 -t 0 -d /opt/logs c.041257.log
user1234    1432  0.0  0.0 296264   164 ?        Sl   Feb20   0:07 /opt/bin/log_proxy -c 644 -n 0 -s 0 -t 0 -d /opt/logs d.041257.log
redissvs    1466  0.0  0.0 296264   168 ?        Sl   Feb20   0:07 /opt/bin/log_proxy -c 644 -n 0 -s 0 -t 0 -d /opt/logs e.041257.log
dataser+    3604  0.0  0.2 366620 68908 ?        Sl   Feb20   0:51 /opt/bin/log_proxy -c 644 -n 0 -s 0 -t 0 -d /opt/logs f.041259.log
redissvs    3605  2.0  6.0 3562760 1911792 ?     Sl   Feb20  34:44 /opt/bin/log_proxy -c 644 -n 0 -s 0 -t 0 -d /opt/logs g.041259.log
root      317774  0.0  0.0 296264   204 ?        Sl   Feb20   0:06 /opt/bin/log_proxy -c 644 -n 0 -s 0 -t 0 -d /opt/logs h.102941.log
user1234  333011  0.0  0.0 296264   204 ?        Sl   Feb20   0:05 /opt/bin/log_proxy -c 644 -n 0 -s 0 -t 0 -d /opt/logs i.105938.log
root      376175  0.0  0.0 296264   204 ?        Sl   Feb20   0:05 /opt/bin/log_proxy -c 644 -n 0 -s 0 -t 0 -d /opt/logs k.120100.log
root      376180  0.0  0.0 296264   200 ?        Sl   Feb20   0:05 /opt/bin/log_proxy -c 644 -n 0 -s 0 -t 0 -d /opt/logs l.120100.log
user1234  508291  4.3 12.2 4188824 3893692 ?     Sl   Feb20  43:26 /opt/bin/log_proxy -c 644 -n 0 -s 0 -t 0 -d /opt/logs m.160201.log
user1234  537435  0.0  0.0 296264  1044 ?        Sl   Feb20   0:02 /opt/bin/log_proxy -c 644 -n 0 -s 0 -t 0 -d /opt/logs n.161501.log
dataser+  537437  0.0  0.0 296264  1000 ?        Sl   Feb20   0:02 /opt/bin/log_proxy -c 644 -n 0 -s 0 -t 0 -d /opt/logs o.161501.log
nginx     537441  0.0  0.0 296264  1048 ?        Sl   Feb20   0:02 /opt/bin/log_proxy -c 644 -n 0 -s 0 -t 0 -d /opt/logs p.161501.log
user1234  558255  3.0  8.1 2881436 2586316 ?     Sl   Feb20  29:28 /opt/bin/log_proxy -c 644 -n 0 -s 0 -t 0 -d /opt/logs q.163000.log
user1234  559646  4.4 12.3 4236592 3941460 ?     Sl   Feb20  42:52 /opt/bin/log_proxy -c 644 -n 0 -s 0 -t 0 -d /opt/logs r.163300.log
user1234  561055  0.3  0.6 511164 216008 ?       Sl   Feb20   2:55 /opt/bin/log_proxy -c 644 -n 0 -s 0 -t 0 -d /opt/logs s.163900.log
user1234  561360  0.0  0.0 296264  1064 ?        Sl   Feb20   0:02 /opt/bin/log_proxy -c 644 -n 0 -s 0 -t 0 -d /opt/logs t.164100.log
user1234  561366  2.0  4.9 1870932 1575812 ?     Sl   Feb20  19:20 /opt/bin/log_proxy -c 644 -n 0 -s 0 -t 0 -d /opt/logs u.164100.log
user1234  565690  0.0  0.0 296264  1096 ?        Sl   Feb20   0:02 /opt/bin/log_proxy -c 644 -n 0 -s 0 -t 0 -d /opt/logs v.164500.log
user1234  568826  0.0  0.0 296264  1052 ?        Sl   Feb20   0:02 /opt/bin/log_proxy -c 644 -n 0 -s 0 -t 0 -d /opt/logs w.165100.log
user1     662463  0.0  0.0 296264  1356 ?        Sl   Feb20   0:02 /opt/bin/log_proxy -c 644 -n 0 -s 0 -t 0 -d /opt/logs x.192412.log
user1234  815311  8.3 12.2 4205172 3910344 ?     Sl   00:00  43:39 /opt/bin/log_proxy -c 644 -n 0 -s 0 -t 0 -d /opt/logs y.000001.log
user1     969050  0.0  0.0 296264  1320 ?        Sl   04:21   0:00 /opt/bin/log_proxy -c 644 -n 0 -s 0 -t 0 -d /opt/logs z.042118.log
user1234 1046991  0.0  0.0 296264  1580 ?        Sl   07:31   0:00 /opt/bin/log_proxy -c 644 -n 0 -s 0 -t 0 -d /opt/logs 1.073107.log

It's the same log_proxy binary on all hosts. Any idea on what would cause this or how to debug?

[ajw1980]

I think I see what caused it. The machine ran out of disk space. It seems like log_proxy does not like that.

[thebaptiste]

So I think we can close this issue ?

[ajw1980]

When the machine runs out of memory log_proxy will start eating memory and cpu and I don't think it can be released without a restart of log_proxy so I'm not sure this should be closed. I recompiled with this change to stop this from happening:

diff --git a/src/log_proxy.c b/src/log_proxy.c
index 9e01bba..c742dec 100644
--- a/src/log_proxy.c
+++ b/src/log_proxy.c
@@ -263,7 +263,7 @@ int main(int argc, char *argv[])
                 if (write_status == FALSE) {
                     g_warning("error during write on: %s", log_file);
                     init_or_reinit_output_channel(log_file, use_locks);
-                    continue;
+                    break;
                 }
                 break;
             }