[Idea] Send Limit and Log - GLPI-Injector

[erique-souza]

Your idea

I have a park of approximately thirty thousand computers that contact a single glpi proxy server through glpi-agent, these inventories are being saved in a specific folder, for example /inventario, and to send these .json files saved in this folder, I created several crons in Linux that run glpi-injector every 1 hour on the GLPI server

What I'm noticing with this situation is that glpi-injector can't handle 3-4 thousand files in a single glpi-injector task

My suggestion here for a glpi-injector feature would be to set a sending limit that can be set via parameter, so that it doesn't get overloaded or trying to send forever

I also didn't find any log that records glpi-injector's actions, I know that there are --debug and --verbose parameters, but if it is being run via cron, this information is lost as far as I noticed.

[g-bougard]

Hi @erique-souza

before having to modify glpi-injector, I would better like to understand what really happens.

When run from a crontab, you should redirect stderr output to a file and before running again the injector you should probably have to rotate the generated file, just renaming it with a number or even a date.

Assuming you're using /var/glpi-agent/inventorio folder and you want to have logs into /var/glpi-agent/logs

You can for example use the following script (and for the example, let say it is /usr/local/sbin/glpi-injector-rotate-logs):

#! /bin/bash

let MAX=10

LOG=/var/glpi-agent/logs/injector.log

let i=MAX
while (( --i > 0 ))
do
    [ -e "$LOG.$i" ] || continue
    mv -f "$LOG.$i" "$LOG.$((i+1))"
done
[ -e "$LOG" ] && mv -f "$LOG" "$LOG.1"

exit 0

You can also use logrotate, rotatelogs or other dedicated commands but this is out of the subject and up to you.

Than you can use follow this crontab example:

00 * * * * /usr/local/sbin/glpi-injector-rotate-logs >/dev/null 2>&1
01 * * * * /usr/bin/glpi-injector -d /var/glpi-agent/inventorio -u http://your-glpi-server/ -v --debug >/var/glpi-agent/logs/injector.log 2>&1

By always try to rotate logs the minute before running glpi-injector, you should keep the last 10 logs of glpi-injector output.

Then maybe this can help you to find what's the real issue.

[erique-souza]

Hi @erique-souza

before having to modify glpi-injector, I would better like to understand what really happens.

When run from a crontab, you should redirect stderr output to a file and before running again the injector you should probably have to rotate the generated file, just renaming it with a number or even a date.

Assuming you're using /var/glpi-agent/inventorio folder and you want to have logs into /var/glpi-agent/logs

You can for example use the following script (and for the example, let say it is /usr/local/sbin/glpi-injector-rotate-logs):

#! /bin/bash

let MAX=10

LOG=/var/glpi-agent/logs/injector.log

let i=MAX
while (( --i > 0 ))
do
    [ -e "$LOG.$i" ] || continue
    mv -f "$LOG.$i" "$LOG.$((i+1))"
done
[ -e "$LOG" ] && mv -f "$LOG" "$LOG.1"

exit 0

You can also use logrotate, rotatelogs or other dedicated commands but this is out of the subject and up to you.

Than you can use follow this crontab example:

00 * * * * /usr/local/sbin/glpi-injector-rotate-logs >/dev/null 2>&1
01 * * * * /usr/bin/glpi-injector -d /var/glpi-agent/inventorio -u http://your-glpi-server/ -v --debug >/var/glpi-agent/logs/injector.log 2>&1

By always try to rotate logs the minute before running glpi-injector, you should keep the last 10 logs of glpi-injector output.

Then maybe this can help you to find what's the real issue.

I made the .sh script in /etc/glpi-agent/glpi-injector-rotate-logs.sh to rotate the logs and do some tests

And I created the log in /var/log/injector.log

What I've already noticed with some manual runs is the following which is probably one of the problems I'm noticing:

• Cron call for glpi-injector execution at 00:00, it is running, as there are many files it was unable to finish before the next call at 01:00, running in parallel
• The GLPI-Injector Cron call happens at 01:00, it manages to send a few files and then stops sending because it tried to send a file but in the middle of the process the file no longer existed, resulting from the 00:00 cron call that is still in progress execution
  If this same 00h call continues running for hours, all the next cron calls start and finish in a short time due to the conflict of file sending attempts, causing the new ones to stop working and the first 00h call to continue almost infinitely , because the server will not stop receiving new files to send

I believe this is the main problem when we are talking about a very large sending park

I left it running for some time to get the log, attached as an example, I used the final parameters:
-v --debug >/var/log/injector.log 2>&1
injector.log

The cron that was already running before running did not have debug enabled, this log represents what I forced to run in parallel

Sometimes I come across error 504 when trying to make the injector for GLPI, specifically about this I tried to increase the php.ini timeout on the GLPI server to try to resolve it, but it still happens sometimes

[g-bougard]

Hi @erique-souza

glpi-injector is not expected to run concurrently with itself: it will abort as soon as an expected file is removed before it can read it for sending. And if the glpi server is overloaded, you should definitively avoid concurrency launches.

Can I suggest you to use a script which will launch glpi-injector only if the previous one has finished ? The most simple way would be to simply lock the folder creating a ".lck" file and don't start glpi-injector if the file still exists. For example:

#! /bin/sh

FOLDER="/var/glpi-agent/inventorio"
LOCK="$FOLDER/injecting.lck"

if [ -e "$LOCK" ]; then
    echo "$FOLDER is still locked for glpi-injector, leaving"
    exit 0
fi

touch "$LOCK"

glpi-injector -d "$FOLDER" -u http://your-glpi-server/ -v --debug

rm -f "$LOCK"

This is not the best way of doing but it should work for scripts only started one time every hours.

If you're sure all the files should have been sent before midnight, you can force a deletion of the lock file at 23h59, just in case the script has been aborted before it removed the file.

Another way of doing would be to move all the inventory file from the folder to a temporary folder that can be accessed by only one glpi-injector, but I won't recommend this way if you're overloading glpi server. Another way could also be to run a loop by yourself in the script to start glpi-injector for one inventory at a time: this way, you can verify only one injector is run for one file and you can even sleep between each injection to limit glpî server overloading.

[erique-souza]

I have great news on the topic

I managed to apply the script and it worked as expected, it created the block so that simultaneous glpi-injector calls no longer occur, I took a screenshot of the top command of the processes to get an idea of ​​what I had seen from a glpi-cron cron. injector every hour trying to execute, they accumulated, ran over each other in execution due to the amount of inventories, and even brought down the GLPI database due to the high number of requests.

The script only modifies the directories and some commands:

#! /bin/sh

FOLDER="/inventario"
LOCK="$FOLDER/injecting.lck"

if [ -e "$LOCK" ]; then
echo "$FOLDER is still locked for glpi-injector, leaving"
exit 0
fi

touch "$LOCK"

glpi-injector --directory "$FOLDER" --remove --url https://URLServerGLPI/ --debug --no-ssl-check --verbose --debug >/var/log/injector.log 2>&1

rm -f "$LOCK"

With the script, now there is really only 1 task running, and if it is still running when called again it is interrupted by the injecting.lck file, which I thought was a great idea, too bad it doesn't register in the log, but it's not necessary

Now talking about the initial problem that was the reason for opening this topic here on git, it is the errors that I mentioned that cause glpi-injector to be unable to send all inventories at once:
I did an isolated send to catch some cases and this is the result:
injector.log

These errors of 504 Gateway Timeout and 500 Internal Server Error are very common, at the same time that these messages appear when trying to send the file via glpi-injector, I check if the GLPI is up and running, and yes it is up and running, which leads me to believe that the GLPI server blocks the server that is trying to send the glpi-injector to it, as if it were a spam attempt

Regarding these errors, I have already tried to increase the timeout in php.ini on the GLPI server, but I don't know exactly which parameter I could change on the GLPI server to try to minimize these errors that eventually occur, because as sending via glpi-injector daily would be around 14 thousand computers, the scale of errors is very large