Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add filter to curl request to prevent oom exceptions #45

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

c-kr
Copy link

@c-kr c-kr commented Apr 13, 2022

Sometimes the apis response gets very long because it logs every http requests by default. On our test machine the following grep cmd filled up the 24gb memory within seconds:

if [[ -n $user ]] || [[ -n $(echo $esstatus | grep -i authentication) ]] ; then

We were able to mitigate it by adding a filter to the api requests to include only those values which are needed. This probably needs some more testing before merging, we only tested the memory mode

@Napsty
Copy link
Owner

Napsty commented Apr 20, 2022

Lets give #42 a bit more time to rebase the PR, then adjust this PR and get a new version out.
Nevermind, we'll go ahead with this one first and then we can do #42.
Very interesting that your Elasticsearch shows so much data though. Could it be a misconfigured ES at the cause?
On which check type did you see the huge data in the API response?

@c-kr
Copy link
Author

c-kr commented Apr 28, 2022

I am not an elastic guy but i don't think it is misconfigured, just heavily used which leads to logging of a lot open http connections in the api output. It only happens sometimes a day.

We see this huge data in all checks used by the getstatus() method with endpoint (/stats).

@Napsty
Copy link
Owner

Napsty commented Apr 28, 2022

Thanks for the info @c-kr

We see this huge data in all checks used by the getstatus() method with endpoint (/stats).

That is certainly strange. I will do some tests with my Elasticsearch instances and maybe I can reproduce this sometime.
What kind of Elasticsearch did you see that on? Local self installed or a Cloud instance? Which version?

Nevertheless, I think your contribution makes sense and will be merged after tests. 👍

@c-kr
Copy link
Author

c-kr commented Apr 30, 2022

What kind of Elasticsearch did you see that on? Local self installed or a Cloud instance? Which version?

Self Installed Elasticsearch Version 7.17.1

Nevertheless, I think your contribution makes sense and will be merged after tests. 👍

Thanks

@Napsty Napsty self-requested a review May 13, 2022 12:41
@Napsty
Copy link
Owner

Napsty commented May 13, 2022

disk check does not work anymore with that change.

Before PR:

$ ./check_es_system.sh -H myes.eu-central-1.aws.cloud.es.io -u monitoring -p secret -S -P 9243 -t disk
ES SYSTEM OK - Disk usage is at 9% (41 G from 450 G)|es_disk=44708304152B;386547056640;459024629760;0;483183820800

When applying the PR:

$ ./check_es_system.sh -H myes.eu-central-1.aws.cloud.es.io -u monitoring -p secret -S -P 9243 -t disk
expr: non-integer argument
expr: non-integer argument
expr: non-integer argument
expr: non-integer argument
./check_es_system.sh: line 390: [: 44708304152: unary operator expected
./check_es_system.sh: line 393: [: 44708304152: unary operator expected
ES SYSTEM OK - Disk usage is at % (41 G from  G)|es_disk=44708304152B;;;0;null

Maybe the filter needs to be adjusted. Can you check please?

Copy link
Owner

@Napsty Napsty left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

needs some changes so that "disk" check works again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants