Can't change temp directory using environment variables #71
Comments
|
More info: It seems like they still do use the ES_TMPDIR in new versions, but only for the elasticsearch binary itself and not the other cli scripts, since the other scripts do not need a temp directory. Perhaps the performance-analyzer-agent-cli script could adopt a similar approach since it similarly requires a temp directory. |
aesgithub
added
enhancement
|
Thanks @shawnz for reporting this issue. We will update the CLI to honor ES_TMPDIR if it is set. We are also thinking of using PA_AGENT_JAVA_OPTS env variable as-is if it is set. Let us know what you think about setting PA_AGENT_JAVA_OPTS from your end. |
|
Hi @sendkb, thank you for your time looking into this issue. Either option would be just fine for me. |
|
Just a follow up to your previous question: it would indeed be helpful to override other parts of PA_AGENT_JAVA_OPTS as well because that would also let me change the log4j.configurationFile property. This has recently become an important concern for me because the default location of the log4j config file is such that the file gets overwritten after any elasticsearch updates. So if I could instead put the log4j configuration in a location of my choosing, it would work around that issue. |
|
Is there any update on this? I am in the same boat as the OP here. We are using SELinux, marking our tmp dirs as NOEXEC, which is leading to elasticsearch.service failing to start up. .We're in AWS using centos 7 VMs, and installing opendistro by rpm. Adding -Djava.io.tmpdir=/etc/elasticsearch/tmp to the end of PA_AGENT_JAVA_OPT and we are still unable to get past this issue. It looks like no matter where I place or set $ES_TMPDIR or "-Djava.io.tmpdir=/etc/elasticsearch/tmp", elasticsearch ignores the setting and always attempts to write to /tmp for something or another. Any place that I have defined the new temp directy and gotten elasticsearch to start up I tried adding "ES_TMPDIR=/etc/elasticsearch/tmp" to the /etc/sysconfig/elasticsearch file, which does seem to get the logs to show that the change was honored, but that causes the JVM to fail and elasticsearch fails to come up. Unless I mark the /tmp directory as executable which is not a viable solution for us, nothing I have tried has seemed to work. Is there a time frame for this fix or another work around I could try? We cant move forward transitioning to OpenDistro if we have to have our tmp directory execuable. Thanks! |
|
Hey there @rdecuir , there are actually two issues at play here. Elasticsearch fails to start with a noexec tmp directory, and also the performance analyzer agent fails to start with a noexec temp directory. In order to fix both problems, my setup is as follows:
So what you are describing should be working as far as I am aware. Can you show me the error message? I am wondering if the problem is maybe that "/etc/elasticsearch" is not a suitable location. I am not sure why but in the past I have had issues where elasticsearch refuses to start if I put foreign files in there. |
|
Thanks for getting back to me! @shawnz error message below, and I did try a few things based on your suggestion.
But still when I run systemctl start elasticsearch.service it fails to start, I even tried as the elasticsearch user to kick off .../bin/elasticsearch, to try elasticsearch without the performance-anaylzer, but still see the same failure. |
|
Interesting. Based on this https://discuss.elastic.co/t/elasticsearch-service-getting-aborted/221857/2 maybe try also setting I guess that would mean, altogether in /etc/sysconfig/elasticsearch, you would have to comment out ES_TMPDIR and instead set ES_JAVA_OPTS to I will also note that I am only using JDK 11, not sure if that is relevant to the issue. I am also using Shenandoah GC instead of G1 GC. EDIT: I also notice you are using 32gb of heap which is not really recommended (although I'm sure that's not related). See here: https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html (note the section about compressed oops) |
|
Hi @shawnz and @rdecuir. I just started following this issue yesterday as I ran into he same problem with the noexec on tmp. Here are the changes I had to make based on @shawnz workaround. /etc/elasticsearch/tmp is the directory I've chosen to use for this. My efforts were to try and try Wazuh installed. /etc/sysconfig/elasticsearch /usr/share/elasticsearch/bin/performance-analyzer-agent-cli |
|
Thanks for the suggestion @shawnz and @jordan. Tried both suggestions and some other things and now trying to narrow the workflow to target issues. Since its both a performance analyzer and elasticsearch problem I've now started testing the following: I now become the elasticsearch user and am executing the /usr/share/elasticsearch/bin/elasticsearch script. What I have noticed is:
Figured working just with elasticsearch might be easier to start with but still having issues. It literally seems like when elasticsearch does pick up and change the tmp dir to what I want, assigning a new tmp dir it breaks elasticsearch. For more infor:
Any more suggestions or ideas are appreciated, thank you so much for you help thus far as its driving me nuts haha. Error when I don't set ES_TMPDIR, regardless if I set Djava.io.tmpdir=/opt/elasticsearch/tmp and/or -Djna.tmpdir=/opt/elasticsearch/tmp: |
|
@rdecuir - do you have the elasticsearch service running? What happens if you start the service as opposed to starting the binary as the user? This is what journalctl shows when I start my service: |
|
@jordanenglish so, if I attempt to run the service it fails and logs look like so: |
|
hi, i got same problem.. that file do not exist, in data folder there is only rca_enabled.conf file i try to create that file with true and false, and i got new errors, so seems that is not only the problem.. |
did you find a fix for that? i got the same Problem right now after a fresh install on a Ubuntu |
|
Ok so here is what I did to get this to work: Created dir: /etc/elasticsearch/tmp Configured the following files: /etc/sysconfig/elasticsearch /etc/systemd/system/elasticsearch.service.d/elasticsearch.conf /usr/share/elasticsearch/bin/performance-analyzer-agent-cli /etc/elasticsearch/jvm.options |
|
I'm not having any issues at the moment @rdecuir, but do you have a reason for your 32g in Java? I'm just curious. your /etc/elasticsearch/jvm.options my /etc/elasticsearch/jvm.options Were you seeing performance issues that required you to increase this? |
|
Note that it is a bad idea to set the java heap above ~30 GB because you lose out on an important optimization called compressed oops. You will likely get worse performance by doing that unless you set it MUCH larger than 30gb, for example 50-60gb. Elastic generally recommends setting the min/max heap size to be half the physical ram on the system, but not more than ~30gb (assuming elasticsearch is all that's running on the system). |
The machines that are running elasticsearch are 64gb machines so we configured them to half that number. From what you are saying would you suggest trying to lower that to 30gb then? I will read the link shortly. We ingest 5-10TB into ES a day with a 20+ cluster. |
Mostly using previous configurations that were determined before my time, that made sense to me given my limited ES "tuning" experience. |
|
Indeed if you have 64gb of ram you will probably get better performance to use slightly less than 32gb of heap, so that your JVM is eligible to use compressed oops. Compressed oops basically allows Java to use 32-bit pointers even on 64-bit systems, thus saving RAM. Since Java is very object-heavy, cutting the size of pointers in half can provide huge RAM savings in a large Java application. If you are not able to use compressed oops, then you might expect almost double the memory usage in some cases (that's why it's not recommended to exceed 32GB of heap unless you can exceed it by a huge amount, like 64GB). |
Thank you, after I read more of the link you sent I'll drop our setting down and give it a shot. |
|
@jordanenglish correct, we have FIPS enabled. |
Can you elaborate on your setup some? I had to get a new ODFE server stood up and can't get it working unless I disable FIPS on RHEL8. |
So, We are running on a AWS with FIPS enabled centos7 boxes. It is required that we do not disable FIPS and keep noexec on the /temp dir. I've installed opendistro via yum after adding it to our repo. I got i working which I provided details above in the comments. It took a while of touching everything but it resolved the issues I was seeing. Let me know if you need more details or what you're looking for, not sure I can help in a different environment but I can def tell you what I went through and my process. |
Thanks for your feedback. I think that RHEL7 handles signatures on unsigned RPMs differently than RHEL8. Even with the repo available on my RHEL8 box I cannot do a I have to do the following: |
|
I think the better way should be to have a config file for the plugin instead of cheating around with env variable and modifying stuff in service files. |
Hi there, as documented in issue #70 it is not possible to use the performance-analyzer-agent-cli with a temp directory marked noexec. The elasticsearch startup scripts provide the possibility of using the "ES_TMPDIR" variable to override the temp directory path, however this does not work for the performance-analyzer-agent-cli script because using ES_TMPDIR causes ES_JAVA_OPTS to be clobbered.
See how on line 9 of performance-analyzer-agent-cli, ES_JAVA_OPTS is set with values necessary to boot the performance analyzer, before calling elasticsearch-cli:
https://github.com/opendistro-for-elasticsearch/performance-analyzer/blob/master/packaging/performance-analyzer-agent-cli#L9
Elasticsearch-cli then calls elasticsearch-env to set up the environment:
https://github.com/elastic/elasticsearch/blob/v7.1.1/distribution/src/bin/elasticsearch-cli#L5
Finally elasticsearch-env clobbers ES_JAVA_OPTS when ES_TMPDIR is set:
https://github.com/elastic/elasticsearch/blob/v7.1.1/distribution/src/bin/elasticsearch-env#L88
Note that this ES_TMPDIR is gone in the latest version of the scripts, so maybe they now intend for us to use ES_JAVA_OPTS to set java.io.tmpdir with a -D parameter. However this approach has a different problem: when
/etc/sysconfig/elasticsearchis sourced in elasticsearch-env, it similarly clobbers the ES_JAVA_OPTS that performance-analyzer-agent-cli defines. Thus the settings from PA_AGENT_JAVA_OPTS, such as the log4j configuration file, are lost.The workaround which we are using, as described in issue #70, is to edit the performance-analyzer-agent-cli script and add the string
-Djava.io.tmpdir=/apps/elasticsearch/tmpto the end of PA_AGENT_JAVA_OPTS (where /apps/ is a volume that we control). We then also set the other components of elasticsearch to use that same temp directory with the ES_TMPDIR variable (NOT the ES_JAVA_OPTS variable, otherwise the PA_AGENT_JAVA_OPTS get lost as noted above).The text was updated successfully, but these errors were encountered: