Tracing requests to Elasticsearch with the specialagent?

[jgoeres]

Hi,

I managed to configure the specialagent for most microservices in our stack and I am seeing cross-service traces now.
Alas, one important component in our stack is Elasticsearch, and requests to it do not show up at all. Our microservices use the Apache HTTP Client and/or the Elasticsearch high level client, so I have the two instrumentations enabled for those, i.e.

    -Dsa.instrumentation.plugin.apache:httpclient.enable 
    -Dsa.instrumentation.plugin.elasticsearch:*.enable 

(Note: these are the 1.6.x properties, since I am having trouble with 1.7.x due to #589)
I assumed that this is because I have not instrumented ES itself (even though I had hoped to see the outbound requests as they are being sent by the microservice). So I tried adding the specialagent to ES, which didn't work too well, as ES's security manager gets in the way:

java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "createSecurityManager")
   at java.base/java.security.AccessControlContext.checkPermission(Unknown Source)
   at java.base/java.security.AccessController.checkPermission(Unknown Source)
   at java.base/java.lang.SecurityManager.checkPermission(Unknown Source)
   at java.base/java.lang.SecurityManager.<init>(Unknown Source)
   at io.opentracing.contrib.specialagent.AgentRuleUtil$CallingClass.<init>(AgentRuleUtil.java:101)
   at io.opentracing.contrib.specialagent.AgentRuleUtil$CallingClass.<init>(AgentRuleUtil.java:101)
   at io.opentracing.contrib.specialagent.AgentRuleUtil.getExecutionStack(AgentRuleUtil.java:119)
   at io.opentracing.contrib.specialagent.AgentRule$1.childValue(AgentRule.java:100)
   at io.opentracing.contrib.specialagent.AgentRule$1.childValue(AgentRule.java:89)
   at java.base/java.lang.ThreadLocal$ThreadLocalMap.<init>(Unknown Source)
[...]

I am seeing this as soon as I enable any instrumentation for ES, only if I disable them all (but still add the special agent), ES will start.
Alas, it seems that disabling the security manager in ES, or using a relaxed security policy, is not supported anymore.

Does anyone have a proposal how to get ES requests to show up in traces? Client-side would be fine, but if there is a way to instrument ES itself, that would also be interesting.

THX
J