How to best include full queries in span

[daldoyle]

Hi there,

I've been working with OTLP for a bit now, focused primarily on Elasticsearch but likely to branch out to other areas soon. Overall the experience has been great! As I continue to expand and actually use this in my day-to-day, I noticed that the spans generated by ElasticsearchInstrumentor().instrument() strip out a lot of the query information. After some reading up I understand why this is and fully support the decision to make that a default behavior.

My question is about best practices when I DO want to store the whole query. Often times this is a very useful debugging tool for operational issues, such as understanding why a particular query may have been slow or failed. There is no option (at least in the ES instrumentor) to toggle this behavior that I could find, so I have come up with some "clever" solutions like defining a local response_hook and doing uninstrument/instrument right before the call to Elastic to allow me to scope in the query.

example -

query = "some_awesome_es_query_that_I_build"
 
def _otlp_response_hook(span, response):
    # overwrite the sanitized db.statement with my scoped in query variable
    span.set_attribute('db.statement', query)

instrumentor = ElasticsearchInstrumentor()

# disconnect any previous instrumentation
if instrumentor.is_instrumented_by_opentelemetry:
    instrumentor.uninstrument()

# attach my new local hook to the auto instrumentation            
instrumentor.instrument(response_hook=_otlp_response_hook)

my_es_object.search(query)

This works but it feels hacky. I've dug through the source code and the overhead of doing instrument/uninstrument doesn't seem to be significant, however this is also a lingering concern.

The other option I have come up with is to add more spans, but this makes later review of the results a bit clunky because now for every ES query I have two spans to look at with partial data in each. One generated by the auto-instrumentation, and the other generated by me manually.

I could of course use another span attribute key besides "db.statement" but now I'm breaking the naming convention and integration with existing tooling.

The final option is I could of course manually instrument the whole ES call and include all of the same info that the auto-instrumentor does (index, method, url, timing, etc), but now I'm just re-implementing the wheel.

Is there a general guidance here? Am I completely overlooking something? Is this a possible / reasonable thing to be doing?