Is it possible to filter spans of a particular trace?

[jcarres-mdsol]

We have recently installed Grafana+Tempo and we are evaluating it.
Our traces are huge, multiple services and lots of DB calls, maybe ~10.000 spans
Our product UI returns the traceid in headers and engineers use this to search. That works well.

But an engineer asked me about what's the syntaxt to show only the spans in this particular traceid which have failed.

For my experiments, I'd say this is not supported in the query language right now, is that so?

[mapno]

Hi! The trace visualisation in Grafana supports span filters: https://grafana.com/docs/grafana/latest/explore/trace-integration/#span-filters

[jcarres-mdsol]

My grafana does not have this, looking at screenshots in the doc, my installation does not look the same.

I'm using grafana 10.0 and tempo 2.2, and although 10.1 has just released with some changes, span filters should be there, I'm going to check the config. Definitely looks like this would solve the issue.

[joe-elliott]

To add to what @mapno said:

This is somewhat supported using TraceQL now and is also a direction we'd like to grow in. By default in the search results Tempo returns 3 spans per spanset. This is configurable by passing a parameter: spss. Grafana does not yet support it though. Once supported this parameter would allow the user some control over how many spans they received back from the TraceQL query.

Then your users could do a query like this to see all errored database statements and the database they were connecting to without digging into a 10k span trace. The Grafana UI will, in this case, also provide nice deep links into the trace to the selected spans.

{ status = error && span.db.name != "" } | select(span.db.name, span.db.statement)

[jcarres-mdsol]

The idea is that engineers already have a trace id from some error response, the kind of query that would be useful would be

24dcaeea6891de7dad58646aa096ff1 | { status = error}

or

{ traceid=24dcaeea6891de7dad58646aa096ff1 && status = error }

[joe-elliott]

Currently increasing spss is the best we can offer, but like I said this is an area we want to grow into so all feedback is welcome.

[jcarres-mdsol]

Makes sense.
I understand technologically it may be a very different thing but at least for me , as naive to this language the first thing I went for is traceid=.... similar to status or name . It looks natural language-wise that traceid is an entity like those.

or in any other way that it is not special cased I think will make a good design in the future.