Ability to search all tenants in multi-tenant setups

[zackman0010]

I'd like the ability to search all tenants at once in a multi-tenant setup. At least when pulling a trace by trace ID. We have multiple departments who communicate with each other, and each department is set up as its own tenant. If a trace goes from tenant A->B->A, it will appear as a broken trace when querying tenant A and as a root span missing trace when querying tenant B.

[joe-elliott]

This already exists in our enterprise tracing product: GET. We have been discussing open sourcing it, but we weren't sure if the community was interested. Thanks for the feature request!

I will bring it up internally and then report back.

[joe-elliott]

So we've discussed this internally and we do intend to eventually open source it. However, there is unfortunately no timeline I can share for this. I would expect to revisit this decision in the next few months.

FWIW, I'm working to open source it and it will happen as soon as we can get alignment internally.

[aaguilartablada]

Hi! I am also interested in this feature. Is there any news? Thanks!!!

[AleksuKey]

would be great to have this feature implemented

[JorTurFer]

We would love to have this feature! ❤️

[luissb40]

Is a very interesting feature!!

[ykapilesh]

Hi.. this is a very useful feature to see the end to end trace spanning multiple systems. With Loki we are able to do this while for tempo this feature is missing.

[sakthiraam]

This is a very useful feature to really see the power of distributed tracing. We are already able to make use of this feature in Mimir and Loki. Hope to see this feature soon in OSS Tempo ❤️

[cheesengoon]

Hi @joe-elliott, we love to have this feature in OSS Tempo for distributed tracing!

[alex5517]

+1

[joe-elliott]

I gather this is an in demand request. The enterprise version referenced above is buried in a component which does not exist in OSS Tempo. I would like this feature as well, but we are not currently prioritizing it. If someone from the community would like to work on it, I will help review a PR.

There are many levels this could be tackled at, but the easiest would be to add a piece of middleware in the query frontend that uses dskit to parse multiple tenant ids in X-Scope-OrgID.

This middleware would then pass a downstream request once for each tenant.

[mghildiy]

I am working on this issue. I want to test my changes. What's the correct way of doing it?

[joe-elliott]

Manually:
You could modify this example to send to 2 different tenants on the backend and then curl Tempo with the appropriate multi-tenant header:

https://github.com/grafana/tempo/tree/main/example/docker-compose/otel-collector-multitenant

Testing
I'd take an e2e test like this:

tempo/integration/e2e/e2e_test.go

Line 41 in bc3f543

func TestAllInOne(t *testing.T) {

• Rip out the "for each backend" part
• Split the test trace created here and push to 2 different tenants.
• Adjust the code that queries the trace back to pass the multi-tenant header and confirm that the retrieved trace is complete.

[mghildiy]

When I run e2e_test.TestAllInOne, I see error as:

tempo: failed parsing config: failed to parse configFile /shared/config.yaml: yaml: unmarshal errors:
tempo: line 45: field user_configurable_overrides not found in type overrides.Limits
panic: test timed out after 30s

Is some configuration change needed to make it run locally?

[joe-elliott]

Is some configuration change needed to make it run locally?

e2e tests use the image grafana/tempo:latest. To confirm you are using the correct image run make docker-tempo from the root. It will build grafana/tempo:latest with the current commit (which should match the e2e tests).

Also, if you run make test-e2e from the root of the repo it will force rebuild the image correctly.

[mghildiy]

A WIP PR:
#2854

[jtama]

any news on this ?

[roobalimsab]

Hello Team,

Any timelines to have this released?

[electron0zero]

@roobalimsab we have #3087 up, we are hoping to include it in next tempo release.

[roobalimsab]

Great, thanks for the update. looking forward