[feature] Observability - Checkpoint 1

[jp-agenta]

Description

• Replaces custom tracing capture, processors, and exporters with standard Open Telemetry SDK.
• Replaces custom tracing receiver with standard OTLP receiver.
• Add app example with instrumentation library.

Changes

• WIP

Relates Issues

• AGE-736 Replace in standard tracing in with OTel tracing in SDK
• AGE-737 Add custom inline trace exporter (OTel -> inline traces) in SDK
• AGE-738 Add custom trace processor (spans -> trace) in SDK
• AGE-739 Add custom trace receiver (route) to observability API and service
• AGE-740 Add custom trace adapter (new -> old) to observability service
• AGE-741 Add example with existing OTel instrumentation library

QA

Regression

• Test existing examples (litellm, rag, etc)
  • Applications run without issues in oss
  • Applications run without issues in cloud, inline trace includes spans as before, and Observability includes spans as before

Features

• Test example with OTel instrumentation library
  • Application runs without issues in oss
  • Application runs without issues in cloud, inline trace includes spans from instrumentation library, and Observability includes spans from instrumentation library

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
agenta	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Sep 12, 2024 3:02pm
agenta-documentation	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Sep 12, 2024 3:02pm

[mmabrouk]

Thanks for the quick fixes @jp-agenta
Overall lgtm. I've added a couple of comments here and there, please check them out. There is also the point about load dot env (but totally not a blocker).

@aybruhm I think we're ready to QA.
@jp-agenta Could you please do a quick update to the docs (no need to overinvest, just make sure we don't have wrong information there, I will take over the rewrite soon)
@jp-agenta @aybruhm can you please update any examples / docs that uses the old version, so that we don't have any example or code that does not work out.

I guess this also means deprecation of things, right? This would require some proper notice I guess (although we're explicitly in beta). @jp-agenta can you give it a thought.