Consider using RubyLLMs new instrumentation layer

[sandstrom]

RubyLLM just released a better way to instrument it's behavior. Similar to Rails instrumentation.

Maybe this would make it easier for this gem to support instrumentation of RubyLLM?

crmne/ruby_llm@04414d5

[delner]

Hey sandstrom ! Sorry for the late reply, and thanks for sharing this!

I think this is a good suggestion. RubyLLM's new instrumentation layer is a cleaner, officially-supported foundation than the internal patching we do today, and we'd prefer vendor-native hooks. As a bonus it'd also let us add embedding tracing.

That said, evaluating it against our current integration, adopting it right now would be roughly a side-grade: the robustness gain comes with a couple of regressions we'd need to close first:

1. Streaming "time-to-first-token" metric: the event API wraps the whole completion and exposes no per-chunk hook, so we'd lose TTFT for streaming responses (a metric we record today). This is something we'd probably want to push upstream but could probably monkey-patch in the interim.
2. *Per-client isolation: we currently let users target instrumentation on a single client instance with our instrument!() API. RubyLLM's event layer is configured via config.instrumenter (global or per-context), so isolating one chat is more difficult.

I'm generally inclined to accept the direction and put it on the roadmap at lower priority, adopting it once these two items can be handled cleanly. Thanks again for pointing this out!

[sandstrom]

Sounds great!

I think the developer of RubyLLM is pretty approachable, and since Braintrust is well-known, you'd probably get more attention that I get.

So if these are easiest to patch upstream, then I'd try to knock down the two you mention in the upstream repo first (just opening issues is a good first step, unless there already are open issues).

Regarding isolation, I would also mention that I think Braintrust SDK can be improved in this direction too 😄

See: #161

[delner]

Regarding isolation, I would also mention that I think Braintrust SDK can be improved in this direction too 😄

Oh yes, this is a topic that's been on my list to tackle. It's the next major improvement I'd like to make.

[sandstrom]

Great! 💯