LLM Langchain wrap call in chain to display it in sentry AI tab #13905
Conversation
paris is already the capital
|
@Luke31 is attempting to deploy a commit to the Sentry Team on Vercel. A member of the Team first needs to authorize it. |
convert to chain
indentation
Luke31
changed the title
Add hint to enable tracing to view data in LLM tab
indentations
| import sentry_sdk | ||
|
|
||
| sentry_sdk.init(...) # same as above | ||
|
|
||
| llm = ChatOpenAI(model="gpt-3.5-turbo-0125", temperature=0, api_key="bad API key") | ||
| with sentry_sdk.start_transaction(op="ai-inference", name="The result of the AI inference"): | ||
| response = llm.invoke([("system", "What is the capital of paris?")]) | ||
| chain = (llm | StrOutputParser()).with_config({"run_name": "test-run"}) |
There was a problem hiding this comment.
I'm admittedly not super familiar with Langchain, could you explain what this change does differently than what we had before? I get that the run_name allows it to show up in Sentry, but what is the StrOutputParser() for?
There was a problem hiding this comment.
@szokeasaurusrex Thank you for the review!
To be honest the StrOutputParser() is more or less syntactic sugar for response.content. Here it is mainly used in combination with the pipe | to create a chain.
The important difference here is that the syntax (X | Y) creates a LangChain chain/Runnable using LangChain Expression Language LCEL. This is the new syntax of the deprecated LLMChain class.
As this example has no prompt and is just invoked directly with a question, no prompt is passed and the chain is just created by adding an output-parser.
I have tried following code:
llm = llm.with_config({"run_name": "llm_raw"})
resp_llm_raw = llm.invoke([("system", "What is the capital of France?")])
print(f"llm response raw without StrOutputParser:\n{resp_llm_raw}\n\n")
print(f"llm response raw content:\n{resp_llm_raw.content}\n\n")
chain_parsed = (llm_session_summary | StrOutputParser()).with_config({"run_name": "chain_parsed"})
resp_chain_parsed = chain_parsed.invoke([("system", "What is the capital of France?")])
print(f"chain response with StrOutputParser:\n{resp_chain_parsed}\n\n")
resulting in this output:
llm response raw without StrOutputParser:
content='The capital of France is Paris.' additional_kwargs={'refusal': None} response_metadata={'token_usage': {'completion_tokens': 7, 'prompt_tokens': 14, 'total_tokens': 21, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}, 'model_name': 'gpt-4o-mini-2024-07-18', 'system_fingerprint': 'XXXXX', 'id': 'XXXXX', 'finish_reason': 'stop', 'logprobs': None} id='run-343435-4d8c-4e85-9ca5-3414e396e6f4-0' usage_metadata={'input_tokens': 14, 'output_tokens': 7, 'total_tokens': 21, 'input_token_details': {'audio': 0, 'cache_read': 0}, 'output_token_details': {'audio': 0, 'reasoning': 0}}
llm response raw content:
The capital of France is Paris.
chain response with StrOutputParser:
The capital of France is Paris.
Only chain_parsed is visible in AI-tab, llm_raw is not visible:
I dug a bit deeper in the sentry-python sdk and found
- only spans with
op-prefixai.pipeline.will get an ai-pipeline name and appear in the AI-tab: https://github.com/getsentry/sentry-python/blob/51db87ca6fd3c6cfc6eadd8ba800d72537a55d29/sentry_sdk/integrations/langchain.py#L150 - for LangChain only root-chains get that prefix, which is set in
on_chain_startlangchain callback: https://github.com/getsentry/sentry-python/blob/51db87ca6fd3c6cfc6eadd8ba800d72537a55d29/sentry_sdk/integrations/langchain.py#L313 / LANGCHAIN_PIPELINE = "ai.pipeline.langchain"
So a plain llm-call will not appear as an AI pipeline.
TL;DR:
- Only Langchain root-chains appear in the sentry AI tab.
- From my point of view that makes sense as an AI-pipeline is the same concept as a chain. It just wasn't clear from the documentation.
| @@ -36,6 +36,8 @@ An additional dependency, `tiktoken`, is required to be installed if you want to | |||
|
|
|||
| In addition to capturing errors, you can monitor interactions between multiple services or applications by [enabling tracing](/concepts/key-terms/tracing/). You can also collect and analyze performance profiles from real users with [profiling](/product/explore/profiling/). | |||
|
|
|||
| Tracing is required to see AI pipelines in sentry's [AI tab](https://sentry.io/insights/ai/llm-monitoring/). | |||
There was a problem hiding this comment.
Thanks for adding this, definitely an oversight that we forgot to add this before!
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
1 Skipped Deployment
|
DESCRIBE YOUR PR
tackles part of issue #13167 and getsentry/sentry-python#3007 (comment)
Current documentation works great for traces and profiles. However if setting up Langchain as described in the documentation, the AI LLM-Monitoring tab in sentry remains empty.
Update Langchain example to introduce a chain with a custom run-name which will be displayed as a pipeline in the AI dashboard.
This will show with the error as
including the message
Also change Paris to France as Paris has no capital🇫🇷
IS YOUR CHANGE URGENT?
Help us prioritize incoming PRs by letting us know when the change needs to go live.
SLA
Thanks in advance for your help!
PRE-MERGE CHECKLIST
Make sure you've checked the following before merging your changes:
LEGAL BOILERPLATE
Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. and is gonna need some rights from me in order to utilize my contributions in this here PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.
EXTRA RESOURCES