[MLOB-5067] Simplify access to Goal Completeness #33646
Open
+61
−64
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tillwf
force-pushed
the
till.wohlfarth/MLOB-5067/Simplify_gc_access
branch
from
44971e0 to
aa2a7eb
Compare
Contributor
Preview links (active after the
|
estherk15
approved these changes
Jan 5, 2026
| | Evaluated on LLM spans | Evaluated using LLM | Checks whether the agent resolved the user’s intent by analyzing full session spans. Runs only on sessions marked as completed. | | ||
|
|
||
| ##### How to Use | ||
| <div class="alert alert-info">Goal completeness is only available for OpenAI and Azure OpenAI.</div> |
Contributor
There was a problem hiding this comment.
Suggested change
| <div class="alert alert-info">Goal completeness is only available for OpenAI and Azure OpenAI.</div> | |
| <div class="alert alert-info">Goal Completeness is only available for OpenAI and Azure OpenAI.</div> |
|
|
||
| The evaluation requires sending a span with a specific tag when the session ends. This signal allows the evaluation to identify session boundaries and trigger the completeness assessment: | ||
|
|
||
| For optimal evaluation accuracy and cost control, it is preferable to send a tag when the session is finished and configure the evaluation to run only on session with this tag. The evaluation returns a detailed breakdown including resolved intentions, unresolved intentions, and reasoning for the assessment. A session is considered incomplete if more than 50% of identified intentions remain unresolved. |
Contributor
There was a problem hiding this comment.
Suggested change
| For optimal evaluation accuracy and cost control, it is preferable to send a tag when the session is finished and configure the evaluation to run only on session with this tag. The evaluation returns a detailed breakdown including resolved intentions, unresolved intentions, and reasoning for the assessment. A session is considered incomplete if more than 50% of identified intentions remain unresolved. | |
| For optimal evaluation accuracy and cost control, it is preferable to send a tag when the session is finished and configure the evaluation to run only on sessions with this tag. The evaluation returns a detailed breakdown including resolved intentions, unresolved intentions, and reasoning for the assessment. A session is considered incomplete if more than 50% of identified intentions remain unresolved. |
|
|
||
| 1. Go to the **Goal Completeness** settings | ||
| 2. Configure the evaluation data: | ||
| - Select **spans** as the data type since Goal Completeness runs on LLM spans which contains the full session history. |
Contributor
There was a problem hiding this comment.
Suggested change
| - Select **spans** as the data type since Goal Completeness runs on LLM spans which contains the full session history. | |
| - Select **spans** as the data type since Goal Completeness runs on LLM spans which contain the full session history. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes MLOB-5067