Skip to content

Commit 214ea21

Browse files
LothiraldanLothiraldan
authored
[NA] [DOCS] Improve filter_string parameter documentation (#3217)
1 parent f2eb048 commit 214ea21

File tree

6 files changed

+233
-49
lines changed

6 files changed

+233
-49
lines changed

‎apps/opik-documentation/documentation/fern/docs/evaluation/evaluate_threads.mdx‎

Lines changed: 36 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
---
22
subtitle: Step-by-step guide on how to evaluate conversation threads
33
---
4+
45
When you are running multi-turn conversations using frameworks that support LLM agents, the Opik integration will
56
automatically group related traces into conversation threads using parameters suitable for each framework.
67

@@ -57,20 +58,39 @@ filter_string='id = "0197ad2a" AND status = "inactive"'
5758

5859
**Supported filter fields and operators**
5960

60-
The `evaluate_threads` function supports the following filter fields in the `filter_string` and
61-
operators to be applied to the corresponding fields:
62-
63-
| Field | Type | Operators |
64-
|--------------------|----------|-------------------------------|
65-
| id | string | ``=, contains, not_contains`` |
66-
| status | string | ``=, contains, not_contains`` |
67-
| start_time | datetime | ``=, >, <, >=, <=`` |
68-
| end_time | datetime | ``=, >, <, >=, <=`` |
69-
| feedback_scores | dict | ``=, >, <, >=, <=`` |
70-
| tags | list | ``contains`` |
71-
| duration | number | ``=, >, <, >=, <=`` |
72-
| number_of_messages | number | ``=, >, <, >=, <=`` |
73-
| created_by | string | ``=, contains, not_contains`` |
61+
The `evaluate_threads` function supports the following filter fields in the `filter_string` using Opik Query Language (OQL).
62+
All fields and operators are the same as those supported by `search_traces` and `search_spans`:
63+
64+
| Field | Type | Operators |
65+
| ------------------------- | ---------- | --------------------------------------------------------------------------- |
66+
| `id` | String | `=`, `!=`, `contains`, `not_contains`, `starts_with`, `ends_with`, `>`, `<` |
67+
| `name` | String | `=`, `!=`, `contains`, `not_contains`, `starts_with`, `ends_with`, `>`, `<` |
68+
| `created_by` | String | `=`, `!=`, `contains`, `not_contains`, `starts_with`, `ends_with`, `>`, `<` |
69+
| `thread_id` | String | `=`, `!=`, `contains`, `not_contains`, `starts_with`, `ends_with`, `>`, `<` |
70+
| `type` | String | `=`, `!=`, `contains`, `not_contains`, `starts_with`, `ends_with`, `>`, `<` |
71+
| `model` | String | `=`, `!=`, `contains`, `not_contains`, `starts_with`, `ends_with`, `>`, `<` |
72+
| `provider` | String | `=`, `!=`, `contains`, `not_contains`, `starts_with`, `ends_with`, `>`, `<` |
73+
| `status` | String | `=`, `contains`, `not_contains` |
74+
| `start_time` | DateTime | `=`, `>`, `<`, `>=`, `<=` |
75+
| `end_time` | DateTime | `=`, `>`, `<`, `>=`, `<=` |
76+
| `input` | String | `=`, `contains`, `not_contains` |
77+
| `output` | String | `=`, `contains`, `not_contains` |
78+
| `metadata` | Dictionary | `=`, `contains`, `>`, `<` |
79+
| `feedback_scores` | Numeric | `=`, `>`, `<`, `>=`, `<=` |
80+
| `tags` | List | `contains` |
81+
| `usage.total_tokens` | Numeric | `=`, `!=`, `>`, `<`, `>=`, `<=` |
82+
| `usage.prompt_tokens` | Numeric | `=`, `!=`, `>`, `<`, `>=`, `<=` |
83+
| `usage.completion_tokens` | Numeric | `=`, `!=`, `>`, `<`, `>=`, `<=` |
84+
| `duration` | Numeric | `=`, `!=`, `>`, `<`, `>=`, `<=` |
85+
| `number_of_messages` | Numeric | `=`, `!=`, `>`, `<`, `>=`, `<=` |
86+
| `total_estimated_cost` | Numeric | `=`, `!=`, `>`, `<`, `>=`, `<=` |
87+
88+
**Rules:**
89+
90+
- String values must be wrapped in double quotes
91+
- DateTime fields require ISO 8601 format (e.g., "2024-01-01T00:00:00Z")
92+
- Use dot notation for nested objects: `metadata.model`, `feedback_scores.accuracy`
93+
- Multiple conditions can be combined with `AND` (OR is not supported)
7494

7595
The `feedback_scores` field is a dictionary where the keys are the metric names and the values are the metric values.
7696
You can use it to filter threads based on their feedback scores. For example, if you want to evaluate only threads
@@ -95,9 +115,10 @@ Once the evaluation is complete, you can access the evaluation results in the Op
95115
Threads are automatically marked as inactive after the timeout period and you can also manually mark a thread as inactive via UI or via SDK.
96116

97117
You can only evaluate/score threads that are inactive.
118+
98119
</Note>
99120

100121
## Next steps
101122

102123
For more details on what metrics can be used to score conversational threads, refer to
103-
the [conversational metrics](/evaluation/metrics/conversation_threads_metrics) page.
124+
the [conversational metrics](/evaluation/metrics/conversation_threads_metrics) page.

‎apps/opik-documentation/documentation/fern/docs/prompt_engineering/prompt_management.mdx‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -171,6 +171,25 @@ for prompt in filtered:
171171
print(prompt.name, prompt.commit, prompt.prompt)
172172
```
173173

174+
The `filter_string` parameter uses Opik Query Language (OQL) with the format:
175+
`"<COLUMN> <OPERATOR> <VALUE> [AND <COLUMN> <OPERATOR> <VALUE>]*"`
176+
177+
**Supported columns for prompts:**
178+
179+
| Column | Type | Operators |
180+
| ------------ | ------ | --------------------------------------------------------------------------- |
181+
| `id` | String | `=`, `!=`, `contains`, `not_contains`, `starts_with`, `ends_with`, `>`, `<` |
182+
| `name` | String | `=`, `!=`, `contains`, `not_contains`, `starts_with`, `ends_with`, `>`, `<` |
183+
| `created_by` | String | `=`, `!=`, `contains`, `not_contains`, `starts_with`, `ends_with`, `>`, `<` |
184+
| `tags` | List | `contains` |
185+
186+
**Examples:**
187+
188+
- `tags contains "production"` - Filter by tag
189+
- `name contains "summary"` - Filter by name substring
190+
- `created_by = "[email protected]"` - Filter by creator
191+
- `tags contains "alpha" AND tags contains "beta"` - Multiple tag filtering
192+
174193
`search_prompts` returns the **latest** version for each matching prompt.
175194

176195
## Linking prompts to Experiments

‎apps/opik-documentation/documentation/fern/docs/tracing/export_data.mdx‎

Lines changed: 34 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -54,36 +54,44 @@ traces = client.search_traces(
5454
traces = [trace.dict() for trace in traces]
5555
```
5656

57-
The `filter_string` parameter should be a string in the following format:
57+
The `filter_string` parameter should be a string in the following format using Opik Query Language (OQL):
5858

5959
```
60-
"<COLUMN> <OPERATOR> <VALUE> [and <COLUMN> <OPERATOR> <VALUE>]*"
60+
"<COLUMN> <OPERATOR> <VALUE> [AND <COLUMN> <OPERATOR> <VALUE>]*"
6161
```
6262

63-
where:
64-
65-
1. `<COLUMN>`: The column name to filter on, these can be:
66-
- `name`
67-
- `input`
68-
- `output`
69-
- `start_time`
70-
- `end_time`
71-
- `metadata`
72-
- `feedback_scores`
73-
- `tags`
74-
- `usage.total_tokens`
75-
- `usage.prompt_tokens`
76-
- `usage.completion_tokens`.
77-
2. `<OPERATOR>`: The operator to use for the filter, this can be `=`, `!=`, `>`, `>=`, `<`, `<=`, `contains`, `not_contains`. Not that not all operators are supported for all columns.
78-
3. `<VALUE>`: The value to use in the comparison to `<COLUMN>`. If the value is a string, you will need to wrap it in double quotes.
79-
80-
You can add as many `and` clauses as required.
81-
82-
If a `<COLUMN>` item refers to a nested object, then you can use the
83-
dot notation to access contained values by using its key. For example,
84-
you could use:
85-
86-
`"feedback_scores.accuracy > 0.5"`
63+
**Supported columns and operators:**
64+
65+
| Column | Type | Operators |
66+
| ------------------------- | ---------- | --------------------------------------------------------------------------- |
67+
| `id` | String | `=`, `!=`, `contains`, `not_contains`, `starts_with`, `ends_with`, `>`, `<` |
68+
| `name` | String | `=`, `!=`, `contains`, `not_contains`, `starts_with`, `ends_with`, `>`, `<` |
69+
| `created_by` | String | `=`, `!=`, `contains`, `not_contains`, `starts_with`, `ends_with`, `>`, `<` |
70+
| `thread_id` | String | `=`, `!=`, `contains`, `not_contains`, `starts_with`, `ends_with`, `>`, `<` |
71+
| `type` | String | `=`, `!=`, `contains`, `not_contains`, `starts_with`, `ends_with`, `>`, `<` |
72+
| `model` | String | `=`, `!=`, `contains`, `not_contains`, `starts_with`, `ends_with`, `>`, `<` |
73+
| `provider` | String | `=`, `!=`, `contains`, `not_contains`, `starts_with`, `ends_with`, `>`, `<` |
74+
| `status` | String | `=`, `contains`, `not_contains` |
75+
| `start_time` | DateTime | `=`, `>`, `<`, `>=`, `<=` |
76+
| `end_time` | DateTime | `=`, `>`, `<`, `>=`, `<=` |
77+
| `input` | String | `=`, `contains`, `not_contains` |
78+
| `output` | String | `=`, `contains`, `not_contains` |
79+
| `metadata` | Dictionary | `=`, `contains`, `>`, `<` |
80+
| `feedback_scores` | Numeric | `=`, `>`, `<`, `>=`, `<=` |
81+
| `tags` | List | `contains` |
82+
| `usage.total_tokens` | Numeric | `=`, `!=`, `>`, `<`, `>=`, `<=` |
83+
| `usage.prompt_tokens` | Numeric | `=`, `!=`, `>`, `<`, `>=`, `<=` |
84+
| `usage.completion_tokens` | Numeric | `=`, `!=`, `>`, `<`, `>=`, `<=` |
85+
| `duration` | Numeric | `=`, `!=`, `>`, `<`, `>=`, `<=` |
86+
| `number_of_messages` | Numeric | `=`, `!=`, `>`, `<`, `>=`, `<=` |
87+
| `total_estimated_cost` | Numeric | `=`, `!=`, `>`, `<`, `>=`, `<=` |
88+
89+
**Rules:**
90+
91+
- String values must be wrapped in double quotes
92+
- Multiple conditions can be combined with `AND` (OR is not supported)
93+
- DateTime fields require ISO 8601 format (e.g., "2024-01-01T00:00:00Z")
94+
- Use dot notation for nested objects: `metadata.model`, `feedback_scores.accuracy`
8795

8896
Here are some full examples of using `filter_string` values in searches:
8997

‎sdks/python/src/opik/api_objects/opik_client.py‎

Lines changed: 96 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -977,7 +977,41 @@ def search_traces(
977977
978978
Args:
979979
project_name: The name of the project to search traces in. If not provided, will search across the project name configured when the Client was created which defaults to the `Default Project`.
980-
filter_string: A filter string to narrow down the search. If not provided, all traces in the project will be returned up to the limit.
980+
filter_string: A filter string to narrow down the search using Opik Query Language (OQL).
981+
The format is: "<COLUMN> <OPERATOR> <VALUE> [AND <COLUMN> <OPERATOR> <VALUE>]*"
982+
983+
Supported columns include:
984+
- `id`, `name`, `created_by`, `thread_id`, `type`, `model`, `provider`: String fields with full operator support
985+
- `status`: String field (=, contains, not_contains only)
986+
- `start_time`, `end_time`: DateTime fields (use ISO 8601 format, e.g., "2024-01-01T00:00:00Z")
987+
- `input`, `output`: String fields for content (=, contains, not_contains only)
988+
- `metadata`: Dictionary field (use dot notation, e.g., "metadata.model")
989+
- `feedback_scores`: Numeric field (use dot notation, e.g., "feedback_scores.accuracy")
990+
- `tags`: List field (use "contains" operator only)
991+
- `usage.total_tokens`, `usage.prompt_tokens`, `usage.completion_tokens`: Numeric usage fields
992+
- `duration`, `number_of_messages`, `total_estimated_cost`: Numeric fields
993+
994+
Supported operators by column:
995+
- `id`, `name`, `created_by`, `thread_id`, `type`, `model`, `provider`: =, !=, contains, not_contains, starts_with, ends_with, >, <
996+
- `status`: =, contains, not_contains
997+
- `start_time`, `end_time`: =, >, <, >=, <=
998+
- `input`, `output`: =, contains, not_contains
999+
- `metadata`: =, contains, >, <
1000+
- `feedback_scores`: =, >, <, >=, <=
1001+
- `tags`: contains (only)
1002+
- `usage.total_tokens`, `usage.prompt_tokens`, `usage.completion_tokens`, `duration`, `number_of_messages`, `total_estimated_cost`: =, !=, >, <, >=, <=
1003+
1004+
Examples:
1005+
- `start_time >= "2024-01-01T00:00:00Z"` - Filter by start date
1006+
- `start_time > "2024-01-01T00:00:00Z" AND start_time < "2024-02-01T00:00:00Z"` - Date range
1007+
- `input contains "question"` - Filter by input content
1008+
- `usage.total_tokens > 1000` - Filter by token usage
1009+
- `feedback_scores.accuracy > 0.8` - Filter by feedback score
1010+
- `tags contains "production"` - Filter by tag
1011+
- `metadata.model = "gpt-4"` - Filter by metadata field
1012+
- `thread_id = "thread_123"` - Filter by thread ID
1013+
1014+
If not provided, all traces in the project will be returned up to the limit.
9811015
max_results: The maximum number of traces to return.
9821016
truncate: Whether to truncate image data stored in input, output, or metadata
9831017
"""
@@ -1015,7 +1049,41 @@ def search_spans(
10151049
Args:
10161050
project_name: The name of the project to search spans in. If not provided, will search across the project name configured when the Client was created which defaults to the `Default Project`.
10171051
trace_id: The ID of the trace to search spans in. If provided, the search will be limited to the spans in the given trace.
1018-
filter_string: A filter string to narrow down the search.
1052+
filter_string: A filter string to narrow down the search using Opik Query Language (OQL).
1053+
The format is: "<COLUMN> <OPERATOR> <VALUE> [AND <COLUMN> <OPERATOR> <VALUE>]*"
1054+
1055+
Supported columns include:
1056+
- `id`, `name`, `created_by`, `thread_id`, `type`, `model`, `provider`: String fields with full operator support
1057+
- `status`: String field (=, contains, not_contains only)
1058+
- `start_time`, `end_time`: DateTime fields (use ISO 8601 format, e.g., "2024-01-01T00:00:00Z")
1059+
- `input`, `output`: String fields for content (=, contains, not_contains only)
1060+
- `metadata`: Dictionary field (use dot notation, e.g., "metadata.model")
1061+
- `feedback_scores`: Numeric field (use dot notation, e.g., "feedback_scores.accuracy")
1062+
- `tags`: List field (use "contains" operator only)
1063+
- `usage.total_tokens`, `usage.prompt_tokens`, `usage.completion_tokens`: Numeric usage fields
1064+
- `duration`, `number_of_messages`, `total_estimated_cost`: Numeric fields
1065+
1066+
Supported operators by column:
1067+
- `id`, `name`, `created_by`, `thread_id`, `type`, `model`, `provider`: =, !=, contains, not_contains, starts_with, ends_with, >, <
1068+
- `status`: =, contains, not_contains
1069+
- `start_time`, `end_time`: =, >, <, >=, <=
1070+
- `input`, `output`: =, contains, not_contains
1071+
- `metadata`: =, contains, >, <
1072+
- `feedback_scores`: =, >, <, >=, <=
1073+
- `tags`: contains (only)
1074+
- `usage.total_tokens`, `usage.prompt_tokens`, `usage.completion_tokens`, `duration`, `number_of_messages`, `total_estimated_cost`: =, !=, >, <, >=, <=
1075+
1076+
Examples:
1077+
- `start_time >= "2024-01-01T00:00:00Z"` - Filter by start date
1078+
- `start_time > "2024-01-01T00:00:00Z" AND start_time < "2024-02-01T00:00:00Z"` - Date range
1079+
- `input contains "question"` - Filter by input content
1080+
- `usage.total_tokens > 1000` - Filter by token usage
1081+
- `feedback_scores.accuracy > 0.8` - Filter by feedback score
1082+
- `tags contains "production"` - Filter by tag
1083+
- `metadata.model = "gpt-4"` - Filter by metadata field
1084+
- `thread_id = "thread_123"` - Filter by thread ID
1085+
1086+
If not provided, all spans in the project/trace will be returned up to the limit.
10191087
max_results: The maximum number of spans to return.
10201088
truncate: Whether to truncate image data stored in input, output, or metadata
10211089
"""
@@ -1211,13 +1279,36 @@ def get_all_prompts(self, name: str) -> List[Prompt]:
12111279
)
12121280
return self.get_prompt_history(name)
12131281

1214-
def search_prompts(self, filter_string: Optional[str] = None) -> List[Prompt]:
1282+
def search_prompts(
1283+
self, name: Optional[str] = None, filter_string: Optional[str] = None
1284+
) -> List[Prompt]:
12151285
"""
12161286
Retrieve the latest prompt versions for the given search parameters.
12171287
12181288
Parameters:
1219-
filter_string: A filter string using Opik Query Language. It will be parsed and
1220-
converted into a stringified list of filters expected by the backend.
1289+
name: The substring of the prompt name to search for. If you have an exact name, consider using the `get_prompt` method instead since the name is a unique identifier.
1290+
filter_string: A filter string to narrow down the search using Opik Query Language (OQL).
1291+
The format is: "<COLUMN> <OPERATOR> <VALUE> [AND <COLUMN> <OPERATOR> <VALUE>]*"
1292+
1293+
Supported columns include:
1294+
- `id`, `name`: String fields
1295+
- `tags`: List field (use "contains" operator only)
1296+
- `created_by`: String field
1297+
1298+
Supported operators by column:
1299+
- `id`: =, !=, contains, not_contains, starts_with, ends_with, >, <
1300+
- `name`: =, !=, contains, not_contains, starts_with, ends_with, >, <
1301+
- `created_by`: =, !=, contains, not_contains, starts_with, ends_with, >, <
1302+
- `tags`: contains (only)
1303+
1304+
Examples:
1305+
- `tags contains "alpha"` - Filter by tag
1306+
- `tags contains "alpha" AND tags contains "beta"` - Filter by multiple tags
1307+
- `name contains "summary"` - Filter by name substring
1308+
- `created_by = "[email protected]"` - Filter by creator
1309+
- `id starts_with "prompt_"` - Filter by ID prefix
1310+
1311+
If not provided, all prompts matching the name filter will be returned.
12211312
12221313
Returns:
12231314
List[Prompt]: A list of Prompt instances found.

0 commit comments

Comments
 (0)