Implement Grafana Query Builder #329

archef2000 · 2025-07-04T13:43:56Z

Add support for all operations of LogsQL in the Grafana Query Builder.
@Loori-R

#48

Loori-R

Tested on Grafana 11.3.1.

An error occurs when adding filters:

Open Explorer
Go to Operations → Filters → Words
This issue happens not only with Words, but also with other filters.

archef2000 · 2025-07-07T17:11:27Z

I am currently on 12.0.2 and everything is working will test version 11

archef2000 · 2025-07-07T19:41:33Z

The problem exists up to including v 11.4.6 and is fixed in 11.5.0.
The source problem is that the custom Field Editors get undefined as props

Loori-R · 2025-07-08T12:44:57Z

It looks like the issue is with the Combobox component - it doesn't work correctly with older versions of Grafana.
The Select component (which was its alternative) is now considered deprecated.

I think we should consider raising the minimum supported Grafana version.
What do you think, @dmitryk-dk, @hagen1778?

Loori-R · 2025-07-08T12:47:35Z

@archef2000, Could you please take a look at the tests and fix them?
Here’s the link to the failing workflow:
https://github.com/VictoriaMetrics/victorialogs-datasource/actions/runs/16075336610/job/45477811316?pr=329

dk · 2025-07-08T12:56:21Z

It looks like the issue is with the Combobox component - it doesn't work correctly with older versions of Grafana.
The Select component (which was its alternative) is now considered deprecated.

I think we should consider raising the minimum supported Grafana version.
What do you think, @dk, @roman?

In my experience query builders are fairly hard to use manually, - click click click hundred times, - I prefer plain old text for that. But what do I know, I am probably a wrong dk :)

Loori-R · 2025-07-08T13:02:32Z

In my experience query builders are fairly hard to use manually, - click click click hundred times, - I prefer plain old text for that. But what do I know, I am probably a wrong dk :)

Oops, looks like I tagged the wrong person 😅
Thanks a lot for sharing your thoughts - and sorry for the ping!

dmitryk-dk · 2025-07-08T13:09:03Z

It looks like the issue is with the Combobox component - it doesn't work correctly with older versions of Grafana. The Select component (which was its alternative) is now considered deprecated.

I think we should consider raising the minimum supported Grafana version. What do you think, @dmitryk-dk, @hagen1778?

As for me, to up the version is ok, but we need to test everything properly

We could have another option to use, use the old Combobox, but I am not sure it would work in the newest versions.

11.3.1 as @Loori-R mentioned, is not an old version and many users are using versions 10-11.

Loori-R · 2025-07-08T13:44:07Z

There is no "old" Combobox - it was introduced as a new component and is currently marked as Alpha. View source
The Select component is marked as Deprecated. View source

As an option, we could use Select for now since it’s more stable, and plan to migrate to Combobox in the future.
Guide: Migrating from Select to Combobox

dmitryk-dk · 2025-07-08T13:47:57Z

There is no "old" Combobox - it was introduced as a new component and is currently marked as Alpha. View source

The Select component is marked as Deprecated. View source

As an option, we could use Select for now since it’s more stable, and plan to migrate to Combobox in the future. Guide: Migrating from Select to Combobox

I think using Select instead of Combobox is a better idea. We do not need to change the supported version, and we do not need to use the Alpha component.

archef2000 · 2025-07-08T14:02:03Z

@Loori-R Can comfirm that with the Select component everything works.

Loori-R · 2025-07-08T14:16:49Z

@Loori-R Can comfirm that with the Select component everything works.

@archef2000 Would you mind replacing Combobox with Select?

archef2000 · 2025-07-08T14:22:57Z

Already on it

Loori-R · 2025-07-08T14:52:07Z

Already on it

It looks like there are still some Combobox components in the code:
https://github.com/VictoriaMetrics/victorialogs-datasource/pull/329/files#diff-35151418f87b2fad4cd2f8e9b4515cd9685e1c922d771a348463efc8672b15ecR12-R22

archef2000 · 2025-07-08T14:53:06Z

I am not finished yet and will make a commit once all are changed and tested

archef2000 · 2025-07-08T15:22:53Z

This query should give me the value types of ClientPort right?
/select/logsql/field_values?query=_msg%3A*+%7C+uniq+by+ClientPort+%7C+block_stats+%7C+uniq+type&start=1751839200000&end=1752011999999&field=type

query: _msg:* | uniq by ClientPort | block_stats | uniq type
field: type

Loori-R · 2025-07-09T10:22:01Z

This query should give me the value types of ClientPort right? /select/logsql/field_values?query=_msg%3A*+%7C+uniq+by+ClientPort+%7C+block_stats+%7C+uniq+type&start=1751839200000&end=1752011999999&field=type

query: _msg:* | uniq by ClientPort | block_stats | uniq type field: type

Not sure, but it seems like uniq by ClientPort and uniq type might overlap.
@dmitryk-dk, could you please help clarify?

Loori-R · 2025-07-09T10:42:12Z

Seems like the query won’t return ClientPort types — it just gets unique type values from logs matching the full query as a text filter, because /field_values doesn’t run the pipeline logic.

dmitryk-dk · 2025-07-09T11:55:30Z

_msg:* | uniq by ClientPort | block_stats | uniq type

Sorry guys missed you question

query _msg:* | uniq by ClientPort | block_stats will return

https://play-vmlogs.victoriametrics.com/select/vmui/#/?query=_msg%3A*+%7C+uniq+by+ClientPort+%7C+block_stats&g0.range_input=5m&g0.end_input=2025-07-09T11%3A56%3A01&g0.relative_time=last_5_minutes

field: ClientPorttype: constvalues_bytes: 0bloom_bytes: 0dict_items: 0dict_bytes: 0rows: 1part_path: inmemory

the next filter will get only type so the query like
_msg:* | uniq by ClientPort | block_stats | uniq type will return only type: const
https://play-vmlogs.victoriametrics.com/select/vmui/#/?query=_msg%3A*+%7C+uniq+by+ClientPort+%7C+block_stats+%7C+uniq+type&g0.range_input=5m&g0.end_input=2025-07-09T11%3A55%3A21&g0.relative_time=last_5_minutes

archef2000 · 2025-07-09T12:03:37Z

I was in the assumption that /field_values will return the values of the field that are available after the query. Just noticed it in the value_type operation, but then I will have to change the logic for all. Should I change the getFieldList function with an additional type?

Loori-R · 2025-07-09T12:52:45Z

I don't quite understand what you're trying to achieve.

For the expression:
_msg:* | uniq by ClientPort | block_stats | uniq type

/fields_values can only be used for _msg (I'm not sure how usable it is for _msg, but still).
To substitute values for uniq, you need to use /field_names

archef2000 · 2025-07-09T13:04:15Z

For the value_type operation I want to get the availabe value types of the field specified. The field_values is explained so that I get all field values of the specified field from the result of the query. The query gives me the correct result, but the endpoint seems to ignore the query.

Loori-R · 2025-07-09T14:04:52Z

Maybe we could just hardcode the supported value types (like const, dict, string, int64, float64, etc.) and allow users to enter a custom type if needed - that might be a simpler and more flexible solution.

archef2000 · 2025-09-26T22:11:49Z

@arturminchukov @Loori-R I am now finished with implementing all the operations and fixed all errors I could find.

arturminchukov · 2025-09-29T11:58:38Z

Should Stream operation work?

I cannot set stream via builder, but can do it via code input, and when I switched back to builder, I had predefined fields in stream operation. Is it expected behavior?

Should we not to wrap with ` the stream field? Cause when I tried to use exact filter with stream field I got an error.

So it looks like now I don't have the opportunity to set the stream field.
Tested it on VL playground data.

arturminchukov · 2025-09-29T13:50:04Z

One more strange behavior of filter for me provided in the video

Screen.Recording.2025-09-29.at.17.46.37.mov

archef2000 · 2025-09-29T14:29:04Z

@arturminchukov

I get the same problem and traced it to the request select/logsql/stream_field_names just returning Hello from VM data source!
The stream field does not work with the exact query. It fails because of the = not the "`"

arturminchukov · 2025-09-29T14:09:53Z

src/components/QueryEditor/QueryBuilder/utils/stringSplitter.ts

+
+export type SplitString = SplitStringValue | SplitStringBracket | SplitStringComment;
+
+export const splitString = (str: string): SplitString[] => {


Could you please add some JS Doc for this function with description what this function do? Cause I didn't guess the functionality just by the name. And maybe add some examples to description.

src/components/QueryEditor/QueryBuilder/Editors/FieldsEditorWithPrefix.tsx

arturminchukov · 2025-09-29T14:42:58Z

src/components/QueryEditor/QueryBuilder/Editors/LogicalFilterEditor.tsx

+  if (prevFieldsName !== fieldName) { // change all defaultFields in the visQuery to the new fieldName
+    const oldVisQueryOps = visQuery.operations.map((v) => ({ ...v, disabled: false })); // render all operations even when disabled
+    const oldQueryModeller = createQueryModellerWithDefaultField(prevFieldsName, [VictoriaLogsQueryOperationCategory.Filters, VictoriaLogsQueryOperationCategory.Operators]);
+    const oldExpr = oldQueryModeller.renderQuery({ operations: oldVisQueryOps }); // old so that old defaultField doesn't get rendered
+    const newVisQuery = parseExprToVisualQuery(oldExpr, fieldName, queryModeller).query;
+    for (let i = 0; i < visQuery.operations.length; i++) {
+      const op = visQuery.operations[i];
+      if (op.disabled) {
+        newVisQuery.operations[i].disabled = op.disabled; // keep disabled operations disabled
+      }
+    }
+    onChange(index, { expr, visQuery: newVisQuery, fieldName } as unknown as string);
+  };


Looks like we should wrap it with useEffect() hook here

The entire if block only runs if the field name of the logic operation gets changed. And the useEffect would need all the dependencies of that block.

arturminchukov · 2025-09-29T15:10:45Z

src/components/QueryEditor/QueryBuilder/Operations.tsx

Should I just create one helper file or split it up even more?

I prefer group util functions into files by their functionality, because I think it improves readability.

arturminchukov · 2025-09-30T08:36:15Z

I get the same problem and traced it to the request select/logsql/stream_field_names just returning Hello from VM data source!

Yeah, you're right! Right now VL datasource doesn't have such endpoints. Created the issue to add it. So could you wait until I add these endpoints and after it check if stream operation is working?

archef2000 · 2025-09-30T09:23:54Z

@arturminchukov It did exist and the filter did work.

arturminchukov · 2025-09-30T10:32:14Z

Yeah, you're right! Right now VL datasource doesn't have such endpoints. Created the issue to add it. So could you wait until I add these endpoints and after it check if stream operation is working?

Looks like just need to add

mux.HandleFunc("/select/logsql/streams", ds.VLAPIQuery)
mux.HandleFunc("/select/logsql/stream_field_names", ds.VLAPIQuery)
mux.HandleFunc("/select/logsql/stream_field_values", ds.VLAPIQuery)

to here. Could you add this code to your pr?

arturminchukov · 2025-10-01T10:20:34Z

Yeah, you're right! Right now VL datasource doesn't have such endpoints. Created the issue to add it. So could you wait until I add these endpoints and after it check if stream operation is working?

Looks like just need to add
mux.HandleFunc("/select/logsql/streams", ds.VLAPIQuery)
mux.HandleFunc("/select/logsql/stream_field_names", ds.VLAPIQuery)
mux.HandleFunc("/select/logsql/stream_field_values", ds.VLAPIQuery)
to here. Could you add this code to your pr?

UPD. @archef2000 , I merged pr with these changes. Could you just make rebase on updated main branch?

arturminchukov · 2025-10-06T13:49:26Z

/build

github-actions · 2025-10-06T13:49:49Z

## 🚀 Build Started

**PR:** #329
**Source:** Fork (`archef2000/victorialogs-datasource`)
**Triggered by:** @arturminchukov

[⏳ View build progress](https://github.com/VictoriaMetrics/victorialogs-datasource/actions/runs/18283109893)

github-actions · 2025-10-06T13:56:30Z

✅ Preview Build Completed!

Version: v0.21.0-pr329-20251006-135624-55f7fbe
PR: #329
Source: Fork (archef2000/victorialogs-datasource)
Commit: 55f7fbe9b0b28cadd15274035f8d74d45d69ab24
Triggered by: @arturminchukov

📦 Build Artifacts

📦 victoriametrics-logs-datasource-heads-main-0-g55f7fbe.zip (67.42 MB)

📥 Download

⬇️ Download from Actions Run

Artifact name: victoriametrics-logs-datasource-pr329-20251006-135624

🔐 Attestation

Build provenance attestation: View details

🧪 How to test this build

Download the artifact from the Actions run link above
Unzip the downloaded file

Copy the plugin to your Grafana plugins directory:

# Local Grafana
unzip victoriametrics-logs-datasource-*.zip
cp -r victoriametrics-logs-datasource /var/lib/grafana/plugins/

# Docker Grafana
docker cp victoriametrics-logs-datasource grafana:/var/lib/grafana/plugins/

Restart Grafana:

# Systemd
sudo systemctl restart grafana-server

# Docker
docker restart grafana

Verify the plugin in Grafana: Configuration → Plugins

📋 Build Details

Workflow: PR Build on Command
Branch: main
Is Fork: ✅ Yes
Retention: 14 days

⚠️ Note: This is a preview build for testing only. Artifacts will be automatically deleted after 14 days.

arturminchukov · 2025-10-06T17:04:54Z

/build

github-actions · 2025-10-06T17:05:17Z

## 🚀 Build Started

**PR:** #329
**Source:** Fork (`archef2000/victorialogs-datasource`)
**Triggered by:** @arturminchukov

[⏳ View build progress](https://github.com/VictoriaMetrics/victorialogs-datasource/actions/runs/18288559359)

github-actions · 2025-10-06T17:11:48Z

✅ Preview Build Completed!

Version: v0.21.0-pr329-20251006-171143-55f7fbe
PR: #329
Source: Fork (archef2000/victorialogs-datasource)
Commit: 55f7fbe9b0b28cadd15274035f8d74d45d69ab24
Triggered by: @arturminchukov

📦 Build Artifacts

📦 victoriametrics-logs-datasource-heads-main-0-g55f7fbe.tar.gz (67.43 MB) 🔐 Signed ✓ SHA1
📦 victoriametrics-logs-datasource-heads-main-0-g55f7fbe.zip (67.42 MB) 🔐 Signed ✓ SHA1

📥 Download

⬇️ Download from Actions Run

Artifact name: victoriametrics-logs-datasource-pr329-20251006-171143

🔐 Attestation

Build provenance attestation: View details
🧪 How to test this build
1. Download the artifact from the Actions run link above
2. Unzip the downloaded file
3. Copy the plugin to your Grafana plugins directory:
```
# Local Grafana
unzip victoriametrics-logs-datasource-*.zip
cp -r victoriametrics-logs-datasource /var/lib/grafana/plugins/

# Docker Grafana
docker cp victoriametrics-logs-datasource grafana:/var/lib/grafana/plugins/
```
4. Restart Grafana:
```
# Systemd
sudo systemctl restart grafana-server

# Docker
docker restart grafana
```
5. Verify the plugin in Grafana: Configuration → Plugins
📋 Build Details
- Workflow: PR Build on Command
- Branch: main
- Is Fork: ✅ Yes
- Retention: 14 days
⚠️ Note: This is a preview build for testing only. Artifacts will be automatically deleted after 14 days.

AndrewChubatiuk · 2025-10-07T06:41:26Z

for stats functions stats word is not prepended to result query, is it a bug?

archef2000 · 2025-10-07T06:53:07Z

The stats word is only prepended when the stats pipe is used.

AndrewChubatiuk · 2025-10-08T09:29:24Z

can stats pipe be excluded on a stage, when no filters were selected yet?
i mean in general query builder should help to build query for those who is not familiar with logsql, but it confuses instead by allowing to select building blocks, which produce query that 100% fails

archef2000 · 2025-10-08T10:21:57Z

No as the OperationList component provided by grafana/ui just asks the queryModeller for all operation categories without any context.

szibis · 2025-10-16T10:32:45Z

Quick question how far we are with releasing this version of datasource officially?

arturminchukov · 2025-10-16T10:39:20Z

Quick question how far we are with releasing this version of datasource officially?

@szibis Right now I don't have the exact ETA for releasing this feature. If I have any updates, I will let you know.

AndrewChubatiuk · 2025-10-16T10:48:16Z

@szibis
in a current state query builder doesn't simplify procedure of building logsql expressions.
we've evaluated it internally and despite logsql syntax knowledge some people failed to build successful query during the first attempt to use it and only reverse conversion from raw expression to builder mode helped to understand why it failed. In my opinion builder should not allow to select options that produce absolutely invalid query. For example here value query.expr should impact a subset of categories that is passed to QueryModeller, which are allowed to choose on the next step (don't allow to select pipes, when filter is not set, don't allow setting filter, when pipes are set, etc)

@szibis Have you tried it already?

archef2000 · 2025-10-16T10:59:10Z

The QueryModeller is pretty heavy to create and it would need more than just this change as there are many places where a full QueryModeller is used. Then there is the problem that nearly all operations that are placed result in a query that fails. You first need to fill it with information so that also requires knowledge. It also means limiting advanced users that might create a query out of order. And what happens when I drag a pipe to the start? The error code of vl-select is very informative and should be very easy to understand. There is always a learning curve of a new tool and after the first failed query it should be clear what is allowed.

AndrewChubatiuk · 2025-10-16T12:58:10Z

understand your concerns since it's a huge feature and it took significant time to implement, but i cannot agree with your statement regarding "learning curve". Builder is an intuitive tool, which should require no learning at all. by the way have you checked loki builder? it has required non-draggable filter, reusing this approach in current PR could improve UX a lot

archef2000 added 2 commits July 4, 2025 13:38

Implement Grafana Query Builder

c2a6cdb

delete unused files

201f871

Loori-R requested review from Loori-R and arturminchukov July 4, 2025 16:46

Loori-R requested changes Jul 7, 2025

View reviewed changes

archef2000 added 3 commits July 10, 2025 08:29

Make Editors compatible with lower versions

66b3d5b

Add error handling for unsupported field types in LogsQlLanguageProvider

ffac882

fix tests

b96a69c

Merge branch 'main' into main

3d9de16

fix options generation for subqueries

3270538

fix logic error in subquery editor

55d2517

arturminchukov reviewed Sep 30, 2025

View reviewed changes

archef2000 added 3 commits October 1, 2025 14:05

Merge branch 'VictoriaMetrics:main' into main

7f3d219

fix stream field parsing and field prediction

cc26956

remove else

55f7fbe


		export type SplitString = SplitStringValue \| SplitStringBracket \| SplitStringComment;

		export const splitString = (str: string): SplitString[] => {

Implement Grafana Query Builder #329

Are you sure you want to change the base?

Implement Grafana Query Builder #329

Uh oh!

Conversation

archef2000 commented Jul 4, 2025

Uh oh!

Loori-R left a comment

Choose a reason for hiding this comment

Uh oh!

archef2000 commented Jul 7, 2025

Uh oh!

archef2000 commented Jul 7, 2025

Uh oh!

Loori-R commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Loori-R commented Jul 8, 2025

Uh oh!

dk commented Jul 8, 2025

Uh oh!

Loori-R commented Jul 8, 2025

Uh oh!

dmitryk-dk commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Loori-R commented Jul 8, 2025

Uh oh!

dmitryk-dk commented Jul 8, 2025

Uh oh!

archef2000 commented Jul 8, 2025

Uh oh!

Loori-R commented Jul 8, 2025

Uh oh!

archef2000 commented Jul 8, 2025

Uh oh!

Loori-R commented Jul 8, 2025

Uh oh!

archef2000 commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

archef2000 commented Jul 8, 2025

Uh oh!

Loori-R commented Jul 9, 2025

Uh oh!

Loori-R commented Jul 9, 2025

Uh oh!

dmitryk-dk commented Jul 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

archef2000 commented Jul 9, 2025

Uh oh!

Loori-R commented Jul 9, 2025

Uh oh!

archef2000 commented Jul 9, 2025

Uh oh!

Loori-R commented Jul 9, 2025

Uh oh!

archef2000 commented Sep 26, 2025

Uh oh!

arturminchukov commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arturminchukov commented Sep 29, 2025

Uh oh!

archef2000 commented Sep 29, 2025

Uh oh!

arturminchukov Sep 29, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

arturminchukov Sep 29, 2025

Choose a reason for hiding this comment

Uh oh!

archef2000 Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arturminchukov Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Loori-R commented Jul 8, 2025 •

edited

Loading

dmitryk-dk commented Jul 8, 2025 •

edited

Loading

archef2000 commented Jul 8, 2025 •

edited

Loading

dmitryk-dk commented Jul 9, 2025 •

edited

Loading

arturminchukov commented Sep 29, 2025 •

edited

Loading

archef2000 Oct 1, 2025 •

edited

Loading

arturminchukov Sep 29, 2025 •

edited

Loading

arturminchukov commented Oct 1, 2025 •

edited

Loading

AndrewChubatiuk commented Oct 7, 2025 •

edited

Loading

archef2000 commented Oct 8, 2025 •

edited

Loading

AndrewChubatiuk commented Oct 16, 2025 •

edited

Loading