Improved patterns command with new algorithm #3263

songkant-aws · 2025-01-24T14:49:46Z

Description

This PR introduces enhancement on original patterns command. It keeps the patterns command naming and rebuild it with specific window functions to parse log messages with different pattern method(log parser algorithms). See this RFC: #3251

The sample query input and output will be like:

New brain algorithm log parser:

Original regex based patterns command will have minor change:

Caveat:
For now, this is a draft PR for initial review so that we can collect major feedbacks to be addressed later.

Related Issues

Resolves #3251

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Songkan Tang <[email protected]>

YANG-DB · 2025-01-24T20:07:24Z

Thanks for the initiative!!

penghuo

In general,

add user facing doc to explain new BRAIN alg with example will be helpfull.
add UT could help reviwer understand the logic also. @dai-chen did a good example. https://github.com/opensearch-project/sql/blob/main/core/src/main/java/org/opensearch/sql/expression/window/frame/PeerRowsWindowFrame.java
add benchmark to compare BRAIN and SIMPLE also help a lot.

penghuo · 2025-01-25T01:06:41Z

ppl/src/main/java/org/opensearch/sql/ppl/parser/AstBuilder.java

+                new Function(
+                    ctx.pattern_method != null
+                        ? ctx.pattern_method.getText().toLowerCase(Locale.ROOT)
+                        : BuiltinFunctionName.BRAIN.name(), // By default, use new algorithm


Could we introduce a setting to control the default pattern algorithms?

Ok, I see. Like a java property to specify default algorithm name? I can add that.

penghuo · 2025-01-25T01:35:19Z

...src/main/java/org/opensearch/sql/expression/window/patterns/StreamPatternWindowFunction.java

+                    ((NamedArgumentExpression) expression).getValue().valueOf().stringValue())
+            .findFirst()
+            .orElse("");
+    return new StreamPatternRowWindowFrame(


do we need StreamPatternRowWindowFrame? could we just use CurrentRowWindowFrame instead? patternExpression could be member of StreamPatternWindowFunction?

Sure, I will remove the need of StreamPatternRowWindowFrame in the next revision.

penghuo · 2025-01-25T01:50:54Z

core/src/main/java/org/opensearch/sql/expression/window/frame/PeerRowsWindowFrame.java

    List<Expression> sortFields =
        windowDefinition.getSortList().stream().map(Pair::getRight).collect(Collectors.toList());

    ExprValue last = peers.get(peers.size() - 1);
    return resolve(sortFields, last).equals(resolve(sortFields, next));
  }

-  private boolean isSamePartition(ExprValue next) {
+  protected boolean isSamePartition(ExprValue next) {


revert unnecessary change.

penghuo · 2025-01-25T01:54:52Z

core/src/main/java/org/opensearch/sql/expression/window/frame/BufferPatternRowsWindowFrame.java

+                  return value.stringValue();
+                })
+            .toList();
+    this.preprocessedMessages.addAll(logParser.preprocessAllLogs(logMessages));


BufferPatternRowsWindowFrame should have it own spec, doet it means over all rows? @dai-chen WindowFrame definition and How to use WIndowFrame should be seperate, right?

For simplicity, I set the spec to empty partition by and empty sort by during AST tree parsing unresolved WindowFunction, which treats the window frame range is all rows on coordinator node. Because I haven't seen requirements on sorting and partitioning on other columns.

I think we can add partition by and sort by syntax if we see values in case users want to specify them. Thoughts?

Yes, I think users are free to use pattern functions added with any window frame definition.

Signed-off-by: Songkan Tang <[email protected]>

dai-chen · 2025-01-27T22:10:50Z

core/src/main/java/org/opensearch/sql/expression/window/frame/BufferPatternRowsWindowFrame.java

+                  return value.stringValue();
+                })
+            .toList();
+    this.preprocessedMessages.addAll(logParser.preprocessAllLogs(logMessages));


Yes, I think users are free to use pattern functions added with any window frame definition.

dai-chen · 2025-01-27T23:18:54Z

core/src/main/java/org/opensearch/sql/expression/window/WindowFunctions.java

+    repository.register(brain());
+    repository.register(simplePattern());


I'm thinking are these new algorithm really window function? If users specify order by or partition by, does it still generate meaningful result?

dai-chen · 2025-01-27T23:20:25Z

core/src/main/java/org/opensearch/sql/ast/dsl/AstDSL.java

+      UnresolvedExpression sourceField,
+      String alias,
+      java.util.Map<String, Literal> arguments) {
+    return new Pattern(


Just wonder if pattern is mostly syntax sugar for pattern window function, is new logical operator still required?

After the integration with LogicalWindow, I see they are quite similar. Yeah, I think Pattern operator is probably not needed.

I don't find the Window unresolvedPlan. So replace Pattern with Window instead.

dai-chen · 2025-01-27T23:42:32Z

...src/main/java/org/opensearch/sql/expression/window/patterns/BufferPatternWindowFunction.java

+import org.opensearch.sql.expression.window.frame.WindowFrame;
+
+@EqualsAndHashCode(callSuper = true)
+public class BufferPatternWindowFunction extends FunctionExpression


order by expression in window definition decides who're peers right? What's the order by expression for buffer (offline) pattern function? I didn't find it and may miss it in design doc:

Project[message#1, patterns_field#2] +- Window[brain(message#1), windowsSpec(partitionBy=null, frame=PeerRowsWindowFrame)] +- OpenSearchIndexScan

dai-chen

A high-level question about the long-term approach: Currently, the pattern command always generates a new field, which offers flexibility as users can perform aggregation (stats) or other operations on it afterward. However, for large datasets, I assume most users would prefer functionality similar to CloudWatch Logs Query Syntax - Pattern.

Do you see any potential issues with the current implementation, which is based on SQL window functions combined with aggregate functions, as we scale this feature in the future?

songkant-aws · 2025-01-28T01:57:34Z

Do you see any potential issues with the current implementation, which is based on SQL window functions combined with aggregate functions, as we scale this feature in the future?

@dai-chen It's a good question. Actually, it's one of my previous thought to use a new specific operator to directly return grouped logs with sample count or grouped samples.

It's more like a compound aggregation operator with a default group by. But current aggregation abstraction only supports row by row iteration. The log pattern algorithms cover both streaming and buffering computing paradigm. Then I figure out this is more close to window operator's abstraction. To achieve what Cloudwatch has done, I think we could extend pattern functions from AggregateWindowFunction that has a special aggregator state structure.

As to partition by and sort by specification, my initial plan is to deliver a simple version without them because that's probably enough.
partition by is probably more useful than sort by. Imagine users may want to see different log patterns for INFO, WARN, ERROR partitions.
I haven't seen a use case for sort by yet. Maybe they need to see latest logs sort by date?

songkant-aws · 2025-01-28T02:10:17Z

To scale the pattern function, I think other issues are pending investigation.

How to push down operator to OpenSearch data node? Is Painless script one of option?
OpenSearch doesn't have shuffle mechanism for now, how to perform large dataset aggregation like operation with simple map reduce? If data volume is quite large, spilling additional data to disk is probably required during execution.
To log pattern, streaming algorithms are probably more suitable to large scale dataset. We may need to introduce more by then.

Signed-off-by: Songkan Tang <[email protected]>

songkant-aws · 2025-01-29T09:30:16Z

AvgTime benchmark:

Benchmark                                          (testDataType)  Mode  Cnt  Score    Error  Units
ComparisonOperatorBenchmark.testEqualOperator                 int  avgt    3  0.002 ±  0.001  ms/op
ComparisonOperatorBenchmark.testEqualOperator              string  avgt    3  0.005 ±  0.001  ms/op
ComparisonOperatorBenchmark.testEqualOperator                date  avgt    3  0.001 ±  0.001  ms/op
ComparisonOperatorBenchmark.testGreaterOperator               int  avgt    3  0.002 ±  0.001  ms/op
ComparisonOperatorBenchmark.testGreaterOperator            string  avgt    3  0.005 ±  0.002  ms/op
ComparisonOperatorBenchmark.testGreaterOperator              date  avgt    3  0.001 ±  0.001  ms/op
ComparisonOperatorBenchmark.testLessOperator                  int  avgt    3  0.002 ±  0.001  ms/op
ComparisonOperatorBenchmark.testLessOperator               string  avgt    3  0.006 ±  0.001  ms/op
ComparisonOperatorBenchmark.testLessOperator                 date  avgt    3  0.002 ±  0.001  ms/op
PatternsWindowFunctionBenchmark.testBrain                     N/A  avgt    3  0.131 ±  0.002  ms/op
PatternsWindowFunctionBenchmark.testSimplePattern             N/A  avgt    3  0.001 ±  0.001  ms/op

Signed-off-by: Songkan Tang <[email protected]>

songkant-aws · 2025-01-29T09:35:30Z

This PR still needs some action items.

Pending adding setting to switch default log pattern algorithm
Pending discussion whether we want to deliver partition by and order by given its effect. (Log patterns are probably special window function that is not sensitive to partition by and order by)

Improved patterns command with new algorithm

9d93991

Signed-off-by: Songkan Tang <[email protected]>

songkant-aws requested review from ps48, kavithacm, derek-ho, joshuali925, dai-chen, YANG-DB, mengweieric, Swiddis, penghuo, seankao-az, MaxKsyunz, Yury-Fridlyand, anirudha, forestmvey, acarbonetto, GumpacG, ykmr1224, LantaoJin and noCharger as code owners January 24, 2025 14:49

penghuo reviewed Jan 25, 2025

View reviewed changes

penghuo added the enhancement New feature or request label Jan 25, 2025

songkant-aws added 2 commits January 26, 2025 16:29

Minor change log parser default configurations

6874a5f

Signed-off-by: Songkan Tang <[email protected]>

Merge branch 'main' into log-pattern-command

dd9373f

dai-chen reviewed Jan 27, 2025

View reviewed changes

songkant-aws added 3 commits January 28, 2025 16:12

Refactor a bit and add partial unit tests

c62b25b

Signed-off-by: Songkan Tang <[email protected]>

Add more unit tests

9c6327a

Signed-off-by: Songkan Tang <[email protected]>

Amend patterns user facing doc

bda53df

Signed-off-by: Songkan Tang <[email protected]>

Add average time benchmark for patterns window functions

786dcee

Signed-off-by: Songkan Tang <[email protected]>

songkant-aws changed the title ~~[DRAFT] Improved patterns command with new algorithm~~ Improved patterns command with new algorithm Jan 29, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improved patterns command with new algorithm #3263

Improved patterns command with new algorithm #3263

songkant-aws commented Jan 24, 2025 •

edited

Loading

YANG-DB commented Jan 24, 2025

penghuo left a comment

penghuo Jan 25, 2025

songkant-aws Jan 27, 2025 •

edited

Loading

penghuo Jan 25, 2025

songkant-aws Jan 27, 2025

penghuo Jan 25, 2025

penghuo Jan 25, 2025

songkant-aws Jan 27, 2025

dai-chen Jan 27, 2025

dai-chen Jan 27, 2025

dai-chen Jan 27, 2025 •

edited

Loading

dai-chen Jan 27, 2025

songkant-aws Jan 28, 2025

songkant-aws Jan 29, 2025

dai-chen Jan 27, 2025 •

edited

Loading

dai-chen left a comment •

edited

Loading

songkant-aws commented Jan 28, 2025 •

edited

Loading

songkant-aws commented Jan 28, 2025 •

edited

Loading

songkant-aws commented Jan 29, 2025

songkant-aws commented Jan 29, 2025

		repository.register(brain());
		repository.register(simplePattern());

Improved patterns command with new algorithm #3263

Are you sure you want to change the base?

Improved patterns command with new algorithm #3263

Conversation

songkant-aws commented Jan 24, 2025 • edited Loading

Description

Related Issues

Check List

YANG-DB commented Jan 24, 2025

penghuo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

songkant-aws Jan 27, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dai-chen Jan 27, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dai-chen Jan 27, 2025 • edited Loading

Choose a reason for hiding this comment

dai-chen left a comment • edited Loading

Choose a reason for hiding this comment

songkant-aws commented Jan 28, 2025 • edited Loading

songkant-aws commented Jan 28, 2025 • edited Loading

songkant-aws commented Jan 29, 2025

songkant-aws commented Jan 29, 2025

songkant-aws commented Jan 24, 2025 •

edited

Loading

songkant-aws Jan 27, 2025 •

edited

Loading

dai-chen Jan 27, 2025 •

edited

Loading

dai-chen Jan 27, 2025 •

edited

Loading

dai-chen left a comment •

edited

Loading

songkant-aws commented Jan 28, 2025 •

edited

Loading

songkant-aws commented Jan 28, 2025 •

edited

Loading