Add log type include filter by jorbaum · Pull Request #714 · cloudfoundry/loggregator-agent-release

jorbaum · 2026-01-05T14:06:49Z

Description

Addresses #711 :

Support both ?include-source-types= and ?exclude-source-types (calling it source type instead of log type as log type is too generic and is already used in this code base)
Throw error in case both are configured. Do not throw an error on misconfigured values for the new config option (consistent with previous HTTP query param usage in binding config), but log a warning (new).
Only exclude filter shall pass unknown/nil source type, include filter will discard

I still need to verify this, but I think a feature flag is not necessary as I tried to make string parsing more efficient by:

running only when source type tag and log type filter is present
strings.IndexByte might take some time, but it is used only with a single char / and strings.IndexByte() should be more efficient than strings.Index()
Looking up content in a map logTypePrefixes rather than in each log line, which should be more efficient

Related PRs:

Disclaimer: Claude helped me in crafting this code.

Type of change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Testing performed?

Unit tests
Integration tests
Acceptance tests

Checklist:

This PR is being made against the main branch, or relevant version branch
I have made corresponding changes to the documentation
I have added testing for my changes

chombium

@jorbaum Thanks for the changes.

Generally it looks good, but I have some smaller objections which should be addressed.

src/pkg/egress/syslog/filtering_drain_writer.go

src/pkg/egress/syslog/filtering_drain_writer_test.go

src/pkg/egress/syslog/syslog_connector.go

src/pkg/ingress/bindings/binding_config.go

src/pkg/ingress/bindings/binding_config_test.go

chombium · 2026-01-09T17:05:15Z

@ZPascal Wdyt about this PR? It addresses #711

jorbaum · 2026-01-12T08:04:26Z

@chombium I left a bunch of replies for points that are not yet clear to me. All the rest look good.

Thanks for the thorough review! You found a bunch of nice things.

jorbaum

Worked in some comments.

src/pkg/egress/syslog/filtering_drain_writer.go

src/pkg/ingress/bindings/binding_config.go

jorbaum

Worked in most comments. Still 2 left to clarify.

src/pkg/ingress/bindings/binding_config.go

src/pkg/ingress/bindings/binding_config_test.go

jorbaum · 2026-01-19T15:30:27Z

Feel free to take another look at it.

src/pkg/egress/syslog/log.go

chombium

Hi @jorbaum. Thanks for the improvements. Unfortunately, I've found few more problematic things.

src/pkg/egress/syslog/log_source.go

src/pkg/egress/syslog/source.go

src/pkg/egress/syslog/filtering_drain_writer_test.go

src/pkg/egress/syslog/source.go

src/pkg/ingress/bindings/binding_config.go

src/pkg/ingress/bindings/binding_config_test.go

jorbaum · 2026-02-18T14:45:29Z

Thanks for the review. I replied to a bunch of comments for some clarifying remarks. Also a quick discussion about uppercase/lowercase config values would help me.

I noticed that as my changes depend on `RawQuery` not being empty, but actually they were.

Also rename remaining instances of log type to source type

jorbaum

Addressed hopefully all comments.

I got a bunch of things to clarify. Biggest question that is left is likely if we should recommend users uppercase or lowercase values.

src/pkg/egress/syslog/filtering_drain_writer_test.go

src/pkg/egress/syslog/source.go

src/pkg/ingress/bindings/binding_config.go

src/pkg/ingress/bindings/binding_config_test.go

src/pkg/egress/syslog/log_source.go

src/pkg/egress/syslog/source.go

chombium

@jorbaum It looks much better now and we're close to mergnig.
Please fix the small things which I've commented on.

I will have to test everything before merging.

src/pkg/egress/syslog/filtering_drain_writer_test.go

chombium · 2026-03-16T14:01:40Z

src/pkg/egress/syslog/filtering_drain_writer_test.go

+			}
+		})
+
+		It("filters out the log with missing source_type when include filter is configured", func() {


The filter works based on the values of the source_type tag and here the code defines the behavior of what should happen.

Is this behavior documented somewhere?

No it is not. The best place for that is likely https://docs.cloudfoundry.org/devguide/services/log-management.html ?

Yes, that's the place. Please open a PR. Btw, why is the description of the parameters already in the docs?

Yes, that's the place. Please open a PR.

Opened up a new PR at cloudfoundry/docs-dev-guide#584

Btw, why is the description of the parameters already in the docs?

The previous PR has been merged a bit sooner than expected.

src/pkg/egress/syslog/filtering_drain_writer_test.go

jorbaum

Should all be addressed now. CI is failing due to new golangci-lint changes. Those have been addressed in the two newest commits of #717 .

jorbaum · 2026-03-17T16:37:18Z

src/pkg/egress/syslog/filtering_drain_writer_test.go

+			}
+		})
+
+		It("filters out the log with missing source_type when include filter is configured", func() {


Yes, that's the place. Please open a PR.

Opened up a new PR at cloudfoundry/docs-dev-guide#584

Btw, why is the description of the parameters already in the docs?

The previous PR has been merged a bit sooner than expected.

cf-foundation-community-automation bot added this to Application Runtime Platform Working Group Jan 5, 2026

cf-foundation-community-automation bot moved this to Inbox in Application Runtime Platform Working Group Jan 5, 2026

jorbaum force-pushed the add-log-type-filter branch 2 times, most recently from 9fcc372 to bccd92c Compare January 8, 2026 09:43

chombium requested changes Jan 9, 2026

View reviewed changes

jorbaum commented Jan 12, 2026

View reviewed changes

src/pkg/egress/syslog/filtering_drain_writer.go Outdated Show resolved Hide resolved

src/pkg/ingress/bindings/binding_config.go Outdated Show resolved Hide resolved

src/pkg/ingress/bindings/binding_config.go Outdated Show resolved Hide resolved

jorbaum commented Jan 13, 2026

View reviewed changes

src/pkg/ingress/bindings/binding_config.go Outdated Show resolved Hide resolved

src/pkg/ingress/bindings/binding_config_test.go Outdated Show resolved Hide resolved

jorbaum commented Feb 4, 2026

View reviewed changes

src/pkg/egress/syslog/log.go Outdated Show resolved Hide resolved

jorbaum force-pushed the add-log-type-filter branch 3 times, most recently from 108740d to ef16bed Compare February 11, 2026 14:33

jorbaum marked this pull request as ready for review February 16, 2026 10:18

jorbaum requested a review from a team as a code owner February 16, 2026 10:18

This was referenced Feb 16, 2026

Document new log-source-type syslog drain config cloudfoundry/docs-dev-guide#576

Merged

Document more source types cloudfoundry/docs-dev-guide#577

Draft

chombium requested changes Feb 17, 2026

View reviewed changes

jorbaum added 10 commits February 20, 2026 11:30

Add log type include filter

a5b1ad3

Support exclude and include

a2088ec

Error when using both include and exclude types

a17e86b

Introduce logging to binding config

8da9a70

Add some more unit tests

7fc1b84

Fix wrong word

b961719

Create LogType as string, move to package

110fb27

Prefix log type constants with LOG_

1820deb

Rename value to sourceType

3447483

Only emit one warning on wrong drain config

1e96b21

jorbaum added 17 commits February 20, 2026 11:30

Rename DrainParameterParser in test

bb0368a

Add tests for the include and exclude filters

2dfc90f

Move new log filter test to table test

95ff14f

Move next test to table

6cda574

Move drain data for old parameter test to table

d0ae8a3

Move log filter validation to binding_fetcher

db8afe4

Dereference pointer to make an actual copy

31b2c90

I noticed that as my changes depend on `RawQuery` not being empty, but actually they were.

Rename log type to source type where possible

91407bd

Only exclude filter shall pass unknown source type

1f98d37

Also rename remaining instances of log type to source type

Add SourceType PROXY, HEALTH, SYS and STATS

e9f1e1b

Make missing source_type tests clearer

a3eda41

Add tests for no filtering value configured

31b219b

Rename source to log source where sensible

d7e830c

Remove unused method

0775898

Add -log to uri query param

3ba2a27

No longer accept URI drains with space in values

29c6830

Move log source type parsing into common function

ecab32a

jorbaum commented Feb 20, 2026

View reviewed changes

jorbaum force-pushed the add-log-type-filter branch from ef16bed to ecab32a Compare February 20, 2026 13:38

chombium requested changes Mar 16, 2026

View reviewed changes

Group tests where source_type is present

a312cd4

jorbaum mentioned this pull request Mar 17, 2026

Improve log type filter cloudfoundry/docs-dev-guide#584

Draft

jorbaum commented Mar 17, 2026

View reviewed changes

Conversation

jorbaum commented Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Testing performed?

Checklist:

Uh oh!

chombium left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chombium commented Jan 9, 2026

Uh oh!

jorbaum commented Jan 12, 2026

Uh oh!

jorbaum left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jorbaum left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jorbaum commented Jan 19, 2026

Uh oh!

Uh oh!

chombium left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jorbaum commented Feb 18, 2026

Uh oh!

jorbaum left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chombium left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chombium Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

jorbaum Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

chombium Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

jorbaum commented Jan 5, 2026 •

edited

Loading

chombium left a comment •

edited

Loading