strict_mode top-level config option #166
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jem-davies
force-pushed
the
strict_mode_1
branch
from
fa9efbb
to
a41b78f
Compare
Signed-off-by: Jem Davies <[email protected]>
jem-davies
force-pushed
the
strict_mode_1
branch
from
a41b78f
to
b058924
Compare
jem-davies
changed the title
Signed-off-by: Jem Davies <[email protected]>
gregfurman
reviewed
Nov 17, 2024
public/service/stream_builder.go
Outdated
| @@ -89,6 +90,7 @@ func NewStreamBuilder() *StreamBuilder { | |||
| configSpec: tmpSpec, | |||
| env: globalEnvironment, | |||
| envVarLookupFn: os.LookupEnv, | |||
| strictMode: false, | |||
There was a problem hiding this comment.
this'll be false by default so unsure this is necessary. See how lintingDisabled is never set.
website/docs/configuration/about.md
Outdated
|
|
||
| Introduced in v1.4.0. | ||
|
|
||
| The default behavior of Bento is to attempt to send messages that have errored to the configured sink. The `strict_mode` option can be set to `true` to override this default behavior of Bento, and instead will reject any message batches with messages that have errors in them. Ultimately this will propagate a Nack to the input layer, which then depending on the component configured will either be handled or the message will be reprocessed from scratch. |
There was a problem hiding this comment.
Suggested change
| The default behavior of Bento is to attempt to send messages that have errored to the configured sink. The `strict_mode` option can be set to `true` to override this default behavior of Bento, and instead will reject any message batches with messages that have errors in them. Ultimately this will propagate a Nack to the input layer, which then depending on the component configured will either be handled or the message will be reprocessed from scratch. | |
| When `strict_mode` is set to `true`, Bento will reject all batches containing messages with errors, propagating a `nack` to the input layer. The input component's `nack` behaviour will then determine how these rejected messages are handled, otherwise re-processing them from scratch. |
There was a problem hiding this comment.
Great explanation. Think we simplify a bit though
There was a problem hiding this comment.
Updated - think that we should keep the first sentence for context - rewrote the other sentences - with the suggestion in mind.
Comment on lines
270
to
286
| func (w *AsyncWriter) getStrictMode() bool { | ||
| v := reflect.ValueOf(w.mgr) | ||
|
|
||
| if v.Kind() == reflect.Pointer && !v.IsNil() { | ||
| v = v.Elem() | ||
| } | ||
|
|
||
| if v.Kind() == reflect.Struct { | ||
| field := v.FieldByName("StrictMode") | ||
| if field.IsValid() && field.Kind() == reflect.Bool { | ||
| return field.Bool() | ||
| } | ||
| } | ||
|
|
||
| return false | ||
| } | ||
|
|
There was a problem hiding this comment.
How come we are using reflect here?
| return nil | ||
| }) | ||
| if hasErrors { | ||
| w.log.Error("Failed to send message to %v: message batch contains errors\n", w.typeStr) |
There was a problem hiding this comment.
Not sure the \n is necessary
Suggested change
| w.log.Error("Failed to send message to %v: message batch contains errors\n", w.typeStr) | |
| w.log.Error("Failed to send message to %v: message batch contains errors", w.typeStr) |
There was a problem hiding this comment.
The other calls in this func to w.log.Error - do have a \n.
internal/config/schema.go
Outdated
| @@ -59,6 +61,7 @@ func observabilityFields() docs.FieldSpecs { | |||
| }), | |||
| docs.FieldString(fieldSystemCloseDelay, "A period of time to wait for metrics and traces to be pulled or pushed from the process.").HasDefault("0s"), | |||
| docs.FieldString(fieldSystemCloseTimeout, "The maximum period of time to wait for a clean shutdown. If this time is exceeded Bento will forcefully close.").HasDefault("20s"), | |||
| docs.FieldBool(fieldStrictMode, "Set strict_mode to true to override the default behavior of sending messages batches that have errored messages to the configured sink, and instead propagate a nack back to the source.").HasDefault(false).Advanced().AtVersion("1.4.0"), | |||
There was a problem hiding this comment.
Suggested change
| docs.FieldBool(fieldStrictMode, "Set strict_mode to true to override the default behavior of sending messages batches that have errored messages to the configured sink, and instead propagate a nack back to the source.").HasDefault(false).Advanced().AtVersion("1.4.0"), | |
| docs.FieldBool(fieldStrictMode, "Set `strict_mode` to `true` to override the default behavior of sending messages batches containing errored messages to the `output` layer, instead rejecting the batch and propagating a `nack` back to the source.").HasDefault(false).Advanced().AtVersion("1.4.0"), |
Signed-off-by: Jem Davies <[email protected]>
Signed-off-by: Jem Davies <[email protected]>
Signed-off-by: Jem Davies <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.