failed to seek to position where writer left off

[sorrowcode]

A note for the community

vconf.zip

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

We have a working configuration were we use opentelemetry as a source and two aws_s3 as a sinks. Vector compiles it correctly but out of nowhere it starts to break with the following output:

2026-06-17T12:03:04.189735Z ERROR vector::topology::builder: Configuration error. error=Sink "backflush-bucket": error occurred when building buffer: failed to build individual stage 0: failed to seek to position where writer left off: failed to validate the last written record: failed to decoded record: InvalidProtobufPayload internal_log_rate_limit=false
2026-06-17T12:03:04.189758Z ERROR vector::topology::builder: Configuration error. error=Sink "unified-firewall": error occurred when building buffer: failed to build individual stage 0: failed to seek to position where reader left off: failed to decoded record: InvalidProtobufPayload internal_log_rate_limit=false
stream closed: EOF for telemetry-hubs/vector-1 (vector)

It already happened more than twice. We're running Vector in Kubernetes and are also using disk buffering via PVCs. The only resolution for now was killing Pods with PVCs which results in data loss so thats not really an option.

vconf.zip

Configuration

Version

0.55.0

Debug Output

Example Data

2026-06-17T12:03:03.373025Z  INFO vector::app: Log level is enabled. level="info"
2026-06-17T12:03:03.374957Z  INFO vector::app: Loading configs. paths=["/etc/vector/vector.yaml", "/etc/vector/dynamic"]
2026-06-17T12:03:03.390993Z  WARN vector::config::loading: Transform "route-config._unmatched" has no consumers
2026-06-17T12:03:03.391017Z  WARN vector::config::loading: Internal log [Transform "route-config._unmatched" has no consumers] is being suppressed to avoid flooding.
2026-06-17T12:03:03.391039Z  INFO vector::app: Starting watcher. paths=["/etc/vector/vector.yaml", "/etc/vector/dynamic"]
2026-06-17T12:03:03.391058Z  INFO vector::app: Components to watch. paths=[ComponentConfig { config_paths: [], component_key: ComponentKey { id: "attributor_logs" }, component_type: Transform }, ComponentConfig { config_paths: [], component_key: ComponentKey { id: "backflush-transform" }, component_type: Transform }, ComponentConfig { config_paths: [], component_key: ComponentKey { id: "route-config" }, component_type: Transform }, ComponentConfig { config_paths: [], component_key: ComponentKey { id: "seahash-config" }, component_type: Transform }, ComponentConfig { config_paths: [], component_key: ComponentKey { id: "prom_exporter" }, component_type: Sink }, ComponentConfig { config_paths: [], component_key: ComponentKey { id: "backflush-bucket" }, component_type: Sink }, ComponentConfig { config_paths: [], component_key: ComponentKey { id: "shard-7f80aec0-2322-4947-a9c4-8dfd4a33dff7" }, component_type: Sink }, ComponentConfig { config_paths: ], component_key: ComponentKey { id: "unified-firewall" }, component_type: Sink }]
2026-06-17T12:03:03.391069Z  INFO vector::config::watcher: Creating configuration file watcher.
2026-06-17T12:03:03.392867Z  INFO vector::config::watcher: Watching configuration files.
2026-06-17T12:03:04.189735Z ERROR vector::topology::builder: Configuration error. error=Sink "backflush-bucket": error occurred when building buffer: failed to build individual stage 0: failed to seek to position where writer left off: failed to validate the last written record: failed to decoded record: InvalidProtobufPayload internal_log_rate_limit=false
2026-06-17T12:03:04.189758Z ERROR vector::topology::builder: Configuration error. error=Sink "unified-firewall": error occurred when building buffer: failed to build individual stage 0: failed to seek to position where reader left off: failed to decoded record: InvalidProtobufPayload internal_log_rate_limit=false
stream closed: EOF for telemetry-hubs/vector-1 (vector)

Additional Context

No response

References

No response