Skip to content

Centralised pipelines don't honour the same configuration directives as static ones #16729

Open
@garethhumphriesgkc

Description

@garethhumphriesgkc

Logstash information:

Please include the following information:

  1. Logstash version: 8.15.3
  2. Logstash installation source: docker
  3. How is Logstash being run: docker

OS version (uname -a if on a Unix-like system):
ubuntu 22.04

Description of the problem including expected versus actual behaviour:
Raised off https://discuss.elastic.co/t/custom-pipeline-settings-with-centralised-pipeline-management/371001

When using the API to create centralised logstash pipelines, you can add all the settings documented, however the majority of these settings are not honoured when the pipeline runs:

[logstash.configmanagement.elasticsearchsource] Ignoring unsupported or unknown pipeline settings 'dead_letter_queue.enable'
[logstash.configmanagement.elasticsearchsource] Ignoring unsupported or unknown pipeline settings 'dead_letter_queue.retain.age'
[logstash.configmanagement.elasticsearchsource] Ignoring unsupported or unknown pipeline settings 'dead_letter_queue.flush_interval'
[logstash.configmanagement.elasticsearchsource] Ignoring unsupported or unknown pipeline settings 'dead_letter_queue.storage_policy'
[logstash.configmanagement.elasticsearchsource] Ignoring unsupported or unknown pipeline settings 'queue.drain'

This means that centralised pipelines can't be used to replace all but the most basic of logstash pipelines.

It's unclear why the settings wouldn't be honoured, but if not they should be rejected by the API, and the documentation should reflect this limitation.

For completeness the discuss article above also mentions:

  • If you subsequently edit them through Kibana, all those custom settings are lost.
  • The "Last Modified" time can't be set (and doesn't default to) "now" - you have to enter the entire timestamp each time.

Steps to reproduce:

Create a centralised pipeline with custom settings. Not that your setting don't apply.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions