Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Centralised pipelines don't honour the same configuration directives as static ones #16729

Open
garethhumphriesgkc opened this issue Nov 26, 2024 · 0 comments

Comments

@garethhumphriesgkc
Copy link

Logstash information:

Please include the following information:

  1. Logstash version: 8.15.3
  2. Logstash installation source: docker
  3. How is Logstash being run: docker

OS version (uname -a if on a Unix-like system):
ubuntu 22.04

Description of the problem including expected versus actual behaviour:
Raised off https://discuss.elastic.co/t/custom-pipeline-settings-with-centralised-pipeline-management/371001

When using the API to create centralised logstash pipelines, you can add all the settings documented, however the majority of these settings are not honoured when the pipeline runs:

[logstash.configmanagement.elasticsearchsource] Ignoring unsupported or unknown pipeline settings 'dead_letter_queue.enable'
[logstash.configmanagement.elasticsearchsource] Ignoring unsupported or unknown pipeline settings 'dead_letter_queue.retain.age'
[logstash.configmanagement.elasticsearchsource] Ignoring unsupported or unknown pipeline settings 'dead_letter_queue.flush_interval'
[logstash.configmanagement.elasticsearchsource] Ignoring unsupported or unknown pipeline settings 'dead_letter_queue.storage_policy'
[logstash.configmanagement.elasticsearchsource] Ignoring unsupported or unknown pipeline settings 'queue.drain'

This means that centralised pipelines can't be used to replace all but the most basic of logstash pipelines.

It's unclear why the settings wouldn't be honoured, but if not they should be rejected by the API, and the documentation should reflect this limitation.

For completeness the discuss article above also mentions:

  • If you subsequently edit them through Kibana, all those custom settings are lost.
  • The "Last Modified" time can't be set (and doesn't default to) "now" - you have to enter the entire timestamp each time.

Steps to reproduce:

Create a centralised pipeline with custom settings. Not that your setting don't apply.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant