Description
Logstash information:
Please include the following information:
- Logstash version: 8.15.3
- Logstash installation source: docker
- How is Logstash being run: docker
OS version (uname -a
if on a Unix-like system):
ubuntu 22.04
Description of the problem including expected versus actual behaviour:
Raised off https://discuss.elastic.co/t/custom-pipeline-settings-with-centralised-pipeline-management/371001
When using the API to create centralised logstash pipelines, you can add all the settings documented, however the majority of these settings are not honoured when the pipeline runs:
[logstash.configmanagement.elasticsearchsource] Ignoring unsupported or unknown pipeline settings 'dead_letter_queue.enable'
[logstash.configmanagement.elasticsearchsource] Ignoring unsupported or unknown pipeline settings 'dead_letter_queue.retain.age'
[logstash.configmanagement.elasticsearchsource] Ignoring unsupported or unknown pipeline settings 'dead_letter_queue.flush_interval'
[logstash.configmanagement.elasticsearchsource] Ignoring unsupported or unknown pipeline settings 'dead_letter_queue.storage_policy'
[logstash.configmanagement.elasticsearchsource] Ignoring unsupported or unknown pipeline settings 'queue.drain'
This means that centralised pipelines can't be used to replace all but the most basic of logstash pipelines.
It's unclear why the settings wouldn't be honoured, but if not they should be rejected by the API, and the documentation should reflect this limitation.
For completeness the discuss article above also mentions:
- If you subsequently edit them through Kibana, all those custom settings are lost.
- The "Last Modified" time can't be set (and doesn't default to) "now" - you have to enter the entire timestamp each time.
Steps to reproduce:
Create a centralised pipeline with custom settings. Not that your setting don't apply.