Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failing to provide a + in the date pattern of the index property silently fails. #1082

Open
nicenemo opened this issue Sep 1, 2022 · 0 comments
Labels

Comments

@nicenemo
Copy link

nicenemo commented Sep 1, 2022

Logstash information:

  1. Using bundled JDK: /usr/share/logstash/jdk logstash 8.4.1 But also fails on older versions
  2. Debian packages from Elastic. But also fails on Mac.
  3. Either as service or as command line
  4. plugin came packaged with logstash

We run Logstash both as systemd service as well as commandline tool.
We run on Debian Bullseye and MacOS.
Debian is ran on AWS ARM & Intel based systems. It is run on Windows WSL2 based Debian instances too.

JVM (e.g. java -version):

Bundled

Steps to reproduce:

Use a config like this:

input {
  # We read from backup files. Note that the start of the path MUST be absolute
  file {
    path => "/opt/data/backup/ecorp-fluffy-data-2022.33.json"
    sincedb_path => "/tmp/sincedb_restore-ecorp.db"
    start_position => "beginning"
    file_chunk_size => 268435456
    mode => "read"
    codec => "json"
    file_completed_action => "log"
    file_completed_log_path => "/opt/data/logstash/logs/restore-ecorp-fluffy-data.log"
  }
}
filter {
  if "_jsonparsefailure" in [tags] {
    drop { }
  }
# sort
  json {
    source => "message"
  }
}

output {
  # We write to the "new" cluster
  elasticsearch {
    manage_template => false
    sniffing => false
    ilm_enabled => false
     # does not work and fails silent because of missing +.
     index => "ecorp-fluffy-%{YYYY.MM}.01"
     # Does work. the added fixed 01 is to work around another bug/feature
     # index => "ecorp-fluffy-%{+YYYY.MM}.01"
    doc_as_upsert => true
    action => "update"
    document_id => "%{identifier}"
    http_compression => true
    ssl => true
    ssl_certificate_verification => false
    hosts => ["https://localhost:9200"]
    api_key => " not going to tell you"
  }
  # We print dots to see it in action
  stdout {
    codec => "dots"
  }
}

It will fail silently.

Provide logs (if relevant):

That's the problem.... Logstash is silent ;)

@nicenemo nicenemo added the bug label Sep 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant