Add ability to retry on unknown exceptions

[borissnd]

Logstash drops batches of logs when it encounters connection resets or read time outs.

Manticore retryable exceptions are hardcoded in the plugin. Log sending is not retried on all other Manticore exceptions.
Retryable error codes parameter does not help in this case, because these types of errors are raised as exceptions.

The plugin should allow the consumer to configure whether it would retry on unknown (or unmapped) exceptions.

Examples:
:headers=>{"Content-Type"=>"application/json", "Content-Encoding"=>"gzip"}, :message=>"Read timed out", :class=>"Manticore::UnknownException", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:37:in block in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:79:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-http-5.2.4/lib/logstash/outputs/http.rb:239:in send_event'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-http-5.2.4/lib/logstash/outputs/http.rb:175:in send_events'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-http-5.2.4/lib/logstash/outputs/http.rb:124:in multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:118:in multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:101:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:243:in block in start_workers'"], :will_retry=>false}

[HTTP Output Failure] Could not fetch URL {:url=>"https://smartcollector.command.cysiv.com/logs", :method=>:post, :headers=>{"Content-Type"=>"application/json", "Content-Encoding"=>"gzip"}, :message=>"Connection reset", :class=>"Manticore::UnknownException", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:37:in block in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:79:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-http-5.2.4/lib/logstash/outputs/http.rb:239:in send_event'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-http-5.2.4/lib/logstash/outputs/http.rb:175:in send_events'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-http-5.2.4/lib/logstash/outputs/http.rb:124:in multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:118:in multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:101:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:243:in block in start_workers'"], :will_retry=>false}

[parkertim]

Any update from dev on this issue? I am running into this problem a lot as well where my log messages aren't being retried when I get "Connection reset" errors. This is a problem that needs to be resolved so a user can decide if it should retry or not.

[mpreddy77]

@robbavey any update on this issue? Its affecting a majority of our integrations between Logstash and ES in V7.17.5. Any workaround to trap this case would be mighty helpful. Cheers!

[tszming]

@mpreddy77 connection reset issue is addressed in #127