Improve retry handling and semantics

[jsvd]

It's unclear how the automatic_retries setting works, and if it applies to network failures, to non 200 status codes, or both (or neither).

Plugins using this mixin don't have visibility if requests are retried at all.

For example here's an endpoint that returns a 503 and doesn't seem to be retried:

/tmp/logstash-7.11.1
❯ echo 1 | bin/logstash -e "filter { http { url => 'http://httpstat.us/503' automatic_retries => 100 } }"
[..]
[2021-02-19T18:14:38,262][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2021-02-19T18:14:38,297][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2021-02-19T18:14:38,457][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2021-02-19T18:14:39,220][ERROR][logstash.filters.http    ][main][bd532599e066920b19b659b8756ad4a5751741b31fc6994b60c993f22fc75bbc] error during HTTP request {:url=>"http://httpstat.us/503", :code=>503, :response=>""}
{
          "tags" => [
        [0] "_httprequestfailure"
    ],
          "host" => "joaos-mbp.lan",
       "message" => "1",
          "type" => "stdin",
      "@version" => "1",
    "@timestamp" => 2021-02-19T18:14:38.288Z
}
[2021-02-19T18:14:39,405][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main", "uptime"=>1887}

And, here's a dns failure, that doesn't seem to be retried either:

/tmp/logstash-7.11.1
❯ echo 1 | bin/logstash -e "filter { http { url => 'http://httpstat.us/503' automatic_retries => 100 } }"
[2021-02-19T18:15:35,758][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.11.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.9+11 on 11.0.9+11 +indy +jit [darwin-x86_64]"}
[2021-02-19T18:15:37,967][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2021-02-19T18:15:38,006][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2021-02-19T18:15:38,177][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2021-02-19T18:15:38,373][ERROR][logstash.filters.http    ][main][bd532599e066920b19b659b8756ad4a5751741b31fc6994b60c993f22fc75bbc] error during HTTP request {:url=>"http://httpstat.us/503", :body=>nil, :client_error=>"httpstat.us: nodename nor servname provided, or not known"}
{
          "host" => "Joaos-MacBook-Pro.local",
    "@timestamp" => 2021-02-19T18:15:37.992Z,
       "message" => "1",
          "type" => "stdin",
          "tags" => [
        [0] "_httprequestfailure"
    ],
      "@version" => "1"
}
[2021-02-19T18:15:38,536][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main", "uptime"=>1252}
[2021-02-19T18:15:38,565][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:main}
[2021-02-19T18:15:38,597][INFO ][logstash.runner          ] Logstash shut down.