Accessing event field references with json_batch is causing exception

[kishoreck]

• Version: Logstash 5.5 [with logstash output http patch with json_batch enabled]
• Operating System: Centos
• Config File (if you have sensitive info, please remove it):
  if [@metadata][@Index]{
  http {
  format => "json_batch"
  http_method => "post"
  codec => "json"
  url => "https://myendpoint/%{[@metadata][@index]}"
  pool_max_per_route => 2000
  pool_max => 2000
  automatic_retries => 2
  retry_non_idempotent => true
  socket_timeout => 60
  connect_timeout => 60
  }
  }

With json_batch , not able to refer to the event's field references as specified in the logstash config. This is throwing an exception as follows

Exception in thread "[main]>worker1" java.net.URISyntaxException: Malformed escape pair at index 56: https://myendpoint/%{[@metadata][@index]}
at org.apache.http.client.utils.URIBuilder.(org/apache/http/client/utils/URIBuilder.java:82)
at Manticore::Client.uri_from_url_and_options(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.1-java/lib/manticore/client.rb:450)
at Manticore::Client.uri_from_url_and_options(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.1-java/lib/manticore/client.rb:450)
at Manticore::Client.request_from_options(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.1-java/lib/manticore/client.rb:457)
at Manticore::Client.request_from_options(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.1-java/lib/manticore/client.rb:457)
at Manticore::Client.request(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.1-java/lib/manticore/client.rb:418)
at Manticore::Client.request(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.1-java/lib/manticore/client.rb:418)
at Manticore::Client.post(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.1-java/lib/manticore/client.rb:265)
at Manticore::Client.post(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.1-java/lib/manticore/client.rb:265)
at LogStash::Outputs::Http.send_json_batch(/usr/share/logstash/logstash-output-http/lib/logstash/outputs/http.rb:145)
at LogStash::Outputs::Http.send_json_batch(/usr/share/logstash/logstash-output-http/lib/logstash/outputs/http.rb:145)
at LogStash::Outputs::Http.multi_receive(/usr/share/logstash/logstash-output-http/lib/logstash/outputs/http.rb:121)
at LogStash::Outputs::Http.multi_receive(/usr/share/logstash/logstash-output-http/lib/logstash/outputs/http.rb:121)
at LogStash::OutputDelegatorStrategies::Shared.multi_receive(/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:13)
at LogStash::OutputDelegatorStrategies::Shared.multi_receive(/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:13)
at LogStash::OutputDelegator.multi_receive(/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:47)
at LogStash::OutputDelegator.multi_receive(/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:47)
at LogStash::Pipeline.output_batch(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:420)
at LogStash::Pipeline.output_batch(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:420)
at org.jruby.RubyHash.each(org/jruby/RubyHash.java:1342)
at LogStash::Pipeline.output_batch(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:419)
at LogStash::Pipeline.output_batch(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:419)
at RUBY.worker_loop(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:365)
at RUBY.start_workers(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:330)

PS: Accessing the event data [for example "Url" in this case] is working with json mode. Is this the expected behavior? Shouldn't "json_batch" also mock the "json" format with respect to accessing the event data?

[jsvd]

@andrewvc it seems the new json_batch sender method doesn't perform sprintf on the @url, I don't see a reason not to have it, thoughts?

[jsvd]

@kishoreck what is your expectation for field references in url when using json_batch?
a) given a list of events, each event's url field will be sprinted and multiple sub batches are sent to different urls?
b) you assume all events in a batch will sprintf to the same url?

[andrewvc]

The reason it doesn't do a sprintf is that a batch contains multiple events. We could sprintf all the events and send multiple batches I suppose

…

On Mar 14, 2018 11:57 AM, "João Duarte" ***@***.***> wrote: @kishoreck <https://github.com/kishoreck> what is your expectation for field references in url when using json_batch? a) given a list of events, each event's url field will be sprinted and multiple sub batches are sent to different urls? b) you assume all events in a batch will sprintf to the same url? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#84 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAIBY3Ws0ZOgDwWH9mB6BTLVa7a1Wb3zks5teUvhgaJpZM4SmkWE> .

[kishoreck]

@jsvd - We are looking for something similar to option "a", as the events that are coming in will have different values and dynamic in nature and will have to pick/form the appropriate URL based on the input event's fields.

[jsvd]

The json_batch mode was made so that the received batch of events could be sent in a single HTTP request. If you need custom/dynamic urls then you can use the normal modes.
We could in the future implement the ability evaluate all elements in the batch and spawn sub batches, but I don't see this coming in the near future (and it slows down the current json_batch format)