How to get total json message without truncate

[alwalaupendra]

Hi Team,
We are using fluentd is a shipping agent for sending logs from application team to logstash server. We are not getting total json message which is above 16kb in size. I am suspecting it is truncated from fluentd side. Could you please guide me how can we get total json message without truncate?

[bdragoiu]

Not part of the team, but what is the input plugin? Can you post a sample message from something like sudo tcpdump -A port 5141

If it's in_syslog, then I was fighting with something similar just now and found that specifying the parser to be string instead of regexp worked for me. I think something with PID extraction was going wrong.

<source>
	@type syslog
	port 5141
	bind 0.0.0.0
	tag rsyslog
	message_format auto
	facility_key facility
	severity_key severity
	source_address_key source_address
	<parse>
		@type syslog
		parser_type string
		with_priority true
	</parse>
</source>

To get more insight you should maybe raw output the incoming messages. For the above example this could be a starting point.

<source>
  @type udp
  tag rsyslog
  <parse>
    @type regexp
	expression /^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[^ :\[]*) *(?<message>.*)$/
time_format "%b %d %H:%M:%S"
  </parse>
  port 5141
  bind 0.0.0.0
  message_length_limit 1MB
</source>