in_prometheus_scrape never reuses TCP connections; root cause also affects flb_input_upstream_set()

[labros-mediaalpha]

Bug Report

Investigation and this writeup were AI-assisted (Claude Code).

Describe the bug

prometheus_scrape opens a brand new TCP connection and tears it down
(FIN) on every single scrape cycle, regardless of net.keepalive.
Confirmed via packet capture: a full SYN/SYN-ACK/ACK handshake followed
by FIN immediately after each response, every scrape_interval, never
a reused connection.

This is actually two separate, independently-necessary bugs:

• Bug 1: in_prometheus_scrape never calls flb_input_upstream_set()
plugins/in_prometheus_scrape/prom_scrape.c calls flb_upstream_create()
  but never calls flb_input_upstream_set(), so net.* properties are
  parsed into ins->net_setup but never applied to the plugin's upstream.
  Same class of bug already fixed for in_kubernetes_events (in_kubernetes_events: add support for configuring input upstream network setup #11188) and
  in_calyptia_fleet (in_calyptia_fleet: add support for net.* properties for the upstream connection. #10998).

• Bug 2: flb_input_upstream_set() itself is missing the FLB_IO_TCP_KA flag
  Adding the call above is not sufficient on its own (verified — see below).
  flb_upstream_conn_get() sets a new connection's initial recycle value
  from flb_stream_is_keepalive(), which checks the FLB_IO_TCP_KA bit on
  stream->flags — not net.keepalive directly.

  flb_output_upstream_set() (src/flb_output.c) sets this flag correctly:

  if (ins->net_setup.keepalive == FLB_TRUE) {
      flags |= FLB_IO_TCP_KA;
  }
  flb_stream_enable_flags(&u->base, flags);

  flb_input_upstream_set() (src/flb_input.c) only does the net.*
  memcpy — it never sets FLB_IO_TCP_KA. So conn->recycle starts
  FALSE on every new connection, and flb_upstream_conn_release()'s
  conn->recycle == FLB_TRUE check always fails, forcing a fresh
  connection every time regardless of net.keepalive.

  This affects every input plugin that calls flb_input_upstream_set() —
  likely including in_kubernetes_events and in_calyptia_fleet despite
  their existing fixes, though I have not independently reproduced that;
  it's an inference from shared code, not a verified repro.

To Reproduce

• Steps to reproduce the problem:
  1. Configure prometheus_scrape against any HTTP/1.1 keep-alive-capable
     endpoint.
  2. Capture loopback traffic:

     tcpdump -i lo -n -tttt 'tcp port <target_port>'
     

  3. Observe a new SYN/FIN pair every scrape_interval, never a reused
     connection.

Expected behavior

Connection is established once and reused across scrape cycles, per
net.keepalive (default: on).

Screenshots

N/A — see packet-capture description above.

Your Environment

• Version used: v5.0.7
• Configuration: prometheus_scrape input scraping two independent
  HTTP/1.1 endpoints on loopback
• Environment name and version: N/A (not Kubernetes)
• Server type and version: Amazon Linux 2023, x86_64
• Operating System and version: Amazon Linux 2023, x86_64
• Filters and plugins: in_prometheus_scrape

Additional context

Fix verified locally, tested in 4 configurations (v5.0.7, x86_64):

Configuration	Result
Unpatched (baseline)	New connection every cycle
Bug 1 fix only	New connection every cycle
Bug 2 fix only	New connection every cycle
Both fixes together	Connection created once, reused

Confirmed via -vv debug log: with both fixes, connections show
"KA connection #N ... is connected" once, then
"has been assigned (recycled)" on every subsequent scrape. With either
fix alone, no "KA connection" line ever appears for these endpoints.

I'll open two PRs against this issue: one for Bug 1 (scoped to
in_prometheus_scrape, mirrors the existing in_kubernetes_events fix)
and one for Bug 2 (core flb_input_upstream_set() fix, broader impact —
worth extra scrutiny and possibly a look at the other two plugins).

[patrick-stephens]

Can you update to match the required description template as well? I know you used some AI to assist but it also should be able to cope with following a template so if there's anything that can be tweaked to ensure that for future AI contributions then be good to capture that too.

[labros-mediaalpha]

@patrick-stephens done, let me know if it missed anything