APM Server 400 decode error with python agent Flask /w HTTPHeader <nil> value

[onesolpark]

APM Server version (apm-server version): checked with 8.15.0 and 8.15.2

Description of the problem including expected versus actual behavior:
Flask transaction data are not gathered due to 400 decode error with HTTPHeader value
Dependency and metric data are gathered and shown on dashboard but transaction data are rejected.

Steps to reproduce:

OS: Ubuntu Linux (Container to be exact)
Python version: 3.10
Framework and version [e.g. Django 2.1]: flask-3.0.2
APM Server version: 8.15
Agent version: 6.23.0

Using all default values.

Provide logs (if relevant):

Python Agent Logs

2024-10-04 01:50:43.785 7 DEBUG elasticapm.metrics [-] Collecting metrics base_metrics.py:97 collect 
2024-10-04 01:50:44.176 7 DEBUG elasticapm.transport [-] flushing due to time since last flush 10.549s > max_flush_time 10.549s base.py:165 _process_queue
2024-10-04 01:50:44.183 7 DEBUG elasticapm.transport.http [-] Sent request, url=http://[apm-server-address]:8200/intake/v2/events size=25.74kb status=400 http.py:97 send 
2024-10-04 01:50:44.184 7 ERROR elasticapm.transport [-] Failed to submit message: 'HTTP 400: {"accepted":32,"errors":[{"message":"decode error: data read error: v2.transactionRoot.Transaction: v2.transaction.Context: v2.context.Request: v2.contextRequest.URL: v2.contextRequestURL.Headers: invalid input for HTTPHeader: \\\\u003cnil\\\\u003e","document":"{\\\\"transaction\\\\": {\\\\"id\\\\": \\\\"956ba79c937359c5\\\\", \\\\"trace_id\\\\": \\\\"8930f92212c7229641b769754c1f13c5\\\\", \\\\"name\\\\": \\\\"GET [REDACTED]\\\\", \\\\"type\\\\": \\\\"request\\\\", \\\\"duration\\\\": 244.316, \\\\"result\\\\": \\\\"HTTP 2xx\\\\", \\\\"timestamp\\\\": 1728006633627826, \\\\"outcome\\\\": \\\\"success\\\\", \\\\"sampled\\\\": true, \\\\"span_count\\\\": {\\\\"started\\\\": 3, \\\\"dropped\\\\": 0}, \\\\"sample_rate\\\\": 1.0, \\\\"context\\\\": {\\\\"request\\\\": {\\\\"env\\\\": {\\\\"REMOTE_ADDR\\\\": \\\\"[REDACTED]\\\\", \\\\"
APM Server Logs

APM Server logs

{"log.level":"error","@timestamp":"2024-10-04T01:51:25.015Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":59},"message":"data validation error","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-python/6.23.0 (APP_NAME)","source.address":"[REDACTED]","http.request.id":"89ce0a55-95f1-4dd1-8b75-417a3bc5c990","event.duration":3209908,"http.request.body.bytes":20497,"http.response.status_code":400,"error.message":"decode error: data read error: v2.transactionRoot.Transaction: v2.transaction.Context: v2.context.Request: v2.contextRequest.URL: v2.contextRequestURL.Headers: invalid input for HTTPHeader: <nil>, decode error: data read error: v2.transactionRoot.Transaction: v2.transaction.Context: v2.context.Request: v2.contextRequest.URL: v2.contextRequestURL.Headers: invalid input for HTTPHeader: <nil>, decode error: data read error: v2.transactionRoot.Transaction: v2.transaction.Context: v2.context.Request: v2.contextRequest.URL: v2.contextRequestURL.Headers: invalid input for HTTPHeader: <nil>, decode error: data read error: v2.transactionRoot.Transaction: v2.transaction.Context: v2.context.Request: v2.contextRequest.URL: v2.contextRequestURL.Headers: invalid input for HTTPHeader: <nil>, decode error: data read error: v2.transactionRoot.Transaction: v2.transaction.Context: v2.context.Request: v2.contextRequest.URL: v2.contextRequestURL.Headers: invalid input for HTTPHeader: <nil>","ecs.version":"1.6.0"}

Seems like agent is sending HTTPHeader as and apm server errors out when trying to decode the request.
I've redacted sensitive data tell me if you need more information.

Made a similar bug issue on python apm agent github because I wasn't sure if it's an agent bug or apm server bug
elastic/apm-agent-python#2134

[simitt]

Initally this reads like an agent issue rather than a server problem. I see that a conversation started on the agent github card, so will close this for now. Feel free to reopen if confirmed that this is a server issue nevertheless.