feat: add request diagnostics

[tma]

Summary

This adds the request diagnostics intentionally left out of #10.

• Log request identity, retry decisions, and lifecycle timing.
• Distinguish upstream attempts from coalesced followers.
• Track retries, recovered retries, and failure streaks.
• Expose degradation details through the health endpoint.
• Keep write payloads out of logs.

Request logs

Upstream lifecycle logs include:

• slave ID, function, address, quantity, and read/write state
• attempt number and total attempts
• queue, attempt, reconnect, coalescing wait, and total duration
• error classification and retry decision
• upstream and downstream exception codes where applicable

A retryable first transport failure is logged at DEBUG with will_retry=true. Successful retries are logged at DEBUG with attempts=2. Final failures and genuine upstream Modbus exceptions are logged at WARN.

Coalesced followers report their own wait and result without pretending they made an independent upstream attempt.

Health diagnostics

The health response now includes:

• total and recovered retry counts
• consecutive first-attempt and final-failure counts
• last first-attempt success
• last successful request
• current and sustained degradation state

One recovered retry sets degraded=true, but health remains HTTP 200 with status ok. If no first-attempt success clears that state for one minute, status changes to degraded while remaining HTTP 200.

A later first-attempt success clears the degradation state. Final upstream failures continue to return an unhealthy response.

Safety

This does not change the request-safety behavior from #10. End-to-end budgets, read-only retries, write ambiguity handling, exception mapping, request pacing, and cache generation remain unchanged.

Write payloads are never included in logs.

Testing

• Docker Go 1.24 targeted race tests repeated 20 times
• Full format, vet, and race-enabled test suite
• Health-state and recovery tests
• Request timing and log-field tests
• Coalescing attribution and stale-fallback tests
• Write-payload redaction tests
• docker build --target test .

[Copilot]

Pull request overview

Adds structured request diagnostics across cache coalescing, upstream execution, and the health endpoint so operators can attribute retries/failures and understand degradation without logging write payloads.

Changes:

• Add upstream lifecycle diagnostics (identity, attempt/retry decisions, and disjoint timing fields) including explicit coalescing follower vs replacement-leader attribution.
• Track retry/recovery and degradation state in the Modbus client and expose those diagnostics via the health endpoint (while keeping availability semantics unchanged).
• Extend cache coalescing to return leader/follower status and wait attribution, with corresponding tests.

Show a summary per file

File	Description
SPEC.md	Documents new diagnostics/health behavior and updates log examples.
README.md	Adds “Production Diagnostics” section describing structured logs and health fields.
internal/proxy/proxy.go	Uses coalescing status to log follower attribution and prevent stale fallback after context end.
internal/proxy/proxy_test.go	Adds tests for proxy health delegation and coalescing/stale-fallback logging behavior.
internal/modbus/server.go	Adds structured downstream failure logging with coalescing attribution and timing fields.
internal/modbus/server_test.go	Adds tests for server-side diagnostics fields, levels, and payload redaction.
internal/modbus/errors.go	Extends RequestError to carry queue/attempt/reconnect/total durations.
internal/modbus/client.go	Implements timing capture, retry diagnostics, degradation counters, and health reporting helpers.
internal/modbus/client_test.go	Adds tests for timing separation, log fields, payload redaction, and degradation state transitions.
internal/health/health.go	Extends health JSON to include optional diagnostic details and prefer atomic health snapshots.
internal/health/health_test.go	Adds tests for degraded/atomic report behavior and diagnostics inclusion on unhealthy.
internal/cache/cache.go	Introduces CoalesceWithStatus and attributed coalescing errors for leader/follower/wait attribution.
internal/cache/cache_test.go	Adds tests validating coalescing attribution for followers and replacement leaders.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

• Files reviewed: 13/13 changed files
• Comments generated: 2
• Review effort level: Low

[Copilot]

Review details

Comments suppressed due to low confidence (3)

internal/modbus/client.go:598

• waitForRequestSlot computes the delay using time.Until(c.nextRequestAt), which is tied to the process wall clock. Since this client now supports an injected c.now() clock, the delay should be computed relative to c.now() to keep request pacing consistent with the same clock used for diagnostics and nextRequestAt.

func (c *Client) waitForRequestSlot(ctx context.Context) error {
	delay := time.Until(c.nextRequestAt)
	if delay <= 0 {
		return contextError(ctx)
	}

internal/modbus/client.go:537

• responseCompletedAt uses time.Now() while the rest of the request timing/pacing logic is based on the injected c.now() clock. This makes diagnostics and nextRequestAt inconsistent when c.now is overridden (tests/future determinism) and can skew pacing relative to the logged durations.

This issue also appears on line 594 of the same file.

		attemptStart := c.now()
		resp, err := c.executeRequest(ctx, req)
		responseCompletedAt := time.Now()
		timing.attempt += c.now().Sub(attemptStart)

internal/proxy/proxy.go:192

• The variable name coalesced is a cache.CoalesceResult struct, not a boolean. Using a boolean-sounding name for a multi-field status object makes later checks like coalesced.Coalesced harder to read and easy to misinterpret; consider renaming it to coalesceResult/coalesceStatus (and updating uses in this function).

	rangeKey := fmt.Sprintf("%d:%s", generation, cache.RangeKey(req.SlaveID, req.FunctionCode, req.Address, req.Quantity))
	data, coalesced, err := p.cache.CoalesceWithStatus(ctx, rangeKey, func(ctx context.Context) ([]byte, error) {
		data, err := p.client.Execute(ctx, req)
		if err != nil {

• Files reviewed: 13/13 changed files
• Comments generated: 0 new
• Review effort level: Low