c/scheduling: fix allocation underflow on no-op reallocate

[joe-redpanda]

Summary

Fixes a partition-allocator accounting bug that aborts a node during
decommission on:

assert - Assert failure: (src/v/cluster/scheduling/allocation_node.cc)
'_allocated_partitions > allocation_capacity{0}'
unable to deallocate partition at node {node: N, ..., allocated: 0, ...}

The bug is latent in dev; it is surfaced frequently by the event-driven
health_manager introduced in #30779, which is why it showed up as a
NodeCrash in ShadowLinkingRandomOpsTest.test_node_operations on that PR's
CI.

The bug

partition_allocator::allocate() builds an allocated_partition per
requested partition and only calls allocated_partition::add_replica() for
partitions that need new replicas (do_allocate's in_progress loop
gates on replication_factor > current replica count).

add_replica() is what lazily initializes _original_node2shard. So when a
request contains a partition that needs no new replicas
(replication_factor == existing replica count), add_replica() is never
called and _original_node2shard stays nullopt.

release_new_partition() then decides which replicas are "new" (and therefore
own an allocation that must be released when the allocation_units RAII object
is destroyed):

if (!_original_node2shard || !_original_node2shard->contains(bs.node_id)) {
    added_replicas.push_back(bs);
}

With _original_node2shard == nullopt, the !_original_node2shard || branch
classifies every existing replica as new. ~allocation_units then calls
remove_allocation() on each of them — decrements that were never matched by a
corresponding allocate()/increment. The per-node _allocated_partitions
counter drops below the true replica count, and the next legitimate
remove_allocation() (a decommission finish_moving_partition_replicas_cmd)
underflows the assert.

Why #30779 triggers it

health_manager::increase_replication_factor →
do_increase_replication_factor builds an allocation request for all
partitions of an internal topic at rf = target. For ct_l1_domain (born
RF=1, then reconciled to RF=3), once some partitions are already at target,
each health_manager pass re-requests them with num_new_replicas == 0 and hits
the path above. #30779 makes health_manager run on every membership-change
notification, so this fires repeatedly during decommission churn. Observed in a
minimal repro to underflow even for a node in state: active, confirming the
defect is the no-op reallocate, independent of which node is being
decommissioned.

The fix

release_new_partition should only treat a replica as new when
add_replica() actually allocated it, i.e. when _original_node2shard has a
value:

if (_original_node2shard && !_original_node2shard->contains(bs.node_id)) {
    added_replicas.push_back(bs);
}

A nullopt map means nothing was allocated, so nothing should be released.
This matches has_changes() and ~allocated_partition, which already treat a
nullopt _original_node2shard as "no changes / nothing to undo".

How the tests reproduce it

Deterministic unit test — topic_updates_dispatcher_test.cc: rf_noop_allocate_spurious_dealloc. Creates an RF=3 single-partition topic (3
allocations), then calls allocate() again for the same partition at the same
RF (num_new_replicas == 0) and lets the returned allocation_units go out of
scope. Without the fix the per-node total drops 3 -> 0
(BOOST_CHECK_EQUAL(total_allocated(), 3) fails); with the fix it stays 3.
No concurrency, sub-second.

Integration reproducer — tests/rptest/tests/l1_decom_repro_test.py.
Single cluster (no shadow linking) with the ingredients that open the race in
the real system: cloud topics enabled (so ct_l1_domain is born RF=1 and
health-manager-reconciled), a rate-limited background producer keeping the
metastore raft groups busy, an AdminOperationsFuzzer issuing concurrent
topic/RF-change commands, and a decommission/recommission/add node-op loop.
Against the #30779 build this reproduced the underflow within the first repeat
iteration; with this fix applied it ran the full node-op loop with zero
underflows.

Validation

	before fix	after fix
unit test rf_noop_allocate_spurious_dealloc	total 3 → 0, FAIL	PASS
ducktape reproducer (2 iters, 40 node-ops)	16 underflow events (crash)	0

🤖 Generated with Claude Code