Skip to content

Add Stopped node status - #18596

Open
d-wang-commit wants to merge 1 commit into
apache:masterfrom
d-wang-commit:node_stopped_status
Open

Add Stopped node status#18596
d-wang-commit wants to merge 1 commit into
apache:masterfrom
d-wang-commit:node_stopped_status

Conversation

@d-wang-commit

Copy link
Copy Markdown

Gracefully stopped nodes report their shutdown and are shown as Stopped instead of Unknown: shutdown reports mark Stopped unless the node is Removing, the heartbeat caches keep Stopped sticky against failure-driven Unknown while still allowing revival by a live heartbeat, and every consumer that treats Unknown as unavailable (region allocation candidates, region maintenance retries, failed task transfers, subscription runtime sensitivity, fenced filtering, removal safety checks, runtime writer selection) now treats Stopped the same way.

Why it's done this way

Stopped means "known, intentionally stopped" while Unknown means "cannot tell why it is unreachable". The distinction only refines the displayed diagnosis — every scheduling decision keeps treating
reaches the leader, heartbeat timeout still marks it Unknown (unchanged fallback).

Description

  • New NodeStatus.Stopped value, treated as not readable (like Unknown).
  • DataNode / ConfigNode shutdown hooks best-effort report their shutdown; the leader marks the node Stopped unless it is Removing (Removing keeps the highest priority). If the report never reaches the leader, heartbeat timeout still marks it Unknown
    (unchanged fallback).
  • Heartbeat caches keep Stopped sticky: failure-driven or periodic Unknown updates cannot refresh it, while a live heartbeat (e.g. after restart) revives the node to Running.
  • All consumers that treat Unknown as unavailable now treat Stopped the same way: region allocation fallback candidates, failed-task region transfer, region-maintenance short retries, subscription runtime sensitivity, fenced DataNode filtering, removal safety
    checks, and runtime writer selection.

Tests

  • UT: NodeCacheTest (sticky / revival / Removing priority for both cache types); ConfigManagerShutdownReportTest (report marks Stopped, skips Removing); ProcedureManagerTest (removal refusedwhen related node is
    Stopped); RegionBalancerTest; RetryFailedTasksThreadTest; RegionMaintainHandlerTest; SchemaUtilsTest; SubscriptionRuntimeCoordinatorTest; ConfigNodeProcedureEnvTest (Stopped excluded from active runtime writers).
  • IT: shutdown-hook and restart ITs (IoTDBClusterNodeShutdownHookIT, IoTDBClusterNodeErrorStartUpIT); partition allocation ITs (IoTDBPartitionCreationIT, IoTDBPartitionDurableIT, Io - DataNode / ConfigNode shutdown hooks best-effort report their shutdown; the leader marks the node Stoppedunless it isRemoving(Removing keeps the highest priority). If the report never reaches the leader, heartbeat timeout still marks itUnknown` (unchanged fallback).
  • Heartbeat caches keep Stopped sticky: failure-driven or periodic Unknown updates cannot refresh it, while a live heartbeat (e.g. after restart) revives the node to Running.
  • All consumers that treat Unknown as unavailable now treat Stopped the same way: region allocation fallback candidates, failed-task region transfer, region-maintenance short retries, subscription
    runtime sensitivity, fenced DataNode filtering, removal safety checks, and runtime writer selection.

This PR has:

  • been self-reviewed.
    • concurrent read
    • concurrent write
    • concurrent read and write
  • added documentation for new or modified features or behaviors.
  • added Javadocs for most classes and all non-trivial methods.
  • added or updated version, license, or notice information
  • added comments explaining the "why" and the intent of the code wherever would not be obvious
    for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold
    for code coverage.
  • added integration tests.
  • been tested in a test IoTDB cluster.

Key changed/added classes (or packages if there are too many classes) in this PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant