Skip to content

Add ReadOnly status reason and StatusReason display - #18594

Open
d-wang-commit wants to merge 1 commit into
apache:masterfrom
d-wang-commit:readonly_status_reason
Open

Add ReadOnly status reason and StatusReason display#18594
d-wang-commit wants to merge 1 commit into
apache:masterfrom
d-wang-commit:readonly_status_reason

Conversation

@d-wang-commit

Copy link
Copy Markdown

Classify ReadOnly trigger points with priority-ordered reasons (Stopping > Manual > UnrecoverableError > DiskFull > unclassified), propagate the reason through heartbeats as a separate statusReason field, and show it as a dedicated StatusReason column in SHOW CLUSTER / SHOW CONFIGNODES / SHOW DATANODES and the information_schema.NODES view instead of the merged Status(reason) string. UnrecoverableError reasons carry the truncated exception message with a class-name fallback.

Description

Classifies ReadOnly trigger points with reasons and displays the reason in a dedicated column instead of merging
it into the Status string.

Reason state machine (CommonConfig)

  • ReadOnly entries carry stable tokens with priority Stopping > Manual > UnrecoverableError > DiskFull, plus a
    legacy unclassified (null) reason at the lowest priority.
  • Same-status writes never downgrade the reason (a DiskFull alarm cannot overwrite a Manual ReadOnly); a
    same-priority UnrecoverableError keeps the first reason. Status and reason are written atomically.

Trigger point classification

  • Disk-full sampling / folder exhaustion → DiskFull
  • SET SYSTEM TO READONLY (local or cluster-wide) → Manual
  • Shutdown flush phase → Stopping
  • Storage / WAL / checkpoint unrecoverable errors → UnrecoverableError, <timestamp>, <message>; the message is
    truncated to 256 chars with ... and falls back to the exception class name when empty

Reason propagation & display

  • DataNode heartbeats carry statusReason alongside status, read from one snapshot to avoid torn pairs.
  • SHOW CLUSTER / SHOW CLUSTER DETAILS / SHOW CONFIGNODES / SHOW DATANODES insert a StatusReason column
    right after Status only when at least one node has a reason; nodes without one show NULL, and the Status
    column keeps the plain status (no more merged ReadOnly(DiskFull) strings).
  • information_schema.NODES gains a status_reason column.
  • New thrift fields are optional: responses from an old ConfigNode (field unset) keep the legacy column layout.

Tests

  • UT: CommonConfigTest (priority chain, first-UnrecoverableError-wins, unclassified-reason semantics, truncation / class-name fallback, same-status reason preservation); DataNodeInternalRPCServiceImplDiskTest (auto-recovery only for an exact
    ReadOnly+DiskFull state); DataNodeInternalRPCServiceImplSetSystemStatusTest (Manual classification); Show*TaskTest (dynamic column layouts incl. old-ConfigNode fallback); NodeCacheTest (reason propagation; a failure-detector Unknown never carries a reason);
    FolderManagerTest; DataNodeShutdownHookTest.
  • IT: IoTDBSetSystemStatusTableIT (end-to-end Manual reason through SHOW DATANODES, plain status in the NODES view); IoTDBDatabaseIT (NODES schema).

Negative effects

  • SHOW statements gain one extra column only when a reason exists; clients parsing by column name are unaffected.
  • Each reason is bounded to 256 chars and only travels in heartbeats / SHOW responses, so the payload overhead is negligible.

This PR has:

  • been self-reviewed.
    • concurrent read
    • concurrent write
    • concurrent read and write
  • added documentation for new or modified features or behaviors.
  • added Javadocs for most classes and all non-trivial methods.
  • added or updated version, license, or notice information
  • added comments explaining the "why" and the intent of the code wherever would not be obvious
    for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold
    for code coverage.
  • added integration tests.
  • been tested in a test IoTDB cluster.

Key changed/added classes (or packages if there are too many classes) in this PR

Classify ReadOnly trigger points with priority-ordered reasons (Stopping > Manual > UnrecoverableError > DiskFull > unclassified), propagate the reason through heartbeats as a separate statusReason field, and show it as a dedicated StatusReason column in SHOW CLUSTER / SHOW CONFIGNODES / SHOW DATANODES and the information_schema.NODES view instead of the merged Status(reason) string. UnrecoverableError reasons carry the truncated exception message with a class-name fallback.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant