Graph database corruption

[jogc]

(Did not report this as a bug yet since it could be a bug in chantools, in lnd or in neither)

I have a small node I haven't paid much attention to for many months and I suddenly realized something is very wrong with its graph database (see below). Good news is my channels are still open with a recent latest update time according to third-party nodes and the channel points have not been spent on-chain so I guess the node is still kinda functioning from an outside perspective. lncli listchannels shows the channels as open and active and lncli describegraph returns a graph with a reasonable number of edges and nodes but excluding any of my channels.

I did a chantools dropchannelgraph on this node in July but I don't have logs further back than late December which is extremely annoying so I can't say if this started right after the dropchannelgraph or if it was later. I have done dropchannelgraph on other nodes before without any problem (the node rebuilds the graph without any issues). What is the best course of action? Do a dropchannelgraph --fix_only and hope for the best? Nuke all data and restart completely? If I have to close the channels I guess that's fine but I would like to keep the node identity.

What I'm really worrying about is that the real cause is data corruption due to some hardware issue since I moved the node to a brand new device at the same time I did the dropchannelgraph. Would be nice to exclude that as an option.

Details:

It spits out messages like these once a minute, one for each of my channels:

[ERR] NANN: Unable to retrieve chan status for Channel(X:n): edge not found: op=X:n

(slightly censored) and once every few minutes, again one for each of my channels:

[INF] CNCT: ChannelArbitrator(X:n) no close event

On top of that I also get a lot of other suspicious messages. Here is the frequency of all WRN and ERR messages (that are not obviously unrelated) from the logs I got:

692940 [WRN] GRDB: Channel=X not found in graph cache
484718 [WRN] GRDB: Node=X not found in graph cache
157638 [ERR] NANN: Unable to retrieve chan status for Channel(X:n): edge not found: op=X:n
 34733 [WRN] PEER: Peer(X): Error processing remote msg *lnwire.ChannelUpdate1: recently rejected
 22860 [WRN] BTWL: Received non-standard input sig=X, witness=[X]
 19933 [WRN] PEER: Peer(X): Error processing remote msg *lnwire.ChannelUpdate1: incorrect pubkey to resurrect zombie with chan_id=A:b:c
 18611 [WRN] PEER: Peer(X): Error processing remote msg *lnwire.ChannelAnnouncement1: recently rejected
 14967 [INF] CNCT: ChannelArbitrator(X:n) no close event
 11574 [WRN] PEER: Peer(X): Error processing remote msg *lnwire.ChannelAnnouncement1: channel output has been spent: unable to fetch utxo for chan_id=X, chan_point=X:n: target output has been spent
  9429 [ERR] DISC: ignoring closed channel A:b:c
  8755 [WRN] PEER: Peer(X): Error processing remote msg *lnwire.ChannelAnnouncement1: ignoring closed channel A:b:c
  1615 [WRN] PEER: Peer(X): Error processing remote msg *lnwire.ChannelUpdate1: unable to validate channel update announcement for short_chan_id=(lnwire.ShortChannelID) A:b:c
  1615 [ERR] DISC: unable to validate channel update announcement for short_chan_id=(lnwire.ShortChannelID) A:b:c
  1094 [WRN] PEER: Peer(X): Error processing remote msg *lnwire.ChannelAnnouncement1: ignoring remote ChannelAnnouncement1 for own channel
  1094 [WRN] DISC: ignoring remote ChannelAnnouncement1 for own channel
   119 [WRN] PEER: Peer(X): Unable to find our forwarding policy for channel X:n, using default values
    20 [WRN] PEER: Peer(X): Error processing remote msg *lnwire.NodeAnnouncement: Ignoring outdated announcement for X
    15 [WRN] BTWL: Received non-standard input sig=, witness=[]
    12 [WRN] NANN: Unable to find channel policies for X:n, skipping. This is typical if the channel is in the process of closing.
    12 [WRN] BTWL: Received non-standard input sig=X, witness=[]
    10 [WRN] PEER: Peer(X): Error processing remote msg *lnwire.ChannelUpdate1: unable to add channel: edge not found
    10 [ERR] GRPH: unable to add channel: edge not found
    10 [ERR] GRPH: process network updates got: unable to add channel: edge not found
    10 [ERR] GRDB: UpdateEdgePolicy faild: edge not found
    10 [ERR] DISC: Update edge for short_chan_id(X) got: unable to add channel: edge not found

I do still have the log from chantools:

2025-07-20 14:46:16.639 [INF] CHAN: chantools version v0.14.0 commit 
2025-07-20 14:46:16.691 [INF] CHDB: Checking for schema update: latest_version=33, db_version=31
2025-07-20 14:46:16.691 [INF] CHDB: Performing database schema migration
2025-07-20 14:46:16.691 [INF] CHDB: Applying migration #32
2025-07-20 14:46:16.691 [INF] CHDB: Migrating Mission Control store to use a more minimal encoding for routes
2025-07-20 14:46:16.691 [INF] CHDB: Applying migration #33
2025-07-20 14:46:16.691 [INF] CHDB: Migrating Mission Control store to use namespaced results
2025-07-20 14:46:16.696 [INF] CHDB: Checking for optional update: prune_revocation_log=false, db_version=empty
2025-07-20 14:46:16.698 [INF] CHAN: Dropping all graph related buckets
2025-07-20 15:28:53.877 [INF] CHAN: chantools version v0.14.0 commit 
2025-07-20 15:29:58.221 [INF] CHAN: chantools version v0.14.0 commit 
2025-07-20 15:31:56.741 [INF] CHAN: chantools version v0.14.0 commit 

I also managed to find the exact command I used in .bash_history:
chantools dropchannelgraph --channeldb data/graph/mainnet/channel.db --node_identity_key PK
(I have double checked that the PK I used is the correct node pubkey of this node, to the letter)

Lnd version is amd64-v0.19.1-beta (been the same since I did the dropchannelgraph)

[bitromortac]

What was your motivation to delete the graph? You could try lncli updatechanpolicy --create_missing_edge which should create the missing edges, but not sure this is already available for v0.19.

[jogc]

Will do tomorrow and report back! Thanks so far

[jogc]

Upgraded to v0.20.1. Managed to fix one of the channels. It actually seems like the reason create_missing_edge did not work with v0.19.1 is that you only get one chance to get it right. You have to specify all the values for the updatechanpolicy and it's not apparent what all of them are supposed to be. So if you for example put a too big value for max_htlc_msat so you get something like this back:

{
    "failed_updates": [
        {
            "outpoint": "X:x",
            "reason": "UPDATE_FAILURE_INVALID_PARAMETER",
            "update_error": "max htlc size of X mSAT is above max pending amount of X mSAT"
        }
    ]
}

you are screwed. updatechanpolicy has now put the channel edge in some kind of irrecoverable state that is not fixable. But if you get everything right immediately it does manage to recreate the edge. v0.20.1 has the same behavior. I will report this as a bug. Anyway, the end result is two permanently borked channels and one seemingly healthy channel. Well, I'm still getting a [INF] CNCT: ChannelArbitrator(<CHANNEL_POINT>) no close event message for the fixed channel every time a new block arrives. Should I care?

[bitromortac]

It would be good if you could open a bug report, thank you. The CNCT message seems to be harmless as it checks on each block if the channel was closed, but it wasn't.

[jogc]

Created #10585

[jogc]

Found this PR that seems related and could possibly be a fix: #10516