Implement a consumer chain halt system

[tbruyelle]

Following #9, we want to halt a consumer chain if the consumer pool has not enought funds for a block. Before implementing this system, we need to design it, let's use this issue for that.

We have essentially 2 different ways to implement the consumer halt :

1. the consumer chain panics like if it was waiting for an upgrade

• pro: the cost for running this consumer chain becomes 0 or negligible
• cons: might be more difficult to recover from that. Once the consumer maintainers have refunded the consumer pool, how to restart the chain since it is totally stopped ?

2. the consumer chain still runs but does not accept transactions

• pro: easy recover. Once the consumer maintainers have refunded the consumer pool, the provider chain can send an IBC message to the consumer saying it can accept transaction again.
• cons: the consumer chain still produce blocks, so there is still a cost for the validators.

We could probably design a more nuanced system, essentially 2) followed by 1) after a timeout. I think this would be more acceptable, since option 1 is a bit too severe.

Please comment and share your point of view.

[giuliostramondo]

Just brainstorming here.

Could we use the OS to stop (SIGSTOP) and restart (SIGCONT) the consumer chain process?
We would need to track the PIDs of the consumer chains, but this way we could halt block production, but still have a way to restart once the conditions are right...

I think this library allows to do this in a cross-platform way gopsutil

[Pantani]

I like the timeout and panic chain after some blocks. We can also create two more phases, HALTED_SOFT and HALTED_HARD. So we keep block production during soft halt, and escalate only after timeout.

[Pantani]

Just brainstorming here.

Could we use the OS to stop (SIGSTOP) and restart (SIGCONT) the consumer chain process? We would need to track the PIDs of the consumer chains, but this way we could halt block production, but still have a way to restart once the conditions are right...

I think this library allows to do this in a cross-platform way gopsutil

Using SIGSTOP/SIGCONT is an OS-level trick, not a protocol-level mechanism, so the provider cannot reliably control remote validator processes across machines. PID-based control is fragile and non deterministic.
Halt/resume should be implemented on-chain with explicit state and messages, while process management stays an optional local operator action.

I think it's only possible if we create something like Cosmosvisor to manage it, but it's one more layer.

[Pantani]

One thing we can do is implement a Credit Model or a Debt Model.

In a Credit Model, consumers prepay “block credits”; each block reduces the number of credits. Halt thresholds are based on credits, not the live balance. Very deterministic and easy to reason about. Requires a new accounting state.

In the Debt Model, we allow a negative balance up to max_debt, apply a penalty, and halt only when the cap is exceeded.
Avoids abrupt outages but adds economic and legal complexity.

[Pantani]

Another idea is if the consumer pool is empty, a shared provider-side reserve automatically covers N blocks, then the chain must repay. Great for reliability, but introduces moral hazard and governance policy overhead.

[julienrbrt]

The problem with panicking is the reboot recovery. Once validators restart the node, in what state are you? Won't IBC channel expire and you'll be stuck in that state if you are down for too long?

I think the second option of having an ante handler blacklist msgs until the chain payment state is confirmed is way better. For validators this shouldn't matter to much as they system is already configured and you don't introduce a manual step for them (restarting the node), because a chain forgot to pay.

[tbruyelle]

I think the second option of having an ante handler blacklist msgs until the chain payment state is confirmed is way better.

I prefer it as well, I am just wondering how expensive is a chain that produces empty blocks vs no process.

[Pantani]

If we don't panic, the chain is better, but if we do, we need to alert the user somehow and give some time to fix it before panicking

[tbruyelle]

If we don't panic, the chain is better, but if we do, we need to alert the user somehow and give some time to fix it before panicking

I have never seen a blockchain that sends notification, is it even possible ? Probably it needs to be an off-chain thing.

[Pantani]

We can show in the logs that for each block we sent a warning log

[tbruyelle]

About the timeout, maybe we can rely on a social timeout. Once the consumer chain has stopped to accept transactions for too long, validators will start to shut it down by themselves.