Re-design processor/coordinator interactions

[kayabaNerve]

The current message queue is an interactive ordered process not amenable to rebuilding one but not both.

Interactions should be designed as an append-only series of data blobs queriable at any time in a way either can successfully be rebuilt. This would also parallelize the ordering, improving performance and preventing the risk of deadlocks.

[kayabaNerve]

I'm unsure we can get to this prior to mainnet :///

[kayabaNerve]

processor-smash is premised on reading from the message-queue, then internally dispatching without locks/blocking yet queueing. It accordingly doesn't mean this issue would improve performance, and has already resolved the deadlock concerns.

The ability to rebuild remains its own active question.

[kayabaNerve]

The Processor, if rebuilding, should be able to reset the message-queue's counter and simply re-read the full event stream. The once concern is if the old Processor sent preprocesses, and then the new Processor sent preprocesses (de-duplicated), and then the new Processor sent signature shares. We'd need to mark all in-process signing attempts as invalid.

The Coordinator, if rebuilding, has to wait for a new Tendermint block and only start participating then. The Coordinator only needs to handle key_gen::ProcessorMessage and substrate::ProcessorMessage, but can similarly reset the counter and re-read the event stream.

If both are rebuilding, it should be the existing commentary, but with the option to reset the message-queue's message history entirely.

[kayabaNerve]

Provided transactions announcing a Batch occur by receiving a preprocess from the Processor for that Batch. The above commentary is inaccurate accordingly.