Duplicate events broadcasted to same API server

[CharlieC3]

Describe the bug
When an API server is configured in both a events_observer block in the stacks-node's config TOML, and the STACKS_EVENT_OBSERVER environment variable, stacks-node will broadcast transactions to the target API server twice, causing it to fail over time.

Steps To Reproduce

1. Start a stacks-blockchain-api server
2. Export the STACKS_EVENT_OBSERVER env variable pointing to the API server.
3. Run stacks-node with a config TOML which has a events_observer block pointing to the API server

Expected behavior
The stacks-node should detect if there are duplicate event server entries between the events_observer block in the TOML and the STACKS_EVENT_OBSERVER env variable. If duplicate entries are found, print a log message saying so and exit.

Environment (please complete the following information):
stacks-node version 2.0.11.2.0

More info can be found here.

[gregorycoppola]

Sounds like something we should be doing!

[pseudozach]

is there any way to get out of this? And does this lead to node sync to fail? I'm having this issue on my testnet node.

I've unset the env var but behavior is the same.

ERRO [1631209353.116014] [testnet/stacks-node/src/event_dispatcher.rs:124] [chains-coordinator] Event dispatcher: POST 127.0.0.1:3700/http://127.0.0.1:3700/new_block failed with error Response { status: InternalServerError, headers: Headers { headers: {HeaderName("content-length"): [HeaderValue { inner: "12" }], HeaderName("date"): [HeaderValue { inner: "Thu, 09 Sep 2021 17:42:33 GMT" }], HeaderName("content-type"): [HeaderValue { inner: "application/json; charset=utf-8" }], HeaderName("x-powered-by"): [HeaderValue { inner: "Express" }], HeaderName("connection"): [HeaderValue { inner: "keep-alive" }], HeaderName("etag"): [HeaderValue { inner: "W/"c-3Rk1bge0s6VuJbi+S2m2iU0UGdY"" }], HeaderName("keep-alive"): [HeaderValue { inner: "timeout=5" }]} }, version: None, sender: Some(Sender { .. }), receiver: Receiver { .. }, body: Body { reader: "", length: Some(12) }, local: TypeMap }

[CharlieC3]

@pseudozach Unfortunately once a data failure occurs on the stacks-blockchain-api side of things and all subsequent events fail to broadcast, it is very difficult to resolve, and would involve manually editing the stacks-node's database.

Your best bet at the moment would be to copy the data from another stacks-node and API to the corrupted pair if you have one, or sync from genesis.

[aulneau]

We just ran into this with two of our api/nodes which is really unfortunate -- curious if there has been any update on this? cc @gregorycoppola

[gregorycoppola]

I actually lost track of this bug.

@kantai @jcnelson Is this an easy thing to fix?

[kantai]

@aulneau: We just ran into this with two of our api/nodes which is really unfortunate -- curious if there has been any update on this?

Did you run into this issue because the same server was configured twice (i.e., either multiple entries for the same server in the config file or one in the config file and one in the environment variable?)

Just trying to figure out if that's related to this issue (which is about a duplication in the event observer list).

@gregorycoppola: Is this an easy thing to fix?

Probably! I think if we just did some simple deduplication when the event observer is instantiated, that would resolve this issue.

[aulneau]

@kantai I asked @wileyj and actually our configuration is not set up that way -- there is only one event emitter configured (via the toml).

[kantai]

@kantai I asked @wileyj and actually our configuration is not set up that way -- there is only one event emitter configured (via the toml).

Then this is probably unrelated to this issue.

[jcnelson]

Temporarily assigning to @kantai. Please feel free to handle however you see fit.

[kantai]

This seems like it could be somewhat straightforward to fix -- the event_emitter can just check as its broadcasting whether or not a particular endpoint appears twice in the notification chain (or use a Set struct).

This seems like it could be a good first issue, so I'll tag as such. If nobody has cycles for this by next week, I'll also tag as icebox.