Failing to acquire a replication slot puts server into a loop

[balegas]

When unable to acquire a replication slot, the server gets into a loop, eventually exhausting CPU. A problem with a replication slot shouldn't bring down the entire instance.

Consider adding an exponential backoff or another approach to handle the situation more gracefully.

00:35:27.572 [error] :gen_statem {Electric.Registry.Processes, {Electric.Postgres.ReplicationClient, :default, "1bbcafee-cc1c-429d-b89a-1a14578ec67a"}} terminating
--
** (Postgrex.Error) ERROR 55006 (object_in_use) replication slot "electric_slot_production_us_east_1" is active for PID 679
(stdlib 6.0.1) gen_statem.erl:3242: :gen_statem.loop_state_callback_result/11
(stdlib 6.0.1) proc_lib.erl:329: :proc_lib.init_p_do_apply/3
Queue: [info: :start_streaming]
Postponed: []
00:35:27.574 [info] Postgres server version = 160004, system identifier = 7431946106013976826, timeline_id = 1

[alco]

This shouldn't normally happen because Electric acquires an advisory lock before opening a replication connection, so no two Electric instances may try using the same slot simultaneously.

I was able to reproduce this issue only after I had commented out the lock acquisition logic from the code.

Regardless, there shouldn't be a spin loop when opening a replication connection fails for any reason. Looking into that.