Processing chain data work unstable

[EvgeniyZZ]

Versions
Skaled: 3.16.0-beta.3(4)
IMA: 1.3.5-beta.0
IMA contract mainnet: ima.1.3.4-rc.0( for both version 1.3.5-beta.0)
Skale Manager : 1.9.2-stable.0

Environment
20 active nodes on the network
4 schains up and running

Preconditions
Update process to release versions

Steps to Reproduce/Description
Observe logs after update

Expected behavior
During init of IMA agent SNB should collect all information about other schain each hour:
2023-01-23 02:39:41.749: SNB WORKER Full init compete for in-worker SNB server skale_observer_worker_server
2023-01-23 03:39:43.613: SNB WORKER Full init compete for in-worker SNB server skale_observer_worker_server
2023-01-23 04:39:45.392: SNB WORKER Full init compete for in-worker SNB server skale_observer_worker_server
2023-01-23 05:39:47.213: SNB WORKER Full init compete for in-worker SNB server skale_observer_worker_server

Actual result
In some cases SNB init failed, see more logs below.
2023-01-23 05:39:47.213: SNB WORKER Full init compete for in-worker SNB server skale_observer_worker_server
2023-01-23 06:39:49.155: SNB WORKER Full init compete for in-worker SNB server skale_observer_worker_server
2023-01-23 08:39:52.559: SNB WORKER Full init compete for in-worker SNB server skale_observer_worker_server
2023-01-23 09:39:54.263: SNB WORKER Full init compete for in-worker SNB server skale_observer_worker_server
Init failed at 07:39
NOTE: Not related to connect ECONNREFUSED

SNBFail.txt
SNBSuccess.txt

Tasks

Preview Give feedback

1. Fix BLS signatures gathering stuck #1595
   IMA bug release:2.2 +3
   

[PolinaKiporenko]

https://github.com/skalenetwork/IMA/blob/develop/docs/SNB.md

[EvgeniyZZ]

Not fixed: 1.3.5-beta.1
2023-02-12 05:07:25.001: SNB WORKER Full init compete for in-worker SNB server skale_observer_worker_server
2023-02-12 06:07:27.384: SNB WORKER Full init compete for in-worker SNB server skale_observer_worker_server
2023-02-12 09:08:16.794: SNB WORKER Full init compete for in-worker SNB server skale_observer_worker_server
2023-02-12 11:07:46.033: SNB WORKER Full init compete for in-worker SNB server skale_observer_worker_server
2023-02-12 12:07:49.828: SNB WORKER Full init compete for in-worker SNB server skale_observer_worker_server
2023-02-12 21:08:31.600: SNB WORKER Full init compete for in-worker SNB server skale_observer_worker_server
2023-02-13 08:09:41.705: SNB WORKER Full init compete for in-worker SNB server skale_observer_worker_server
2023-02-13 12:09:43.303: SNB WORKER Full init compete for in-worker SNB server skale_observer_worker_server

[sergiy-skalelabs]

See PR here #1463

[EvgeniyZZ]

Add information when:

• request info(method, request ID, timestamp) for parallel SNB(didn't see request in logs)
• response in case if parallel SNB was successful or unsuccessful
• request info for single SNB request per each attempt
• response in case if single SNB was successful or unsuccessful

[EvgeniyZZ]

Still reproducible:
IMA Agent: 2.0.0-beta.7
Can be related: https://github.com/skalenetwork/internal-support/issues/855

[DmytroNazarenko]

IMA: 2.0.0-beta.11

[EvgeniyZZ]

Verified: Not fixed
IMA Agent: 2.0.0-beta.12

frameImaB12.txt

Log below describe that only S2M transfer are working.
M2S, S2S not started on particular schain

[DmytroNazarenko]

IMA: 2.0.0-beta.16

[EvgeniyZZ]

Verified: Reopen. In case if first attempt of SNB was unsuccessful cache will not updated at all.
Logs below describe behavior of different chains on the same node.
IMA Agent: 2.0.0-beta.16

imaRefreshCache.txt

[EvgeniyZZ]

Verified: Agent worked several days without any time/preference gaps
IMA Agent: 2.0.0-beta.17