Monitor and Alert when routers not correctly filling transfers #4982

    
           await Promise.all( 
        
             transfersToPublish.map(async (transfer) => { 
        
               // new request context with the transfer id 
        
               const { requestContext: _requestContext, methodContext: _methodContext } = createLoggingContext( 
        
                 "pollSubgraph", 
        
                 undefined, 
        
                 transfer!.transferId, 
        
               ); 
        
               try { 
        
                 await mqClient.publish<OriginTransfer>(MQ_EXCHANGE, { 
        
                   body: transfer as OriginTransfer, 
        
                   type: XCALL_MESSAGE_TYPE, 
        
                   routingKey: XCALL_QUEUE, 
        
                 }); 
        
                 await cache.transfers.setBidStatus(transfer?.transferId as string); 
        
                 logger.debug("Published transfer to mq", _requestContext, _methodContext, { transfer }); 
        
               } catch (err: unknown) { 
        
                 logger.error("Error publishing to mq", _requestContext, _methodContext, jsonifyError(err as Error)); 
        
               } 
        
             }), 
        
           );

Overall Cause:

Our routers don’t run rabbitMQ in a HA environment.

MinistroDolar · 2024-02-06T16:19:19Z

Transaction:
https://connextscan.io/tx/0xfd1b7fc34491d0ada8afde841d3491e261e2e96fa519ba71557d125c2fa25c5f

Routers:
https://connextscan.io/router/0xc4ae07f276768a3b74ae8c47bc108a2af0e40eba https://connextscan.io/router/0x6fd84ba95525c4ccd218f2f16f646a08b4b0a598

Ticket:
https://discord.com/channels/454734546869551114/1204446970480042035

MinistroDolar · 2024-02-08T13:46:27Z

Transaction:

https://connextscan.io/tx/0xfd1b7fc34491d0ada8afde841d3491e261e2e96fa519ba71557d125c2fa25c5f

Routers:

https://connextscan.io/router/0xc4ae07f276768a3b74ae8c47bc108a2af0e40eba https://connextscan.io/router/0x6fd84ba95525c4ccd218f2f16f646a08b4b0a598

Ticket:

https://discord.com/channels/454734546869551114/1204446970480042035

Adding 2 more. Same destination chain and routers

Transaction:
https://connextscan.io/tx/0xbf10c1f5ff68959ed45ea595d28b719a5c6d80a67cb9409005117d66172e1928

Ticket:
https://discord.com/channels/454734546869551114/1205144727813619744

Transaction:
https://connextscan.io/tx/0x50f3bd5d77009b420e095e3647f1839e2ffd4d3ec0b71bc3d3dcf431109b0a14

Ticket:
https://discord.com/channels/454734546869551114/1205235577650413639

MinistroDolar · 2024-02-14T15:04:41Z

Case 1: 3.2 Metis from Andromeda to Mainnet

Transaction:
https://connextscan.io/tx/0x48781fdf83b7036b46a7993b685d1c37a2341f5ee36f65410035293e7f6a4c73

Routers:
https://connextscan.io/router/0x8cb19ce8eedf740389d428879a876a3b030b9170

Ticket:
https://discord.com/channels/454734546869551114/1207316640471977994

Case 2: 304 USDT from BNB to Arbitrum

Transaction:
https://connextscan.io/tx/0x287a9edb884664bcc5d5d71843593f43fa852d00a04178fe2446f0a9949b3d80

Routers that could fulfill the txn:
https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4
https://connextscan.io/router/0x49a9e7ec76bc8fdf658d09557305170d9f01d2fa
https://connextscan.io/router/0xc4ae07f276768a3b74ae8c47bc108a2af0e40eba
https://connextscan.io/router/0xf26c772c0ff3a6036bddabdaba22cf65eca9f97c

Ticket:
https://discord.com/channels/454734546869551114/1207336207097528360

Tommy-R8 · 2024-02-14T18:05:51Z

Transaction: https://connextscan.io/tx/0xdcae3368f11ee562ba51dbc57ef0b62d4eaaa3d1a8ab6d675e9bef3606863905
Ticket 152 on Intercom

Transaction: https://connextscan.io/tx/0x032a1173d7e868b3cd0376d537253bdb3401984eba0efc85e64bd6b0c8715106
Ticket 154 on Intercom

Transaction: https://connextscan.io/tx/0xad762da074eeafd097160ea9af9fd926e0571f4957d6478c46b69433ef17864
Ticket 150 on Intercom

Transaction: https://connextscan.io/tx/0x0440af350352d17d8b9468b538f62e646857139b724ed16005e492f5f8a02942
Ticket 153 on Intercom

MinistroDolar · 2024-03-20T19:15:01Z

USDC from BNB to Metis

Tx: https://connextscan.io/tx/0x4abaeed58105c2ddd9756077b91a30aae2fb45ed2d48ec37bc81bb2140a6cbab

Router: https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4

Ticket: https://discord.com/channels/454734546869551114/1220087029002403960

MinistroDolar · 2024-03-20T19:52:16Z

WETH from Polygon to Mode

Tx: https://connextscan.io/tx/0x2e71a209f83ecf76ea07f105efc56527b525ffaeeab0bb31d96b62cf56296887

Router: https://connextscan.io/router/0x8cb19ce8eedf740389d428879a876a3b030b9170

Ticket: https://discord.com/channels/454734546869551114/1220094862041550898

Tommy-R8 · 2024-03-21T18:01:53Z

DAI from Arbitrum to Gnosis
Transaction: https://connextscan.io/tx/0xc06fc8a855260ae31360822a112671b6f463f1fc84bcb284cf34c47a6df10e01
Routers: https://connextscan.io/router/0x048a5ecc705c280b2248aeff88fd581abbeb8587
https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4
Ticket 321 on Discord

Tommy-R8 · 2024-03-21T19:03:02Z

USDT from BNB to Gnosis
Transaction: https://connextscan.io/tx/0x1d400f73726db87ac71078fc3e24ddc1cdab1c146e4ff7f250322a3b3e9da693
Router: https://connextscan.io/router/0x048a5ecc705c280b2248aeff88fd581abbeb8587
Ticket 322 on Intercom

MinistroDolar · 2024-03-22T21:24:09Z

USDC from Optimism to Polygon

Transaction: https://connextscan.io/tx/0xe2e0a43d6f11e101bab14cb9167a4c50509ac28b5b61960a4545738fbca17f56

Router:
https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4
https://connextscan.io/router/0xc4ae07f276768a3b74ae8c47bc108a2af0e40eba
https://connextscan.io/router/0x33b2ad85f7dba818e719fb52095dc768e0ed93ec
https://connextscan.io/router/0x58507fed0cb11723dfb6848c92c59cf0bbeb9927

Ticket: https://discord.com/channels/454734546869551114/1220816651587551243

MinistroDolar · 2024-03-22T22:25:44Z

USDT from BNB to Linea

Tx: https://connextscan.io/tx/0x69054b755d9f9e06c4d8b0a253ad304bb05e96ccb3097246c0c54ecadf63347c

Router: https://connextscan.io/router/0x58507fed0cb11723dfb6848c92c59cf0bbeb9927

Ticket: https://discord.com/channels/454734546869551114/1220829143260860546

MinistroDolar · 2024-03-23T13:59:17Z

WETH from Mainnet to Linea

Tx: https://connextscan.io/tx/0x6b7531d26a00b411769ee2fb6e3991381438d84b6840a0dc6c944bcfd9b3327b

Router: https://connextscan.io/router/0x5d527765252003acee6545416f6a9c8d15ae8402

Ticket: https://discord.com/channels/454734546869551114/1221092571846283266

MinistroDolar · 2024-03-23T16:35:54Z

WETH from Arbitrum to Linea - this is a weird one because there are FOUR routers with more than enough liquidity and no one is bidding it.

Tx: https://connextscan.io/tx/0x036539cedfe5dca06f83a829f14e0af9ebbb60b6d9eb1c11aefefdac5d83c2ec

Routers:
https://connextscan.io/router/0x58507fed0cb11723dfb6848c92c59cf0bbeb9927
https://connextscan.io/router/0x5d527765252003acee6545416f6a9c8d15ae8402
https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4
https://connextscan.io/router/0xeca085906cb531bdf1f87efa85c5be46aa5c9d2c

Ticket: https://discord.com/channels/454734546869551114/1221132819573637322

Tommy-R8 · 2024-03-24T15:34:49Z

USDC from Arbitrum to Polygon

Transaction: https://connextscan.io/tx/0x7f461965893e29a2f64d4fa943009802b64f6605a67b90147c1e15ee58116ff3?source=search
Routers: https://connextscan.io/router/0xc4ae07f276768a3b74ae8c47bc108a2af0e40eba
https://connextscan.io/router/0x33b2ad85f7dba818e719fb52095dc768e0ed93ec
https://connextscan.io/router/0x7ce49752ffa7055622f444df3c69598748cb2e5f
Ticket: https://discord.com/channels/454734546869551114/1221481029932552243/1221481153169592331

Tommy-R8 · 2024-03-25T02:17:10Z

USDC from Polygon to BNB

Transaction: https://connextscan.io/tx/0x33c6f752aa1b500f20fd8a475778d9ab589737c2e12cf0153c80384d1f81e836
Router: https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4
Ticket: https://discord.com/channels/454734546869551114/1221643363527430304/1221643424218742805

wanglonghong · 2024-03-25T15:48:36Z

The sequencer endpoint to monitor router status is live:

Request: 
https://sequencer.mainnet.connext.ninja/router-status/{ROUTER_ADDRESS}

Response:
{
"lastActiveTimestamp": 0,
"lastBidTimestamp": {...}
}

Need further discussion to figure out the best place where we integrate it.

MinistroDolar · 2024-03-29T14:27:25Z

ezETH from BNB Chain to Mode

Tx: https://connextscan.io/tx/0xb6967b2bd79a88723556e500790a87a1fc8e7142982cfebdd709a80ae5213818

Routers: (all of them seems online but for some reasons which idk they're not bidding this tx)
https://connextscan.io/router/0x6fd84ba95525c4ccd218f2f16f646a08b4b0a598
https://connextscan.io/router/0x8cb19ce8eedf740389d428879a876a3b030b9170
https://connextscan.io/router/0xeca085906cb531bdf1f87efa85c5be46aa5c9d2c

Ticket: Intercom 375

MinistroDolar · 2024-03-29T14:41:02Z

WETH from Arbitrum to Linea

tx: https://connextscan.io/tx/0x7c6348731cbe30c490cef3f5513ab6e71d1dbeb00c460738a14ad40d07d706f3

Routers:
https://connextscan.io/router/0x5d527765252003acee6545416f6a9c8d15ae8402 (this router seems not bidding txns higher than 1 eth)

Ticket: Intercom 375

MinistroDolar · 2024-03-30T18:45:09Z

USDC from BNB to Mainnet

Tx: https://connextscan.io/tx/0xdecb46aeffa2b8ccfd3168573b106b214781ad8b6076c24074595b0e35ee4d30

Router: https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4

Ticket: https://discord.com/channels/454734546869551114/1223691495215730808

MinistroDolar · 2024-03-30T18:50:18Z

WETH from Base to Mode

Tx: https://connextscan.io/tx/0x39bea430f93c56a14f3e65ae473992eea6ee9b60cb878036bb4e3ed7ad532ddd

Routers:

https://connextscan.io/router/0x6fd84ba95525c4ccd218f2f16f646a08b4b0a598

https://connextscan.io/router/0x8cb19ce8eedf740389d428879a876a3b030b9170

Ticket: https://discord.com/channels/454734546869551114/1223642596073799823

MinistroDolar · 2024-03-30T19:36:57Z

WETH from Optimism to Linea

Tx1: https://connextscan.io/tx/0xcca1adf36cf9861e1755df8185bad31ed9e39a72fbbaf6384526af65a7891156

Tx2: https://connextscan.io/tx/0xb875da77c4443f484dee4ca078e0246a852a6d47e2eaccc311aebd864e3f957d

Routers: https://connextscan.io/router/0x5d527765252003acee6545416f6a9c8d15ae8402

Ticket: https://discord.com/channels/454734546869551114/1223713534093824010

MinistroDolar · 2024-03-31T14:43:21Z

WETH from Ethereum to BNB Chain

tx: https://connextscan.io/tx/0x72dc641a40393ea8d33e5d5113c1446ef67bf0118e33f3cd3f8b76264845f78f

routers:
https://connextscan.io/router/0x6fd84ba95525c4ccd218f2f16f646a08b4b0a598
https://connextscan.io/router/0xf26c772c0ff3a6036bddabdaba22cf65eca9f97c

discord: https://discord.com/channels/454734546869551114/1224002837558591499

MinistroDolar · 2024-03-31T17:58:47Z

WETH from Arbitrum to Linea

TX: https://connextscan.io/tx/0x0139dd60817e3d7fc44a81b12d285d54df025eebc48fb03015ea78b2aeeb64ef

Router: https://connextscan.io/router/0x5d527765252003acee6545416f6a9c8d15ae8402

Ticket: Intercom 375

MinistroDolar · 2024-03-31T18:24:49Z

ezETH from Ethereum to Mode (this txn should be fulfilled by at least 2 routers)

tx: https://connextscan.io/tx/0xfcbc79292298008de6f38de6428d9316c1fc7fb3a8d9f27b7c18dd27f819503f

Routers:
https://connextscan.io/router/0x6fd84ba95525c4ccd218f2f16f646a08b4b0a598
https://connextscan.io/router/0xeca085906cb531bdf1f87efa85c5be46aa5c9d2c
https://connextscan.io/router/0x8cb19ce8eedf740389d428879a876a3b030b9170

Ticket: https://discord.com/channels/454734546869551114/1224052652992626719

MinistroDolar · 2024-04-01T21:50:52Z

USDC from Polygon to Metis

Tx: https://connextscan.io/tx/0x54c7bf3a8630180b6d1468670fd1838de7540087234d1e1228e1a01e501f9725?source=search

Router: https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4

Ticket: https://discord.com/channels/454734546869551114/1224469492352946256

MinistroDolar · 2024-04-02T19:40:50Z

WETH from Arbitrum to Linea

Anyone knows why this txn isn't getting bids? there are 4 online routers with enough ETH on Linea.

tx:
https://connextscan.io/tx/0xa338b877374d592b38c5f887e33a2f7a5fc1336cad8cb4bdc64cd50c4f7ec816

routers:
https://connextscan.io/router/0x58507fed0cb11723dfb6848c92c59cf0bbeb9927
https://connextscan.io/router/0x5d527765252003acee6545416f6a9c8d15ae8402
https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4
https://connextscan.io/router/0xeca085906cb531bdf1f87efa85c5be46aa5c9d2c

discord: https://discord.com/channels/454734546869551114/1224426322940661831

@just-a-node @wanglonghong

alexwhte · 2024-04-04T18:06:50Z

Were looking into this

MinistroDolar · 2024-04-06T15:28:54Z

ezETH from Linea to Mode

tx: https://connextscan.io/tx/0xde25907d24629049200108196e4604d8e67949b1710bb27a8418571e4a94faa8

routers (there are 4 active routers with enough liquidity, 2 of them also support Linea):

https://connextscan.io/router/0x8cb19ce8eedf740389d428879a876a3b030b9170
https://connextscan.io/router/0x6fd84ba95525c4ccd218f2f16f646a08b4b0a598
https://connextscan.io/router/0xeca085906cb531bdf1f87efa85c5be46aa5c9d2c
https://connextscan.io/router/0xc770ec66052fe77ff2ef9edf9558236e2d1c41ef

ticket: https://discord.com/channels/454734546869551114/1226187490868859060

MinistroDolar · 2024-04-09T11:56:31Z

ETH from Arbitrum to Mode

tx: https://connextscan.io/tx/0x6180ee5bc2fdb15d0eb57890d57dc0fa998cd2bb0decc2b6e5827dd05d0d43ea

Routers:
https://connextscan.io/router/0x6fd84ba95525c4ccd218f2f16f646a08b4b0a598
https://connextscan.io/router/0x8cb19ce8eedf740389d428879a876a3b030b9170
(both are working fine, i don't understand why they don't bid this txn)

ticket: https://discord.com/channels/454734546869551114/1227202633626816604

MinistroDolar · 2024-04-11T18:38:49Z

ETH from Polygon to Metis

tx: https://connextscan.io/tx/0xd2f698802d30ff553c64cb3bdd2a296b83971f4c2815d81d4da07384d9ac9f5e

routers: https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4 (this router is active and bidding txns going to Metis, I'm not sure of what's happening with the txn above)

discord: https://discord.com/channels/454734546869551114/1228041371500154933

MinistroDolar · 2024-04-12T23:19:35Z

USDT from BNB to Ethereum

This is a special case, the user sent a high amount (55.700 USDT) and there are enough router liquidity to complete it, but it is distributed among 3 diff (active) routers. Multirouter bidding seems not working properly in this case.

Tx: https://connextscan.io/tx/0xe0422ea721f45fd4b4f9e3e713a17ac4eb78c841c296577c248353e5a32c6452

Routers:
https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4
https://connextscan.io/router/0xf26c772c0ff3a6036bddabdaba22cf65eca9f97c
https://connextscan.io/router/0xc4ae07f276768a3b74ae8c47bc108a2af0e40eba

Discord: https://discord.com/channels/454734546869551114/1228470019390832751

@oncall

MinistroDolar · 2024-04-15T19:46:40Z

USDC from BNB to Metis

Only 3.34 USDC txn and 2 active routers, but it is going through the slow path

tx: https://connextscan.io/tx/0x3e968ba73a0216ef9bf208b1f854c4b63eebfb0a2bda0bf851019bde58f3b932

routers:
https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4
https://connextscan.io/router/0xf26c772c0ff3a6036bddabdaba22cf65eca9f97c

Discord: https://discord.com/channels/454734546869551114/1229507135998791681

preethamr added the Enhance 🧘 Issue type: Enhancement Request label Oct 7, 2023

alexwhte added the core label Oct 23, 2023

alexwhte removed the core label Nov 15, 2023

alexwhte added Bug Issue type: Bug and removed Enhance 🧘 Issue type: Enhancement Request labels Jan 29, 2024

alexwhte changed the title ~~Monitor and Alert when a activate router stops bidding~~ Monitor and Alert when routers not correctly filling transfers Jan 29, 2024

alexwhte assigned just-a-node and wanglonghong Feb 2, 2024

MinistroDolar mentioned this issue Feb 8, 2024

[ST] Slow path longer than expected #5680

Closed

wanglonghong closed this as completed Mar 25, 2024

alexwhte reopened this Mar 25, 2024

0xanedi pinned this issue Mar 31, 2024

Monitor and Alert when routers not correctly filling transfers #4982

Monitor and Alert when routers not correctly filling transfers #4982

Comments

preethamr commented Oct 7, 2023

Problem

Impact

Proposed Solution

Acceptance Criteria

just-a-node commented Jan 13, 2024

Tommy-R8 commented Jan 16, 2024

MinistroDolar commented Jan 17, 2024

Tommy-R8 commented Jan 19, 2024

Tommy-R8 commented Jan 19, 2024

MinistroDolar commented Jan 22, 2024

Tommy-R8 commented Jan 23, 2024

Tommy-R8 commented Jan 23, 2024

Tommy-R8 commented Jan 23, 2024

Tommy-R8 commented Jan 26, 2024

Tommy-R8 commented Jan 27, 2024

ValerywArz commented Jan 27, 2024

Tommy-R8 commented Jan 28, 2024

alexwhte commented Jan 31, 2024

Tommy-R8 commented Feb 1, 2024

Tommy-R8 commented Feb 2, 2024

preethamr commented Feb 3, 2024

MinistroDolar commented Feb 6, 2024

MinistroDolar commented Feb 8, 2024 • edited Loading

MinistroDolar commented Feb 14, 2024

Tommy-R8 commented Feb 14, 2024

MinistroDolar commented Mar 20, 2024

MinistroDolar commented Mar 20, 2024

Tommy-R8 commented Mar 21, 2024

Tommy-R8 commented Mar 21, 2024

MinistroDolar commented Mar 22, 2024

MinistroDolar commented Mar 22, 2024

MinistroDolar commented Mar 23, 2024

MinistroDolar commented Mar 23, 2024

Tommy-R8 commented Mar 24, 2024

Tommy-R8 commented Mar 25, 2024

wanglonghong commented Mar 25, 2024 • edited Loading

MinistroDolar commented Mar 29, 2024

MinistroDolar commented Mar 29, 2024

MinistroDolar commented Mar 30, 2024

MinistroDolar commented Mar 30, 2024

MinistroDolar commented Mar 30, 2024

MinistroDolar commented Mar 31, 2024

MinistroDolar commented Mar 31, 2024

MinistroDolar commented Mar 31, 2024

MinistroDolar commented Apr 1, 2024

MinistroDolar commented Apr 2, 2024

alexwhte commented Apr 4, 2024

MinistroDolar commented Apr 6, 2024

MinistroDolar commented Apr 9, 2024

MinistroDolar commented Apr 11, 2024

MinistroDolar commented Apr 12, 2024

MinistroDolar commented Apr 15, 2024

MinistroDolar commented Feb 8, 2024 •

edited

Loading

wanglonghong commented Mar 25, 2024 •

edited

Loading