grandine stops all validator activity when slasher constructs slashings

[InvisibleSymbol]

network: holesky
validators: 3.3k attached through web3signer

some of them appear to be invalid (i.e the same attester index doesnt appear in both logged attestations), but there are indeed valid detected double votes, such as validator 967119 for example.

noticed that to they have been detected after they were already slashed on the chain? beaconcha.in says the slashing for 967119 was included in slot 889285, but the below logs are at slot 889349? thats exactly 64 slots later.

anyways this caused all attached validators to be offline until restarted

[2024-01-30T00:29:37.523+00:00] [INFO] [validator::validator] started work on execution payload with id Capella(0x02d2c34f5115bade) for head 0x9fd0b5e41a7b8cdeaf6ca125f935db1a25ec8da72592f9632c865754ed64cbf3 at slot 889349
[2024-01-30T00:29:39.000+00:00] [INFO] [validator::validator] ProposeFourth tick in slot 889348
[2024-01-30T00:29:40.000+00:00] [INFO] [validator::validator] Attest tick in slot 889348
[2024-01-30T00:29:40.129+00:00] [INFO] [validator::validator] validators [1316016, 1316864, 1316129, 1315144, 1315758, 1316669, 1316325, 1317236, 1316831, 1315085, 1316020, 1315244, 1315087, 1316177, 1316882, 1317008, 1315030, 1317107, 1317423, 1317202, 1317508, 1315176, 1316002, 1317138, 1315819, 1315150, 1316353, 1315211, 1318077, 1316036, 1315179, 1315980, 1316065, 1317208, 1316040, 1315533, 1315364, 1315370, 1317989, 1315004, 1316575, 1315485, 1315497, 1317131, 1317193, 1315692, 1318138, 1316548, 1317873, 1315623, 1317514, 1318060, 1315543, 1317327, 1315396, 1317333, 1317268, 1315771, 1318058, 1316241, 1317655, 1317123, 1316027, 1317394, 1317080, 1316420, 1318074, 1316725, 1315882, 1315261, 1316391, 1315329, 1317175, 1318168, 1318001, 1316778, 1317069, 1316403, 1318084, 1315339, 1317792, 1316105, 1316008, 1315489, 1315208, 1316784, 1316702, 1315200, 1317062, 1315734, 1318160, 1315378, 1315897, 1316127, 1315262, 1315919, 1316017, 1316908, 1315305, 1316792, 1317850, 1317300, 1315937, 1315051, 1317815, 1317302, 1315357, 1316518, 1317766, 1316868, 1316641] attesting in slot 889348
[2024-01-30T00:29:41.534+00:00] [INFO] [eth1::eth1_chain] started Eth1 block download task
[2024-01-30T00:29:41.536+00:00] [INFO] [eth1::download_manager] downloaded 1 Eth1 blocks from block 836233 to block 836233
[2024-01-30T00:29:43.001+00:00] [INFO] [validator::validator] AttestFourth tick in slot 889348
[2024-01-30T00:29:44.001+00:00] [INFO] [validator::validator] Aggregate tick in slot 889348
[2024-01-30T00:29:44.003+00:00] [INFO] [validator::validator] validators [1315758, 1316725, 1316105] aggregating in slot 889348
[2024-01-30T00:29:46.538+00:00] [INFO] [eth1::eth1_chain] started Eth1 block download task
[2024-01-30T00:29:47.001+00:00] [INFO] [validator::validator] AggregateFourth tick in slot 889348
[2024-01-30T00:29:48.094+00:00] [INFO] [validator::validator] Propose tick in slot 889349
[2024-01-30T00:29:50.284+00:00] [INFO] [p2p::network] [Peers: 249/256] received beacon block as gossip (slot: 889349, root: 0x03d7a4cef2a668618b54c0ba91e40918bf4ee28b626d8f454dd69c8f780df23e)
[2024-01-30T00:29:50.645+00:00] [INFO] [validator::validator] started work on execution payload with id Capella(0x02a931aac064c2fe) for head 0x03d7a4cef2a668618b54c0ba91e40918bf4ee28b626d8f454dd69c8f780df23e at slot 889350
[2024-01-30T00:29:51.001+00:00] [INFO] [validator::validator] ProposeFourth tick in slot 889349
[2024-01-30T00:29:51.540+00:00] [INFO] [eth1::eth1_chain] started Eth1 block download task
[2024-01-30T00:29:51.544+00:00] [INFO] [eth1::download_manager] downloaded 1 Eth1 blocks from block 836234 to block 836234
[2024-01-30T00:29:52.001+00:00] [INFO] [validator::validator] Attest tick in slot 889349
[2024-01-30T00:29:52.177+00:00] [INFO] [validator::validator] validators [1315802, 1315777, 1316411, 1316063, 1317331, 1317689, 1315739, 1316584, 1315573, 1316705, 1316911, 1316821, 1315559, 1317556, 1316979, 1317493, 1315048, 1317094, 1316846, 1317602, 1316216, 1316313, 1318047, 1315740, 1317400, 1316220, 1317132, 1316441, 1317524, 1317857, 1315584, 1317951, 1317943, 1318171, 1316513, 1315591, 1317491, 1315574, 1317976, 1316333, 1316218, 1315059, 1316737, 1315231, 1316149, 1315173, 1316734, 1315912, 1316095, 1317323, 1317450, 1317337, 1316061, 1315215, 1315005, 1316157, 1316787, 1316497, 1316351, 1317097, 1316529, 1315228, 1316021, 1316929, 1317271, 1315546, 1315526, 1315913, 1316773, 1316327, 1317927, 1316223, 1317999, 1315983, 1315995, 1318146, 1315035, 1318139, 1317624, 1315071, 1316053, 1316245, 1317324, 1315587, 1315006, 1315952, 1316416, 1318153, 1317142, 1315205, 1317710, 1318274, 1317286, 1315227, 1315997, 1317283, 1317784, 1318174, 1317718, 1315188, 1317146, 1317795, 1316515, 1316532, 1315956, 1316907, 1316736, 1318180, 1316756, 1317677, 1315308, 1317950, 1315814, 1316273, 1317104, 1315924, 1315326, 1318188, 1318073, 1317073, 1316268, 1315307] attesting in slot 889349
[2024-01-30T00:29:54.162+00:00] [INFO] [slasher::slasher] attester slashing constructed: ExplainedAttesterSlashing { slashing: AttesterSlashing { attestation_1: IndexedAttestation { attesting_indices: [946254], data: AttestationData { slot: 889249, index: 11, beacon_block_root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1, source: Checkpoint { epoch: 27788, root: 0x4bae73a8f746a022a42b49cd5a192da5112d87a3e1f493ebd9afef4dad221fff }, target: Checkpoint { epoch: 27789, root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1 } }, signature: 0x977f7b0b51b21bb5f1d440e836cf061c00130e7c57a7d200dc2d3a6798dbcb8cb7ef85d0824f1dbaeb81da64f2405e7810c9c95ab7f11108800b3d4ce6c87c991367ef584a3d9fa6ace80e873f0d13db58e88b6cf08b5a405ed0e0acb657b8c7 }, attestation_2: IndexedAttestation { attesting_indices: [985739], data: AttestationData { slot: 889249, index: 11, beacon_block_root: 0x694d5b71e186639dc54e0b509997744a0abbb7e56dde613139c7893602bf11d2, source: Checkpoint { epoch: 27788, root: 0x4bae73a8f746a022a42b49cd5a192da5112d87a3e1f493ebd9afef4dad221fff }, target: Checkpoint { epoch: 27789, root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1 } }, signature: 0xae78c6d5614c2382f8fe7563cd592cfc88754fa7bf61e29b20dc34cce4996964bcb38235d8697ea77fd1786d9a520fc11122a3b76d0924b30a3f4ed30eb354c2405546a01d0830a20231857b11789f208436851f33734a78fde94b7c58e34185 } }, reason: DoubleVote }
[2024-01-30T00:29:54.162+00:00] [INFO] [slasher::slasher] attester slashing constructed: ExplainedAttesterSlashing { slashing: AttesterSlashing { attestation_1: IndexedAttestation { attesting_indices: [972765], data: AttestationData { slot: 889249, index: 39, beacon_block_root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1, source: Checkpoint { epoch: 27788, root: 0x4bae73a8f746a022a42b49cd5a192da5112d87a3e1f493ebd9afef4dad221fff }, target: Checkpoint { epoch: 27789, root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1 } }, signature: 0xb8aef84c3def06d5d50221cb3386620d7da8daa07cd46af3694875740df5ad2a62dbe268ef0271d45dac10a6e0f1db540c35c6be6890a5065213cd1a0e0b3c0817f509d410110d498246fba873c44c792cefad49ffc6185ad7f7b8ce9a679af4 }, attestation_2: IndexedAttestation { attesting_indices: [960499], data: AttestationData { slot: 889249, index: 39, beacon_block_root: 0x694d5b71e186639dc54e0b509997744a0abbb7e56dde613139c7893602bf11d2, source: Checkpoint { epoch: 27788, root: 0x4bae73a8f746a022a42b49cd5a192da5112d87a3e1f493ebd9afef4dad221fff }, target: Checkpoint { epoch: 27789, root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1 } }, signature: 0xae096981ddcb8e0f3159b9b097e4e6e97b3d05515294d0decfdd6a5dd13d03f2607a164bea71679da43d1813ba64c1e9113ba26f6a0146ef4d9cdaa73a68dbd7092e072a32bf24cd26caa1265c3507cf1766f28f7ca5cbaaefcac212505651ae } }, reason: DoubleVote }
[2024-01-30T00:29:54.165+00:00] [WARN] [fork_choice_control::mutator] attester slashing rejected (error: no attesters slashed, origin: Own)
[2024-01-30T00:29:54.165+00:00] [WARN] [fork_choice_control::mutator] attester slashing rejected (error: no attesters slashed, origin: Own)
[2024-01-30T00:29:55.405+00:00] [INFO] [slasher::slasher] attester slashing constructed: ExplainedAttesterSlashing { slashing: AttesterSlashing { attestation_1: IndexedAttestation { attesting_indices: [967119], data: AttestationData { slot: 889249, index: 21, beacon_block_root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1, source: Checkpoint { epoch: 27788, root: 0x4bae73a8f746a022a42b49cd5a192da5112d87a3e1f493ebd9afef4dad221fff }, target: Checkpoint { epoch: 27789, root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1 } }, signature: 0xa3a5347a40cdf788dde1b14fc6d79c1f3e2db7b84d5e6529990c5b1f2b89b4a24773bec1ebe346a00dd10f0753e5cf7e11ac55cfe67ded16a0d5312f9652b19fa49218037bbff0120eea1f52be52cd06ce0993bf17d7f0f80beb082c11e4167a }, attestation_2: IndexedAttestation { attesting_indices: [967119], data: AttestationData { slot: 889249, index: 21, beacon_block_root: 0x694d5b71e186639dc54e0b509997744a0abbb7e56dde613139c7893602bf11d2, source: Checkpoint { epoch: 27788, root: 0x4bae73a8f746a022a42b49cd5a192da5112d87a3e1f493ebd9afef4dad221fff }, target: Checkpoint { epoch: 27789, root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1 } }, signature: 0x9140a0243f531391a4b090f91d74e9e027958a69a373ca1055f5be9501a855a4be5ed7bc3bd97efb1dcc2cec501f58300573138e260046f230ee5f1f945e34812dc6a602789937cf3fa730855801db6a9c106c978ffe94d11a8e9900a8c3317e } }, reason: DoubleVote }
[2024-01-30T00:29:55.407+00:00] [WARN] [fork_choice_control::mutator] attester slashing rejected (error: no attesters slashed, origin: Own)
[2024-01-30T00:29:55.408+00:00] [INFO] [slasher::slasher] attester slashing constructed: ExplainedAttesterSlashing { slashing: AttesterSlashing { attestation_1: IndexedAttestation { attesting_indices: [944920], data: AttestationData { slot: 889249, index: 44, beacon_block_root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1, source: Checkpoint { epoch: 27788, root: 0x4bae73a8f746a022a42b49cd5a192da5112d87a3e1f493ebd9afef4dad221fff }, target: Checkpoint { epoch: 27789, root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1 } }, signature: 0x828b46381ba93a3d0b82cec2278feb605e3067c627a02d3c5d36a58a4a9001fe9fb1e5c021ac83a60ac97921e81e18c60bcab1afce32a48b04044e9fb246b983de0e470d0a76ea9555eb32611934368d143f60ba3be1cc2b9d3c67501df72278 }, attestation_2: IndexedAttestation { attesting_indices: [944920], data: AttestationData { slot: 889249, index: 44, beacon_block_root: 0x694d5b71e186639dc54e0b509997744a0abbb7e56dde613139c7893602bf11d2, source: Checkpoint { epoch: 27788, root: 0x4bae73a8f746a022a42b49cd5a192da5112d87a3e1f493ebd9afef4dad221fff }, target: Checkpoint { epoch: 27789, root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1 } }, signature: 0xa25ac0a17263e93a6fd6744fecf6b0903253f50ea7f3f0b28a9ba8b08ba58bc9a9afca83aa5b0b19fa8bba03fde2f4d0023d4aa62777d7888ec4c39f3e23cd1dc20dadf20e0e412180178b162c2cfb01b83a62bb59bf348949b28a6fba49a32b } }, reason: DoubleVote }
[2024-01-30T00:29:55.408+00:00] [INFO] [slasher::slasher] attester slashing constructed: ExplainedAttesterSlashing { slashing: AttesterSlashing { attestation_1: IndexedAttestation { attesting_indices: [979682], data: AttestationData { slot: 889249, index: 41, beacon_block_root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1, source: Checkpoint { epoch: 27788, root: 0x4bae73a8f746a022a42b49cd5a192da5112d87a3e1f493ebd9afef4dad221fff }, target: Checkpoint { epoch: 27789, root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1 } }, signature: 0xa5a8de5ead44cabca0d238ace4358425fdf0c32a2dd1b614262502271b8302210cb4ef0086e132aa0656916c58375d180398276bbe651f96f5005551efed66d850a2189137ac2267f96f10db39fac2e2489d1670ec66f5457eaf9352df0bc2e3 }, attestation_2: IndexedAttestation { attesting_indices: [979682], data: AttestationData { slot: 889249, index: 41, beacon_block_root: 0x694d5b71e186639dc54e0b509997744a0abbb7e56dde613139c7893602bf11d2, source: Checkpoint { epoch: 27788, root: 0x4bae73a8f746a022a42b49cd5a192da5112d87a3e1f493ebd9afef4dad221fff }, target: Checkpoint { epoch: 27789, root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1 } }, signature: 0x8057b0f5a98283cf6f292c769610c1f9ca020670c4fcc6eaa1c30d474a5a46cfba18b1ff166038ea29386915dab35bff1189670c5ebe28dd7d8b2d9a8e27bcc93ba90c59b374b239fd3b341587fa5958ab6f67d39261093011013a3308c3862e } }, reason: DoubleVote }
[2024-01-30T00:29:55.411+00:00] [WARN] [fork_choice_control::mutator] attester slashing rejected (error: no attesters slashed, origin: Own)
[2024-01-30T00:29:55.412+00:00] [WARN] [fork_choice_control::mutator] attester slashing rejected (error: no attesters slashed, origin: Own)
[2024-01-30T00:29:55.576+00:00] [INFO] [slasher::slasher] attester slashing constructed: ExplainedAttesterSlashing { slashing: AttesterSlashing { attestation_1: IndexedAttestation { attesting_indices: [944920], data: AttestationData { slot: 889249, index: 44, beacon_block_root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1, source: Checkpoint { epoch: 27788, root: 0x4bae73a8f746a022a42b49cd5a192da5112d87a3e1f493ebd9afef4dad221fff }, target: Checkpoint { epoch: 27789, root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1 } }, signature: 0x828b46381ba93a3d0b82cec2278feb605e3067c627a02d3c5d36a58a4a9001fe9fb1e5c021ac83a60ac97921e81e18c60bcab1afce32a48b04044e9fb246b983de0e470d0a76ea9555eb32611934368d143f60ba3be1cc2b9d3c67501df72278 }, attestation_2: IndexedAttestation { attesting_indices: [931404], data: AttestationData { slot: 889249, index: 44, beacon_block_root: 0x694d5b71e186639dc54e0b509997744a0abbb7e56dde613139c7893602bf11d2, source: Checkpoint { epoch: 27788, root: 0x4bae73a8f746a022a42b49cd5a192da5112d87a3e1f493ebd9afef4dad221fff }, target: Checkpoint { epoch: 27789, root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1 } }, signature: 0xb6dbc20388ce0bf8a3b79ea01a76c791036592fe9cfab6da9d271757aa17bff594a64cb1370509c78efaae025a02a2730eae6a80076c284bbbe60960acdd1675302a824e8e426e116ce8ac3b05b42c7b78fa090343201eba707d2c9f363a41b3 } }, reason: DoubleVote }
[2024-01-30T00:29:55.578+00:00] [WARN] [fork_choice_control::mutator] attester slashing rejected (error: no attesters slashed, origin: Own)
[2024-01-30T00:29:56.546+00:00] [INFO] [eth1::eth1_chain] started Eth1 block download task
[2024-01-30T00:30:01.547+00:00] [INFO] [eth1::eth1_chain] started Eth1 block download task
[2024-01-30T00:30:02.546+00:00] [INFO] [p2p::network] [Peers: 249/256] received beacon block as gossip (slot: 889350, root: 0x8a7287a9a1b4f6c6c291056ec71fb55394ad2184439e4dc7ab52d26db5973aa6)
[2024-01-30T00:30:06.549+00:00] [INFO] [eth1::eth1_chain] started Eth1 block download task
[2024-01-30T00:30:11.549+00:00] [INFO] [eth1::eth1_chain] started Eth1 block download task
[2024-01-30T00:30:12.963+00:00] [INFO] [p2p::network] [Peers: 249/256] received beacon block as gossip (slot: 889351, root: 0xe85eccd4f0a65d4505e8a5d93fa34d6875c87c2f19b72698360b89bc39f43ceb)
[2024-01-30T00:30:14.721+00:00] [INFO] [metrics::service] metrics sent to external service
[2024-01-30T00:30:16.551+00:00] [INFO] [eth1::eth1_chain] started Eth1 block download task
[2024-01-30T00:30:21.552+00:00] [INFO] [eth1::eth1_chain] started Eth1 block download task
[2024-01-30T00:30:26.285+00:00] [INFO] [p2p::network] [Peers: 252/256] received beacon block as gossip (slot: 889352, root: 0x7a404547bf5d17fd76f105308250b4c993f6063ebf431c3bcedfcfe527e6c022)
[2024-01-30T00:30:26.553+00:00] [INFO] [eth1::eth1_chain] started Eth1 block download task
[2024-01-30T00:30:31.554+00:00] [INFO] [eth1::eth1_chain] started Eth1 block download task
[2024-01-30T00:30:32.401+00:00] [WARN] [p2p::network] [Peers: 251/256] request 107043 to peer 16Uiu2HAmGmFmaKxGkmDTLjA4WAsocfb7GLcSh3FvK9BDvpWvxPSC failed: Stream Timeout
[2024-01-30T00:30:36.555+00:00] [INFO] [eth1::eth1_chain] started Eth1 block download task
[2024-01-30T00:30:36.687+00:00] [INFO] [p2p::network] [Peers: 252/256] received beacon block as gossip (slot: 889353, root: 0x27b8695db5580da2b7b7d9d1c64894ac235ddf46bdcc4c7911409b159e8ec3c1)
[2024-01-30T00:30:41.557+00:00] [INFO] [eth1::eth1_chain] started Eth1 block download task
[2024-01-30T00:30:46.558+00:00] [INFO] [eth1::eth1_chain] started Eth1 block download task
[2024-01-30T00:30:51.560+00:00] [INFO] [eth1::eth1_chain] started Eth1 block download task
[2024-01-30T00:30:56.562+00:00] [INFO] [eth1::eth1_chain] started Eth1 block download task
[2024-01-30T00:31:01.564+00:00] [INFO] [eth1::eth1_chain] started Eth1 block download task

[sauliusgrigaitis]

Thanks for the report. I'm afraid that we even forgot that we have slasher and didn't run it for more than 2 years. We will look at it at some point, but for now it's best to not activate built-in slasher.