Started using Distributed data - get ddatareplicator dead letter messages

[giovanni-ferrari]

I started using Distributed Data to store temporary information that should be resilient across the cluster (i'm using distributed data in conjunction with sharding).

Since then I started seeing some [INFO] logs from akka.net, that repeats continuosly.

Sep 27 09:36:07 ubuntu service.service[3831814]: [INFO][09/27/2024 07:36:07.329Z][Thread 0017][akka://Service/user/ddataReplicator] Message [Status] from [akka.tcp://[email protected]:9997/user/ddataReplicator#43346742] to [akka://Service/user/ddataReplicator] was not delivered. [9] dead letters encountered. If this is not an expected behavior then [akka://Service/user/ddataReplicator] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. Message content: Status(chunk=0, totalChunks=1, digest={Data1->Google.Protobuf.ByteString,Data2->Google.Protobuf.ByteString,Data3->Google.Protobuf.ByteString,})

Any idea why this is happening, and if it can be related to any configuration issue?

[Aaronontheweb]

Is the replicator not running on your current node? Because that's why you're getting the dead letter - the actor at your local path akka://Service/user/ddataReplicator doesn't exist. If the replicators are syncing based on role types, the issue might be that the replicator on xxx.xxx.xxx.xx0:9997 expects there to be one running locally on your other node as well, but there isn't one. If you didn't specify a role setting for the replicator, then that would also cause this issue - no role means everyone gets all the the replication messages.

[giovanni-ferrari]

Thank you Aaron, I confirm the problem was related to roles configuration

[Aaronontheweb]

Glad to hear you got to the bottom of it

[giovanni-ferrari]

Hi Aaron,
After the configuration was fixed, I did not receive the "dead letters" error message for a long time, but it has recently started appearing again.
What happens now is that the application runs correctly for about 10 hours, then the dead letter messages start showing up again (only on one node in a cluster of three). Even if the node is restarted, the issue persists.

I also tried setting 'RecreateOnFailure' to true, but it had no effect.

Message [Status] from [akka.tcp://service@service1:9997/user/DataReplicator#973567920] to [akka://Service/user/DataReplicator] was not delivered. [168] dead letters encountered, of which 157 were not logged. The counter will be reset now. If this is not an expected behavior then [akka://Service/user/DataReplicator] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. Message content: Status(chunk=0, totalChunks=1, digest={Missions->Google.Protobuf.ByteString,Charging->Google.Protobuf.ByteString,Alarms->Google.Protobuf.ByteString,}),

[Aaronontheweb]

@giovanni-ferrari was there a recent upgrade or change inside your system that could have prompted this?

[giovanni-ferrari]

It happened on a new installation - but the code that initializes the data replicator and use it to store/retrieve data has not changed. Neither the roles configuration for the cluster nodes.

[Aaronontheweb]

Got it - would you mind sharing what one of your DData writes / reads looks like? It could be innocuous: write needs a majority + 1 quorum and gets it, thus it completes, but other nodes who received the write request are still sending back ACKs that don't need to be processed any longer.