App remains connected to wrong Redis node after failover which leads to writing against read-only replica

[mariusstaicu]

We're using spring boot app with redis support via lettuce to connect to a redis sentinel deployment in kubernetes. The spring-boot version is 2.7.12.

There are 3 sentinel nodes with 3 redis nodes in different pods.

There is a very specific scenario with a 30-40% reproduction rate in which the spring app remains connected to the former master node after a failover which leads to being unable to write to readonly replica.

The scenario is as follows:

1. with the app up and running and connected to the sentinels
2. we need to restart master redis node (not re-create it) -> to do this, ssh into master node and kill redis process; same result is achieved when kubernetes is OOM killing the pod
3. sentinels detect the master restart and wait for it to respond
4. master restarts immediately, but not in master mode
5. the app reconnects to the node that just restarted
6. sentinels trigger a failover since the master is not responding anymore
7. another node becomes master
8. in 30-40% of the cases the app does not switch the new redis master and cannot write anymore to the node it is connected, because it is connected to a replica

The implementation of the spring app is standard and it connects to a k8s service pointing to the sentinel nodes.

spring:
  redis:
    sentinel:
      nodes: rfr-redisfailover.infrastructure.svc:26379
      master: mymaster

There is a little customization for spring redis that overrides default serializer used by RedisTemplate and replaces it with Jackson and another one that sets up a message listener for some redis keys expiration events, however I don't think this is relevant here.

Logs:

1. sentinel logs

08 Aug 2023 15:02:19.068 * +reboot master mymaster 10.42.0.15 6379
08 Aug 2023 15:02:20.743 * +reboot master mymaster 10.42.0.15 6379
08 Aug 2023 15:02:21.250 * +reboot master mymaster 10.42.0.15 6379
08 Aug 2023 15:02:44.089 # +sdown master mymaster 10.42.0.15 6379
08 Aug 2023 15:02:45.795 # +sdown master mymaster 10.42.0.15 6379
08 Aug 2023 15:02:45.872 # +odown master mymaster 10.42.0.15 6379 #quorum 2/2
08 Aug 2023 15:02:45.872 # +new-epoch 26
08 Aug 2023 15:02:45.872 # +try-failover master mymaster 10.42.0.15 6379
08 Aug 2023 15:02:45.881 # +vote-for-leader 999b68d9cd15e085dbff865fc31dcaf6382874ff 26
08 Aug 2023 15:02:45.888 # +new-epoch 26
08 Aug 2023 15:02:45.888 # +new-epoch 26
08 Aug 2023 15:02:45.895 # +vote-for-leader 999b68d9cd15e085dbff865fc31dcaf6382874ff 26
08 Aug 2023 15:02:45.895 # 9b6940dffbaaaadf8ba30068f854d46974ee1d41 voted for 999b68d9cd15e085dbff865fc31dcaf6382874ff 26
08 Aug 2023 15:02:45.895 # +vote-for-leader 999b68d9cd15e085dbff865fc31dcaf6382874ff 26
08 Aug 2023 15:02:45.895 # a91e3195f1db1d05b3abf7b7f209e3d80ef04ce3 voted for 999b68d9cd15e085dbff865fc31dcaf6382874ff 26
08 Aug 2023 15:02:45.936 # +elected-leader master mymaster 10.42.0.15 6379
08 Aug 2023 15:02:45.936 # +failover-state-select-slave master mymaster 10.42.0.15 6379
08 Aug 2023 15:02:45.992 # +selected-slave slave 10.42.0.16:6379 10.42.0.16 6379 @ mymaster 10.42.0.15 6379
08 Aug 2023 15:02:45.992 * +failover-state-send-slaveof-noone slave 10.42.0.16:6379 10.42.0.16 6379 @ mymaster 10.42.0.15 6379
08 Aug 2023 15:02:46.050 * +failover-state-wait-promotion slave 10.42.0.16:6379 10.42.0.16 6379 @ mymaster 10.42.0.15 6379
08 Aug 2023 15:02:46.165 # +odown master mymaster 10.42.0.15 6379 #quorum 2/2
08 Aug 2023 15:02:46.165 # Next failover delay: I will not start a failover before Tue Aug  8 15:03:06 2023
08 Aug 2023 15:02:46.265 # +sdown master mymaster 10.42.0.15 6379
08 Aug 2023 15:02:46.320 # +odown master mymaster 10.42.0.15 6379 #quorum 3/2
08 Aug 2023 15:02:46.320 # Next failover delay: I will not start a failover before Tue Aug  8 15:03:06 2023
08 Aug 2023 15:02:46.599 # +promoted-slave slave 10.42.0.16:6379 10.42.0.16 6379 @ mymaster 10.42.0.15 6379
08 Aug 2023 15:02:46.600 # +failover-state-reconf-slaves master mymaster 10.42.0.15 6379
08 Aug 2023 15:02:46.643 * +slave-reconf-sent slave 10.42.0.13:6379 10.42.0.13 6379 @ mymaster 10.42.0.15 6379
08 Aug 2023 15:02:46.643 # +config-update-from sentinel 999b68d9cd15e085dbff865fc31dcaf6382874ff 10.42.0.253 26379 @ mymaster 10.42.0.15 6379
08 Aug 2023 15:02:46.643 # +switch-master mymaster 10.42.0.15 6379 10.42.0.16 6379
08 Aug 2023 15:02:46.644 # +config-update-from sentinel 999b68d9cd15e085dbff865fc31dcaf6382874ff 10.42.0.253 26379 @ mymaster 10.42.0.15 6379
08 Aug 2023 15:02:46.644 # +switch-master mymaster 10.42.0.15 6379 10.42.0.16 6379
08 Aug 2023 15:02:46.644 * +slave slave 10.42.0.13:6379 10.42.0.13 6379 @ mymaster 10.42.0.16 6379
08 Aug 2023 15:02:46.644 * +slave slave 10.42.0.11:6379 10.42.0.11 6379 @ mymaster 10.42.0.16 6379
08 Aug 2023 15:02:46.644 * +slave slave 10.42.0.13:6379 10.42.0.13 6379 @ mymaster 10.42.0.16 6379
08 Aug 2023 15:02:46.644 * +slave slave 10.42.0.15:6379 10.42.0.15 6379 @ mymaster 10.42.0.16 6379
08 Aug 2023 15:02:46.644 * +slave slave 10.42.0.15:6379 10.42.0.15 6379 @ mymaster 10.42.0.16 6379
08 Aug 2023 15:02:46.968 # -odown master mymaster 10.42.0.15 6379
08 Aug 2023 15:02:47.593 * +slave-reconf-inprog slave 10.42.0.13:6379 10.42.0.13 6379 @ mymaster 10.42.0.15 6379
08 Aug 2023 15:02:47.593 * +slave-reconf-done slave 10.42.0.13:6379 10.42.0.13 6379 @ mymaster 10.42.0.15 6379
08 Aug 2023 15:02:47.648 # +failover-end master mymaster 10.42.0.15 6379
08 Aug 2023 15:02:47.648 # +switch-master mymaster 10.42.0.15 6379 10.42.0.16 6379
08 Aug 2023 15:02:47.648 * +slave slave 10.42.0.13:6379 10.42.0.13 6379 @ mymaster 10.42.0.16 6379
08 Aug 2023 15:02:47.648 * +slave slave 10.42.0.15:6379 10.42.0.15 6379 @ mymaster 10.42.0.16 6379
08 Aug 2023 15:02:50.570 # +reset-master master mymaster 10.42.0.16 6379
08 Aug 2023 15:02:50.655 # +set master mymaster 10.42.0.16 6379 down-after-milliseconds 5000
08 Aug 2023 15:02:50.664 # +set master mymaster 10.42.0.16 6379 failover-timeout 10000
08 Aug 2023 15:02:50.671 # +set master mymaster 10.42.0.16 6379 down-after-milliseconds 5000
08 Aug 2023 15:02:50.677 # +set master mymaster 10.42.0.16 6379 failover-timeout 10000
08 Aug 2023 15:02:50.684 # +set master mymaster 10.42.0.16 6379 down-after-milliseconds 5000
08 Aug 2023 15:02:50.691 # +set master mymaster 10.42.0.16 6379 failover-timeout 10000
08 Aug 2023 15:02:50.774 * +sentinel sentinel 999b68d9cd15e085dbff865fc31dcaf6382874ff 10.42.0.253 26379 @ mymaster 10.42.0.16 6379
08 Aug 2023 15:02:51.329 * +sentinel sentinel 9b6940dffbaaaadf8ba30068f854d46974ee1d41 10.42.0.254 26379 @ mymaster 10.42.0.16 6379
08 Aug 2023 15:02:51.359 * +slave slave 10.42.0.13:6379 10.42.0.13 6379 @ mymaster 10.42.0.16 6379
08 Aug 2023 15:02:51.367 * +slave slave 10.42.0.15:6379 10.42.0.15 6379 @ mymaster 10.42.0.16 6379

2. spring app logs

2023-08-08 15:02:14.659  INFO 1 --- [xecutorLoop-1-5] i.l.core.protocol.ConnectionWatchdog     : Reconnecting, last destination was 10.42.0.15/10.42.0.15:6379
2023-08-08 15:02:14.659  INFO 1 --- [xecutorLoop-1-4] i.l.core.protocol.ConnectionWatchdog     : Reconnecting, last destination was 10.42.0.15/10.42.0.15:6379
2023-08-08 15:02:14.670  WARN 1 --- [llEventLoop-4-1] i.l.core.protocol.ConnectionWatchdog     : Cannot reconnect to [10.42.0.15/<unresolved>:6379]: finishConnect(..) failed: Connection refused: /10.42.0.15:6379
2023-08-08 15:02:14.671  WARN 1 --- [llEventLoop-4-2] i.l.core.protocol.ConnectionWatchdog     : Cannot reconnect to [10.42.0.15/<unresolved>:6379]: finishConnect(..) failed: Connection refused: /10.42.0.15:6379
2023-08-08 15:02:17.463  INFO 1 --- [llEventLoop-4-6] i.l.core.protocol.ReconnectionHandler    : Reconnected to 10.42.0.15/<unresolved>:6379
2023-08-08 15:02:17.463  INFO 1 --- [llEventLoop-4-5] i.l.core.protocol.ReconnectionHandler    : Reconnected to 10.42.0.15/<unresolved>:6379
2023-08-08 15:03:47.031  WARN 1 --- [container-0-C-1] r.e.r.t.device.DeviceEventHandler   : failed to save batch: [] with cause: Error in execution; nested exception is io.lettuce.core.RedisReadOnlyException: READONLY You can't write against a read only replica.

[mp911de]

Thanks for reaching out. I'm not exactly sure how well the timestamps line up.

However, I see that at 15:02:17.463 the reconnect has succeeded but the reconfiguration happened at 15:02:50.570. With the Redis client being connected, there's no indication to the client that the master node has changed.

The simple mode detects reconfiguration from disconnects.

You could switch to the advanced Master/Replica mode that subscribes to Sentinels via Pub/Sub but that will add more complexity to the setup and additional Redis connections. You can achieve this by configuring a LettuceConnectionFactory with a customized LettuceClientConfiguration that has ReadFrom configured (e.g. LettuceClientConfiguration.builder().readFrom(ReadFrom.MASTER).build()).

[mariusstaicu]

Ok, I will try that, thanks.

[mariusstaicu]

Ok, this seems to be working as suggested, however I'm curious how to switch to master/replica mode with sentinels subscription ?
I am very confused on how to achieve that looking into spring docs and into spring data code.
Lettuce docs seem straightforward, however I cannot seem to understand how to customize spring.

Many thanks.

[jxblum]

Hi @mariusstaicu -

DISCLAIMER: First, I am no Redis expert as I am still learning, so please take my answer with some caution. My hope is to at least point you in the right direction.

After reviewing your earlier/initial issue(s) and reading Mark's comments, this is what I take from the conversation.

Regarding...

I'm curious how to switch to master/replica mode with sentinels subscription?

With Spring Data Redis in the context of Spring Boot (in your case, 2.7.x), as you know, simply by using the spring.redis.sentinel.* properties you are configuring your Spring Boot, Spring Data Redis application to be aware of and use Redis Sentinel.

Under-the-hood, this is going to use the Spring Data (SD) Redis, RedisSentinelConfiguration class to configure your Spring Boot, SD Redis application, and ultimately Lettuce, to use and be Redis Sentinel aware.

TIP: You can follow the Spring Boot auto-configuration for Redis Sentinel from here, then here, and finally, to here. In other words, if any spring.redis.sentinel.* properties have been declared, then the Sentinel config will not be null (this), thereby enabling support for Redis Sentinel.

NOTE: Please be aware that in Spring Boot 3.0 and later, the spring.redis.* properties have been changed to spring.data.redis.*. See here.

If you need programmatical control of the RedisSentinelConfiguration, then Spring Boot allows you to declare a bean of type RedisSentinelConfiguration:

@SpringBootConfiguration
class ApplicationRedisConfiguration { 

  @Bean
  RedisSentinelConfiguration redisSentinelConfiguration(..) {
    // ...
  }
}

That is the fundamental premise of the logic behind this along with this. Otherwise, Spring Boot simply creates a new RedisSentinelConfiguration on your behalf, configured from Spring Boot's spring.redis.sentinel.* properties, of course.

TIP: You should refer to the Javadoc for RedisSentinelConfiguration on the Redis Sentinel configuration offered by SD Redis.

As you know, Redis Sentinel is in charge of the master and replicas making up your topology (as determined (in part) by the spring.redis.sentinel.master and spring.redis.sentinel.nodes configuration properties.

In addition, Lettuce automatically subscribes to Sentinels on your behalf, as documented. Also see here along with this. Therefore, after reading this, and TMK, there is no additional configuration required using SD Redis to "subscribe" to Sentinels when using Lettuce.

Finally, it is the Lettuce's ReadFrom class that determines how and where reads (and writes?) are performed in a Sentinel topology.

Mark referred to setting the ReadFrom configuration using: LettuceClientConfiguration.builder().readFrom(ReadFrom.MASTER).build().

However, when using Spring Boot, you would simply declare a LettuceClientConfigurationBuilderCustomizer as follows:

@SpringBootConfiguration
class ApplicationRedisConfiguration {

  @Bean
  LettuceClientConfigurationBuilderCustomizer lettuceClientConfigurationBuilderCustomizer(..) {

    return lettuceClientConfigurationBuilder -> 
        lettuceClientConfigurationBuilder.readFrom(ReadFrom.MASTER);
  }
}

Clearly, you can customize the LettuceClientConfiguration produced by the builder further with this "customizer" provided by Spring Boot (see Javadoc).

Please refer to the Lettuce ReadFrom class (Javadoc) for further details on the appropriate setup for master/replica reads and writes (involving automatic detection and failure handling).

WARNING: Honestly, I don't know how Lettuce's ReadFrom class relates to writes at all. It seems to me, Redis simply always tries to write to a Master for consistency's sake. In a quick Google search, while it seems possible to write to Replicas now, Replicas are read-only for the most part.

NOTE: I mention the WARNING above because your original issue was on writes to a Replica node after recovery, which was previously a failed (restarted) Master, hence your statement: "6. sentinels trigger a failover since the master is not responding anymore 7. another node becomes master 8. in 30-40% of the cases the app does not switch the new redis master and cannot write anymore to the node it is connected, because it is connected to a replica". So, I am assuming Mark knows that the ReadFrom class and its array of configuration options also helps in the detection of topology changes in the context of Redis Sentinel, thereby leading to writes on Masters only. Don't quote me on this!

I hope this helps.

For the time being, I am going to close this ticket as complete.

For any additional questions, please continue the conversation StackOverflow for all users to benefit. Things can easily get lost in SD Redis Issue tickets and PRs.

If you have additional issues, please re-open this ticket, or better yet, open a new ticket, as necessary.

Thank you

[mariusstaicu]

Wow, @jxblum thanks for the very detailed and elaborate response, I very much appreciate it. I didn't have time to get in all the links you posted but I soon will.
From Mark's response and by looking into the SB & lettuce code I understood that, when using normal spring.redis.sentinel.* properties (or spring.data.redis.sentinel.* properties in 3.x), the case that I described in the issue is not covered.

What I did not get the first time is that, by using ReadFrom.MASTER, it will actually use the Master/Replica mode that will subscribe to events from sentinel and be updated in real time of any Redis master/replica changes. This is what I needed and it works perfectly now.

Thank you again for your elaborate answer and for taking the time to explain. :)

[jxblum]

Hi @mariusstaicu - Glad to hear that everything is working correctly for you now! Also glad to help.

I gather that Redis always makes writes to the Master node, unless configured otherwise. So, by using a setting of ReadOnly.MASTER, then this effectively ensures that your client application gets notification of the topology change and the newly elected Master.

In any case, I am glad you are on the right track now.

Cheers!