Read Only Command Preference & Dynamic Load Balancing

[proost]

Summary

Adding to read only command preference option to rueidis. read only command preference can distributes read only commands to primary and replicas evenly.

Problem Definition

Background

Handling read only command via replica makes valkey/redis cluster scalable. When using replica, it is important to distribute loads in a each node. Because uneven load balancing makes cluster less-scalable and inefficient in terms of resource usage especially read-heavy workload.

Current rueidis can distribute command using “SendToReplicas”. But problem is “SendToReplicas” doesn’t consider how many nodes in a each shard. If a valkey/redis cluster uses auto scaling, “SendToReplicas” shows less efficient load balancing.

Goals

• Load balancing strategy to read only command.
• Dynamic Load Balancing Strategy to read only command.

Keywords Description

• Read only command preference: Load Balancing strategy applied to only read only command. This preference determines which node should be handle read only command in a slot range.

Design

Load Balancing Strategy

Reader node selection.

One of important thing in load balancing is readerNodeSelector . readerNodeSelector chooses which node is reader each slot. readerNodeSelector runs in _refresh function, when setting conn per each slot.

// readerNodeSelector selects a reader node from a list of connections.
type readerNodeSelector interface {
	// Select selects a reader node from a list of connections based on the read preference.
	Select(primaryConn connrole, replicaConns []connrole) (connrole, error)
}

User can configure readerNodeSelector via ReadPreference option in ClientOption .

	// ReadPreference is the read preference for the cluster client.
	// NOTE: This function can't be used with ReplicaOnly or SendToReplicas.
	ReadPreference ReadPreference

// ReadPreference defines which node to read from in a cluster
type ReadPreference int

Using factory method, readerNodeSelector implementation is returned internally.

func newReaderNodeSelector(preference ReadPreference) (readerNodeSelector, error)

Routing to reader node.

If ReadPreference is configured in ClientOption , then sendToReadershould be set to SendReplicas function. sendToReader sends read only command to selected reader node.

var sendToReader = func(cmd Completed) bool {
	if cmd.IsReadOnly() {
		return true
	}
	return false
}

Dynamic Load Balancing

We can achieve Any read preference and ShardsRefreshInterval .

If Any is set, then newReaderNodeSelector returns anyNodeSelector . anyNodeSelector chooses connection randomly. So If replica is added because of auto scaling, then after refreshed cluster connection cache, we can distribute read requests to each node evenly.

const (
	PreferenceUndefined ReadPreference = iota // PreferenceUndefined is the default read preference
	Any                 ReadPreference = iota // Any reads from any node
)

// anyNodeSelector selects any node from a list of connections.
type anyNodeSelector struct{}

// Select selects any node from a list of connections.
func (s *anyNodeSelector) Select(primaryConn connrole, replicaConns []connrole) (connrole, error) {
	i := util.FastRand(len(replicaConns) + 1)
	if i == 0 {
		return primaryConn, nil
	}
	return replicaConns[i-1], nil
}

Alternatives Considered

I’ve considered making custom SendReplicas . But it is hard. Because In a SendReplicas we can’t know how many nodes in each shard.

[rueian]

I think this generally looks very good, especially the combination of the ShardsRefreshInterval, and can be implemented easily.

But instead of hiding the selector, I prefer exposing the following API to users:

type ReplicaInfo struct {
    Addr string
    // Zone string <- we could need this field in the future, see https://github.com/valkey-io/valkey-glide/pull/2539
}
SlotReplicaSelector: func(slot uint16, replicas []ReplicaInfo) int

1. This function should return an int index in the range of 0~len(replicas). There is no error returned from the function and if the returned index is out of range, we can pick a replica randomly.
2. The slot parameter passed in can let users know which slot they are choosing a replica for. This allows them to monitor their decision.
3. If there are no replicas for a slot, the function will not be called.
4. If SendToReplicas is not configured, this function will not be called either.

[proost]

1. If we stick to above interface, ReaderNodeSelector is more correct. We should consider primary also. Because Including primary to handle read requests in read-heavy environments makes more efficient resource usage. So in that case, if index is out of range, we should pick primary. And picking randomly makes difficult to expect which node selected as a reader node.

2. How about adding primary? Maybe we can make a load based or latency based node selecting later. For example, If we adding a new feature to get metrics from each node regularly and set to NodeInfo.

type NodeInfo struct {
    Addr string
    // Zone string <- we could need this field in the future, see https://github.com/valkey-io/valkey-glide/pull/2539
    Replica bool
}
ReaderNodeSelector: func(slot uint16, nodeInfoList []Nodeinfo) int

[rueian]

I think it makes sense to choose the primary when the returned index is out of range. Do we still need to add the primary into parameters?

[proost]

OK, I agree. we don't need primary node info.

[proost]

Close: #692