[Feature Request] Make segment replication the default for 3.0

[andrross]

Is your feature request related to a problem? Please describe

Segment replication was introduced in the 2.x line as an alternative to document replication. It can offer substantial performance benefits by reducing the amount of CPU-intensive indexing work that has to happen on the cluster.

Describe the solution you'd like

The default indexing method for new indexes in 3.0+ versions of OpenSearch should be SEGMENT instead of DOCUMENT.

Related component

Indexing:Replication

Describe alternatives you've considered

Do nothing; keep document replication as the default.

Additional context

Segment replication is the only option when using remote store and is being used by remote store-based offerings of Amazon OpenSearch Service. However, the node-to-node variant of segment replication likely has less real world production usage. Some things to consider before making this change are:

1. Segment copying from the primary can become a bottleneck with large numbers of replicas configured
2. We'd need a thorough audit to make sure all integration tests are running with segment replication enabled
3. The semantics of wait_for=refresh are weird today. With document replication that means your write will be searchable after the index call completes. With segment replication it means the write will be searchable on the primary shard only, not replicas.

[Bukhtawar]

Thanks @andrross

Segment copying from the primary can become a bottleneck with large numbers of replicas configured

The default number of replica is one, so the default option should work seamlessly

We'd need a thorough audit to make sure all integration tests are running with segment replication enabled

+1

The semantics of wait_for=refresh are weird today. With document replication that means your write will be searchable after the index call completes. With segment replication it means the write will be searchable on the primary shard only, not replicas.

The wait_for=refresh is something that is an explicit option used today, would it be fine to document that this option has a semantic change, that causes refreshes with wait(read after write consistency) to apply exclusively to primary in 3.0?

[andrross]

The wait_for=refresh is something that is an explicit option used today, would it be fine to document that this option has a semantic change, that causes refreshes with wait(read after write consistency) to apply exclusively to primary in 3.0?

@Bukhtawar We definitely can change the semantics of this option. The question is really whether this is good enough. There likely are use cases where its necessary to know all primaries and replicas are up-to-date with respect to a specific write, and today we don't really have any mechanism to do that with segment replication.

[sachinpkale]

@andrross @mch2 @Bukhtawar A bit tangential to this discussion but are we also thinking of removing the mixed mode (SegRep + DocRep) cluster from 3.0?

[krisfreedain]

Catch All Triage - 1

[mch2]

The default number of replica is one, so the default option should work seamlessly

This is true, but you can still fairly easily overload a node with multiple primaries actively taking writes. In the absence of any heat based allocation or even enforcing a hard primary limit per node the experience here isn't great.

A bit tangential to this discussion but are we also thinking of removing the mixed mode (SegRep + DocRep) cluster from 3.0?

Any particular reason to take that option away? I think its a reasonable fall back for node-node users wanting strong reads on an index (system indices are common here) other than routing to primaries.

I wouldn't be opposed to making segrep the default, but I think we need to improve the out of the box experience before that happens.