[BUG] <title>Cross-cluster search that includes self no longer works 2.16-2.19

[kmfreder1]

Describe the bug

We have an Opensearch 2.16.0 installation from where we set up cross cluster search to a handful of other Opensearch 2.16.0 systems. Pre-opensearch 2.16.0, we could create a *:index-name* pattern and view the events of our other Opensearch clusters along with the events from the Opensearch cluster we were searching from. We recently upgraded from Opensearch 2.8.0 where was being done.

As an a trouble-shooting step, I installed a separate Opensearch 2.19 and set up the installation as a cross-cluster search to itself. When I attempted to make an index pattern like remote_host:index-name*, it failed the same as with Opensearch 2.16.0 even though we were doing this successfully with Opensearch <= 2.8.0.

Related component

Search:Remote Search

To Reproduce

1. Go to 'Dev Tools' and set up the local host as a remote cluster by entering the following:

PUT _cluster/settings
{
    "persistent": {
        "cluster.remote": { 
            "my_remote": {
                "seeds": ["localhost:9300"],
                "skip_unavailable": true
            }
        }
    }
}

2. Check that the remote cluster connected by entering the following:
   GET /_remote/info
   The output should show "connected": true as below

{
  "my_remote": {
    "connected": true,
    "mode": "sniff",
    "seeds": [
      "localhost:9300"
    ],
    "num_nodes_connected": 1,
    "max_connections_per_cluster": 3,
    "initial_connect_timeout": "30s",
    "skip_unavailable": true
  }
}

For this test, there should be no other remote cluster set up but the local cluster.
3. Go to Management > Dashboard Management > Index Patterns and click on the Create index pattern button.

4. In the index pattern name field, enter the name of the remote followed by a known index name pattern separated by a colon:my_remote:security-auditlog*

5. See error
   

Expected behavior

The expected behavior is to be able to review events from the local SIEM that has been set up as a remote cluster along with the events from other remote clusters as we were doing in <= Opensearch 2.8.

Additional Details

Plugins
Please list all plugins currently enabled.

/usr/share/opensearch/bin/opensearch-plugin list

opensearch-alerting
opensearch-anomaly-detection
opensearch-asynchronous-search
opensearch-cross-cluster-replication
opensearch-custom-codecs
opensearch-flow-framework
opensearch-geospatial
opensearch-index-management
opensearch-job-scheduler
opensearch-knn
opensearch-ltr
opensearch-ml
opensearch-neural-search
opensearch-notifications
opensearch-notifications-core
opensearch-observability
opensearch-performance-analyzer
opensearch-reports-scheduler
opensearch-security
opensearch-security-analytics
opensearch-skills
opensearch-sql
opensearch-system-templates
query-insights

Screenshots
If applicable, add screenshots to help explain your problem.

This is what happens on 2.8 when I attempt to set up an index pattern to the local "remote-cluster":

This is what happens in 2.19 (and 2.16) wen I do the same:

Host/Environment (please complete the following information):

• OS: Ubuntu
• Version 22.04

[kkewwei]

@kmfreder1 Can you help check is the search failed/timeout or the bug? When you clieck Create index pattern, it will send a very time consuming query(opensearch-project/OpenSearch-Dashboards#8596), maybe it will time out?

I I encountered a similar issue (opensearch-project/OpenSearch-Dashboards#8596, opensearch-project/index-management-dashboards-plugin#1178), this is currently a bug and no reply.

[kmfreder1]

@kkewwei No, it does not timeout. It immediately shows the error above when I type in the above remote cluster name and index pattern.

[HenryTheSir]

Hi,

this should solve your issue
https://forum.opensearch.org/t/ccs-query-to-same-cluster-is-not-working/17995/2

best regards

[kmfreder1]

@HenryTheSir That is a great help! Thanks for sharing.