[system tests] Ingored fields are not detected in stacks 8.15.0 onward

[mrodm]

The current search performed to get the ignored fields in a given data stream is not working successfully for Elastic stack versions: 8.15.0

elastic-package/internal/testrunner/runners/system/tester.go

Lines 46 to 78 in 091c663

	checkFieldsBody = `{
	"fields": ["*"],
	"runtime_mappings": {
	"my_ignored": {
	"type": "keyword",
	"script": {
	"source": "for (def v : params['_fields']._ignored.values) { emit(v); }"
	}
	}
	},
	"aggs": {
	"all_ignored": {
	"filter": {
	"exists": {
	"field": "_ignored"
	}
	},
	"aggs": {
	"ignored_fields": {
	"terms": {
	"size": 100,
	"field": "my_ignored"
	}
	},
	"ignored_docs": {
	"top_hits": {
	"size": 5
	}
	}
	}
	}
	}
	}`

For Elastic stack >= 8.15.0, that my_ignored runtime field is not populated and it is not present in the search response, even for the data streams that contain some ignored field.

Example: this build from windows package (integrations repo) fails with 8.14.0, but it finishes successfully when it runs with 8.15.0 or 8.17.0
https://buildkite.com/elastic/integrations/builds/22752

To be tested if this query works for >=8.15.0

{
  "fields": ["*"],
  "aggs": {
    "all_ignored": {
      "filter": {
        "exists": {
          "field": "_ignored"
         }
      },
      "aggs": {
        "ignored_fields": {
          "terms": {
            "size": 100,
            "field": "_ignored"
          }
        },
        "ignored_docs": {
          "top_hits": {
            "size": 5
          }
        }
      }
    }
  }
}

[jsoriano]

@flash1293 pinging you just in case as you contributed the original query. Do you have an idea on what may have changed in 8.15.0 regarding this?

[flash1293]

I'm not aware of anything - @salvatore-campagna as you worked on this on the Elasticsearch side, are you aware of anything? Did the work on making _ignored aggregatable somehow interfere with accessing them via params['_fields']._ignored.values?

To fix it forward for more recent versions - it should be possible now to just use _ignored directly