[Meta] Validation of transforms defined in packages

[mrodm]

Relates #1402
Relates #2341

Currently, elastic-package performs a validation process for packages containing transform definitions based on validating the documents returned by the transform preview API. This process was introduced in #1402.

As discovered in #2341, validating transforms in a proper way could be problematic as it is now. Main issues found:

• Some packages define logs-* as the source index (see [system tests] Validate fields in transforms are documented based on mappings #2341 (comment)).
  • Difficult to control what documents are going to be ingested in the destination index.
  • Is this a valid usage in an integration package?
  • Does it make sense for a content package?
    • Currently, it is not supported defining transforms in content packages (spec link).
• Some fields are defined as constant_keyword in transforms (e.g. data_stream.dataset).
  • If those fields come from different source data streams, there could be issues processing documents in the transform since those fields could have different values in different documents.
  • Example error:

    Preview of field's value: '59467'; java.lang.IllegalArgumentException: [constant_keyword] field [data_stream.namespace] only accepts values that are equal to the value defined in the mappings [75519], but got [59467]]

• If validation takes into account the documents processed by the transforms (as it was tested with validation based on mappings):
  • Testing times are increased considerably, as it can be checked in [system tests] Validate transforms based on the mappings #2347
    • Details in: [system tests] Validate fields in transforms are documented based on mappings #2341 (comment)
  • Packages containing several system test configuration files (test cases) for the same data stream could not detect all errors/failures:
    • It's likely that the second test case (configuration file) does not wait for the expected documents, since the transform already processed documents for the first test case.
    • More information about this scenario in [system tests] Validate fields in transforms are documented based on mappings #2341 (comment)

Due to the above, it looks like that the testing and validation process for transforms should be revisited. Maybe they should have their own test suite along with pipeline, system, static, and other test suites.

cc @elastic/ecosystem

Related PRs:

• System tests for transforms #2242
• (Attempt to add validation based on mappings): [system tests] Validate transforms based on the mappings #2347

Other issues related (should it be moved as sub-issue here?):

• [system tests] Validate fields in transforms are documented based on mappings #2341

[jsoriano]

Due to the above, it looks like that the testing and validation process for transforms should be revisited.

I'd go further, to say that the whole transforms support in packages should be revisited. It is too open now, and as we have found it is likely not being correctly used in some cases.