[FTR] split configs by target into multiple manifest files #187440

dmlemeshko · 2024-07-03T07:38:28Z

Summary

Split FTR configs manifest into multiple files based on distro (serverless/stateful) and area of testing (platform/solutions)
Update the CI scripts to support the change, but without logic modification

More context:

With this change we will have a clear split of FTR test configs owned by Platform and Solutions. It is a starting point to make configs discoverable, our test pipelines be flexible and run tests based on distro/solution.

github-actions · 2024-07-03T07:38:44Z

A documentation preview will be available soon.

🔨 Buildkite builds
📚 HTML diff
📙 Preview page

Request a new doc build by commenting

Rebuild this PR: run docs-build
Rebuild this PR and all Elastic docs: run docs-build rebuild

_{run docs-build is much faster than run docs-build rebuild. A rebuild should only be needed in rare situations.}

_{If your PR continues to fail for an unknown reason, the doc build pipeline may be broken. Elastic employees can check the pipeline status here.}

…y-target

dmlemeshko · 2024-07-03T07:51:18Z

/ci

dmlemeshko · 2024-07-03T08:03:41Z

/ci

dmlemeshko · 2024-07-03T08:10:30Z

.buildkite/pipeline-utils/ci-stats/pick_test_group_run_order.ts

@@ -16,8 +16,11 @@ import { BuildkiteClient, BuildkiteStep } from '../buildkite';
 import { CiStatsClient, TestGroupRunOrderResponse } from './client';

 import DISABLED_JEST_CONFIGS from '../../disabled_jest_configs.json';
+import { serverless, stateful } from './ftr_configs_manifests.json';


Using json to store manifests paths because importing constant from .buildkite/pipeline-utils into packages/kbn-test won't work and I don't like the idea of solving it with ts-ignore. Open for suggestions here

Regarding living in two different tsconfig.json files?

Regarding living in two different tsconfig.json files?

This code is living on its own, but manifest files loading is also happening in packages/kbn-test. The issue is that pipeline-utils is not a package, otherwise we can easily import it as a dependency.
I think for the start json is ok, but maybe we should change depending on how we plan to access the manifest based on distro/project type

.buildkite/pipeline-utils/ci-stats/pick_test_group_run_order.ts

dmlemeshko · 2024-07-03T08:17:42Z

docs/developer/contributing/development-functional-tests.asciidoc

+- ftr_oblt_stateful_configs.yml
+- ftr_security_stateful_configs.yml
+- ftr_search_stateful_configs.yml
+If you’re writing a plugin outside the {kib} repo, you will have your own config file.


@pheyos This is the only place where we mentioned original manifest file. Do you think we need to add more context here or put this info in other place?

I think that's fine and we don't need to add more context here.

dmlemeshko · 2024-07-03T08:19:43Z

packages/kbn-test/src/functional_test_runner/lib/config/config_loading.ts

@@ -67,7 +67,7 @@ async function getConfigModule({
  ) {
    const rel = Path.relative(REPO_ROOT, resolvedPath);
    throw createFlagError(
-      `Refusing to load FTR Config at [${rel}] which is not listed in [${FTR_CONFIGS_MANIFEST_REL}]. All FTR Config files must be listed there, use the "enabled" key if the FTR Config should be run on automatically on PR CI, or the "disabled" key if it is run manually or by a special job.`
+      `Refusing to load FTR Config at [${rel}] which is not listed in [${ALL_FTR_MANIFEST_REL_PATHS}]. All FTR Config files must be listed in one of manifest files, use the "enabled" key if the FTR Config should be run on automatically on PR CI, or the "disabled" key if it is run manually or by a special job.`
    );


We will print out all the manifests here, but probably we can only print the ones related to the distro (e.g. only serverless manifests) assuming the FTR config has a clear definition where it should be run

Line 70 is a little bit like a wall of text, can we break it up a tiny bit?

Good point, I will split it

.buildkite/pipeline-utils/ci-stats/pick_test_group_run_order.ts

dmlemeshko · 2024-07-03T08:35:11Z

/ci

dmlemeshko · 2024-07-03T11:57:45Z

/ci

dmlemeshko · 2024-07-03T12:00:17Z

packages/kbn-test/src/functional_test_runner/lib/config/run_check_ftr_configs_cli.ts

+          `The following files look like FTR configs which are not listed in one of manifest files:\nstateful: ${manifestPaths.stateful}\nserverless: ${manifestPaths.serverless}\n  - ${invalidList}`
        );
        throw createFailError(
-          `Please add the listed paths to .buildkite/ftr_configs.yml. If it's not an FTR config, you can add it to the IGNORED_PATHS in ${THIS_REL} or contact #kibana-operations`
+          `Please add the listed paths to the correct manifest file. If it's not an FTR config, you can add it to the IGNORED_PATHS in ${THIS_REL} or contact #kibana-operations`


After update the message is:

ERROR The following files look like FTR configs which are not listed in one of manifest files: stateful: .buildkite/ftr_platform_stateful_configs.yml,.buildkite/ftr_oblt_stateful_configs.yml,.buildkite/ftr_security_stateful_configs.yml,.buildkite/ftr_search_stateful_configs.yml serverless: .buildkite/ftr_base_serverless_configs.yml,.buildkite/ftr_oblt_serverless_configs.yml,.buildkite/ftr_security_serverless_configs.yml,.buildkite/ftr_search_serverless_configs.yml - x-pack/test/api_integration/apis/cloud_security_posture/config.ts - x-pack/test/cloud_security_posture_api/config.ts - x-pack/test/cloud_security_posture_functional/config.ts - x-pack/test/fleet_cypress/cli_config.ts - x-pack/test/fleet_cypress/visual_config.ts - x-pack/test/functional/apps/apm/config.ts - x-pack/test/functional/apps/search_playground/config.ts ERROR Please add the listed paths to the correct manifest file. If it's not an FTR config, you can add it to the IGNORED_PATHS in packages/kbn-test/src/functional_test_runner/lib/config/run_check_ftr_configs_cli.ts or contact #kibana-operations

…ibana into ftr/split-configs-by-target

dmlemeshko · 2024-07-03T12:55:53Z

/ci

wayneseymour · 2024-07-03T14:48:09Z

scripts/enabled_ftr_configs.js

+var path = require('path');
+
+var manifestsJsonPath = path.resolve(__dirname, '../.buildkite/ftr_configs_manifests.json');
+console.log(manifestsJsonPath);


Should we use an actual logger, instead of console?

Ahh, it's probably not loaded yet right?

yes, it is a vanilla js script that you added a while ago, my change broke it :)

I added? I can't recall, but I believe ya

Oh I see now.

wayneseymour · 2024-07-03T14:50:33Z

.buildkite/pipeline-utils/ci-stats/pick_test_group_run_order.ts

+        uniqueQueues.add(manifest.defaultQueue);
+      }
+    } catch (_) {
+      const error = _ instanceof Error ? _ : new Error(`${_} thrown`);


Personal happy (opposite of nit) is the _.
Years of fp makes me happy to see this Dima.

No wasted time or thought in naming an uninteresting symbol. Good stuff

I was trying to make as little changes as possible and kept the original error handling by Brian in place, not sure about the code style that was followed here.

Yeah, that's a notation style very common in fp langs, such as Haskell, F#.Net, etc.

and in javascript

mistic

this looks good to me. The current changes are a good first step forward on setting the stones to achieve future ability on running only a current type of tests. Also it doesn't seems like the current changes are going to affect for now the ci stats.

dmlemeshko · 2024-07-17T13:46:45Z

Also it doesn't seems like the current changes are going to affect for now the ci stats

It will affect flaky test runner as manifest file is replaced with multiple ones, but if we merge elastic/kibana-ci-stats/pull/831 first backward compatibility will be in place

This reverts commit 6060ea4.

This reverts commit a940986.

…y-target

elasticmachine · 2024-07-19T12:54:26Z

💚 Build Succeeded

Buildkite Build
Commit: 7d0592a

Metrics [docs]

✅ unchanged

History

💔 Build #222293 failed 6060ea4
💚 Build #222202 succeeded a940986
💛 Build #221651 was flaky 9c0e234
💔 Build #221557 failed 9d6de35
💔 Build #221429 failed f528ea7

kibanamachine · 2024-07-19T13:05:26Z

💔 All backports failed

Status	Branch	Result
❌	8.15	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 187440

Questions ?

Please refer to the Backport tool documentation

## Summary During #187440 we lost some FTR configs, this PR adds it back.

@crespocarlos

## Summary Follow-up to #188825 @crespocarlos reported that some Oblt configs after missing after #187440 I was using `node scripts/check_ftr_configs.js` to validate I did not miss anything and decided to debug the script. We had a pretty strict config file content validation like `testRunner|testFiles`, that was skipping some FTR configs like `x-pack/test/apm_api_integration/basic/config.ts` I extended file content check to look for default export function and also skip test/suite or Cypress-own config files. In the end 7 FTR configs were discovered, but only 2 are with tests. I will ask owners to confirm if it should be enabled/disabled. Script run output: ``` node scripts/check_ftr_configs.js ERROR The following files look like FTR configs which are not listed in one of manifest files: - x-pack/plugins/observability_solution/uptime/e2e/config.ts - x-pack/test/functional_basic/apps/ml/config.base.ts - x-pack/test/functional_basic/apps/transform/config.base.ts - x-pack/test/security_solution_api_integration/config/ess/config.base.trial.ts - x-pack/test_serverless/functional/test_suites/observability/cypress/oblt_config.base.ts Make sure to add your new FTR config to the correct manifest file. Stateful tests: .buildkite/ftr_platform_stateful_configs.yml .buildkite/ftr_oblt_stateful_configs.yml .buildkite/ftr_security_stateful_configs.yml .buildkite/ftr_search_stateful_configs.yml Serverless tests: .buildkite/ftr_base_serverless_configs.yml .buildkite/ftr_oblt_serverless_configs.yml .buildkite/ftr_security_serverless_configs.yml .buildkite/ftr_search_serverless_configs.yml ERROR Please add the listed paths to the correct manifest file. If it's not an FTR config, you can add it to the IGNORED_PATHS in packages/kbn-test/src/functional_test_runner/lib/config/run_check_ftr_configs_cli.ts or contact #kibana-operations ```

kibanamachine · 2024-07-22T21:48:49Z