Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SIEM][Detection Engine] Fixes skipped tests #71347

Merged
merged 6 commits into from
Jul 10, 2020

Conversation

FrankHassanabad
Copy link
Contributor

@FrankHassanabad FrankHassanabad commented Jul 9, 2020

Summary

The errors that were coming back from the failed tests are in the afterEach and look to be network related or another test interfering:

1) detection engine api security and spaces enabled
01:59:54         find_statuses
01:59:54           "after each" hook for "should return a single rule status when a single rule is loaded from a find status with defaults added":
01:59:54       ResponseError: Response Error
01:59:54        at IncomingMessage.response.on (/dev/shm/workspace/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:287:25)
01:59:54        at endReadableNT (_stream_readable.js:1145:12)
01:59:54        at process._tickCallback (internal/process/next_tick.js:63:19)
01:59:54  
01:59:54               -  fail: "detection engine api security and spaces enabled find_statuses "after each" hook for "should return a single rule status when a single rule is loaded from a find status with defaults added""
01:59:54               
01:59:54               -> "after all" hook
01:59:54             -> "after all" hook
01:59:54         
01:59:54         │42 passing (2.0m)
01:59:54         │1 failing

So this should fix it to where the afterEach calls try up to 20 times before giving up and then on giving up they move on with the hope a different test doesn't fail.

Checklist

@FrankHassanabad FrankHassanabad requested review from a team as code owners July 9, 2020 22:49
@elasticmachine
Copy link
Contributor

Pinging @elastic/siem (Team:SIEM)

@FrankHassanabad FrankHassanabad added v9.0.0 v7.9.0 release_note:skip Skip the PR/issue when compiling release notes labels Jul 9, 2020
Copy link
Member

@spong spong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks for the improvements and (hopefully 🤞) taking the flake out of these tests @FrankHassanabad! 🙂

@FrankHassanabad
Copy link
Contributor Author

@elasticmachine merge upstream

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Build metrics

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@FrankHassanabad FrankHassanabad merged commit d8e9327 into elastic:master Jul 10, 2020
@FrankHassanabad FrankHassanabad deleted the fix-skipped-tests branch July 10, 2020 02:36
FrankHassanabad added a commit to FrankHassanabad/kibana that referenced this pull request Jul 10, 2020
## Summary

* elastic#69632
* Adds a retry loop in case of a network outage/issue which should increase the chances of success
* If there is still an issue after the 20th try, then it moves on and there is a high likelihood the tests will continue without issues.
* Adds console logging statements so we know if this flakiness happens again a bit more insight into why the network is behaving the way it is.
* Helps prevent the other tests from being skipped in the future due to bad networking issues. 

The errors that were coming back from the failed tests are in the `afterEach` and look to be network related or another test interfering:

```ts
1) detection engine api security and spaces enabled
01:59:54         find_statuses
01:59:54           "after each" hook for "should return a single rule status when a single rule is loaded from a find status with defaults added":
01:59:54       ResponseError: Response Error
01:59:54        at IncomingMessage.response.on (/dev/shm/workspace/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:287:25)
01:59:54        at endReadableNT (_stream_readable.js:1145:12)
01:59:54        at process._tickCallback (internal/process/next_tick.js:63:19)
01:59:54  
01:59:54               └- ✖ fail: "detection engine api security and spaces enabled find_statuses "after each" hook for "should return a single rule status when a single rule is loaded from a find status with defaults added""
01:59:54               │
01:59:54               └-> "after all" hook
01:59:54             └-> "after all" hook
01:59:54         │
01:59:54         │42 passing (2.0m)
01:59:54         │1 failing
```

So this should fix it to where the afterEach calls try up to 20 times before giving up and then on giving up they move on with the hope a different test doesn't fail.


### Checklist
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
jloleysens added a commit to jloleysens/kibana that referenced this pull request Jul 10, 2020
…11y-overlay

* 'master' of github.com:elastic/kibana: (33 commits)
  address index templates feedback (elastic#71353)
  Upgrade EUI to v26.3.1 (elastic#70243)
  [build] Creates Linux aarch64 archive (elastic#69165)
  [SIEM][Detection Engine] Fixes skipped tests (elastic#71347)
  [SIEM][Detection Engine][Lists] Adds read_privileges route for lists and list items
  [kbn/optimizer] implement "requiredBundles" property of KP plugins (elastic#70911)
  [Security Solution][Exceptions] - Exceptions modal pt 2 (elastic#70886)
  [ML] DF Analytics:  stop status polling when job stopped (elastic#71159)
  [SIEM][CASE] IBM Resilient Connector (elastic#66385)
  jenkins_xpack_saved_objects_field_metrics.sh expects to be run from the KIBANA_DIR in CI
  Deduplication of entries and items before sending to endpoint (elastic#71297)
  [services/remote/webdriver] fix eslint error (elastic#71346)
  send slack notifications on visual baseline failures
  fix visual regression job (elastic#70999)
  [Ingest Manager] Add schema to usageCollector. (elastic#71219)
  [ftr] use typed chromeOptions object, adding TEST_BROWSER_BINARY_PATH (elastic#71279)
  [Ingest Manager] Fix limited packages incorrect response (elastic#71292)
  Support multiple features declaring same properties (elastic#71106)
  [Security_Solution][Resolver]Add beta badge to Resolver panel (elastic#71183)
  [DOCS] Clarify trial subscription levels (elastic#70636)
  ...
FrankHassanabad added a commit that referenced this pull request Jul 10, 2020
## Summary

* #69632
* Adds a retry loop in case of a network outage/issue which should increase the chances of success
* If there is still an issue after the 20th try, then it moves on and there is a high likelihood the tests will continue without issues.
* Adds console logging statements so we know if this flakiness happens again a bit more insight into why the network is behaving the way it is.
* Helps prevent the other tests from being skipped in the future due to bad networking issues. 

The errors that were coming back from the failed tests are in the `afterEach` and look to be network related or another test interfering:

```ts
1) detection engine api security and spaces enabled
01:59:54         find_statuses
01:59:54           "after each" hook for "should return a single rule status when a single rule is loaded from a find status with defaults added":
01:59:54       ResponseError: Response Error
01:59:54        at IncomingMessage.response.on (/dev/shm/workspace/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:287:25)
01:59:54        at endReadableNT (_stream_readable.js:1145:12)
01:59:54        at process._tickCallback (internal/process/next_tick.js:63:19)
01:59:54  
01:59:54               └- ✖ fail: "detection engine api security and spaces enabled find_statuses "after each" hook for "should return a single rule status when a single rule is loaded from a find status with defaults added""
01:59:54               │
01:59:54               └-> "after all" hook
01:59:54             └-> "after all" hook
01:59:54         │
01:59:54         │42 passing (2.0m)
01:59:54         │1 failing
```

So this should fix it to where the afterEach calls try up to 20 times before giving up and then on giving up they move on with the hope a different test doesn't fail.


### Checklist
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
gmmorris added a commit to gmmorris/kibana that referenced this pull request Jul 13, 2020
* master: (78 commits)
  Bump lodash package version (elastic#71392)
  refactor: 💡 use allow-list in AppArch codebase (elastic#71400)
  improve bugfix 7198 test stability (elastic#71250)
  [Security Solution][Ingest Manager][Endpoint] Optional ingest manager (elastic#71198)
  [Metrics UI] Round metric threshold time buckets to nearest unit (elastic#71172)
  [Security Solution][Endpoint] Policy creation callback fixes + Improved error handling in user manifest loop (elastic#71269)
  [Security Solution] Allow to configure Event Renderers settings (elastic#69693)
  Fix a11y keyboard overlay (elastic#71214)
  [APM] UI text updates (elastic#71333)
  [Logs UI] Limit `extendDatemath` to valid ranges (elastic#71113)
  [SIEM] fix tooltip of notes (elastic#71342)
  address index templates feedback (elastic#71353)
  Upgrade EUI to v26.3.1 (elastic#70243)
  [build] Creates Linux aarch64 archive (elastic#69165)
  [SIEM][Detection Engine] Fixes skipped tests (elastic#71347)
  [SIEM][Detection Engine][Lists] Adds read_privileges route for lists and list items
  [kbn/optimizer] implement "requiredBundles" property of KP plugins (elastic#70911)
  [Security Solution][Exceptions] - Exceptions modal pt 2 (elastic#70886)
  [ML] DF Analytics:  stop status polling when job stopped (elastic#71159)
  [SIEM][CASE] IBM Resilient Connector (elastic#66385)
  ...
@MindyRS MindyRS added the Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. label Sep 23, 2021
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@jbudz jbudz added backport:skip This commit does not require backporting and removed v9.0.0 labels Sep 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport:skip This commit does not require backporting release_note:skip Skip the PR/issue when compiling release notes Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:SIEM v7.9.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants