Skip to content

Commit

Permalink
[SIEM][Detection Engine] Fixes skipped tests (#71347) (#71357)
Browse files Browse the repository at this point in the history
## Summary

* #69632
* Adds a retry loop in case of a network outage/issue which should increase the chances of success
* If there is still an issue after the 20th try, then it moves on and there is a high likelihood the tests will continue without issues.
* Adds console logging statements so we know if this flakiness happens again a bit more insight into why the network is behaving the way it is.
* Helps prevent the other tests from being skipped in the future due to bad networking issues. 

The errors that were coming back from the failed tests are in the `afterEach` and look to be network related or another test interfering:

```ts
1) detection engine api security and spaces enabled
01:59:54         find_statuses
01:59:54           "after each" hook for "should return a single rule status when a single rule is loaded from a find status with defaults added":
01:59:54       ResponseError: Response Error
01:59:54        at IncomingMessage.response.on (/dev/shm/workspace/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:287:25)
01:59:54        at endReadableNT (_stream_readable.js:1145:12)
01:59:54        at process._tickCallback (internal/process/next_tick.js:63:19)
01:59:54  
01:59:54               └- ✖ fail: "detection engine api security and spaces enabled find_statuses "after each" hook for "should return a single rule status when a single rule is loaded from a find status with defaults added""
01:59:54               │
01:59:54               └-> "after all" hook
01:59:54             └-> "after all" hook
01:59:54         │
01:59:54         │42 passing (2.0m)
01:59:54         │1 failing
```

So this should fix it to where the afterEach calls try up to 20 times before giving up and then on giving up they move on with the hope a different test doesn't fail.


### Checklist
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
  • Loading branch information
FrankHassanabad authored Jul 10, 2020
1 parent 1628ce4 commit e89fbd8
Show file tree
Hide file tree
Showing 2 changed files with 76 additions and 22 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,7 @@ export default ({ getService }: FtrProviderContext): void => {
const supertest = getService('supertest');
const es = getService('es');

// FLAKY: https://github.com/elastic/kibana/issues/69632
describe.skip('find_statuses', () => {
describe('find_statuses', () => {
beforeEach(async () => {
await createSignalsIndex(supertest);
});
Expand Down
95 changes: 75 additions & 20 deletions x-pack/test/detection_engine_api_integration/utils.ts
Original file line number Diff line number Diff line change
Expand Up @@ -235,50 +235,105 @@ export const getSimpleMlRuleOutput = (ruleId = 'rule-1'): Partial<RulesSchema> =

/**
* Remove all alerts from the .kibana index
* This will retry 20 times before giving up and hopefully still not interfere with other tests
* @param es The ElasticSearch handle
*/
export const deleteAllAlerts = async (es: Client): Promise<void> => {
await es.deleteByQuery({
index: '.kibana',
q: 'type:alert',
wait_for_completion: true,
refresh: true,
body: {},
});
export const deleteAllAlerts = async (es: Client, retryCount = 20): Promise<void> => {
if (retryCount > 0) {
try {
await es.deleteByQuery({
index: '.kibana',
q: 'type:alert',
wait_for_completion: true,
refresh: true,
body: {},
});
} catch (err) {
// eslint-disable-next-line no-console
console.log(`Failure trying to deleteAllAlerts, retries left are: ${retryCount - 1}`, err);
await deleteAllAlerts(es, retryCount - 1);
}
} else {
// eslint-disable-next-line no-console
console.log('Could not deleteAllAlerts, no retries are left');
}
};

/**
* Remove all rules statuses from the .kibana index
* This will retry 20 times before giving up and hopefully still not interfere with other tests
* @param es The ElasticSearch handle
*/
export const deleteAllRulesStatuses = async (es: Client): Promise<void> => {
await es.deleteByQuery({
index: '.kibana',
q: 'type:siem-detection-engine-rule-status',
wait_for_completion: true,
refresh: true,
body: {},
});
export const deleteAllRulesStatuses = async (es: Client, retryCount = 20): Promise<void> => {
if (retryCount > 0) {
try {
await es.deleteByQuery({
index: '.kibana',
q: 'type:siem-detection-engine-rule-status',
wait_for_completion: true,
refresh: true,
body: {},
});
} catch (err) {
// eslint-disable-next-line no-console
console.log(
`Failure trying to deleteAllRulesStatuses, retries left are: ${retryCount - 1}`,
err
);
await deleteAllRulesStatuses(es, retryCount - 1);
}
} else {
// eslint-disable-next-line no-console
console.log('Could not deleteAllRulesStatuses, no retries are left');
}
};

/**
* Creates the signals index for use inside of beforeEach blocks of tests
* This will retry 20 times before giving up and hopefully still not interfere with other tests
* @param supertest The supertest client library
*/
export const createSignalsIndex = async (
supertest: SuperTest<supertestAsPromised.Test>
supertest: SuperTest<supertestAsPromised.Test>,
retryCount = 20
): Promise<void> => {
await supertest.post(DETECTION_ENGINE_INDEX_URL).set('kbn-xsrf', 'true').send().expect(200);
if (retryCount > 0) {
try {
await supertest.post(DETECTION_ENGINE_INDEX_URL).set('kbn-xsrf', 'true').send();
} catch (err) {
// eslint-disable-next-line no-console
console.log(
`Failure trying to create the signals index, retries left are: ${retryCount - 1}`,
err
);
await createSignalsIndex(supertest, retryCount - 1);
}
} else {
// eslint-disable-next-line no-console
console.log('Could not createSignalsIndex, no retries are left');
}
};

/**
* Deletes the signals index for use inside of afterEach blocks of tests
* @param supertest The supertest client library
*/
export const deleteSignalsIndex = async (
supertest: SuperTest<supertestAsPromised.Test>
supertest: SuperTest<supertestAsPromised.Test>,
retryCount = 20
): Promise<void> => {
await supertest.delete(DETECTION_ENGINE_INDEX_URL).set('kbn-xsrf', 'true').send().expect(200);
if (retryCount > 0) {
try {
await supertest.delete(DETECTION_ENGINE_INDEX_URL).set('kbn-xsrf', 'true').send();
} catch (err) {
// eslint-disable-next-line no-console
console.log(`Failure trying to deleteSignalsIndex, retries left are: ${retryCount - 1}`, err);
await deleteSignalsIndex(supertest, retryCount - 1);
}
} else {
// eslint-disable-next-line no-console
console.log('Could not deleteSignalsIndex, no retries are left');
}
};

/**
Expand Down

0 comments on commit e89fbd8

Please sign in to comment.