Ensure functional testing is stable and reliable

[schrolla]

💡 Summary

Functional testing is built to be an automated process to quickly and regularly validate that ScubaGear functions as intended. Currently, some of the functional tests are failing regularly which impedes the value of those tests. This epic is meant to collect these issues to support fixes that improve functional test stability (Defender) and possibly coverage (SPO module).

Motivation and context

Automated functional testing saves developers time and improves the overall quality of the ScubaGear tool. Making functional tests more reliable and increasing their coverage will further increase code quality.

Implementation notes

Please provide details for implementation, such as:

• Review and test updates to resolve nightly testing errors with Defender
• Review possible updates to improve SharePoint automated testing related to SPO module
• Resolve other recurring errors in automated functional tests

Acceptance criteria

The following issues are completed

• Add status badge to readme for nightly product testing #883
• Defender functional tests fail due to timing/cache issues #1055
• AAD functional test 3.2 fails against our G3 test tenant #1168
• Sharepoint functional tests for policy 3.2 non applicable cases need to remove unnecessary fields to get rid of warnings #1268 (Does not affect automated tests, warnings only)
• Split nightly functional tests so TCOs can be notified upon failure #1235

[gdasher]

It also looks like the smoke tests need a once over. In particular, the markdown tests reference the wrong location for the baselines and fail on baseline update changes:

Testing/Functional/MarkdownChecks/ValidateMarkdowns.Tests.ps1

[james-garriss]

Ok, but they are distinct from the functional tests, so this problem would not be part of this epic. I will make sure that a new issue is created for the smoke tests.

[james-garriss]

This epic has been completed.