Separate integration tests suite (+ OTEL AWS account)

[ghost]

Instrumentation codebase contains quite a bit integration tests that depend on a behaviour of an external system. Most notable example here is AWS SDK instrumentation (and coupled Apache Camel 2.x propagation testing). In order to keep the test stable and decouple from such external systems we use localstack (and separately elasticmq for SQS). The problem is that with new features regularly coming to AWS stack, our testing stack lags behind and therefore some tests have to be disabled (see SNS and S3 tests).

The idea to solve this is as follows:

• setup an OTEL AWS account for testing (would need a sponsorship for this I guess)
• move all of the tests currently unsupported by localstack to a separate suite running on live AWS
• configure the suite not to fail build even if some tests fail (to avoid flakiness due to ext system dependency) - failures should be reviewed by the developer / approver of course

Benefits:

• modern, unsupported by localstack features still tested (example - X-Ray propagation in AWS)
• reduced flakiness - risky tests moved to a separate suite

/CC @anuraaga @johnbley

[trask]

setup an OTEL AWS account for testing (would need a sponsorship for this I guess)

Check out open-telemetry/community#392

To get access to CNCF AWS account, we need to open community issue and provide rough estimate of cost for our needs.

configure the suite not to fail build even if some tests fail (to avoid flakiness due to ext system dependency) - failures should be reviewed by the developer / approver of course

we could run this in Nightly, which automatically opens a github issue on failure so failures won't go undetected

[ghost]

Thanks for looking into it. I can definitely run the process for getting access to CNCF AWS account, provided we (meaning - instrumentation folks) believe that the proposed test suite split would be beneficial for the project. @iNikem any thoughts on this?

[iNikem]

I agree with Trask.

[ghost]

Thanks @trask and @iNikem for your input, I've just opened a community issue for this: open-telemetry/community#693 As the costs are not high I hope that it will be approved soon :)

[ghost]

Community issue (open-telemetry/community#693) is now closed, I'll proceed with the implementation (move AWS - dependant tests to a new suite run with Nightly).