feat: implement opentelemetry for distributed tracing

[namkyu1999]

What this PR does / why we need it:

This pr implements opentelemetry codes according to this proposal.

Parent context and OTEL endpoint are passed through env(TRACE_PARENT, OTEL_EXPORTER_OTLP_ENDPOINT)

Result is like below

---

relevant PRs

• proposal docs
• chaos-runner
• chaos-operator

Checklist:

• Fixes #
• PR messages has document related information
• Labelled this PR & related issue with breaking-changes tag
• PR messages has breaking changes related information
• Labelled this PR & related issue with requires-upgrade tag
• PR messages has upgrade related information
• Commit has unit tests
• Commit has integration tests
• E2E run Required for the changes

[xavierW]

 邮件已收到，我会尽快阅读并回复，谢谢

[smitthakkar96]

This is a great start however I don't see any place where we are marking span as error in case of failure, do we plan to do this in future iterations?

Based on the experiment context it might be good to explore what span attributes might be relevant.

Lastly to be able to correlate logs with traces can we add traceID to logs?

[namkyu1999]

@smitthakkar96

This is a great start however I don't see any place where we are marking span as error in case of failure, do we plan to do this in future iterations?

I will record errors like below

if err != nil {
	span.SetStatus(codes.Error, "operationThatCouldFail failed")
	span.RecordError(err)
}

But we have over 50 chaos faults, so it can be merged in the next PR (it is a time-consuming job)

Based on the experiment context it might be good to explore what span attributes might be relevant.

Does this mean adding a span attribute to help with future queries by attribute?

Lastly to be able to correlate logs with traces can we add traceID to logs?

sure, I will add it

[smitthakkar96]

Does this mean adding a span attribute to help with future queries by attribute?

Span Attributes can be useful for queries but also they can serve as additional context when reviewing the spans. Let's take an example, you get an http request, you might wanna log the relevant headers, the request body and the path similarly in context of the experiments it might be helpful to log the parameters for the experiment, maybe the targets and any other data you think that might be helpful when looking at an error trace :)

The demo application might be a good inspiration, another source of inspiration is semantic attributes.

Lastly for future I would suggest we increase the coverage of instrumentation (can be a good task for LFX contributors) by creating spans for calls to external services such as AWS, GCP, K8s or any HTTP or DB calls this contrib package has some out of the box shims that can make it easy

But we have over 50 chaos faults, so it can be merged in the next PR (it is a time-consuming job)

Totally understandable, instrumentation quality and coverage can always be improved overtime. Your initiative paves the foundation we can build on top of, my comment was just suggestions/ideas if you have sometime maybe raise log them as Github issues so other contributors can pick them up

[namkyu1999]

Thanks for your detailed comments @smitthakkar96 :) I am going to pave the foundation for collaborating with other contributors. Adopting a contrib package is also good idea

[namkyu1999]

I updated contribute/developer-guide folder for using context.Context

context.Context makes context propagation automatically so i fixed all of the functions to add context.Context as a parameter.

Here is a screen shot of distributed tracing. It's a result of pod-delete chaos fault.

[ispeakc0de]

As we are creating one more ctx context, it will shadow the above context var created with the same name.
Which can cause the defer function not to work. Can we use a different var name here to avoid var shadow?

[namkyu1999]

fixed here fix: avoid shadowing

[ispeakc0de]

We init otel SDK if otelExporterEndpoint != "". Here we are calling Tracer function without checking if the endpoint is set or not. Will it handle negative cases implicitly or should we call this conditionally?

[namkyu1999]

Basically, They do nothing when Otel SDK is not init.

here's a otel docs

and here's a reference PR

[namkyu1999]

enhance coverage

AS-IS
span: InjectXXXChaos

TO-BE
parent span: PrepareXXXFault
child span: InjectXXXFaultIn(Serial/Parallel)Mode