Memory leak of spans with grpc server stats handler

[atoulme]

Description

See open-telemetry/opentelemetry-collector#11875 for original filing.

The gRPC server stats handler set by the OTLP receiver exposes a leak of spans over a long period of time.

From review of the code, it is possible the spans are not always closed by the stats_handler.go.

The span.End() call is made upon receiving the *stats.End event:

opentelemetry-go-contrib/instrumentation/google.golang.org/grpc/otelgrpc/stats_handler.go

Line 205 in eb2064c

span.End()

It seems in light of the leak, given the pprof dumps provided in the original report, that the event is not always received.

[dmathieu]

cc @dashpole as code owner.

[dashpole]

Updated the title and description. Looks like you are using the stats handler, rather than the interceptor.