Skip to content
This repository has been archived by the owner on Jul 31, 2023. It is now read-only.

goroutine leak detected #1191

Open
adamdecaf opened this issue Jan 15, 2020 · 4 comments
Open

goroutine leak detected #1191

adamdecaf opened this issue Jan 15, 2020 · 4 comments
Labels

Comments

@adamdecaf
Copy link

Please answer these questions before submitting a bug report.

What version of OpenCensus are you using?

v0.22.0

What version of Go are you using?

go1.13.6

What did you do?

I was running the new goleak tool on my codebase and ran into a leak coming from OpenCensus.

goleak: Errors on successful test run: found unexpected goroutines:
[Goroutine 37 in state select, with go.opencensus.io/stats/view.(*worker).start on top of the stack:
goroutine 37 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc000143310)
	/Users/adam/code/pkg/mod/[email protected]/stats/view/worker.go:154 +0x100
created by go.opencensus.io/stats/view.init.0
	/Users/adam/code/pkg/mod/[email protected]/stats/view/worker.go:32 +0x57
]

What did you expect to see?

I expected no goroutine leaks in libraries I depend on.

What did you see instead?

A crash from goleaks.

Additional context

I'm pulling this library in via gocloud.dev/secrets v0.17.0, but that doesn't matter here since v0.22.0 of OpenCensus is used in both.

This appears to be coming from internal global state of a default recorder being set.

https://github.com/census-instrumentation/opencensus-go/blob/v0.22.0/stats/view/worker.go#L30-L34

Also, there doesn't seem to be a way to retrieve this reporter to shutdown on application shutdown.

https://github.com/census-instrumentation/opencensus-go/search?q=DefaultRecorder&unscoped_q=DefaultRecorder

@adamdecaf adamdecaf added the bug label Jan 15, 2020
adamdecaf added a commit to adamdecaf/paygate that referenced this issue Jan 15, 2020
We can't fully enable this due to an issue with a downstream library,
but we did find a couple leaks to patch in our code.

Issue: census-instrumentation/opencensus-go#1191
@regularcoder
Copy link

regularcoder commented Apr 7, 2020

Here is a workaround that I used:

ignoreOpenCensus := goleak.IgnoreTopFunction("go.opencensus.io/stats/view.(*worker).start")
defer goleak.VerifyNone(suite.T(), ignoreOpenCensus)

@brandur
Copy link

brandur commented Nov 30, 2022

Just wanted to bump one and add a little context — a lot of us are running goleak in our tests because it's easy to make mistakes around leaking goroutines in Go, and once you have enough code written in a project, you probably are somewhere, and goleak is a great way to hedge against that and protect yourself against bad production problems before they happen.

We're not using this module directly, but it comes in transitively via a Google module we're using, and even when we don't use that Google module directly, this package's goroutine still starts because it's in an init, so we have to protect against its presence in every one of our project's package test suites.

I think you'd find reasonably good consensus (pun intended) out there that it's bad form to (1) starting goroutines in init, and (2) not provide any way of stopping goroutines that've started. Our project sits at around 160 module dependencies including transitive ones, and OpenCensus is the only one that behaves this way.

If you're open to it, even moving the goroutine to start lazily on first use would be an improvement. Google's Go APIs that are using OpenCensus provide a WithTelemetryDisabled option to disable it, but as far as the goroutine is concerned it doesn't make a difference — it starts regardless.

@dashpole
Copy link
Collaborator

dashpole commented Dec 2, 2022

@brandur That all sounds reasonable. Given the state of the project, we have to be careful not to break existing users, but lazily starting the goroutine should be backwards-compatible. Let me know if you are planning to put up a PR to fix it.

brandur added a commit to brandur/opencensus-go that referenced this issue Dec 11, 2022
See discussion in census-instrumentation#1191 for impetus, but in general, it's not
particularly good practice to start up goroutines in package `init`
functions, with one reason being that it means that they're started for
projects that don't even make use of the package, and maybe just have it
as a transitive dependency.

So here, remove the `view` package's default worker in favor of a new
function that'll lazily initialize one when any package-level functions
are called that use the default worker. We use a `sync.Once` that has an
optimized short-circuit path in case the work's already been done, so
new overhead should be negligible, and registering/unregistering views
is probably not a particularly common operation anyway.

This change is backwards compatible, with use of the package's API
staying exactly the same. Test coverage seems to be pretty good, with a
function that previously reset the default worker between test cases
which I've modified to instead reset the default worker back to its
uninitialized state, and thereby verify that the test cases can
initialize it if necessary.

I thought about trying to deinitialize the default worker in case its
last view is unregistered, but decided against it because (1) it's more
invasive, (2) it's not likely to actually help anyone in practice as
views probably stay registered for a program's entire duration anwyay,
and (3) its interaction with the existing package-level `Stop` function
would take some thinking through.
@brandur
Copy link

brandur commented Dec 11, 2022

@dashpole I opened a pull request at #1287 in case you have a change to take a look. I think it fulfills the backwards compatibility requirement.

@adamdecaf One thing I noticed while in there is that there is a package-level view.Stop() function that you can probably use to satisfy the goroutine checker.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

4 participants