[Golang] OpenCensus Agent sample appears to not work

[DazWilkin]

https://opencensus.io/exporters/supported-exporters/go/ocagent/

With a configuration

receivers:
    opencensus:
        address: "127.0.0.1:55678"

exporters:
    datadog:
        namespace: ocagent
        enable_metrics: true
        enable_tracing: false
    prometheus:
        namespace: ocagent
        address: "localhost:9090"
    stackdriver:
        project: "[[REDACTED]]"
        metric_prefix: ocagent
        enable_metrics: true

zpages:
    port: 55679

The agent (!) appears to receive no data from the sample.

The agent isn't verbose (#453) but tweaking the configuration to use a Prometheus receiver and tweaking the sample to use the Prometheus exporter, works better (albeit imperfectly):

receivers:
    opencensus:
        address: "127.0.0.1:55678"
    prometheus:
      config:
        scrape_configs:
        - job_name: 'test'
          scrape_interval: 5s
          static_configs:
          - targets: ['localhost:8889']

exporters:
    datadog:
        namespace: ocagent
        enable_metrics: true
        enable_tracing: false
    prometheus:
        namespace: ocagent
        address: "localhost:9090"
    stackdriver:
        project: "dazwilkin-190221-trillian"
        metric_prefix: ocagent
        enable_metrics: true

zpages:
    port: 55679

and then

oce,err:=prometheus.NewExporter(prometheus.Options{
	Namespace: "ocagent",
})
if err != nil {
	log.Fatalf("Failed to create exporter: %v", err)
}
//trace.RegisterExporter(oce)
view.RegisterExporter(oce)

go func() {
	mux:=http.NewServeMux()
	mux.Handle("/metrics",oce)
	if err:=http.ListenAndServe(":8889",mux);err!=nil{
		log.Fatal(err)
	}
}()

NB retained oce for convenience.

I'm then able to curl the Prometheus exporter's metrics (:9090/metrics) which shows that the agent is proxying them from the receiver (:8889/metrics)

# HELP ocagent_ocagent_opdemo_latency The various latencies of the methods
# TYPE ocagent_ocagent_opdemo_latency histogram
ocagent_ocagent_opdemo_latency_bucket{client="cli",method="repl",le="10"} 0
ocagent_ocagent_opdemo_latency_bucket{client="cli",method="repl",le="50"} 1
ocagent_ocagent_opdemo_latency_bucket{client="cli",method="repl",le="100"} 8
ocagent_ocagent_opdemo_latency_bucket{client="cli",method="repl",le="200"} 12
ocagent_ocagent_opdemo_latency_bucket{client="cli",method="repl",le="400"} 14
ocagent_ocagent_opdemo_latency_bucket{client="cli",method="repl",le="800"} 16
ocagent_ocagent_opdemo_latency_bucket{client="cli",method="repl",le="1000"} 17
ocagent_ocagent_opdemo_latency_bucket{client="cli",method="repl",le="1400"} 17
ocagent_ocagent_opdemo_latency_bucket{client="cli",method="repl",le="2000"} 18
ocagent_ocagent_opdemo_latency_bucket{client="cli",method="repl",le="5000"} 19
ocagent_ocagent_opdemo_latency_bucket{client="cli",method="repl",le="10000"} 19
ocagent_ocagent_opdemo_latency_bucket{client="cli",method="repl",le="15000"} 39
ocagent_ocagent_opdemo_latency_bucket{client="cli",method="repl",le="+Inf"} 39
ocagent_ocagent_opdemo_latency_sum{client="cli",method="repl"} 20416.791576000163
ocagent_ocagent_opdemo_latency_count{client="cli",method="repl"} 39
# HELP ocagent_ocagent_opdemo_line_counts The counts of the lines in
# TYPE ocagent_ocagent_opdemo_line_counts counter
ocagent_ocagent_opdemo_line_counts{client="cli",method="repl"} 60
# HELP ocagent_ocagent_opdemo_line_lengths The lengths of the various lines in
# TYPE ocagent_ocagent_opdemo_line_lengths histogram
ocagent_ocagent_opdemo_line_lengths_bucket{client="cli",method="repl",le="10"} 0
ocagent_ocagent_opdemo_line_lengths_bucket{client="cli",method="repl",le="20"} 0
ocagent_ocagent_opdemo_line_lengths_bucket{client="cli",method="repl",le="50"} 2
ocagent_ocagent_opdemo_line_lengths_bucket{client="cli",method="repl",le="100"} 3
ocagent_ocagent_opdemo_line_lengths_bucket{client="cli",method="repl",le="150"} 9
ocagent_ocagent_opdemo_line_lengths_bucket{client="cli",method="repl",le="200"} 11
ocagent_ocagent_opdemo_line_lengths_bucket{client="cli",method="repl",le="500"} 14
ocagent_ocagent_opdemo_line_lengths_bucket{client="cli",method="repl",le="800"} 88
ocagent_ocagent_opdemo_line_lengths_bucket{client="cli",method="repl",le="+Inf"} 88
ocagent_ocagent_opdemo_line_lengths_sum{client="cli",method="repl"} 31003.999999999767
ocagent_ocagent_opdemo_line_lengths_count{client="cli",method="repl"} 88
# HELP ocagent_ocagent_opdemo_process_counts The various counts
# TYPE ocagent_ocagent_opdemo_process_counts counter
ocagent_ocagent_opdemo_process_counts{client="cli",method="repl"} 20

The Stackdriver Account receives the metrics (created) but no data is received:

GET https://monitoring.googleapis.com/v3/projects/[[REDACTED]]/metricDescriptors?filter=metric.type%3Dstarts_with(%22custom.googleapis.com%2Fopencensus%2Focagent%22)&fields=metricDescriptors%2FdisplayName&key={YOUR_API_KEY}
 
 
Response
 
200
 
- Show headers -
  
{
 "metricDescriptors": [
  {
   "displayName": "ocagent/ocagent_opdemo_latency"
  },
  {
   "displayName": "ocagent/ocagent_opdemo_line_counts"
  },
  {
   "displayName": "ocagent/ocagent_opdemo_line_lengths"
  },
  {
   "displayName": "ocagent/ocagent_opdemo_process_counts"
  }
 ]
}

The Datadog agent receives no data... I think the agent's Datadog metrics functionality is broken (#454)

[odeke-em]

Thank you for this report @DazWilkin!

The problem suffices because I believe the version of your opencensus-proto at $GOPATH/src/github.com/census-instrumentation/opencensus-proto is stale. To get this sample running, please prefix GO111MODULE=on in your environment so

GO111MODULE=on go run example/main.go

/cc @bogdandrutu @songy23 in regards to publishing opencensus-proto updates

[DazWilkin]

Thanks @odeke-em!

I am using the release binary after having problems building the agent from source.

[odeke-em]

@DazWilkin did GO111MODULE=on go run example/main.go make the problem go away?

[DazWilkin]

@odeke-em I'm using one of the release binaries because I had problems building from sources.

[odeke-em]

@DazWilkin yes I got that thanks! However, my question was about running the sample example that you said isn't working

[DazWilkin]

@odeke-em will try this today

[DazWilkin]

No joy but it's probably me and my lack of familiarity with go mod.

I copied the sample from the OpenCensus Agent page to a local directory... no go.mod. I ran GO111MODULE=on go run main.go. No evident change in behavior; not reporting measurements on Stackdriver. I blatted everything except ocagent and my main.go and tried again.

Then I tried getting what appears to be a most recent commit for opencensus-proto FC89

GO111MODULE=on \
go get github.com/census-instrumentation/opencensus-proto@192f9920ad6a40dfc65e2e6732134d0d45d3fc89

Then I tried using modules.... possibly incorrectly:

module github.com/DazWilkin/opencensus

go 1.12

require (
	contrib.go.opencensus.io/exporter/ocagent v0.4.6

	github.com/census-instrumentation/opencensus-proto 192f9920ad6a40dfc65e2e6732134d0d45d3fc89
	go.opencensus.io v0.19.0
)

But the opencensus-proto version appears to be overwritten by go run. Initially it appears to report that it's using it:

go: finding github.com/census-instrumentation/opencensus-proto 192f9920ad6a40dfc65e2e6732134d0d45d3fc89

and it modifies go.mod:

module github.com/DazWilkin/opencensus

go 1.12

require (
	contrib.go.opencensus.io/exporter/ocagent v0.4.6

	github.com/census-instrumentation/opencensus-proto v0.0.3-0.20190108184814-192f9920ad6a
	go.opencensus.io v0.19.0
)

but, by the time the code's running, go.mod is reverted:

module github.com/DazWilkin/opencensus

go 1.12

require (
	contrib.go.opencensus.io/exporter/ocagent v0.4.6

	github.com/census-instrumentation/opencensus-proto v0.1.0-0.20181214143942-ba49f56771b8
	go.opencensus.io v0.19.0
)

If this is just dumb-user error, let's drop it as I don't want to waste more of your time.

If it's not, would a good next step be for you to provide me with a definitive go.mod?

[DazWilkin]

@odeke-em what do you think: drop it or pursue?

I'm interested in revising some existing OpenCensus code to replace backend exporter-specific code (Datadog, Prometheus, Stackdriver) with the Agent Exporter.

I'm unable to pursue this until I can get the Agent Exporter working correctly.

Am I alone in having challenges with it? If so, I'll continue to try to get it to work. I did try again with a clean project, clean install on Thursday but same-same behavior.

[odeke-em]

@DazWilkin wanna have a call?

I copied the sample from the OpenCensus Agent page to a local directory... no go.mod. I ran GO111MODULE=on go run main.go. No evident change in behavior; not reporting measurements on Stackdriver. I blatted everything except ocagent and my main.go and tried again.

The go.mod file is the one in this repository which is what is missing. You had filed this issue for this repository hence the assumption that you were still using the entire repo. I have solved the exact same issue for others just by using this repo as is and running GO111MODULE=on go run example/main.go not by separately downloading files -- this is the step that's missing. Let's have a call though to fix this once and for all. My email is emmanuel at orijtech dot com.

[DazWilkin]

I think I've uncovered an issue.

The sample works after removing (!) the view.Distribution[s].

I revised the sample to use the stackdriver.NewExporter({...}) directly and received errors:

 Failed to export to Stackdriver: rpc error: code = InvalidArgument desc = Field timeSeries[0].points[0].distributionValue had an invalid value: Distribution value has 14 |bucket_counts| fields, which is more than the 13 buckets allowed by the bucketing options.

No matter how I tweaked the distribution buckets, the error merely revised 9 |bucket_counts| ... 8 allowed.

So I removed the view.Distribution[s] entirely and the sample now works with both the Stackdriver Exporter directly and indirectly with the Agent.

[DazWilkin]

Created an issue for this in the OpenCensus Go Exportoer for Stackdriver:
census-ecosystem/opencensus-go-exporter-stackdriver#98

[songy23]

#490 bumps up the oc-proto version.

[DazWilkin]

I'll keep an eye on it, thank you @songy23!

When it's merged, you are confident this will then work for me?

[songy23]

Agent-Exporter should work fine with Agent after that PR. Though we may need to do a bit more debugging with census-ecosystem/opencensus-go-exporter-stackdriver#98.

[pjanotti]

On the proto (used internally on the service) it is expected that if there are N-1 bucket bounds, there should be N bucket counts so this really seems an issue for the exporter (incorrect translation from proto).

[DazWilkin]

Folks -- willing to do legwork to get this issue resolved but, I'd really like to wait until there's assurance that the agent works with Stackdriver. What can I do to help get this progressed?