Skip to content

Investigating performance issues

Jump to bottom
Alexander Emelin edited this page Jul 11, 2024 · 7 revisions

First step to investigate what's going on with your Centrifugo is looking at information provided by info command. There are 2 ways to get this information:

  • send info command with HTTP API client
  • send info command from actions tab of admin web interface

In general, you should always configure Centrifugo monitoring to periodically collect Centrifugo metrics, save it to external monitoring system (Prometheus, Graphite) and see the state of system in Grafana. Metrics provide great insights.

But sometimes looking at metrics is not enough. In this case you can additionally capture CPU and heap profiles from running Centrifugo node.

To be able to do this Centrifugo must be run with debug configuration option enabled:

centrifugo --config=config.json --debug

Or with "debug": true in configuration file.

When debug enabled extra HTTP handlers become available on internal port.

Assuming your Centrifugo address is https://centrifugo.example.com then go to https://centrifugo.example.com/debug/pprof/ and you will find debug information.

If you modified internal address or port when starting Centrifugo you should use that custom address and port to access debug data.

To capture CPU profile:

curl "https://centrifugo.example.com/debug/pprof/profile" > /tmp/cpu.profile

To capture heap memory profile:

curl "https://centrifugo.example.com/debug/pprof/heap" > /tmp/heap.profile

To capture goroutine dump:

curl "https://centrifugo.example.com/debug/pprof/goroutine?debug=1" > /tmp/goroutine1.txt

And also capture another one (with ?debug=2):

curl "https://centrifugo.example.com/debug/pprof/goroutine?debug=2" > /tmp/goroutine2.txt

All profiles can provide some insights to what's going on and why Centrifugo consumes a lot of memory or lot of CPU time. Goroutine profiles can help to find goroutine leaks or deadlocks.

There many articles describing how to work with profiles in Go, some of them:

But you can just show you profiles to us (please put profiles into ZIP archive before uploading). And also provide a configuration file (with obfuscated secrets).

Clone this wiki locally