-
Notifications
You must be signed in to change notification settings - Fork 10
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
9791f03
commit 16c09d4
Showing
7 changed files
with
356 additions
and
771 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -18,7 +18,7 @@ This utility uses similar flags, cancellation and timing snippets for the grpc c | |
| ### Build | ||
|
|
||
| You can either build from source | ||
| You can either build from source or with `bazel` | ||
|
|
||
| ```bash | ||
| go build -o grpc_health_proxy main.go | ||
|
|
@@ -146,6 +146,16 @@ Configuration options for HTTTP(s) listener supports TLS and mTLS | |
| | **`-https-listen-verify`** | option to enable mTLS for HTTPS requests | | ||
| | **`-https-listen-ca`** | trust CA for mTLS | | ||
|
|
||
|
|
||
| ## Prometheus Options | ||
|
|
||
| Configuration option for the Prometheus metrics listener endpoint and path | ||
|
|
||
| | Option | Description | | ||
| |:------------|-------------| | ||
| | **`-metrics-http-path`** | metrics endpoint path (default: /metics") | | ||
| | **`-metrics-http-listen-addr`** |http host:port for metrics endpoint (default: localhost:9000") | | ||
|
|
||
| ---- | ||
|
|
||
| [hc]: https://github.com/grpc/grpc/blob/master/doc/health-checking.md | ||
|
|
@@ -475,4 +485,88 @@ $ cosign verify [email protected] --certificate-oid | |
| # $ rekor-cli get --rekor_server https://rekor.sigstore.dev --log-index $LogIndex --format=json | jq '.' | ||
| ``` | ||
|
|
||
| These images were built using bazel so you should get the same container hash (i.e., deterministic builds) | ||
| #### Bazel | ||
|
|
||
| These images were built using bazel so you should get the same container hash (i.e., deterministic builds) | ||
|
|
||
| ```bash | ||
| bazel run :gazelle -- update-repos -from_file=go.mod -prune=true -to_macro=repositories.bzl%go_repositories | ||
|
|
||
| bazel run :server-image | ||
|
|
||
| bazel run :main -- --http-listen-addr localhost:8080 \ | ||
| --http-listen-path=/healthz \ | ||
| --grpcaddr localhost:50051 \ | ||
| --service-name echo.EchoServer \ | ||
| --logtostderr=1 -v 10 | ||
| ``` | ||
|
|
||
| ### Metrics | ||
|
|
||
| The healthcheck hander also exposes service statistics through prometheus endpoint. | ||
|
|
||
| * `grpc_health_check_seconds`: Histogram for the overall latency to http healtcheck endpoint (eg `/healthz`) | ||
| * `grpc_health_check_service_duration_seconds`: Histogram for the latency per serviceName | ||
| * `grpc_health_check_service_requests`: Counter and status per serviceName | ||
|
|
||
|
|
||
| To see this locally, run prometheus (i'm using docker here) | ||
|
|
||
| ```bash | ||
| docker run \ | ||
| --net=host \ | ||
| -p 9090:9090 \ | ||
| -v `pwd`/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml \ | ||
| prom/prometheus | ||
| ## ui: http://localhost:9090/graph?g0.expr=grpc_health_check_service_requests&g0.tab=1&g0.display_mode=lines&g0.show_exemplars=0&g0.range_input=1h | ||
|
|
||
| ## then send requests | ||
| # for ((i=1;i<=500;i++)); do curl -s --resolve 'http.domain.com:8080:127.0.0.1' "http://http.domain.com:8080/healthz"; sleep 1; done | ||
|
|
||
| ## or also attach grafana to the prometheus service | ||
| # docker run --net=host -p 3000:3000 grafana/grafana | ||
| ## ui: http://localhost:3000 | ||
| ``` | ||
|
|
||
| The sample metrics displayed at `http://localhost:9000/metrics` shows | ||
|
|
||
| ```log | ||
| # HELP grpc_health_check_seconds Duration of HTTP requests. | ||
| # TYPE grpc_health_check_seconds histogram | ||
| grpc_health_check_seconds_bucket{path="/healthz",le="0.005"} 447 | ||
| grpc_health_check_seconds_bucket{path="/healthz",le="0.01"} 448 | ||
| grpc_health_check_seconds_bucket{path="/healthz",le="0.025"} 448 | ||
| grpc_health_check_seconds_bucket{path="/healthz",le="0.05"} 448 | ||
| grpc_health_check_seconds_bucket{path="/healthz",le="0.1"} 448 | ||
| grpc_health_check_seconds_bucket{path="/healthz",le="0.25"} 448 | ||
| grpc_health_check_seconds_bucket{path="/healthz",le="0.5"} 448 | ||
| grpc_health_check_seconds_bucket{path="/healthz",le="1"} 448 | ||
| grpc_health_check_seconds_bucket{path="/healthz",le="2.5"} 448 | ||
| grpc_health_check_seconds_bucket{path="/healthz",le="5"} 448 | ||
| grpc_health_check_seconds_bucket{path="/healthz",le="10"} 448 | ||
| grpc_health_check_seconds_bucket{path="/healthz",le="+Inf"} 448 | ||
| grpc_health_check_seconds_sum{path="/healthz"} 0.8027037920000004 | ||
| grpc_health_check_seconds_count{path="/healthz"} 448 | ||
|
|
||
| # HELP grpc_health_check_service_duration_seconds Duration of HTTP requests per service. | ||
| # TYPE grpc_health_check_service_duration_seconds histogram | ||
| grpc_health_check_service_duration_seconds_bucket{service_name="echo.EchoServer",le="0.005"} 447 | ||
| grpc_health_check_service_duration_seconds_bucket{service_name="echo.EchoServer",le="0.01"} 448 | ||
| grpc_health_check_service_duration_seconds_bucket{service_name="echo.EchoServer",le="0.025"} 448 | ||
| grpc_health_check_service_duration_seconds_bucket{service_name="echo.EchoServer",le="0.05"} 448 | ||
| grpc_health_check_service_duration_seconds_bucket{service_name="echo.EchoServer",le="0.1"} 448 | ||
| grpc_health_check_service_duration_seconds_bucket{service_name="echo.EchoServer",le="0.25"} 448 | ||
| grpc_health_check_service_duration_seconds_bucket{service_name="echo.EchoServer",le="0.5"} 448 | ||
| grpc_health_check_service_duration_seconds_bucket{service_name="echo.EchoServer",le="1"} 448 | ||
| grpc_health_check_service_duration_seconds_bucket{service_name="echo.EchoServer",le="2.5"} 448 | ||
| grpc_health_check_service_duration_seconds_bucket{service_name="echo.EchoServer",le="5"} 448 | ||
| grpc_health_check_service_duration_seconds_bucket{service_name="echo.EchoServer",le="10"} 448 | ||
| grpc_health_check_service_duration_seconds_bucket{service_name="echo.EchoServer",le="+Inf"} 448 | ||
| grpc_health_check_service_duration_seconds_sum{service_name="echo.EchoServer"} 0.7760661480000001 | ||
| grpc_health_check_service_duration_seconds_count{service_name="echo.EchoServer"} 448 | ||
|
|
||
| # HELP grpc_health_check_service_requests backend status, partitioned by status code and service_name. | ||
| # TYPE grpc_health_check_service_requests counter | ||
| grpc_health_check_service_requests{code="NOT_SERVING",service_name="echo.EchoServer"} 32 | ||
| grpc_health_check_service_requests{code="SERVING",service_name="echo.EchoServer"} 416 | ||
| ``` | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| global: | ||
| scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute. | ||
| evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute. | ||
| scrape_configs: | ||
|
|
||
| - job_name: 'grpc_health_proxy' | ||
| metrics_path: '/metrics' | ||
| static_configs: | ||
| - targets: ['localhost:9000'] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,47 +1,32 @@ | ||
| cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= | ||
| github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= | ||
| github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw= | ||
| github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b h1:VKtxabqXZkF25pY9ekfRL6a582T4P37/31XEstQ5p58= | ||
| github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= | ||
| github.com/golang/glog v1.2.0 h1:uCdmnmatrKCgMBlM4rMuJZWOkPDqdbZPnrMXDY4gI68= | ||
| github.com/golang/glog v1.2.0/go.mod h1:6AhwSGph0fcJtXVM/PEHPqZlFeoLxhs7/t5UDAwmO+w= | ||
| github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= | ||
| github.com/golang/protobuf v1.2.0 h1:P3YflyNX/ehuJFLhxviNdFxQPkGK5cDcApsge1SqnvM= | ||
| github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= | ||
| github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek= | ||
| github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps= | ||
| github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= | ||
| golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= | ||
| golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= | ||
| golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= | ||
| golang.org/x/net v0.0.0-20190620200207-3b0461eec859 h1:R/3boaszxrf1GEUWTVDzSKVwLmSJpwZ1yqXm8j0v2QI= | ||
| golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= | ||
| golang.org/x/net v0.22.0 h1:9sGLhx7iRIHEiX0oAJ3MRZMUCElJgy7Br1nO+AMN3Tc= | ||
| golang.org/x/net v0.22.0/go.mod h1:JKghWKKOSdJwpW2GEx0Ja7fmaKnMsbu+MWVZTokSYmg= | ||
| golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= | ||
| golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f h1:wMNYb4v58l5UBM7MYRLPG6ZhfOqbKu7X5eyFl8ZhKvA= | ||
| golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= | ||
| golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a h1:1BGLXjeY4akVXGgbC9HugT3Jv3hCI0z56oJR5vAMgBU= | ||
| golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= | ||
| golang.org/x/sys v0.18.0 h1:DBdB3niSjOA/O0blCZBqDefyWNYveAYMNF1Wum0DYQ4= | ||
| golang.org/x/sys v0.18.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= | ||
| golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg= | ||
| golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= | ||
| github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= | ||
| github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= | ||
| github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= | ||
| github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= | ||
| github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= | ||
| github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= | ||
| github.com/golang/glog v1.2.1 h1:OptwRhECazUx5ix5TTWC3EZhsZEHWcYWY4FQHTIubm4= | ||
| github.com/golang/glog v1.2.1/go.mod h1:6AhwSGph0fcJtXVM/PEHPqZlFeoLxhs7/t5UDAwmO+w= | ||
| github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI= | ||
| github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= | ||
| github.com/gorilla/mux v1.8.1 h1:TuBL49tXwgrFYWhqrNgrUNEY92u81SPhu7sTdzQEiWY= | ||
| github.com/gorilla/mux v1.8.1/go.mod h1:AKf9I4AEqPTmMytcMc0KkNouC66V3BtZ4qD5fmWSiMQ= | ||
| github.com/prometheus/client_golang v1.19.0 h1:ygXvpU1AoN1MhdzckN+PyD9QJOSD4x7kmXYlnfbA6JU= | ||
| github.com/prometheus/client_golang v1.19.0/go.mod h1:ZRM9uEAypZakd+q/x7+gmsvXdURP+DABIEIjnmDdp+k= | ||
| github.com/prometheus/client_model v0.6.1 h1:ZKSh/rekM+n3CeS952MLRAdFwIKqeY8b62p8ais2e9E= | ||
| github.com/prometheus/client_model v0.6.1/go.mod h1:OrxVMOVHjw3lKMa8+x6HeMGkHMQyHDk9E3jmP2AmGiY= | ||
| github.com/prometheus/common v0.53.0 h1:U2pL9w9nmJwJDa4qqLQ3ZaePJ6ZTwt7cMD3AG3+aLCE= | ||
| github.com/prometheus/common v0.53.0/go.mod h1:BrxBKv3FWBIGXw89Mg1AeBq7FSyRzXWI3l3e7W3RN5U= | ||
| github.com/prometheus/procfs v0.14.0 h1:Lw4VdGGoKEZilJsayHf0B+9YgLGREba2C6xr+Fdfq6s= | ||
| github.com/prometheus/procfs v0.14.0/go.mod h1:XL+Iwz8k8ZabyZfMFHPiilCniixqQarAy5Mu67pHlNQ= | ||
| golang.org/x/net v0.24.0 h1:1PcaxkF854Fu3+lvBIx5SYn9wRlBzzcnHZSiaFFAb0w= | ||
| golang.org/x/net v0.24.0/go.mod h1:2Q7sJY5mzlzWjKtYUEXSlBWCdyaioyXzRB2RtU8KVE8= | ||
| golang.org/x/sys v0.19.0 h1:q5f1RH2jigJ1MoAWp2KTp3gm5zAGFUTarQZ5U386+4o= | ||
| golang.org/x/sys v0.19.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= | ||
| golang.org/x/text v0.14.0 h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ= | ||
| golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU= | ||
| golang.org/x/tools v0.0.0-20190311212946-11955173bddd h1:/e+gpKk9r3dJobndpTytxS2gOy6m5uvpg+ISQoEcusQ= | ||
| golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= | ||
| google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= | ||
| google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8 h1:Nw54tB0rB7hY/N0NQvRW8DG4Yk3Q6T9cu9RcFQDu1tc= | ||
| google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= | ||
| google.golang.org/genproto v0.0.0-20240123012728-ef4313101c80 h1:KAeGQVN3M9nD0/bQXnr/ClcEMJ968gUXJQ9pwfSynuQ= | ||
| google.golang.org/genproto v0.0.0-20240123012728-ef4313101c80/go.mod h1:cc8bqMqtv9gMOr0zHg2Vzff5ULhhL2IXP4sbcn32Dro= | ||
| google.golang.org/genproto/googleapis/rpc v0.0.0-20240325203815-454cdb8f5daa h1:RBgMaUMP+6soRkik4VoN8ojR2nex2TqZwjSSogic+eo= | ||
| google.golang.org/genproto/googleapis/rpc v0.0.0-20240325203815-454cdb8f5daa/go.mod h1:WtryC6hu0hhx87FDGxWCDptyssuo68sk10vYjF+T9fY= | ||
| google.golang.org/grpc v1.21.1 h1:j6XxA85m/6txkUCHvzlV5f+HBNl/1r5cZ2A/3IEFOO8= | ||
| google.golang.org/grpc v1.21.1/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM= | ||
| google.golang.org/grpc v1.62.1 h1:B4n+nfKzOICUXMgyrNd19h/I9oH0L1pizfk1d4zSgTk= | ||
| google.golang.org/grpc v1.62.1/go.mod h1:IWTG0VlJLCh1SkC58F7np9ka9mx/WNkjl4PGJaiq+QE= | ||
| google.golang.org/genproto/googleapis/rpc v0.0.0-20240415180920-8c6c420018be h1:LG9vZxsWGOmUKieR8wPAUR3u3MpnYFQZROPIMaXh7/A= | ||
| google.golang.org/genproto/googleapis/rpc v0.0.0-20240415180920-8c6c420018be/go.mod h1:WtryC6hu0hhx87FDGxWCDptyssuo68sk10vYjF+T9fY= | ||
| google.golang.org/grpc v1.63.2 h1:MUeiw1B2maTVZthpU5xvASfTh3LDbxHd6IJ6QQVU+xM= | ||
| google.golang.org/grpc v1.63.2/go.mod h1:WAX/8DgncnokcFUldAxq7GeB5DXHDbMF+lLvDomNkRA= | ||
| google.golang.org/protobuf v1.33.0 h1:uNO2rsAINq/JlFpSdYEKIZ0uKD/R9cpdv0T+yoGwGmI= | ||
| google.golang.org/protobuf v1.33.0/go.mod h1:c6P6GXX6sHbq/GpV6MGZEdwhWPcYBgnhAHhKbcUYpos= | ||
| honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= |
Oops, something went wrong.