Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Better backend health checks #1366

Closed
oliverchang opened this issue Jun 7, 2023 · 2 comments
Closed

Better backend health checks #1366

oliverchang opened this issue Jun 7, 2023 · 2 comments
Assignees
@oliverchang @michaelkedar
Labels
api API-related infrastructure infra infrastructure bugs/FRs priority

Comments

@oliverchang
Copy link
Collaborator

oliverchang commented Jun 7, 2023 •
edited
Loading

Based on our recent issues with the gRPC deadlock (googleapis/python-ndb#887) causing all subsequent requests to fail, we need to come up with a more reliable mechanism to detect dead Cloud Run API instances.
@oliverchang oliverchang added priority infra infrastructure bugs/FRs api API-related infrastructure labels Jun 7, 2023
@oliverchang
Copy link
Collaborator Author

oliverchang commented Jun 7, 2023 •
edited
Loading

We can set up a gRPC probe here: https://cloud.google.com/run/docs/configuring/healthchecks#grpc-liveness-probes

Then we can just point this to a new OSV API gRPC endpoint that performs a trivial Datastore read.

@oliverchang oliverchang self-assigned this Jun 7, 2023
oliverchang added a commit that referenced this issue Jun 7, 2023
@oliverchang
Implement basic gRPC health check protocol.
00eba20 
This health check method does a simple read from Datastore.

Per https://github.com/grpc/grpc/blob/master/doc/health-checking.md.

Watch() is not implemented, and the docs at
https://github.com/grpc/grpc/blob/28692fdcae144446fa84a05ef7d284d71bb3e5a1/src/proto/grpc/health/v1/health.proto#L58
seem to imply this is OK.

Part of #1366.
@oliverchang oliverchang mentioned this issue Jun 7, 2023
Merged
oliverchang added a commit that referenced this issue Jun 8, 2023
@oliverchang
Implement basic gRPC health check protocol. (#1368)
5aea60e 
This health check method does a simple read from Datastore.

Per https://github.com/grpc/grpc/blob/master/doc/health-checking.md.

Watch() is not implemented since it's a little complicated. The docs at

https://github.com/grpc/grpc/blob/28692fdcae144446fa84a05ef7d284d71bb3e5a1/src/proto/grpc/health/v1/health.proto#L58
seem to imply this is OK. This is corroborated by lots of other gRPC
users doing the same.

Part of #1366.
@oliverchang
Copy link
Collaborator Author

This was done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@oliverchang oliverchang

@michaelkedar michaelkedar

Labels
api API-related infrastructure infra infrastructure bugs/FRs priority
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@oliverchang @michaelkedar