Datastore: difficult to decipher gRPC stack trace

[andrewpollock]

Hi,

We've seen some sort of failure in OSV.dev's API backend (in Cloud Run), where the instance's ability to make further Datastore queries was permanently impacted (we had to kill and restart it).

Any assistance on how to further troubleshoot what happened here would be appreciated.

Environment details

• Cloud Run
• Python version and virtual environment information: Python 3.11, per https://github.com/google/osv.dev/blob/master/gcp/api/Dockerfile
• google-cloud-ndb version: 2.1.1

Steps to reproduce

Unfortunately due to not logging the queries, we're unsure of the exact query that triggered this.

Code example

https://github.com/google/osv.dev/blob/aed410e4badbc073a6910c11c8ca57f9fc49310a/gcp/api/server.py#L316

specifically

https://github.com/google/osv.dev/blob/aed410e4badbc073a6910c11c8ca57f9fc49310a/gcp/api/server.py#L610

or

https://github.com/google/osv.dev/blob/aed410e4badbc073a6910c11c8ca57f9fc49310a/gcp/api/server.py#L663

Stack trace

Exception in thread Thread-11366361 (_run):
Traceback (most recent call last): File "/usr/local/lib/python3.11/threading.py", line 1038, in _bootstrap_inner self.run() File "/usr/local/lib/python3.11/threading.py", line 975, in run self._target(*self._args, **self._kwargs) File "src/python/grpcio/grpc/_cython/_cygrpc/thread.pyx.pxi", line 53, in grpc._cython.cygrpc._run_with_context._run File "/usr/local/lib/python3.11/site-packages/grpc/_channel.py", line 1392, in channel_spin
event = state.channel.next_call_event()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 489, in grpc._cython.cygrpc.Channel.next_call_event
File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 171, in grpc._cython.cygrpc._next_call_event
File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 172, in grpc._cython.cygrpc._next_call_event
File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 481, in grpc._cython.cygrpc.Channel.next_call_event.on_success
File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 263, in grpc._cython.cygrpc._process_integrated_call_tag
KeyError: <grpc._cython.cygrpc._BatchOperationTag object at 0x3eb06004e6b0>