Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prevent Execution of Delayed Tasks for Cancelled gRPC Requests #6066

Merged
merged 5 commits into from
Jan 14, 2025

Conversation

minwoox
Copy link
Contributor

@minwoox minwoox commented Jan 11, 2025

Motivation:
When blockingTaskExecutor is used in a gRPC service, tasks handling gRPC requests are queued if the executor is busy with heavy blocking jobs.

blockingExecutor.execute(() -> invokeOnMessage(request, endOfStream));
If these tasks are delayed for too long, the corresponding gRPC requests may time out and get cancelled. Despite the cancellation, the queued tasks are still executed once the executor becomes available, leading to unnecessary processing.
private void invokeOnMessage(I request, boolean halfClose) {
try (SafeCloseable ignored = ctx.push()) {
assert listener != null;
listener.onMessage(request);
if (halfClose) {
listener.onHalfClose();
}
} catch (Throwable cause) {
close(cause);
}
}

Modifications:

  • Updated AbstractServerCall.invokeOnMessage() to check if the request is cancelled before calling listener.onMessage.

Result:

  • Prevents execution of delayed tasks for gRPC requests that have already been cancelled.

Co-authored-by: szymon.habrainski [email protected]

Motivation:
When blockingTaskExecutor is used in a gRPC service, tasks handling gRPC requests are queued if the executor is busy with heavy blocking jobs.
https://github.com/line/armeria/blob/0960d091bfc7f350c17e68f57cc627de584b9705/grpc/src/main/java/com/linecorp/armeria/internal/server/grpc/AbstractServerCall.java#L363
If these tasks are delayed for too long, the corresponding gRPC requests may time out and get cancelled.
Despite the cancellation, the queued tasks are still executed once the executor becomes available, leading to unnecessary processing.
https://github.com/line/armeria/blob/0960d091bfc7f350c17e68f57cc627de584b9705/grpc/src/main/java/com/linecorp/armeria/internal/server/grpc/AbstractServerCall.java#L394-L404

Modifications:
- Updated `AbstractServerCall.invokeOnMessage()` to check if the request is cancelled before calling `listener.onMessage`.

Result:
- Prevents execution of delayed tasks for gRPC requests that have already been cancelled.
@minwoox minwoox added the defect label Jan 11, 2025
@minwoox minwoox added this to the 1.32.0 milestone Jan 11, 2025
@ikhoon ikhoon merged commit 8ae035e into line:main Jan 14, 2025
13 of 14 checks passed
@minwoox minwoox deleted the fix_grpc_blocking branch January 14, 2025 02:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants