Repository navigation
fix(storage): send gRPC keepalive pings on AsyncGrpcClient channels - #19167
Draft
zhixiangli wants to merge 2 commits into
Draft
zhixiangli wants to merge 2 commits into
zhixiangli wants to merge 2 commits into
Conversation
Contributor
There was a problem hiding this comment.
Code Review
This pull request configures HTTP/2 keepalive options for the asynchronous gRPC client to detect dead connections and prevent indefinite hangs during active RPCs or bidirectional streams. It sets keepalive time to 30 seconds, keepalive timeout to 20 seconds, and ping timeout to 20 seconds, while keeping keepalive without calls disabled to avoid triggering GCS frontend protections. Corresponding unit tests have been updated and a new test has been added to verify these configurations. No review comments were provided, and the implementation is clean and well-tested.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Enable client-side HTTP/2 keepalive on the gRPC channels created by
AsyncGrpcClient, so an open RPC or bidi stream fails fast when the connection is dead instead of hanging.Without HTTP/2 keepalive, a pending
recv()on a bidi read/write stream (e.g.AsyncMultiRangeDownloader,AsyncAppendableObjectWriter) can block indefinitely when the path to the server silently drops traffic or the peer stops responding while its kernel still ACKs TCP keepalives. Linux TCP keepalive defaults to ~2 hours and does not catch the latter case.Channel options added (both the default and the DirectPath-over-Interconnect paths):
grpc.keepalive_time_msgrpc.keepalive_timeout_msgrpc.http2.ping_timeout_msgrpc.keepalive_permit_without_callsis intentionally left unset, so idle channels send no pings and don't riskGOAWAY too_many_pingsfrom GCS frontends. When a ping times out, the RPC fails withUNAVAILABLE, which the existing retry/resume logic handles.The sync
GrpcClientis unchanged.Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly: