Skip to content

fix(storage): send gRPC keepalive pings on AsyncGrpcClient channels - #19167

Draft
zhixiangli wants to merge 2 commits into
googleapis:mainfrom
zhixiangli:storage-grpc-keepalive
Draft

zhixiangli wants to merge 2 commits into
googleapis:mainfrom
zhixiangli:storage-grpc-keepalive

Conversation

@zhixiangli

Copy link
Copy Markdown
Contributor

Enable client-side HTTP/2 keepalive on the gRPC channels created by AsyncGrpcClient, so an open RPC or bidi stream fails fast when the connection is dead instead of hanging.

Without HTTP/2 keepalive, a pending recv() on a bidi read/write stream (e.g. AsyncMultiRangeDownloader, AsyncAppendableObjectWriter) can block indefinitely when the path to the server silently drops traffic or the peer stops responding while its kernel still ACKs TCP keepalives. Linux TCP keepalive defaults to ~2 hours and does not catch the latter case.

Channel options added (both the default and the DirectPath-over-Interconnect paths):

Option Value Why
grpc.keepalive_time_ms 30000 Ping every 30 s while an RPC is open
grpc.keepalive_timeout_ms 20000 Drop the connection if a ping is not ACKed within 20 s
grpc.http2.ping_timeout_ms 20000 grpc-python otherwise waits its 60 s default for the ping ACK

grpc.keepalive_permit_without_calls is intentionally left unset, so idle channels send no pings and don't risk GOAWAY too_many_pings from GCS frontends. When a ping times out, the RPC fails with UNAVAILABLE, which the existing retry/resume logic handles.

The sync GrpcClient is unchanged.

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request configures HTTP/2 keepalive options for the asynchronous gRPC client to detect dead connections and prevent indefinite hangs during active RPCs or bidirectional streams. It sets keepalive time to 30 seconds, keepalive timeout to 20 seconds, and ping timeout to 20 seconds, while keeping keepalive without calls disabled to avoid triggering GCS frontend protections. Corresponding unit tests have been updated and a new test has been added to verify these configurations. No review comments were provided, and the implementation is clean and well-tested.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant