Repository navigation
fix(storage): make AsyncMultiRangeDownloader reopen and close cancellation-safe - #19169
zhixiangli wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request improves the robustness of stream handling and resource cleanup in the asynchronous Google Cloud Storage client. Specifically, it raises a retryable ServiceUnavailable exception if a send is attempted on a closed stream multiplexer, and wraps resource cleanup in AsyncMultiRangeDownloader.close() with a try...finally block to ensure references are cleared even if an error occurs. Unit tests are also added to verify these behaviors. Feedback suggests further improving the cleanup logic by wrapping each individual resource's close call in its own try-except block to prevent a failure in one from blocking the closure of others.
| try: | ||
| if self._pool: | ||
| await self._pool.close() | ||
|
|
||
| if self._multiplexer: | ||
| await self._multiplexer.close() | ||
| self._multiplexer = None | ||
| if self._multiplexer: | ||
| await self._multiplexer.close() | ||
|
|
||
| if self.read_obj_str: | ||
| try: | ||
| if getattr(self.read_obj_str, "is_stream_open", True): | ||
| res = self.read_obj_str.close() | ||
| if inspect.isawaitable(res): | ||
| await res | ||
| except ( | ||
| ValueError, | ||
| asyncio.CancelledError, | ||
| exceptions.GoogleAPICallError, | ||
| ): | ||
| pass | ||
| self.read_obj_str = None | ||
| self._primary_stream = None | ||
| self._is_stream_open = False | ||
| if self.read_obj_str: | ||
| try: | ||
| if getattr(self.read_obj_str, "is_stream_open", True): | ||
| res = self.read_obj_str.close() | ||
| if inspect.isawaitable(res): | ||
| await res | ||
| except ( | ||
| ValueError, | ||
| exceptions.GoogleAPICallError, | ||
| ): | ||
| pass | ||
| finally: | ||
| self._pool = None | ||
| self._multiplexer = None | ||
| self.read_obj_str = None | ||
| self._primary_stream = None | ||
| self._is_stream_open = False |
There was a problem hiding this comment.
When closing multiple resources during cleanup, wrap each close call in a try-except block to handle exceptions individually, log any failures as warnings, and use a finally block to clear the resource references. This ensures that a failure in closing one resource does not prevent other resources from being closed.
try:
if self._pool:
await self._pool.close()
except Exception as e:
logger.warning("Failed to close pool: %s", e)
try:
if self._multiplexer:
await self._multiplexer.close()
except Exception as e:
logger.warning("Failed to close multiplexer: %s", e)
try:
if self.read_obj_str:
if getattr(self.read_obj_str, 'is_stream_open', True):
res = self.read_obj_str.close()
if inspect.isawaitable(res):
await res
except Exception as e:
logger.warning("Failed to close stream: %s", e)
finally:
self._pool = None
self._multiplexer = None
self.read_obj_str = None
self._primary_stream = None
self._is_stream_open = FalseReferences
- When closing multiple resources during cleanup, wrap each close call in a try-except block to handle exceptions individually, log any failures as warnings, and use a finally block to clear the resource references.
Fix two cancellation edge cases on
AsyncMultiRangeDownloaderand_StreamMultiplexerwhen callers cancel or time outdownload_ranges()orclose():_StreamMultiplexer.send): When an in-flight stream reopen is cancelled after closing the old stream,_streamon_StreamMultiplexerremains closed at the same_stream_generation. Previously, a subsequentsend()attempted to write to the closed stream and raised a non-retryableValueError("Stream is not open"). Now_StreamMultiplexer.sendchecksself._stream.is_stream_openand raises retryablegoogle.api_core.exceptions.ServiceUnavailableso the caller's retry path reopens the stream cleanly.AsyncMultiRangeDownloader.close): Stop swallowingasyncio.CancelledErrorwhen closingread_obj_str, and wrap teardown intry ... finallyso_pool,_multiplexer,read_obj_str,_primary_stream, and_is_stream_open = Falseare always cleared even ifclose()is cancelled partway through.Changes
packages/google-cloud-storage/google/cloud/storage/asyncio/_stream_multiplexer.py: raise retryableexceptions.ServiceUnavailablein_StreamMultiplexer.sendwhennot self._stream.is_stream_open.packages/google-cloud-storage/google/cloud/storage/asyncio/async_multi_range_downloader.py: stop catchingasyncio.CancelledErrorand wrap teardown intry ... finallyinsideAsyncMultiRangeDownloader.close.packages/google-cloud-storage/tests/unit/asyncio/test_async_multi_range_downloader.py: addtest_close_propagates_caller_cancellationandtest_download_ranges_recovers_after_cancel_during_reopen.packages/google-cloud-storage/tests/unit/asyncio/test_stream_multiplexer.py: addtest_send_after_cancelled_reopen_raises_retryable_error.Testing
pytest tests/unit/asyncio -k 'not (test_create_mrd_with_generation_number or test_unknown_read_id_is_dropped)': 427 passed, 2 deselected (the 2 deselected order-dependent caplog tests are pre-existing failures onupstream/main).ruff checkon touched files: passed.grpc.aioserver prior to rebase (concurrentdownload_rangesrecover cleanly after cancelling an in-flight reopen; 200 open/read/close loops and 20 mid-read cancel/read/close loops passed).Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly: