Skip to content

fix(storage): make AsyncMultiRangeDownloader reopen and close cancellation-safe - #19169

Draft
zhixiangli wants to merge 3 commits into
googleapis:mainfrom
zhixiangli:storage-mrd-cancel-safe-reopen
Draft

zhixiangli wants to merge 3 commits into
googleapis:mainfrom
zhixiangli:storage-mrd-cancel-safe-reopen

Conversation

@zhixiangli

Copy link
Copy Markdown
Contributor

Fix two cancellation edge cases on AsyncMultiRangeDownloader and _StreamMultiplexer when callers cancel or time out download_ranges() or close():

  1. Recover after a cancelled stream reopen (_StreamMultiplexer.send): When an in-flight stream reopen is cancelled after closing the old stream, _stream on _StreamMultiplexer remains closed at the same _stream_generation. Previously, a subsequent send() attempted to write to the closed stream and raised a non-retryable ValueError("Stream is not open"). Now _StreamMultiplexer.send checks self._stream.is_stream_open and raises retryable google.api_core.exceptions.ServiceUnavailable so the caller's retry path reopens the stream cleanly.
  2. Propagate cancellation and finalize closed state (AsyncMultiRangeDownloader.close): Stop swallowing asyncio.CancelledError when closing read_obj_str, and wrap teardown in try ... finally so _pool, _multiplexer, read_obj_str, _primary_stream, and _is_stream_open = False are always cleared even if close() is cancelled partway through.

Changes

  • packages/google-cloud-storage/google/cloud/storage/asyncio/_stream_multiplexer.py: raise retryable exceptions.ServiceUnavailable in _StreamMultiplexer.send when not self._stream.is_stream_open.
  • packages/google-cloud-storage/google/cloud/storage/asyncio/async_multi_range_downloader.py: stop catching asyncio.CancelledError and wrap teardown in try ... finally inside AsyncMultiRangeDownloader.close.
  • packages/google-cloud-storage/tests/unit/asyncio/test_async_multi_range_downloader.py: add test_close_propagates_caller_cancellation and test_download_ranges_recovers_after_cancel_during_reopen.
  • packages/google-cloud-storage/tests/unit/asyncio/test_stream_multiplexer.py: add test_send_after_cancelled_reopen_raises_retryable_error.

Testing

  • pytest tests/unit/asyncio -k 'not (test_create_mrd_with_generation_number or test_unknown_read_id_is_dropped)': 427 passed, 2 deselected (the 2 deselected order-dependent caplog tests are pre-existing failures on upstream/main).
  • ruff check on touched files: passed.
  • Verified against real RAPID GCS and a local grpc.aio server prior to rebase (concurrent download_ranges recover cleanly after cancelling an in-flight reopen; 200 open/read/close loops and 20 mid-read cancel/read/close loops passed).

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request improves the robustness of stream handling and resource cleanup in the asynchronous Google Cloud Storage client. Specifically, it raises a retryable ServiceUnavailable exception if a send is attempted on a closed stream multiplexer, and wraps resource cleanup in AsyncMultiRangeDownloader.close() with a try...finally block to ensure references are cleared even if an error occurs. Unit tests are also added to verify these behaviors. Feedback suggests further improving the cleanup logic by wrapping each individual resource's close call in its own try-except block to prevent a failure in one from blocking the closure of others.

Comment on lines +1002 to +1025
try:
if self._pool:
await self._pool.close()

if self._multiplexer:
await self._multiplexer.close()
self._multiplexer = None
if self._multiplexer:
await self._multiplexer.close()

if self.read_obj_str:
try:
if getattr(self.read_obj_str, "is_stream_open", True):
res = self.read_obj_str.close()
if inspect.isawaitable(res):
await res
except (
ValueError,
asyncio.CancelledError,
exceptions.GoogleAPICallError,
):
pass
self.read_obj_str = None
self._primary_stream = None
self._is_stream_open = False
if self.read_obj_str:
try:
if getattr(self.read_obj_str, "is_stream_open", True):
res = self.read_obj_str.close()
if inspect.isawaitable(res):
await res
except (
ValueError,
exceptions.GoogleAPICallError,
):
pass
finally:
self._pool = None
self._multiplexer = None
self.read_obj_str = None
self._primary_stream = None
self._is_stream_open = False

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

When closing multiple resources during cleanup, wrap each close call in a try-except block to handle exceptions individually, log any failures as warnings, and use a finally block to clear the resource references. This ensures that a failure in closing one resource does not prevent other resources from being closed.

            try:
                if self._pool:
                    await self._pool.close()
            except Exception as e:
                logger.warning("Failed to close pool: %s", e)

            try:
                if self._multiplexer:
                    await self._multiplexer.close()
            except Exception as e:
                logger.warning("Failed to close multiplexer: %s", e)

            try:
                if self.read_obj_str:
                    if getattr(self.read_obj_str, 'is_stream_open', True):
                        res = self.read_obj_str.close()
                        if inspect.isawaitable(res):
                            await res
            except Exception as e:
                logger.warning("Failed to close stream: %s", e)
            finally:
                self._pool = None
                self._multiplexer = None
                self.read_obj_str = None
                self._primary_stream = None
                self._is_stream_open = False
References
  1. When closing multiple resources during cleanup, wrap each close call in a try-except block to handle exceptions individually, log any failures as warnings, and use a finally block to clear the resource references.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant