Repository navigation
GH-49643: [C++][Python] Use AWS_ENDPOINT_URL env vars for all S3Options - #55643
Open
furqan-khan07 wants to merge 1 commit into
Open
furqan-khan07 wants to merge 1 commit into
furqan-khan07 wants to merge 1 commit into
Conversation
…3Options The AWS_ENDPOINT_URL_S3 and AWS_ENDPOINT_URL environment variables were only read in S3Options::FromUri. Options created with Defaults(), Anonymous(), FromAccessKey() or FromAssumeRole(), which is what pyarrow.fs.S3FileSystem() uses, ignored them and connected to AWS. Read the variables in the S3Options constructor instead, so that every way of creating S3Options starts from the same default endpoint. An explicitly set endpoint_override still takes precedence. The S3 region resolver keeps querying AWS regardless of these variables.
furqan-khan07
requested review from
AlenkaF,
pitrou,
raulcd and
rok
as code owners
November 17, 2026 04:47
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
AWS_ENDPOINT_URL_S3andAWS_ENDPOINT_URLare only read inS3Options::FromUriAndOptions. Options created withS3Options::Defaults(),Anonymous(),FromAccessKey()orFromAssumeRole()ignore them.pyarrow.fs.S3FileSystem()uses those factories, so it connects to AWS even when the variables point to MinIO or another S3-compatible endpoint, whileFileSystem.from_uri("s3://...")in the same process uses the endpoint.What changes are included in this PR?
S3Options::FromUriAndOptionsto theS3Optionsconstructor, so all ways of creatingS3Optionsget the same defaultendpoint_override. An explicitly setendpoint_override(C++ field, URI query parameter, Python keyword) still takes precedence.endpoint_override, soResolveS3BucketRegionkeeps querying AWS when the variables are set.endpoint_overridedocs (C++ header, Python docstring,env_vars.rst).Are these changes tested?
Yes.
S3OptionsTest.EndpointFromEnvVarandS3RegionResolutionTest.EndpointEnvVarIgnored.test_s3fs_endpoint_from_env, which runs against the MinIO test server and checksAWS_ENDPOINT_URL, the precedence ofAWS_ENDPOINT_URL_S3, and the precedence of an explicitendpoint_override.Locally (macOS arm64, debug build, aws-sdk-cpp 1.11.905):
arrow-s3fs-test,arrow-s3fs-module-test,arrow-filesystem-testandpyarrow/tests/test_fs.py --enable-s3pass. The newEndpointFromEnvVarand Python tests fail without the change.Are there any user-facing changes?
Yes. When
AWS_ENDPOINT_URL_S3orAWS_ENDPOINT_URLis set,pyarrow.fs.S3FileSystem(...)and C++S3Options::Defaults()/Anonymous()/FromAccessKey()/FromAssumeRole()now use that endpoint unlessendpoint_overrideis set explicitly. Previously only filesystems created from a URI did. Passingendpoint_override=""in Python keeps AWS as default.Was AI used for this PR?
PR code and description written by:
Reviewed before submission by:
Claude (Fable 5.2) was used to investigate the issue and write the fix, tests and this description, and Claude (Opus 5.5) re-checked it. I reviewed the changes, and the tests and the MinIO reproduction were built and run locally.