Skip to content

GH-49643: [C++][Python] Use AWS_ENDPOINT_URL env vars for all S3Options - #55643

Open
furqan-khan07 wants to merge 1 commit into
apache:mainfrom
furqan-khan07:GH-49643-s3-endpoint-env
Open

furqan-khan07 wants to merge 1 commit into
apache:mainfrom
furqan-khan07:GH-49643-s3-endpoint-env

Conversation

@furqan-khan07

Copy link
Copy Markdown

Rationale for this change

AWS_ENDPOINT_URL_S3 and AWS_ENDPOINT_URL are only read in S3Options::FromUriAndOptions. Options created with S3Options::Defaults(), Anonymous(), FromAccessKey() or FromAssumeRole() ignore them. pyarrow.fs.S3FileSystem() uses those factories, so it connects to AWS even when the variables point to MinIO or another S3-compatible endpoint, while FileSystem.from_uri("s3://...") in the same process uses the endpoint.

What changes are included in this PR?

  • Move the existing environment lookup from S3Options::FromUriAndOptions to the S3Options constructor, so all ways of creating S3Options get the same default endpoint_override. An explicitly set endpoint_override (C++ field, URI query parameter, Python keyword) still takes precedence.
  • The default region resolver clears endpoint_override, so ResolveS3BucketRegion keeps querying AWS when the variables are set.
  • Mention the environment variables in the endpoint_override docs (C++ header, Python docstring, env_vars.rst).

Are these changes tested?

Yes.

  • C++: S3OptionsTest.EndpointFromEnvVar and S3RegionResolutionTest.EndpointEnvVarIgnored.
  • Python: test_s3fs_endpoint_from_env, which runs against the MinIO test server and checks AWS_ENDPOINT_URL, the precedence of AWS_ENDPOINT_URL_S3, and the precedence of an explicit endpoint_override.

Locally (macOS arm64, debug build, aws-sdk-cpp 1.11.905): arrow-s3fs-test, arrow-s3fs-module-test, arrow-filesystem-test and pyarrow/tests/test_fs.py --enable-s3 pass. The new EndpointFromEnvVar and Python tests fail without the change.

Are there any user-facing changes?

Yes. When AWS_ENDPOINT_URL_S3 or AWS_ENDPOINT_URL is set, pyarrow.fs.S3FileSystem(...) and C++ S3Options::Defaults() / Anonymous() / FromAccessKey() / FromAssumeRole() now use that endpoint unless endpoint_override is set explicitly. Previously only filesystems created from a URI did. Passing endpoint_override="" in Python keeps AWS as default.

Was AI used for this PR?

PR code and description written by:

  • Human
  • AI

Reviewed before submission by:

  • Human
  • AI
  • Not reviewed

Claude (Fable 5.2) was used to investigate the issue and write the fix, tests and this description, and Claude (Opus 5.5) re-checked it. I reviewed the changes, and the tests and the MinIO reproduction were built and run locally.

…3Options

The AWS_ENDPOINT_URL_S3 and AWS_ENDPOINT_URL environment variables were
only read in S3Options::FromUri. Options created with Defaults(),
Anonymous(), FromAccessKey() or FromAssumeRole(), which is what
pyarrow.fs.S3FileSystem() uses, ignored them and connected to AWS.

Read the variables in the S3Options constructor instead, so that every
way of creating S3Options starts from the same default endpoint. An
explicitly set endpoint_override still takes precedence. The S3 region
resolver keeps querying AWS regardless of these variables.
@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #49643 has been automatically assigned in GitHub to PR creator.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant