Skip to content

GH-55258: [C++] Move Azure and its SDK to its own libarrow_azure.so - #55299

Open
raulcd wants to merge 8 commits into
apache:mainfrom
raulcd:GH-55258
Open

raulcd wants to merge 8 commits into
apache:mainfrom
raulcd:GH-55258

Conversation

@raulcd

@raulcd raulcd commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

Rationale for this change

Trying to reduce the size of libarrow.so and remove Azure SDK on some builds. Allow for users to plug and play based on requirements and divide our functionality into cleaner modules. Same as we have done for GCS and AWS S3.

What changes are included in this PR?

Unconditionally build Azure file system and the Azure SDK into a different module libarrow_azure.so outside of libarrow.so.
Update bindings to link against the new libarrow_azure.so library.
Update the Linux Package jobs to have the new module into a different package.

Are these changes tested?

Yes via CI

Are there any user-facing changes?

Yes, users will need to either link against libarrow_azure.so or register using LoadFileSystemFactories

Was AI used for this PR?

In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.

PR code and description written by:

  • Human
  • AI

Reviewed before submission by:

  • Human
  • AI
  • Not reviewed

@raulcd

raulcd commented Nov 10, 2026

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit wheel-cp315

@github-actions

This comment was marked as outdated.

@raulcd

raulcd commented Nov 10, 2026

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit -g r

@github-actions

This comment was marked as outdated.

@raulcd

raulcd commented Nov 12, 2026

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit -g r

@raulcd

raulcd commented Nov 12, 2026

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit wheel-cp314

@github-actions

This comment was marked as outdated.

@github-actions

This comment was marked as outdated.

@raulcd

raulcd commented Nov 12, 2026

Copy link
Copy Markdown
Member Author

This should be ready for review now but I am going to wait for the following PR to land which fixes the GCS static builds.

Once the above is merged I'll rebase and validate the debian 13 jobs for the static checks for Azure too.
cc @tadeja @kou

@raulcd

raulcd commented Nov 16, 2026

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit test-debian-13-cpp-amd64 test-ubuntu-22.04-cpp-bundled test-ubuntu-24.04-cpp-gcc-13-bundled

@github-actions

Copy link
Copy Markdown

Revision: b494674

Submitted crossbow builds: ursacomputing/crossbow @ actions-3dd5f9292e

Task Status
test-debian-13-cpp-amd64 GitHub Actions
test-ubuntu-22.04-cpp-bundled GitHub Actions
test-ubuntu-24.04-cpp-gcc-13-bundled GitHub Actions

@raulcd

raulcd commented Nov 16, 2026

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit -g r

@raulcd

raulcd commented Nov 16, 2026

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit wheel-cp315

@github-actions

Copy link
Copy Markdown

Revision: b494674

Submitted crossbow builds: ursacomputing/crossbow @ actions-7deaf4bc08

Task Status
r-binary-packages GitHub Actions
r-recheck-most GitHub Actions
test-r-alpine-linux-cran GitHub Actions
test-r-arrow-backwards-compatibility GitHub Actions
test-r-depsource-system GitHub Actions
test-r-dev-duckdb GitHub Actions
test-r-devdocs GitHub Actions
test-r-extra-packages GitHub Actions
test-r-fedora-clang GitHub Actions
test-r-gcc-11 GitHub Actions
test-r-gcc-12 GitHub Actions
test-r-install-local GitHub Actions
test-r-install-local-minsizerel GitHub Actions
test-r-linux-as-cran GitHub Actions
test-r-linux-rchk GitHub Actions
test-r-linux-sanitizers GitHub Actions
test-r-linux-valgrind GitHub Actions
test-r-m1-san GitHub Actions
test-r-macos-as-cran GitHub Actions
test-r-offline-maximal GitHub Actions
test-r-offline-minimal GitHub Actions
test-r-ubuntu-22.04 GitHub Actions
test-r-versions GitHub Actions
test-r-wasm GitHub Actions

@github-actions

Copy link
Copy Markdown

Revision: b494674

Submitted crossbow builds: ursacomputing/crossbow @ actions-46ba0fe6c4

Task Status
wheel-macos-monterey-cp315-cp315-amd64 GitHub Actions
wheel-macos-monterey-cp315-cp315-arm64 GitHub Actions
wheel-macos-monterey-cp315-cp315t-amd64 GitHub Actions
wheel-macos-monterey-cp315-cp315t-arm64 GitHub Actions
wheel-manylinux-2-28-cp315-cp315-amd64 GitHub Actions
wheel-manylinux-2-28-cp315-cp315-arm64 GitHub Actions
wheel-manylinux-2-28-cp315-cp315t-amd64 GitHub Actions
wheel-manylinux-2-28-cp315-cp315t-arm64 GitHub Actions
wheel-musllinux-1-2-cp315-cp315-amd64 GitHub Actions
wheel-musllinux-1-2-cp315-cp315-arm64 GitHub Actions
wheel-musllinux-1-2-cp315-cp315t-amd64 GitHub Actions
wheel-musllinux-1-2-cp315-cp315t-arm64 GitHub Actions
wheel-windows-cp315-cp315-amd64 GitHub Actions
wheel-windows-cp315-cp315t-amd64 GitHub Actions

@raulcd

raulcd commented Nov 16, 2026

Copy link
Copy Markdown
Member Author

@tadeja this doesn't seem to fail our static builds, the CI failures are unrelated (failing on main and being tackled on other issues). This should be ready for review now.

@raulcd
raulcd marked this pull request as ready for review November 16, 2026 11:46
Copilot AI balanced review requested due to automatic review settings November 16, 2026 11:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

5 open issues
What changed in this PR

This PR splits the Azure filesystem (and Azure SDK integration) out of libarrow.so into a dedicated libarrow_azure module, and updates bindings + packaging/docs to consume the new module.

Changes:

  • Add a new C++ module/library (libarrow_azure) with its own CMake + pkg-config package (ArrowAzure / arrow-azure) and symbol visibility controls.
  • Update Python and R build/link logic to link against the new Azure module.
  • Update Linux packaging (RPM/Debian) and C++ build system docs to ship/describe the new module; add tests validating URI handling via the registry/module.
File Description
r/​configure Adds arrow-azure pkg-config name + -larrow_azure fallback linking for R when Azure is enabled.
python/​CMakeLists.txt Finds/links ArrowAzure and bundles libarrow_azure when building PyArrow with Azure.
docs/​source/​cpp/​build_system.rst Documents new ArrowAzure CMake package and arrow-azure pkg-config module.
dev/​tasks/​linux-packages/​apache-arrow/​yum/​arrow.spec.in Introduces RPM subpackages for Azure runtime + devel artifacts.
dev/​tasks/​linux-packages/​apache-arrow/​debian/​* Adds Debian control/install entries for Azure runtime + dev packages.
cpp/​src/​arrow/​filesystem/​meson.build Installs new azure_visibility.h header.
cpp/​src/​arrow/​filesystem/​filesystem.cc Removes built-in abfs/abfss URI handling from core filesystem dispatch.
cpp/​src/​arrow/​filesystem/​azurefs_test.cc Adds a registry-focused FileSystemFromUri test for abfs/abfss.
cpp/​src/​arrow/​filesystem/​azurefs_module_test.cc Adds a module loading test that exercises LoadFileSystemFactories.
cpp/​src/​arrow/​filesystem/​azurefs_module.cc Adds Azure module TU intended to participate in filesystem module behavior.
cpp/​src/​arrow/​filesystem/​azurefs_internal.h Switches exported symbol annotation to ARROW_AZURE_EXPORT.
cpp/​src/​arrow/​filesystem/​azurefs.h Switches exported types to ARROW_AZURE_EXPORT and includes new visibility header.
cpp/​src/​arrow/​filesystem/​azurefs.cc Registers abfs/abfss factories via FileSystemRegistrar.
cpp/​src/​arrow/​filesystem/​azure_visibility.h Adds Azure-specific export/import visibility macros.
cpp/​src/​arrow/​filesystem/​CMakeLists.txt Links azurefs tests against arrow_azure_* and adds a shared-module loading test.
cpp/​src/​arrow/​arrow-azure.pc.in Adds pkg-config template for arrow-azure.
cpp/​src/​arrow/​CMakeLists.txt Moves Azure sources + SDK deps into a new arrow_azure library; emits CMake/pkg-config artifacts.
cpp/​src/​arrow/​ArrowAzureConfig.cmake.in Adds ArrowAzure CMake package config and targets import.
cpp/​cmake_modules/​ThirdpartyToolchain.cmake Adjusts Azure dependency resolution + pc/system dependency variables for the new module.
ci/​scripts/​python_wheel_xlinux_build.sh Ensures libarrow_azure.so is included in visibility checks when Azure is enabled.

Review effort: Lite

Comment thread cpp/src/arrow/filesystem/azurefs.cc Resolved
Comment thread python/CMakeLists.txt Resolved
Comment thread r/configure Resolved
@kou

kou commented Nov 17, 2026

Copy link
Copy Markdown
Member

Can we add ARROW_EXAMPLE_AZURE_STATIC_CHECK to https://github.com/apache/arrow/blob/main/cpp/examples/minimal_build/CMakeLists.txt ?

Copilot AI balanced review requested due to automatic review settings November 17, 2026 10:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread cpp/src/arrow/CMakeLists.txt Resolved
Comment thread cpp/src/arrow/filesystem/azurefs.h Resolved
LibXml2::LibXml2)
endif()
endif()
set(ARROW_AZURE_LIB_SRCS filesystem/azurefs_module.cc filesystem/azurefs.cc)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as GCS. GCS is tracked in #55284 , which covers GCS. @kou, should I create one for Azure for C GLib?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you do it, it's helpful. Or I can create it later. :-)

Copilot AI balanced review requested due to automatic review settings November 17, 2026 10:48

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants