Repository navigation
Conversation
|
@github-actions crossbow submit wheel-cp315 |
This comment was marked as outdated.
This comment was marked as outdated.
|
@github-actions crossbow submit -g r |
This comment was marked as outdated.
This comment was marked as outdated.
|
@github-actions crossbow submit -g r |
|
@github-actions crossbow submit wheel-cp314 |
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
|
@github-actions crossbow submit test-debian-13-cpp-amd64 test-ubuntu-22.04-cpp-bundled test-ubuntu-24.04-cpp-gcc-13-bundled |
|
Revision: b494674 Submitted crossbow builds: ursacomputing/crossbow @ actions-3dd5f9292e
|
|
@github-actions crossbow submit -g r |
|
@github-actions crossbow submit wheel-cp315 |
|
Revision: b494674 Submitted crossbow builds: ursacomputing/crossbow @ actions-7deaf4bc08 |
|
Revision: b494674 Submitted crossbow builds: ursacomputing/crossbow @ actions-46ba0fe6c4 |
|
@tadeja this doesn't seem to fail our static builds, the CI failures are unrelated (failing on main and being tackled on other issues). This should be ready for review now. |
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
5 open issues
With the Azure URI handling removed fromfilesystem.cc,abfs/abfssnow relies entirely on… · New These RPM subpackages (and their%fileslists) appear unconditional and hard-require both shared… · New On the non-MSVC Windows branch,#pragma GCC diagnostic ignored \"-Wattributes\"is not paired… · New The bundling step depends onARROW_AZURE_SHARED_LIB/ARROW_AZURE_IMPORT_LIBvariables being… · New The non-pkg-config fallback (PKG_LIBS_FEATURES_WITHOUT_PC) adds-larrow_azurebut doesn’t add… · New
What changed in this PR
This PR splits the Azure filesystem (and Azure SDK integration) out of libarrow.so into a dedicated libarrow_azure module, and updates bindings + packaging/docs to consume the new module.
Changes:
- Add a new C++ module/library (
libarrow_azure) with its own CMake + pkg-config package (ArrowAzure/arrow-azure) and symbol visibility controls. - Update Python and R build/link logic to link against the new Azure module.
- Update Linux packaging (RPM/Debian) and C++ build system docs to ship/describe the new module; add tests validating URI handling via the registry/module.
| File | Description |
|---|---|
| r/configure | Adds arrow-azure pkg-config name + -larrow_azure fallback linking for R when Azure is enabled. |
| python/CMakeLists.txt | Finds/links ArrowAzure and bundles libarrow_azure when building PyArrow with Azure. |
| docs/source/cpp/build_system.rst | Documents new ArrowAzure CMake package and arrow-azure pkg-config module. |
| dev/tasks/linux-packages/apache-arrow/yum/arrow.spec.in | Introduces RPM subpackages for Azure runtime + devel artifacts. |
| dev/tasks/linux-packages/apache-arrow/debian/* | Adds Debian control/install entries for Azure runtime + dev packages. |
| cpp/src/arrow/filesystem/meson.build | Installs new azure_visibility.h header. |
| cpp/src/arrow/filesystem/filesystem.cc | Removes built-in abfs/abfss URI handling from core filesystem dispatch. |
| cpp/src/arrow/filesystem/azurefs_test.cc | Adds a registry-focused FileSystemFromUri test for abfs/abfss. |
| cpp/src/arrow/filesystem/azurefs_module_test.cc | Adds a module loading test that exercises LoadFileSystemFactories. |
| cpp/src/arrow/filesystem/azurefs_module.cc | Adds Azure module TU intended to participate in filesystem module behavior. |
| cpp/src/arrow/filesystem/azurefs_internal.h | Switches exported symbol annotation to ARROW_AZURE_EXPORT. |
| cpp/src/arrow/filesystem/azurefs.h | Switches exported types to ARROW_AZURE_EXPORT and includes new visibility header. |
| cpp/src/arrow/filesystem/azurefs.cc | Registers abfs/abfss factories via FileSystemRegistrar. |
| cpp/src/arrow/filesystem/azure_visibility.h | Adds Azure-specific export/import visibility macros. |
| cpp/src/arrow/filesystem/CMakeLists.txt | Links azurefs tests against arrow_azure_* and adds a shared-module loading test. |
| cpp/src/arrow/arrow-azure.pc.in | Adds pkg-config template for arrow-azure. |
| cpp/src/arrow/CMakeLists.txt | Moves Azure sources + SDK deps into a new arrow_azure library; emits CMake/pkg-config artifacts. |
| cpp/src/arrow/ArrowAzureConfig.cmake.in | Adds ArrowAzure CMake package config and targets import. |
| cpp/cmake_modules/ThirdpartyToolchain.cmake | Adjusts Azure dependency resolution + pc/system dependency variables for the new module. |
| ci/scripts/python_wheel_xlinux_build.sh | Ensures libarrow_azure.so is included in visibility checks when Azure is enabled. |
Review effort: Lite
|
Can we add |
There was a problem hiding this comment.
🟡 Changes recommended
Incorrect Azure export macros break Windows shared Meson builds and static pkg-config linking, while missing module loading regresses C GLib support for abfs/abfss URIs.
3 open issues
5 resolved since last review
These RPM subpackages (and their%fileslists) appear unconditional and hard-require both shared… With the Azure URI handling removed fromfilesystem.cc,abfs/abfssnow relies entirely on… The non-pkg-config fallback (PKG_LIBS_FEATURES_WITHOUT_PC) adds-larrow_azurebut doesn’t add… The bundling step depends onARROW_AZURE_SHARED_LIB/ARROW_AZURE_IMPORT_LIBvariables being… On the non-MSVC Windows branch,#pragma GCC diagnostic ignored \"-Wattributes\"is not paired…
Review effort: Balanced
| LibXml2::LibXml2) | ||
| endif() | ||
| endif() | ||
| set(ARROW_AZURE_LIB_SRCS filesystem/azurefs_module.cc filesystem/azurefs.cc) |
There was a problem hiding this comment.
If you do it, it's helpful. Or I can create it later. :-)


Rationale for this change
Trying to reduce the size of libarrow.so and remove Azure SDK on some builds. Allow for users to plug and play based on requirements and divide our functionality into cleaner modules. Same as we have done for GCS and AWS S3.
What changes are included in this PR?
Unconditionally build Azure file system and the Azure SDK into a different module libarrow_azure.so outside of libarrow.so.
Update bindings to link against the new libarrow_azure.so library.
Update the Linux Package jobs to have the new module into a different package.
Are these changes tested?
Yes via CI
Are there any user-facing changes?
Yes, users will need to either link against libarrow_azure.so or register using LoadFileSystemFactories
Was AI used for this PR?
In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.
PR code and description written by:
Reviewed before submission by: