Skip to content

GH-55263: [Python][Annotations] Add internal types and core data structures - #55264

Open
rok wants to merge 129 commits into
apache:mainfrom
rok:pyarrow-stubs-pr3-core-data-sans-generic
Open

rok wants to merge 129 commits into
apache:mainfrom
rok:pyarrow-stubs-pr3-core-data-sans-generic

Conversation

@rok

@rok rok commented Oct 26, 2026 •

Copy link
Copy Markdown
Member

Rationale for this change

This is the next step in adding type annotations to PyArrow (#32609). It merges #49103 (internal type system stubs) and #49193 (core data structures) because the two depend on each other and can't be type checked separately.

What changes are included in this PR?

This PR adds stubs for core data structures and types. Annotations live in pyarrow-stubs/ and can be used at development time. Annotations are not packaged into wheels. Annotations are tested in CI with ci/scripts/python_test_type_annotations.sh and can be linted and tested locally with pre-commit.

During review we agreed on no generics for now (could be follow-up) and overloads are used sparingly.

Detailed AI generated description of changes **Type stubs** (`python/pyarrow-stubs/`)
  • Stubs for the parts of pyarrow.lib that hold data types and core data structures: types, array, table, tensor, builder, memory, device, config and error.
  • pyarrow.types (the is_* predicates, with TypeIs/TypeGuard narrowing).
  • pyarrow/__init__.pyi now re-exports the annotated public names explicitly, so pa.array, pa.Table, pa.schema etc. are typed.
  • Placeholders for modules annotated in later PRs: io, ipc, scalar and _compute. Their names resolve to Any, except Buffer and ExtensionScalar, which are empty classes for now. pyarrow/__init__.pyi and lib.pyi also keep a module-level __getattr__ returning Any for names that aren't annotated yet (to be removed before the stubs ship, [CI] Remove all __getattr__(name) -> Any from *.pyi files before next release #48970).

Layout

  • pyarrow-stubs/ is now laid out as a PEP 561 stub-only package (pyarrow-stubs/__init__.pyi instead of pyarrow-stubs/pyarrow/__init__.pyi), so type checkers and IDEs pick it up from arrow/python.
  • The stubs for the .pxi files compiled into pyarrow.lib live in a stub-only pyarrow-stubs/__lib/ package and are re-exported from lib.pyi. A flat layout made pa.array, pa.table and pa.scalar resolve to stub modules instead of functions.

Annotation conventions agreed during review

  • No generics: arrays, scalars and types are not parametrised. Stub-only DataType subclasses in _stubs_typing.pyi describe the results of the type factories.
  • Return types use Self when the runtime returns the receiver's class and a concrete class otherwise. Parameters never use Self.
  • Parameters that need an array take the base Array rather than a specific subclass such as BooleanArray or Int32Array, because user variables are usually inferred as Array.
  • pa.array() is annotated as returning Array. The cases where it returns a ChunkedArray are tracked in [Python] Let array() return a ChunkedArray only when asked #55499. asarray() has overloads.
  • Overloads are used sparingly, only where the un-overloaded annotation would be of little use (for example __getitem__ and asarray).
  • String options (unit, null_placement, ...) are Literal types, as in NumPy and pandas-stubs.
  • NumPy arrays are annotated as NDArray[...].
  • Classes that define __eq__ without __hash__ are marked unhashable.

Type checking setup

  • mypy, pyright and ty are configured in python/pyproject.toml to check pyarrow-stubs and pyarrow-stubs-tests. Modules that aren't annotated yet are treated as Any.
  • python/requirements-stubs.txt pins the type checkers and the stub packages of the dependencies. CI and pre-commit both install from it.
  • New python-stubs pre-commit hooks: ruff lint and format for the .pyi files run on commit; mypy, pyright and ty run on request (pre-commit run --hook-stage manual python-stubs --all-files) and in the Lint CI job.
  • ci/scripts/python_test_type_annotations.sh now also runs the type stub tests and stubtest against the PyArrow built in CI.
  • python/scripts/run_stubtest.py runs mypy.stubtest with the __lib modules mapped to pyarrow.lib, using a small allowlist.

Other changes

  • Small fixes found while reviewing the stubs against the code: docstring corrections in array.pxi, scalar.pxi, tensor.pxi and types.pxi, and comparisons in pyarrow/tests that were missing assert.
  • # type: ignore comments on the optional-import fallbacks in a few pyarrow/tests files.
  • Path updates for the new layout in CMakeLists.txt (commented-out install block), python_wheel_validate_contents.py and update_stub_docstrings.py.
  • The "Type Checking" section of the Python development docs is rewritten.

Are these changes tested?

Yes.

  • The tests are type checked by mypy, pyright and ty, and run with pytest.
  • stubtest compares the stub signatures with the installed PyArrow.

The PyArrow test suite (pyarrow/tests) is not type checked in CI yet.

Yes CI runs (ci/scripts/python_test_type_annotations.sh) which in runs typecheckers and tests in python/pyarrow-stubs-tests/.
Changes are also teste locally via pre-commit. Once we package annotations into wheels we should expand testing.

Are there any user-facing changes?

No. The stubs are still not packaged into wheels.

Was AI used for this PR?

In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.

PR code and description written by:

  • Human
  • AI

Reviewed before submission by:

  • Human
  • AI
  • Not reviewed

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #55263 has been automatically assigned in GitHub to PR creator.

@AlenkaF

AlenkaF commented Oct 27, 2026

Copy link
Copy Markdown
Member

Thanks for continuing to push this forward Rok! I think this PR is a great next step and is quite clear to review.

I have checked out the PR locally and started with running type checkers (mypy, pyright and ty). mypy and ty emit no errors or warnings, but pyright does emit reportMissingImports and reportMissingModuleSource which get fixed with adding _compute.pyi placeholder (similar to existing io.pyi) and reportMissingModuleSource = false under [tool.pyright] in pyproject.toml.

The reason for these warnings and errors in pyright is that locally I have editable pyarrow install where scikit-build-core puts the .so files in a separate folder (not python/pyarrow) without __init__.py and pyright can't find them. As we do not need the compiled modules for the type checkers, I think it is fine to use reportMissingModuleSource = false.

I propose to push the changes directly if we agree the additions makes sense.

Meanwhile I will continue with running the pyarrow-stubs tests.

@AlenkaF

AlenkaF commented Oct 27, 2026

Copy link
Copy Markdown
Member

I have run pytest pyarrow-stubs/tests, no errors found. I have looked through the tests also and tried to break some, everything looks good. I think the tests are easy to follow while still cover a lot of different angles.

I have also run mypy's stubtest with stubtest pyarrow.lib --concise --mypy-config-file pyproject.toml. All of the reported discrepancies are cases of "... is not present in stub" with majority of them being stubs that are not yet included in this PR (scalars, io, etc.). Part of these are also Type_* stubs, for example pyarrow.lib.Type_UINT8. As we already have TypesEnum defined here, might it make sense to also include Type_* stubs now? (I am not inclined towards adding them, just wondering).

Besides the reports mentioned above which will be fixed in subsequent PRs, stubtest did found one bug in pyarrow:

pyarrow.lib.Decima32Array is not present in stub

Note missing "L" in "Decimal". I will work on the fix for this now and would suggest to add it as part of this PR. Can commit directly, if that is ok.

@tadeja

tadeja commented Oct 27, 2026

Copy link
Copy Markdown
Member

I've run pre-commit run --hook-stage manual python-stubs --all-files which currently uses pyarrow==26.0.0 wheel modules in site-packages/pyarrow, and got all passing (Python Stubs Lint (ruff), Python Stubs Type Check (mypy/pyright/ty)) - except for ruff-format which requires 10 files to be reformatted, fyi @rok
Python Stubs Format (ruff)...............................................Failed

Running on editable dev build 27, with pyright==1.1.414 in my conda env, I also hit errors reportMissingImports and warnings reportMissingModuleSource - Suggested changes by @AlenkaF sound good and seem to work fine for me locally.

@rok

rok commented Oct 27, 2026 •

Copy link
Copy Markdown
Member Author

Thanks for continuing to push this forward Rok! I think this PR is a great next step and is quite clear to review.

I have checked out the PR locally and started with running type checkers (mypy, pyright and ty). mypy and ty emit no errors or warnings, but pyright does emit reportMissingImports and reportMissingModuleSource which get fixed with adding _compute.pyi placeholder (similar to existing io.pyi) and reportMissingModuleSource = false under [tool.pyright] in pyproject.toml.

The reason for these warnings and errors in pyright is that locally I have editable pyarrow install where scikit-build-core puts the .so files in a separate folder (not python/pyarrow) without __init__.py and pyright can't find them. As we do not need the compiled modules for the type checkers, I think it is fine to use reportMissingModuleSource = false.

I propose to push the changes directly if we agree the additions makes sense.

That makes sense and please go for it. I think we should aim for pyproject.toml setting to be as close to the state we want to maintain as possible. So far I've tried to exclude modules that are not covered and ignore as few things as possible. reportMissingModuleSource = false seems good, we will want to challenge it when we have complete coverage.

Part of these are also Type_* stubs, for example pyarrow.lib.Type_UINT8. As we already have TypesEnum defined here, might it make sense to also include Type_* stubs now? (I am not inclined towards adding them, just wondering).

Huh. I've never noticed these were missing. Given we're not really using them as types I'd skip them if we can.

Besides the reports mentioned above which will be fixed in subsequent PRs, stubtest did found one bug in pyarrow:

pyarrow.lib.Decima32Array is not present in stub

Note missing "L" in "Decimal". I will work on the fix for this now and would suggest to add it as part of this PR. Can commit directly, if that is ok.

Huh. Let's do the fix as separate PR, but also include a fix here? (And we'll just rebase over the actual fix on main)

I've run pre-commit run --hook-stage manual python-stubs --all-files which currently uses pyarrow==26.0.0 wheel modules in site-packages/pyarrow, and got all passing (Python Stubs Lint (ruff), Python Stubs Type Check (mypy/pyright/ty)) - except for ruff-format which requires 10 files to be reformatted, fyi @rok Python Stubs Format (ruff)...............................................Failed

I'm not sure we want (can) bring in ruff as a CI dependency. However we should autoformat before we merge, so we don't do it later and change history for no good reason. So I propose one of you runs the ruff formatter and commits the change into this branch.

@AlenkaF

AlenkaF commented Oct 27, 2026

Copy link
Copy Markdown
Member

reportMissingModuleSource = false seems good, we will want to challenge it when we have complete coverage.

I think reportMissingModuleSource will have to stay due to local editable installs. Otherwise I agree, we should ignore as few things as possible.
Will push the commit 👍

Huh. Let's do the fix as separate PR, but also include a fix here? (And we'll just rebase over the actual fix on main)

Sounds good. Will open an issue and a PR and add a fix here also.

@AlenkaF

AlenkaF commented Oct 27, 2026

Copy link
Copy Markdown
Member

Part of these are also Type_* stubs, for example pyarrow.lib.Type_UINT8. As we already have TypesEnum defined here, might it make sense to also include Type_* stubs now? (I am not inclined towards adding them, just wondering).

Huh. I've never noticed these were missing. Given we're not really using them as types I'd skip them if we can.

Yeah, they are internal and exposed through TypesEnum which are added to the stubs here. For stubtest we can later add them to allowlist. For the type checkers we will need to work around them once lib stubs loose their def __getattr__(name: str) -> Any: .... Maybe we can revisit this topic later (maybe we see it's easier to simply add them to the stubs)

@tadeja

tadeja commented Oct 27, 2026

Copy link
Copy Markdown
Member

... ruff-format which requires 10 files to be reformatted, fyi @rok Python Stubs Format (ruff)...............................................Failed

I'm not sure we want (can) bring in ruff as a CI dependency. However we should autoformat before we merge, so we don't do it later and change history for no good reason. So I propose one of you runs the ruff formatter and commits the change into this branch.

Ok, I'll do that reformatting in a bit.

@AlenkaF

AlenkaF commented Oct 28, 2026

Copy link
Copy Markdown
Member

I have started with reviewing pyarrow tests with running mypy --check-untyped-defs --warn-unused-ignores pyarrow/tests/test_* and looking at the ignores. The stubs seem to have changed since the last time type checkers were run on pyarrow tests, example:

click for output
$ mypy --check-untyped-defs --warn-unused-ignores pyarrow/tests/test_array.py 
pyarrow/tests/test_array.py:21: error: Unused "type: ignore" comment  [unused-ignore]
pyarrow/tests/test_array.py:22: error: Unused "type: ignore" comment  [unused-ignore]
pyarrow/tests/test_array.py:25: error: Unused "type: ignore" comment  [unused-ignore]
pyarrow/tests/test_array.py:80: error: Unused "type: ignore" comment  [unused-ignore]
pyarrow/tests/test_array.py:790: error: Unused "type: ignore" comment  [unused-ignore]
pyarrow/tests/test_array.py:986: error: Unused "type: ignore" comment  [unused-ignore]
pyarrow/tests/test_array.py:1300: error: List item 3 has incompatible type "None"; expected "int"  [list-item]
pyarrow/tests/test_array.py:1302: error: List item 1 has incompatible type "None"; expected "list[tuple[bytes, int]]"  [list-item]
pyarrow/tests/test_array.py:1313: error: Unused "type: ignore" comment  [unused-ignore]
pyarrow/tests/test_array.py:1320: error: Unused "type: ignore" comment  [unused-ignore]
pyarrow/tests/test_array.py:1324: error: Unused "type: ignore" comment  [unused-ignore]
pyarrow/tests/test_array.py:1348: error: List item 0 has incompatible type "None"; expected "list[tuple[bytes, int]]"  [list-item]
pyarrow/tests/test_array.py:1348: error: List item 1 has incompatible type "None"; expected "list[tuple[bytes, int]]"  [list-item]
pyarrow/tests/test_array.py:1362: error: Unused "type: ignore" comment  [unused-ignore]
pyarrow/tests/test_array.py:2513: error: Need type annotation for "buffers" (hint: "buffers: list[<type>] = ...")  [var-annotated]
pyarrow/tests/test_array.py:2539: error: "object" has no attribute "dtype"  [attr-defined]
pyarrow/tests/test_array.py:2541: error: Value of type "object" is not indexable  [index]
pyarrow/tests/test_array.py:2542: error: Value of type "object" is not indexable  [index]
pyarrow/tests/test_array.py:2543: error: Value of type "object" is not indexable  [index]
pyarrow/tests/test_array.py:2885: error: Unused "type: ignore" comment  [unused-ignore]
pyarrow/tests/test_array.py:2917: error: Unused "type: ignore" comment  [unused-ignore]
pyarrow/tests/test_array.py:2938: error: Unused "type: ignore" comment  [unused-ignore]
pyarrow/tests/test_array.py:2939: error: Unused "type: ignore" comment  [unused-ignore]
pyarrow/tests/test_array.py:3183: error: Value of type "object" is not indexable  [index]
pyarrow/tests/test_array.py:3299: error: Unused "type: ignore" comment  [unused-ignore]
pyarrow/tests/test_array.py:3720: error: Incompatible types in assignment (expression has type "None", variable has type "ndarray[tuple[Any, ...], dtype[Any]]")  [assignment]
pyarrow/tests/test_array.py:3729: error: Value of type "object" is not indexable  [index]
pyarrow/tests/test_array.py:3740: error: Value of type "object" is not indexable  [index]
pyarrow/tests/test_array.py:3817: error: Unused "type: ignore" comment  [unused-ignore]
pyarrow/tests/test_array.py:4004: error: Incompatible types in assignment (expression has type "MyArrayInvalid", variable has type "MyArray")  [assignment]
pyarrow/tests/test_array.py:4017: error: Incompatible types in assignment (expression has type "MyArray2", variable has type "MyArray")  [assignment]
pyarrow/tests/test_array.py:4031: error: Incompatible types in assignment (expression has type "MyArray3", variable has type "MyArray")  [assignment]
pyarrow/tests/test_array.py:4095: error: Unused "type: ignore" comment  [unused-ignore]
pyarrow/tests/test_array.py:4099: error: Unused "type: ignore" comment  [unused-ignore]
pyarrow/tests/test_array.py:4315: error: List item 3 has incompatible type "None"; expected "int"  [list-item]
pyarrow/tests/test_array.py:4319: error: List item 5 has incompatible type "None"; expected "int"  [list-item]
pyarrow/tests/test_array.py:4324: error: List item 2 has incompatible type "None"; expected "int"  [list-item]
pyarrow/tests/test_array.py:4324: error: List item 4 has incompatible type "None"; expected "int"  [list-item]
pyarrow/tests/test_array.py:4395: error: List item 1 has incompatible type "None"; expected "int"  [list-item]
pyarrow/tests/test_array.py:4407: error: List item 0 has incompatible type "None"; expected "int"  [list-item]
Found 40 errors in 1 file (checked 1 source file)

Type checkers are currently configured to skip python/pyarrow folder (too much noise) so the changes are not caught. Maybe we could start with adding mypy --check-untyped-defs --warn-unused-ignores pyarrow/tests/test_* for specific files that can already be type checked.

I propose to temporarily add the mypy command explicitly naming files which bypasses the exclude setting to ci/scripts/python_test_type_annotations.sh. Will also push the updates of ignores in the tests once I go through the ones connected to this PR.

@AlenkaF

AlenkaF commented Oct 28, 2026

Copy link
Copy Markdown
Member

Another thing: I can't run pyright on specific files inside an excluded dir. I guess I would need to be running the checks with pyright only locally (with the pyarrow exclude temporarily removed) until the pyarrow/ exclude in pyproject.toml is removed.

@AlenkaF

AlenkaF commented Oct 28, 2026

Copy link
Copy Markdown
Member

Oh, I was too fast here. Unfortunately def __getattr__(name: str) -> Any: ... in python/pyarrow-stubs/pyarrow/__init__.pyi makes anything in form of pa.* be Any making lots of correct ignores (in with pytest.raises ... blocks) falsely reported as unused.

Would it make sense to remove # type: ignores from the pyarrow tests for now and add type checking of the tests (with the ignores) after the stubs are in?

I will leave this for discussion and start with reading the stubs and comparing them with the APIs, starting with _types.pyi first.

@rok

rok commented Oct 28, 2026

Copy link
Copy Markdown
Member Author

Ignore statements not being needed is more likely due to changes to typecheckers than due to changes in stubs. Or they might be needed once we import other parts of the library. Also note camel cased (e.g. reportArgumentType) ignores are likely coming from pyright. So if mypy thinks it's a redundant ignore you might want to check pyright too.

@rok

rok commented Oct 28, 2026

Copy link
Copy Markdown
Member Author

Oh, I was too fast here. Unfortunately def __getattr__(name: str) -> Any: ... in python/pyarrow-stubs/pyarrow/__init__.pyi makes anything in form of pa.* be Any making lots of correct ignores (in with pytest.raises ... blocks) falsely reported as unused.

Would it make sense to remove # type: ignores from the pyarrow tests for now and add type checking of the tests (with the ignores) after the stubs are in?

Let's rather remove def __getattr__(name: str) -> Any: ... at least for development work? Otherwise we'll just need to add ignores again.

@AlenkaF

AlenkaF commented Oct 28, 2026

Copy link
Copy Markdown
Member

OK, agree, when reviewing pyarrow/tests we should remove def __getattr__(name: str) -> Any: ... from python/pyarrow-stubs/pyarrow/__init__.pyi so we keep the ignores and run mypy and pyright (with updated pyproject.toml) so both pass 👍

I plan to do that later and am focusing on reading the stubs currently.

@tadeja

tadeja commented Oct 28, 2026

Copy link
Copy Markdown
Member

Ehhh... I've fallen into this interesting rabbit hole - There are over 1.5k mypy errors Module not callable in pyarrow/tests because of the change from upstream's zen-xu/pyarrow-stubs having subfolder __lib_pxi to current PR's flat layout without a subfolder (Past discussion rok#45 (comment) was about the naming, but not really why removing subfolder __lib_pxi)

% mypy --check-untyped-defs --warn-unused-ignores pyarrow/tests/test_convert_builtin.py 
pyarrow/tests/test_convert_builtin.py:78: error: Module not callable  [operator]
pyarrow/tests/test_convert_builtin.py:79: error: Module not callable  [operator]
...
pyarrow/tests/test_convert_builtin.py:168: error: Module not callable  [operator]
...
Found 321 errors in 1 file (checked 1 source file)

% mypy --check-untyped-defs --warn-unused-ignores pyarrow/tests/test_*.py
Found 1866 errors in 35 files (checked 44 source files)
% ... | grep -c  "Module not callable"
1562

test_convert_builtin.py:168 : pa.array(...

A stub pyarrow/array.pyi declares pyarrow.array submodule that doesn't exist at runtime because array.pxi gets compiled into lib. Then pa.array, pa.table or pa.scalar can resolve to stub modules instead of functions.
The count of mypy errors may be even higher, depending on module processing order and mypy cache. (optionally run rm -rf .mypy_cache)

So I propose we reintroduce subfolder __lib_pxi/ (or renamed subfolder _lib_pxi/ or just _lib/) in this PR (for files: array.pyi, builder.pyi, config.pyi, device.pyi, error.pyi, io.pyi, memory.pyi, scalar.pyi, table.pyi, tensor.pyi).
Locally that seems to resolve all 1.5k Module not callable errors and still passes type checkers' tests for me.

@tadeja

tadeja commented Oct 29, 2026

Copy link
Copy Markdown
Member

Just to note: with commit eda3425 stubs array, builder, config, device, error, io, memory, scalar, table, tensor got moved into pyarrow/__lib/ subfolder/subpackage for mypy to not regard pa.array/pa.table/pa.scalar as stub modules anymore and so more than 1500 errors Module not callable in pyarrow/tests no longer show up.

@jorisvandenbossche

Copy link
Copy Markdown
Member

@tadeja I suppose the _types.pyi also needs to be moved into the subdirectory (and can then be renamed to types.pyi)?

@tadeja

tadeja commented Oct 29, 2026

Copy link
Copy Markdown
Member

@jorisvandenbossche oh, yes, I think _types.pyi and _ipc.pyi both still need to be moved but there are some extra steps needed (and @AlenkaF has some work in progress on types at the moment, and I may wait for that)

@AlenkaF

AlenkaF commented Oct 30, 2026

Copy link
Copy Markdown
Member

@tadeja @rok I have checked typed.pyi and decided to push a commit (4b97c69) after all =) Instead of suggestions I will post the reasoning behind the changes here:

  • NumPy is also an optional dependency (next to pandas) so I added # type: ignore[..] there too.
    Locally mypy and pyright both pass with only # type: ignore[..] so I removed pyright ignores

  • ty doesn't honor mypy's # type: ignore[...] and # ty: ignore[unresolved-import] fails if numpy/pandas are installed, so I disabled ty unresolved-import in pyproject.toml instead (mypy and pyright still catch broken imports).

  • I removed noqa: F401 on the lib import and ruff didn't complain

  • I have changed some classmethods in stubs as they are defined as static methods in source (types.pxi).
    Example: _import_from_c in DataType, Field and Schema.

  • I propose to use typing_extensions.CapsuleType instead of Any in __arrow_c_schema__.
    Claude added a test for this case, I have reviewed and changed it a bit. typing_extensions dependency had to be added.
    Note: we should leave the Protocols in _stubs_typing.pyi at Any. Polars' public pl.DataFrame and pl.Series, for example, annotate __arrow_c_stream__ -> object so narrowing the Protocol would reject them

    Claude suggested to add a test for Protocols also (under the Table stubs tests, to make sure they do not become to narrow) but I think these should be added later.

  • KeyValueMetadata stubs do not include keys(), values() and items() stubs because they inherit them from Mapping[bytes, bytes]:
    https://github.com/python/typeshed/blob/17cbf254205549ccdcf886fadf8c265e634ad596/stdlib/typing.pyi#L823-L825
    but our code overrides all three as generators, not Mapping views (probably so that duplicate keys are kept):

    arrow/python/pyarrow/types.pxi

    Lines 2409 to 2419 in 9b9fad6

    def keys(self):
    for i in range(self.metadata.size()):
    yield self.metadata.key(i)
    def values(self):
    for i in range(self.metadata.size()):
    yield self.metadata.value(i)
    def items(self):
    for i in range(self.metadata.size()):
    yield (self.metadata.key(i), self.metadata.value(i))

    I have added overrides to the stubs so they are in sync with runtime and added ignores because of

    Return type "Iterator[bytes]" of "keys" incompatible with return type "KeysView[bytes]" in supertype "typing.Mapping" [override]
    

    I have also added a test test_key_value_metadata.
    I think code overrides need a separate issue as we might want to decide on changing the overrides in types.pxi KeyValueMetadata class to return Mapping views instead of generators (and to somehow keep support of duplicate keys).

  • keys_sorted in map_ take bool, None is by truthiness converted to False in Cython but I think it should not be suggested as a possible input so I changed the default from None to False. Same for dictionary().

  • Included the use of np.typing.DTypeLike for parameter type in from_numpy_dtype https://numpy.org/devdocs/reference/typing.html#numpy.typing.DTypeLike

  • I have changed the Field methods that returned Self to return Field as the runtime always builds a plain Field via pyarrow_wrap_field.

Future TODO:

  • string_to_tzinfo, tzinfo_to_string and ensure_metadata are not in the stubs currently (not public). We should add them to the stubtest allowlist at some point or think of adding them to the stubs.

I have run

  • all 3 type checkers on this file with and without numpy/pandas installed
  • the tests
  • ruff linter check

Comment thread python/pyarrow-stubs/__lib/table.pyi Resolved
Comment thread python/pyarrow-stubs/__lib/table.pyi Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Incomplete and overly restrictive stubs reject supported Buffer operations, array operators and constructors, timezone inputs, and DataType arguments.

5 open issues
What changed in this PR

This PR advances PyArrow’s type-checking support by combining the interdependent internal type-system and core data-structure stubs from #49103 and #49193. The stubs remain excluded from distributed wheels while coverage is incomplete.

Changes:

  • Add core API stubs, shared typing helpers, and public re-exports.
  • Add static and runtime annotation tests, stubtest validation, and pinned checking tools.
  • Integrate checks into development workflows and update documentation and existing tests.
File Description
python/​scripts/​update_stub_docstrings.py Resolve internal stubs through pyarrow.lib.
python/​scripts/​run_stubtest.py Compare stubs against installed runtime signatures.
python/​requirements-stubs.txt Centralize annotation-checking dependencies.
python/​pyproject.toml Configure type checkers, annotation tests, and stub linting.
python/​pyarrow/​tests/​test_types.py Add missing assertions and typing suppressions.
python/​pyarrow/​tests/​test_table.py Annotate optional NumPy import fallback.
python/​pyarrow/​tests/​test_sparse_tensor.py Adjust optional imports and SciPy skip conditions.
python/​pyarrow/​tests/​test_schema.py Annotate optional NumPy import fallback.
python/​pyarrow/​tests/​test_convert_builtin.py Annotate optional NumPy import fallback.
python/​pyarrow/​tests/​test_array.py Update scalar assertion and add missing offset assertion.
python/​pyarrow/​tensor.pxi Correct sparse-tensor parameter documentation.
python/​pyarrow/​scalar.pxi Correct documented parameter name.
python/​pyarrow/​array.pxi Fix spacing in a documentation example.
python/​pyarrow-stubs/​types.pyi Annotate type predicates and type identifiers.
python/​pyarrow-stubs/​lib.pyi Re-export internal stubs and define MonthDayNano.
python/​pyarrow-stubs/​_stubs_typing.pyi Define shared aliases, protocols, and synthetic types.
python/​pyarrow-stubs/​_compute.pyi Provide a compute placeholder.
python/​pyarrow-stubs/​__lib/​types.pyi Annotate data types, fields, schemas, and factories.
python/​pyarrow-stubs/​__lib/​tensor.pyi Annotate dense and sparse tensor APIs.
python/​pyarrow-stubs/​__lib/​table.pyi Annotate tables, record batches, and chunked arrays.
python/​pyarrow-stubs/​__lib/​scalar.pyi Provide scalar placeholders.
python/​pyarrow-stubs/​__lib/​memory.pyi Annotate memory pools and allocation controls.
python/​pyarrow-stubs/​__lib/​ipc.pyi Provide a record-batch reader placeholder.
python/​pyarrow-stubs/​__lib/​io.pyi Add a buffer placeholder.
python/​pyarrow-stubs/​__lib/​error.pyi Annotate exceptions and signal handling.
python/​pyarrow-stubs/​__lib/​device.pyi Annotate devices and memory managers.
python/​pyarrow-stubs/​__lib/​config.pyi Annotate build and runtime information.
python/​pyarrow-stubs/​__lib/​builder.pyi Annotate string builders.
python/​pyarrow-stubs/​__lib/​array.pyi Annotate arrays, constructors, and conversions.
python/​pyarrow-stubs/​__lib/​__init__.pyi Separate internal stubs from public factory names.
python/​pyarrow-stubs/​__init__.pyi Expose annotated public APIs.
python/​pyarrow-stubs-tests/​test_types.py Check type factories, schemas, and narrowing.
python/​pyarrow-stubs-tests/​test_tensor.py Check tensor conversions and annotations.
python/​pyarrow-stubs-tests/​test_table.py Check tabular APIs and input protocols.
python/​pyarrow-stubs-tests/​test_device.py Check device properties and hashability.
python/​pyarrow-stubs-tests/​test_array.py Check array operations, constructors, and invalid inputs.
python/​pyarrow-stubs-tests/​stubtest_allowlist.txt Record expected runtime/stub mismatches.
python/​pyarrow-stubs-tests/​conftest.py Supply stub-only types during runtime tests.
python/​pyarrow-stubs-tests/​_check.py Pair static assertions with runtime checks.
python/​CMakeLists.txt Correct stub paths in commented-out installation guidance.
docs/​source/​developers/​python/​development.rst Document annotation development and validation.
ci/​scripts/​python_wheel_validate_contents.py Update the stub source path used for validation.
ci/​scripts/​python_test_type_annotations.sh Run annotation tests and stubtest in CI.
.pre-commit-config.yaml Add stub linting, formatting, and manual type checks.
.github/​workflows/​dev.yml Run manual stub-checking hooks in lint CI.

Review effort: Balanced

Comment thread python/pyarrow-stubs/__lib/array.pyi Resolved
@property
def is_cpu(self) -> bool: ...
@property
def statistics(self) -> ArrayStatistics | None: ...
Comment thread python/pyarrow-stubs/__lib/io.pyi Resolved
Comment thread python/pyarrow-stubs/__lib/types.pyi Resolved
Comment on lines +532 to +551
index_type: IntegerType | _DataTypeAliasInput,
value_type: _DataTypeLike,
ordered: bool = False,
) -> DictionaryType: ...
def struct(
fields: _StructFieldsInput,
) -> StructType: ...
def sparse_union(
child_fields: Iterable[Field], type_codes: Sequence[int] | None = None
) -> SparseUnionType: ...
def dense_union(
child_fields: Iterable[Field], type_codes: Sequence[int] | None = None
) -> DenseUnionType: ...
def union(
child_fields: Iterable[Field],
mode: Literal["sparse", "dense"] | int,
type_codes: Sequence[int] | None = None,
) -> SparseUnionType | DenseUnionType: ...
def run_end_encoded(
run_end_type: _RunEndType | _DataTypeAliasInput,

This comment was marked as low quality.

This comment was marked as low quality.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@tadeja

tadeja commented Nov 17, 2026

Copy link
Copy Markdown
Member

Only a note on 7fa510f - this is to fix strict type checking errors "Missing type arguments" (np.dtype, list, tuple) and errors due to np.typing.DTypeLike.

a) mypy --strict: numpy 2.1.3 had 7 errors, numpy 2.5.5 had 4 errors. With 7fa510f zero errors.
b) pyright (strict): numpy 2.1.3 had 50 errors, numpy 2.5.5 had 39 errors. With 7fa510f 34 errors remaining (mostly due to to-be annotated modules)

Details

a) python % mypy --strict and numpy-2.1.3

pyarrow-stubs/lib.pyi:37: error: Missing type arguments for generic type "tuple"  [type-arg]
pyarrow-stubs/__lib/table.pyi:192: error: Missing type arguments for generic type "dtype"  [type-arg]
pyarrow-stubs/__lib/table.pyi:251: error: Missing type arguments for generic type "dtype"  [type-arg]
pyarrow-stubs/__lib/array.pyi:126: error: Missing type arguments for generic type "list"  [type-arg]
pyarrow-stubs/__lib/array.pyi:126: error: Missing type arguments for generic type "tuple"  [type-arg]
pyarrow-stubs/__lib/array.pyi:230: error: Missing type arguments for generic type "dtype"  [type-arg]
pyarrow-stubs-tests/test_table.py:95: error: Missing type arguments for generic type "list"  [type-arg]

python % mypy --strict and numpy 2.5.5

pyarrow-stubs/lib.pyi:37: error: Missing type arguments for generic type "tuple"  [type-arg]
pyarrow-stubs/__lib/array.pyi:126: error: Missing type arguments for generic type "list"  [type-arg]
pyarrow-stubs/__lib/array.pyi:126: error: Missing type arguments for generic type "tuple"  [type-arg]
pyarrow-stubs-tests/test_table.py:95: error: Missing type arguments for generic type "list"  [type-arg]
Found 4 errors in 3 files (checked 25 source files)

b) pyright (strict) and numpy-2.1.3

  ...
  /arrow/python/pyarrow-stubs/__lib/types.pyi:568:22 - error: Type of parameter "dtype" is unknown (reportUnknownParameterType)
  /arrow/python/pyarrow-stubs/__lib/types.pyi:568:29 - error: Type of "typing" is unknown (reportUnknownMemberType)
  /arrow/python/pyarrow-stubs/__lib/types.pyi:568:29 - error: Type of "DTypeLike" is unknown (reportUnknownMemberType)
  /arrow/python/pyarrow-stubs/__lib/types.pyi:568:32 - error: "typing" is not a known attribute of module "numpy" (reportAttributeAccessIssue)
  ...
50 errors, 0 warnings, 0 informations

pyright (strict) and numpy-2.5.5 errors in CI:
https://github.com/apache/arrow/actions/runs/43902765930/job/133423484153#step:6:11
https://github.com/apache/arrow/actions/runs/43902766009/job/133423485569#step:6:4754
and https://github.com/apache/arrow/actions/runs/43902766096/job/133423485406#step:6:4758

For now we don't have strict type checking but I think it's good to fix errors ahead for potential switch to "strict" sometime later.

typeCheckingMode = "basic"

@tadeja

tadeja commented Nov 17, 2026

Copy link
Copy Markdown
Member

@github-actions crossbow submit preview-docs

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@github-actions

Copy link
Copy Markdown

Revision: d085957

Submitted crossbow builds: ursacomputing/crossbow @ actions-fa2fdfd6a0

Task Status
preview-docs GitHub Actions

@rok

rok commented Nov 17, 2026

Copy link
Copy Markdown
Member Author
Detailed AI generated open questions

Open questions

Summary of what is still open on this PR after going through all comments and review threads (head 374093b).

Needs an answer or a decision

  1. Stubs and non-editable installs (Joris, in the development.rst thread). The docs say "With a non-editable install ... the stubs aren't installed yet". This is because of [Python] type checkers broken for pyarrow.compute and other sub-modules since PyArrow 24.0+ #49831: the CMake block that installs the stubs has been commented out since June, so no build installs them yet. Still unexplained from the same thread: mypy pyarrow/tests/test_table.py finds the same issues with or without pyarrow-stubs on the command line.
  2. IDE support with an editable install. VS Code did not pick up the stubs for Joris on 6 Nov. The layout has changed since (pyarrow-stubs/pyarrow/ became pyarrow-stubs/), which fixed PyCharm, but VS Code has not been confirmed.
  3. Changes to pyarrow/tests. The suggestion was to either type check those tests in CI or leave the edits out of this PR. Most were removed, but 11 # type: ignore lines across six test files remain, and CI doesn't check them.
  4. --follow-untyped-imports for mypy. Raised in review and tested locally, but no decision is recorded and it isn't in pyproject.toml.
  5. dictionary() and run_end_encoded() reject a plain DataType (Copilot). pa.dictionary(arr.type, pa.string()) works at runtime and fails in mypy and pyright, because the parameter is typed with the stub-only integer types. Should the "parameters take the base Array" rule also apply to DataType?
  6. Array operators and Array.from_dlpack are missing from the stubs (Copilot). arr + arr, -arr, arr & arr and pa.Array.from_dlpack(...) work on the nightly and are rejected by both checkers. Add them here or in a follow-up?

Deferred, tracked in an issue

Deferred, no issue yet

  • Squash __lib/*.pyi into lib.pyi, and decide whether _stubs_typing.pyi stays separate. Proposed for after the whole series is merged.
  • Literal versus str for string options (unit etc.). Left strict, as in NumPy and pandas-stubs; to revisit once we see downstream impact.
  • to_pandas return type per class. Left as a TODO.
  • KeyValueMetadata.keys/values/items return generators at runtime rather than mapping views. Worth a separate issue.
  • Type_* constants and private helpers (string_to_tzinfo, tzinfo_to_string, ensure_metadata, table_to_blocks) are not in the stubs. Add them or allowlist them once stubtest stops skipping missing names.
  • Re-check the Self rule before merge ("use Self when it is correct, a concrete type otherwise").
  • Array1D for parameters that only accept 1-d NumPy arrays.
  • Iterable / Sequence / list for array-like inputs. Handled case by case; no general rule written down.
  • Buffer placeholder is an empty class, so buffer.to_pybytes() or len(buffer) are rejected. Waiting for the I/O stubs PR.

Additional findings from a local pass

Checked against the current head with mypy and pyright; not raised in review so far.

  • Dict schemas: pa.schema({"a": pa.int8()}) and pa.struct({...}) are rejected. Mapping[str | bytes, ...] doesn't accept a dict[str, ...], because Mapping is invariant in its key type.
  • Metadata dicts: for the same reason, a dict[str, str] variable is rejected as metadata= in pa.table, pa.record_batch, from_pydict and from_pylist.
  • take / filter with a ChunkedArray: Array.take and Array.filter are typed as returning Array, but the runtime returns a ChunkedArray. RecordBatch.take(ChunkedArray) type checks and raises at runtime.
  • Tuples: rejected for names= in pa.table / pa.record_batch, and in rename_columns, drop_columns and Table.drop, although they work at runtime. RecordBatch.select rejects mixed names and indices, unlike Table.select.
  • Housekeeping: a per-file # mypy: warn-unused-ignores in test_tensor.py and test_device.py (already set globally), pytest.importorskip("numpy") in two stub tests, the _compute.pyi header comment naming pyarrow.lib, and = ... defaults in Bool8Array.to_numpy and binary().
Details

@rok

rok commented Nov 17, 2026

Copy link
Copy Markdown
Member Author

I suppose this is now ready for a wider review! (See #55264 (comment) for potentially useful context)
@dangotbanned @paddyroddy @samwgoldman would you have time to take a look?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants