Skip to content

fix(indexing): keep the last value when setitem gets repeated indices - #4643

Draft
d-v-b wants to merge 4 commits into
zarr-developers:mainfrom
d-v-b:fix/stable-chunk-grouping
Draft

d-v-b wants to merge 4 commits into
zarr-developers:mainfrom
d-v-b:fix/stable-chunk-grouping

Conversation

@d-v-b

@d-v-b d-v-b commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

🤖 AI text below 🤖

Assigning through oindex, vindex or __setitem__ with an index array that repeats a position now stores the last value given for that position, as NumPy does. Previously the stored value depended on the platform, the chunk layout and the number of selected items.

Fixes #2855

Cause and fix

IntArrayDimIndexer and CoordinateIndexer group the selected items by chunk with np.argsort, whose default sort is not stable. Repeated indices therefore reached the in-chunk assignment in an arbitrary order. The agent changed both sorts to kind="stable", so repeats stay in the order given and NumPy's in-chunk assignment stores the last value. The sharding codec builds its inner selections with the same two classes, so it is covered by the same change.

This is not only a Linux problem: NumPy's default sort happens to be stable below 16 elements, so short examples pass on some platforms. On an arm64 Mac, a.vindex[idx] = np.arange(24) with 23 of 24 points on one cell stored 22 instead of 23 before this change.

Relation to #4601

This is an alternative to #4601, which removes duplicates from the selection before any chunk is written. Compared with that approach, this one adds no pass over the selection, no second indexer build and no copy of the value. It also never flattens coordinates over the full array shape, so coordinate writes to arrays with more than 2**63 elements keep working. The change to the property tests (no longer skipping repeated indices) is the same as in #4601.

TensorStore gets the same ordering by construction: it partitions positions into per-chunk buckets in C order and does not remove duplicates. zarr-indexing sorts stably and also removes duplicates per chunk.

Tests
  • test_setitem_repeated_index was skipped; it is now one parametrized test over oindex/vindex, four chunk layouts (including sharded) and four selections, compared against NumPy. 9 of its 32 cases fail without the fix on an arm64 Mac.
  • The property tests test_oindex and test_vindex no longer assume away repeated indices. Both fail without the fix and pass with it on five seeds.
  • tests/test_indexing.py, tests/test_properties.py and tests/test_codecs/test_sharding.py pass locally.
Limits
  • The fix relies on NumPy applying an advanced assignment with repeated indices in order inside a chunk. NumPy does this in practice, but its documentation does not guarantee it.
  • GPU buffers were not tested. CuPy documents the result of an assignment with repeated indices as undefined, so this change may not be enough there. Removing duplicates per chunk before the in-chunk assignment would close both gaps.

🤖 Generated with Claude Code

The indexers group selected items by chunk with an unstable argsort, so
repeated indices reached the in-chunk assignment in an arbitrary order
and the stored value depended on the platform, the chunk layout and the
number of selected items. Sorting stably keeps repeats in the order
given, and the in-chunk NumPy assignment then stores the last value.

Refs zarr-developers#2855

Assisted-by: ClaudeCode:claude-fable-5.2
@codecov

codecov Bot commented Nov 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.90%. Comparing base (95ec04a) to head (91b7c2c).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4643   +/-   ##
=======================================
  Coverage   94.90%   94.90%           
=======================================
  Files          96       96           
  Lines       14051    14055    +4     
=======================================
+ Hits        13335    13339    +4     
  Misses        716      716           
Files with missing lines Coverage Δ
src/zarr/core/indexing.py 96.57% <100.00%> (ø)

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

setitem with repeated indices is not guaranteed in practice

1 participant