Repository navigation
Conversation
The indexers group selected items by chunk with an unstable argsort, so repeated indices reached the in-chunk assignment in an arbitrary order and the stored value depended on the platform, the chunk layout and the number of selected items. Sorting stably keeps repeats in the order given, and the in-chunk NumPy assignment then stores the last value. Refs zarr-developers#2855 Assisted-by: ClaudeCode:claude-fable-5.2
Assisted-by: ClaudeCode:claude-fable-5.2
Assisted-by: ClaudeCode:claude-fable-5.2
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4643 +/- ##
=======================================
Coverage 94.90% 94.90%
=======================================
Files 96 96
Lines 14051 14055 +4
=======================================
+ Hits 13335 13339 +4
Misses 716 716
🚀 New features to boost your workflow:
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
Assigning through
oindex,vindexor__setitem__with an index array that repeats a position now stores the last value given for that position, as NumPy does. Previously the stored value depended on the platform, the chunk layout and the number of selected items.Fixes #2855
Cause and fix
IntArrayDimIndexerandCoordinateIndexergroup the selected items by chunk withnp.argsort, whose default sort is not stable. Repeated indices therefore reached the in-chunk assignment in an arbitrary order. The agent changed both sorts tokind="stable", so repeats stay in the order given and NumPy's in-chunk assignment stores the last value. The sharding codec builds its inner selections with the same two classes, so it is covered by the same change.This is not only a Linux problem: NumPy's default sort happens to be stable below 16 elements, so short examples pass on some platforms. On an arm64 Mac,
a.vindex[idx] = np.arange(24)with 23 of 24 points on one cell stored 22 instead of 23 before this change.Relation to #4601
This is an alternative to #4601, which removes duplicates from the selection before any chunk is written. Compared with that approach, this one adds no pass over the selection, no second indexer build and no copy of the value. It also never flattens coordinates over the full array shape, so coordinate writes to arrays with more than 2**63 elements keep working. The change to the property tests (no longer skipping repeated indices) is the same as in #4601.
TensorStore gets the same ordering by construction: it partitions positions into per-chunk buckets in C order and does not remove duplicates.
zarr-indexingsorts stably and also removes duplicates per chunk.Tests
test_setitem_repeated_indexwas skipped; it is now one parametrized test overoindex/vindex, four chunk layouts (including sharded) and four selections, compared against NumPy. 9 of its 32 cases fail without the fix on an arm64 Mac.test_oindexandtest_vindexno longerassumeaway repeated indices. Both fail without the fix and pass with it on five seeds.tests/test_indexing.py,tests/test_properties.pyandtests/test_codecs/test_sharding.pypass locally.Limits
🤖 Generated with Claude Code