Repository navigation
Keep the last value when setitem gets repeated indices - #4601
bhaskargurram-ai wants to merge 5 commits into
Conversation
An orthogonal or coordinate selection that names the same element more than once used to store whichever value the chunk dispatch and the in-chunk scatter happened to write last, which depends on the chunk layout and on the sort order of the platform's NumPy build. The selection is now reduced, before any chunk is touched, to the last occurrence of each element, with the value array reduced to match, so the stored value is the last one given, as with NumPy, on every platform and chunking. The property tests draw repeated indices again instead of skipping them.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #4601 +/- ##
==========================================
+ Coverage 94.90% 94.92% +0.02%
==========================================
Files 96 96
Lines 14055 14131 +76
==========================================
+ Hits 13339 13414 +75
- Misses 716 717 +1
🚀 New features to boost your workflow:
|
Merging this PR will not alter performance
Comparing Footnotes
|
| return None | ||
| # self.selection is flat and non-negative, grouped by chunk; sel_sort maps each entry | ||
| # back to its position in the selection as given, which decides what "last" means. | ||
| linear = np.ravel_multi_index(self.selection, shape) |
There was a problem hiding this comment.
this will fail for arrays with more than a 64-bit-integer's worth of elements. the old code flattened over the chunk grid, so it was more robust. this is a blocker
np.ravel_multi_index overflows for arrays with more than 2**63 elements. Sort the points coordinate by coordinate with lexsort and compare neighbours instead.
When an index array repeats a position,
oindex,vindexand__setitem__could store any of the values given for it. The issue's example,arr.oindex[np.array([-1, -1, 0, 0])] = [0, 1, 2, 3]on a chunked array, gives[2, 0, 0, 0]on Linux with numpy 2 and[3, 0, 0, 1]on macOS, while NumPy itself gives[3, 0, 0, 1]everywhere. The reason is that the in-chunk scatter receives the repeated positions in whatever order the (unstable) argsort by chunk put them, so the "last" write is platform dependent, and for a single-chunk array the order is different again.This follows the suggestion in the thread: before any chunk is touched, the selection is reduced so that it names each element once, keeping the last occurrence, and the value array is reduced to match. The stored value is then the last one given for the element, which is what sequential NumPy assignment produces, on every platform and for every chunking. For orthogonal selections the reduction is per dimension (that matches
a[np.ix_(i, j)] = v, where the last row wins for a repeatediand the last column for a repeatedj); for coordinate selections the points are reduced on their linear index. Negative and positive spellings of the same index count as the same position. Selections with no repeats take the existing path unchanged: the check is a linear pass for sorted index arrays and onenp.uniqueornp.lexsortotherwise. Masks never repeat a point and are skipped.A value that cannot be broadcast to the selection now raises
ValueError: shape mismatch ...(NumPy's wording) on this path instead of silently writing a prefix of it.Tests: the previously skipped
test_setitem_repeated_indexruns again, parametrised over chunkings and overoindex,vindexand__setitem__; new tests cover repeats along both dimensions of an orthogonal selection (with integer, slice and Boolean selectors on the other dimension, and scalar values), repeated points across chunk boundaries with 1-D and 2-D coordinate arrays and the sorted single-array path, the shape-mismatch error, and thelast_occurrenceshelper. The hypothesis teststest_oindexandtest_vindexno longerassumeaway repeated indices, so they now compare repeated-index writes against NumPy as well; before the changetest_oindexfails within a few examples.Fixes #2855