Skip to content

[mypyc] Fix range() loops with native int bounds and step > 1 near the type's maximum - #22470

Open
Jahongir-Qurbonov wants to merge 1 commit into
python:masterfrom
Jahongir-Qurbonov:mypyc-range-unsigned-step
Open

Jahongir-Qurbonov wants to merge 1 commit into
python:masterfrom
Jahongir-Qurbonov:mypyc-range-unsigned-step

Conversation

@Jahongir-Qurbonov

Copy link
Copy Markdown
Contributor

When the end value of a range() for loop has a native int type, mypyc uses that type for the loop index register. The loop checks i < end and then does i += step. With step 1 this can't overflow, but with a larger step the increment can wrap around past the bounds of the type before the condition gets a chance to stop the loop:

from mypy_extensions import u64

def f(a: u64, b: u64) -> int:
    c = 0
    for i in range(a, b, 2):
        c += 1
    return c

f(2**64 - 2, 2**64 - 1)  # never terminates; expected 1

The same happens with u8 etc., and with signed types (i8/i16/i32/i64) near the maximum value it's signed overflow (undefined behavior in C for i64).

Also, a negative step with an unsigned end value was rejected at compile time, since the step was added as a negative constant of the index type:

def g(a: u64, b: u64) -> None:
    for i in range(a, b, -1):  # error: Value -1 is out of range for "u64"
        ...

Now, for a fixed-width index, the step block adds or subtracts the magnitude of the step, so negative steps work with unsigned types. When the magnitude of the step is greater than 1, it first compares the index against a constant (e.g. MAX - step). If stepping would go past the bounds of the type, it would also go past the end value, so the index is set to the end value instead, which makes the loop condition fail. The common step ±1 case generates the same IR as before (apart from i - 1 instead of i + -1). If the step magnitude doesn't even fit in the native int type (e.g. range(i8(...), i8(...), 200), which used to be a compile error too), the loop falls back to an arbitrary precision int index.

Added a run test that compares loops over u8, u64, i8, i16, i32 and i64 bounds near the minimum and maximum values with various positive and negative steps against Python's range(), plus irbuild tests for a step > 1 (i64) and a negative step with an unsigned end (u64).

…e type's maximum

When the end value of a range() for loop has a native int type, the
loop index uses the same type. With a step whose magnitude is greater
than 1, incrementing the index could overflow past the bounds of the
type, so for example `for i in range(u64(2**64 - 2), u64(2**64 - 1), 2)`
never terminated, and with signed types this was undefined behavior.

Now when |step| > 1, the step block first checks whether stepping would
go past the bounds of the type (by comparing the index against a
constant). If so, it would also go past the end value, so the index is
set to the end value instead, which ends the loop.

Negative steps with unsigned end values were also rejected at compile
time (`Value -1 is out of range for "u64"`), since the step was added
as a negative constant. Subtract the magnitude of the step instead.
If the magnitude of the step doesn't fit in the native int type, fall
back to using an arbitrary precision int index.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant