Skip to content

[mypyc] Regenerate dependent modules when weakrefs, serializable or acyclic is toggled on a base class - #22467

Open
Jahongir-Qurbonov wants to merge 2 commits into
python:masterfrom
Jahongir-Qurbonov:mypyc-weakrefs-separate-compilation
Open

Jahongir-Qurbonov wants to merge 2 commits into
python:masterfrom
Jahongir-Qurbonov:mypyc-weakrefs-separate-compilation

Conversation

@Jahongir-Qurbonov

@Jahongir-Qurbonov Jahongir-Qurbonov commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

@mypyc_attr(weakrefs=True) (added in #22387) adds a weak reference list field to the object struct of the class, right after the vtable pointer, so the attributes of every native subclass move and tp_basicsize grows. With separate compilation, toggling it on a base class didn't regenerate subclasses in other groups. Mypy doesn't record class decorators, so the base module's interface hash didn't change, and the subclass module kept its old C code with the old layout.

Example, compiled with mypycify(["base_mod.py", "sub_mod.py"], separate=True):

# base_mod.py
from mypy_extensions import mypyc_attr

class A:  # @mypyc_attr(weakrefs=True) added before the second build
    def __init__(self) -> None:
        self.x = 111
        self.s = "hello"

# sub_mod.py
from base_mod import A

class B(A):
    def __init__(self) -> None:
        super().__init__()
        self.y = 222
        self.t = "world"

After the second build B.__basicsize__ was 56 instead of 64, and accessing attributes of a B instance and creating a weak reference to it segfaulted. Expected: sub_mod is recompiled and everything works.

The kinds of a module's classes are already part of its interface through MypycPlugin.report_config_data, for native_class=False and @trait (#22136). This adds the weakrefs flag to the tuple class_kinds() returns. When the cache is written it's derived from the tree the same way prepare_class_def sets ClassIR._weakrefs (only for native classes that aren't traits), and when the cache is checked it comes from _weakrefs in the cached IR, so the two agree when nothing changed.

I added testIncrementalCompilationTogglingWeakrefs to run-multimodule.test, following the existing tests for native_class and @trait: it turns weakrefs on and off on a base class in other.py, and in the middle step it changes only native.py to check that other stays fresh. Without the fix, the separate compilation variant crashes in step 2.

The same gap existed for two more @mypyc_attr flags, fixed in a second commit. serializable is inherited, and a serializable class's tp_new doesn't call __init__. After making a base class serializable, a stale subclass in another group still called __init__ without arguments, so copy.copy()/pickle of the subclass failed with a TypeError about missing arguments. acyclic is only allowed when all base classes are acyclic. After making a base class non-acyclic, the error for an acyclic subclass wasn't reported. CPython then made the subclass a GC type with the base's tp_traverse, but its tp_dealloc doesn't untrack instances, which can segfault if a collection runs while an instance is freed. Both flags are now part of the class kinds (_serializable and is_acyclic on the cached-IR path), with a new run-multimodule test for each. allow_interpreted_subclasses doesn't need this: a subclass's generated code doesn't depend on its base having the flag. The only stale case is a base losing it while a subclass still uses it, and then the build fails with a C compile error instead of mypyc's error message.

…se class

@mypyc_attr(weakrefs=True) (added in python#22387) adds a weak reference list
field to the object struct of the class, which moves the attributes of
all native subclasses. With separate compilation, toggling it on a base
class didn't regenerate subclasses in other groups, since mypy doesn't
record class decorators and the interface hash of the base module
stayed the same. The subclass kept its stale layout, and accessing its
attributes or creating a weak reference could segfault.

Include the weakrefs flag in class_kinds(), which MypycPlugin reports
as part of a module's interface. When writing the cache it's derived
from the tree like prepare_class_def sets ClassIR._weakrefs, and when
checking the cache it's read from the cached IR.
…toggled

Like weakrefs, the serializable and acyclic flags of
`@mypyc_attr` affect the generated code of subclasses in other modules,
but mypy ignores the decorator, so toggling them didn't change the
interface of the class's module. With separate compilation, the
subclasses were then left stale:

* `serializable` is inherited, and the tp_new of a serializable class
  doesn't call `__init__`. After making a base class serializable, a
  stale subclass still called `__init__` without arguments from tp_new,
  so `copy.copy()` and `pickle` failed with a `TypeError` about missing
  arguments to `__init__`.

* A class can only be acyclic if all its base classes are. After making
  a base class non-acyclic, the error for an acyclic subclass wasn't
  reported. CPython then gives the subclass the GC flag and tp_traverse
  of the base, but the subclass's tp_dealloc doesn't untrack instances,
  so a collection triggered while an instance was being freed (e.g. from
  a `__del__` of one of its attributes) crashed.

Add both flags to the class kinds that `MypycPlugin.report_config_data`
makes part of a module's interface, using the `_serializable` and
`is_acyclic` fields of the cached IR on the check path.
@Jahongir-Qurbonov Jahongir-Qurbonov changed the title [mypyc] Regenerate dependent modules when weakrefs is toggled on a base class [mypyc] Regenerate dependent modules when weakrefs, serializable or acyclic is toggled on a base class Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant