From 2a2f9fb72b2fe2281956f3ad9f542bff80f302b5 Mon Sep 17 00:00:00 2001 From: Sam Gammon Date: Tue, 17 Nov 2026 03:13:18 -0800 Subject: [PATCH 1/2] Resolve variable-access names lazily from tagged bytecode (#12) * Resolve variable-access names lazily from tagged bytecode Drop the per-code-unit variable-access side table added in #11. It duplicated names the code unit already holds and locations the Bytecode DSL already records: every name occurrence was copied into the code unit, marshaled (UTF-8 name plus five ints), re-interned on load, and kept on the heap whether or not a tool was attached. The ReadVariableTag/WriteVariableTag node object now resolves names on demand from the tagged region's bytecode (TagTreeNode#getBytecodeNode and the enter/return bytecode indices). The access is the last matching load or store outside nested variable tags. Its operands identify the name: frame slots map to varnames, cellvars and freevars; LocalAccessor operands map through the locals table info; cell instructions carry an index; and global/name instructions carry the name constant. Identifier locations come from the tag's own source layers. Fused two-target unpacking now gets one identifier layer per operand, so per-name ranges are preserved. Variable tags no longer overwrite the compiler's currentLocation, which is used for syntax error positions and loop-header trace lines. Bump the bytecode cache version to 39 for the marshal format change (37 on elide-25.5.x; 25.6 was already at 38). Co-Authored-By: Claude Opus 5.5 * Fix JDK 21 download URLs for github jobs Ported from elide-25.5.x; the 25.6-only placeholder code unit in PBytecodeDSLRootNode drops the argument too. (cherry picked from commit 3fa9e560b70ea1486d8ec26d34233bf3373f61e8) * Identify variable accesses by operation class, not instruction names Instruction names are debugging aids and can change at any time. Use InstructionDescriptor#getOperationClass() (GR-81626, backported in elide-dev/graal#241) to recognize the custom name operations, and recognize built-in local loads/stores by their local argument. The access is still the last match outside nested variable tags: store values can contain untagged local loads, such as the self/__class__ loads that super() generates. Pin the Graal imports to the backport commit. Co-Authored-By: Claude Opus 5.5 * Pin Graal imports to the merged getOperationClass() backport (elide-dev/graal#241) Co-Authored-By: Claude Opus 5.5 * Keep instruction locations unchanged inside variable tags tagged variable access. Source sections exist whether or not tags are instrumented, so exception locations, traceback carets and line tables moved to the statement. This broke the tagged CPython tests test_traceback (test_caret_in_type_annotation, test_byte_offset_with_emojis_term_highlight) and test_compile.test_lineno_procedure_call. Open the statement and identifier layers outside the tag, and re-open the previously innermost range inside it. The tag's own location stays at the statement, which keeps debugger caller locations stable (PythonDebugTest), and the tagged instructions keep their previous location. The augmented-assignment read tag now uses the same layout. Co-Authored-By: Claude Opus 5.5 * ci: sync ci/graal/common.json with the pinned Graal revision JVMCI 25.5 (and mx 7.88.0), but ci/graal/common.json still pinned JVMCI 25.4. On that JDK Truffle falls back to the non-optimizing runtime ("Graal expected a JVMCI release version of 25.5, but got 25.4"), so every junit test fails on the -Dpolyglot.engine.CompilationFailureAction option the gate sets. This went unnoticed while the JDK 21 bootstrap download failed before any test ran. Co-Authored-By: Claude Opus 5.5 * Apply mx 7.88 Eclipse formatting to TpSlotDescrGet The style gate's CodeFormatCheck with mx 7.88.0 indents the sealed class's permits continuation line. Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 Co-authored-by: Michael Simacek Ported from elide-25.5.x; the 25.6-only placeholder code unit in PBytecodeDSLRootNode drops the argument too. (cherry picked from commit 09e64c70ac3f50ae976db49701056371acb5783c) --- docs/user/Tooling.md | 4 +- .../test/runtime/VariableAccessTests.java | 70 +++++- .../modules/MarshalModuleBuiltins.java | 18 +- .../bytecode_dsl/BytecodeDSLCompiler.java | 2 +- .../bytecode_dsl/RootNodeCompiler.java | 105 +++++---- .../bytecode_dsl/BytecodeDSLCodeUnit.java | 22 +- .../bytecode_dsl/PBytecodeDSLRootNode.java | 12 +- .../VariableAccessDescriptor.java | 208 ++++++++++++++++-- 8 files changed, 323 insertions(+), 118 deletions(-) diff --git a/docs/user/Tooling.md b/docs/user/Tooling.md index e498423ef9..ecbb141a9e 100644 --- a/docs/user/Tooling.md +++ b/docs/user/Tooling.md @@ -49,7 +49,9 @@ use Truffle's UTF-16 columns, including for non-BMP identifiers. For unexecuted source, attach a source-section load listener before calling the instrument `Env.parse(Source)` API. Retain the returned call target while collecting facts. Reading node objects and their name/location descriptors does not require an entered Python context or guest -values, and supports concurrent readers. Code-unit serialization preserves this metadata. +values, and supports concurrent readers. Names are resolved on demand from the tagged load and store +instructions and the code object's existing name tables, so code units carry no extra metadata and +unmarshaled code reports the same names and locations. This interface describes compiler-emitted accesses, rather than a complete source symbol index. It covers name expressions, assignment and augmented-assignment targets, loop targets and diff --git a/graalpython/com.oracle.graal.python.test/src/com/oracle/graal/python/test/runtime/VariableAccessTests.java b/graalpython/com.oracle.graal.python.test/src/com/oracle/graal/python/test/runtime/VariableAccessTests.java index eda760b817..43cfe519d6 100644 --- a/graalpython/com.oracle.graal.python.test/src/com/oracle/graal/python/test/runtime/VariableAccessTests.java +++ b/graalpython/com.oracle.graal.python.test/src/com/oracle/graal/python/test/runtime/VariableAccessTests.java @@ -57,7 +57,6 @@ import org.graalvm.polyglot.Context; import org.junit.Test; -import com.oracle.graal.python.builtins.objects.code.PCode; import com.oracle.truffle.api.CallTarget; import com.oracle.truffle.api.instrumentation.InstrumentableNode; import com.oracle.truffle.api.instrumentation.SourceSectionFilter; @@ -169,6 +168,63 @@ def method(self, __private): } } + @Test + public void scopesAndNestedAccesses() throws IOException { + try (Context context = Context.newBuilder("python").allowAllAccess(true).build()) { + AnalysisInstrument instrument = context.getEngine().getInstruments().get(ID).lookup(AnalysisInstrument.class); + context.initialize("python"); + context.enter(); + try { + Analysis analysis = instrument.analyze(Source.newBuilder("python", """ + counter = 0 + def bump(): + global counter + counter += 1 + class Holder: + attr = counter + def method(self): + return __class__ + def outer(): + cell = 1 + def inner(): + nonlocal cell + cell += 1 + first = (second := cell) + return inner, first, second + class Child(Holder): + def method(self): + kept = super().method + return lambda: kept + """, "scopes.py").build()); + List accesses = analysis.accesses(); + // module and class bodies: name stores/loads + assertTrue(accesses.contains(new Access("", "counter", true, 1, 1, "counter"))); + assertTrue(accesses.stream().anyMatch(a -> a.name().equals("attr") && a.write() && a.line() == 6 && a.column() == 5)); + assertTrue(accesses.stream().anyMatch(a -> a.name().equals("counter") && !a.write() && a.line() == 6 && a.column() == 12)); + // explicit global, augmented + assertTrue(accesses.contains(new Access("bump", "counter", true, 4, 5, "counter"))); + assertTrue(accesses.contains(new Access("bump", "counter", false, 4, 5, "counter"))); + // free variable read + assertTrue(accesses.contains(new Access("method", "__class__", false, 8, 16, "__class__"))); + // cell store in the defining scope, nonlocal augmented store and load + assertTrue(accesses.contains(new Access("outer", "cell", true, 10, 5, "cell"))); + assertTrue(accesses.contains(new Access("inner", "cell", true, 13, 9, "cell"))); + assertTrue(accesses.contains(new Access("inner", "cell", false, 13, 9, "cell"))); + // a nested walrus store is its own access, not part of the enclosing store + List line14 = accesses.stream().filter(a -> a.line() == 14).toList(); + assertEquals(3, line14.size()); + assertTrue(line14.contains(new Access("outer", "first", true, 14, 5, "first"))); + assertTrue(line14.contains(new Access("outer", "second", true, 14, 14, "second"))); + assertTrue(line14.contains(new Access("outer", "cell", false, 14, 24, "cell"))); + // the store value's implicit loads (__class__, self for super()) are not the access + assertEquals(List.of(new Access("method", "kept", true, 18, 9, "kept"), new Access("method", "super", false, 18, 16, "super")), + accesses.stream().filter(a -> a.line() == 18).toList()); + } finally { + context.leave(); + } + } + } + @Test public void executedTagsPreservePythonBehavior() throws IOException { try (Context context = Context.newBuilder("python").allowAllAccess(true).build()) { @@ -208,7 +264,14 @@ def serialized(arg): return result """, "serialized.py").build(); Analysis analysis = instrument.analyzeSerialized(source); - assertEquals(6, analysis.accesses().stream().filter(a -> a.root().equals("serialized")).count()); + List accesses = analysis.accesses().stream().filter(a -> a.root().equals("serialized")).toList(); + assertEquals(6, accesses.size()); + // The original and the unmarshaled code report the same names and locations, which + // are derived from the serialized bytecode and its source information. + for (Access access : accesses) { + assertEquals(2, accesses.stream().filter(a -> a.name().equals(access.name()) && a.write() == access.write() && a.line() == access.line() && + a.column() == access.column()).count()); + } } finally { context.leave(); } @@ -309,9 +372,6 @@ private Analysis analyze(Source source, boolean execute, boolean serialized) thr cloned = types.FunctionType(marshal.loads(marshal.dumps(serialized.__code__)), globals()) assert cloned(42) == 42 """, "roundtrip.py").build()).call(); - PCode original = (PCode) env.parse(Source.newBuilder("python", "serialized.__code__", "original-code.py").build()).call(); - PCode cloned = (PCode) env.parse(Source.newBuilder("python", "cloned.__code__", "cloned-code.py").build()).call(); - assertEquals(original.getCodeUnit().variableAccesses, cloned.getCodeUnit().variableAccesses); } } finally { binding.dispose(); diff --git a/graalpython/com.oracle.graal.python/src/com/oracle/graal/python/builtins/modules/MarshalModuleBuiltins.java b/graalpython/com.oracle.graal.python/src/com/oracle/graal/python/builtins/modules/MarshalModuleBuiltins.java index f2143da04f..31b88126bd 100644 --- a/graalpython/com.oracle.graal.python/src/com/oracle/graal/python/builtins/modules/MarshalModuleBuiltins.java +++ b/graalpython/com.oracle.graal.python/src/com/oracle/graal/python/builtins/modules/MarshalModuleBuiltins.java @@ -117,7 +117,6 @@ import com.oracle.graal.python.nodes.PRaiseNode; import com.oracle.graal.python.nodes.StringLiterals; import com.oracle.graal.python.nodes.bytecode_dsl.BytecodeDSLCodeUnit; -import com.oracle.graal.python.nodes.bytecode_dsl.BytecodeDSLCodeUnit.VariableAccess; import com.oracle.graal.python.nodes.bytecode_dsl.PBytecodeDSLRootNode; import com.oracle.graal.python.nodes.bytecode_dsl.PBytecodeDSLRootNodeGen; import com.oracle.graal.python.nodes.call.special.LookupAndCallBinaryNode; @@ -1429,16 +1428,10 @@ private BytecodeDSLCodeUnit readBytecodeDSLCodeUnit() { int yieldFromGeneratorIndex = readInt(); int instrumentationDataIndex = readInt(); int maxProfileCEventStackSize = readInt(); - int accessCount = readSize(); - List variableAccesses = new ArrayList<>(accessCount); - for (int i = 0; i < accessCount; i++) { - variableAccesses.add(new VariableAccess(readString(true).toJavaStringUncached(), readInt(), readInt(), readInt(), readInt(), readInt() != 0)); - } BytecodeSupplier provider = new BytecodeSupplier(serialized, bytecodeFile, sourceReference, bytecodeOffset, bytecodeSize, cacheKey); return new BytecodeDSLCodeUnit(name, qualname, argCount, kwOnlyArgCount, positionalOnlyArgCount, flags, names, varnames, cellvars, freevars, constants, - startLine, startColumn, endLine, endColumn, classcellIndex, selfIndex, yieldFromGeneratorIndex, instrumentationDataIndex, maxProfileCEventStackSize, variableAccesses, - provider); + startLine, startColumn, endLine, endColumn, classcellIndex, selfIndex, yieldFromGeneratorIndex, instrumentationDataIndex, maxProfileCEventStackSize, provider); } private void writeCodeUnit(BytecodeDSLCodeUnit code) throws IOException { @@ -1474,15 +1467,6 @@ private void writeBytecodeDSLCodeUnit(BytecodeDSLCodeUnit code) throws IOExcepti writeInt(code.yieldFromGeneratorIndex); writeInt(code.instrumentationDataIndex); writeInt(code.maxProfileCEventStackSize); - writeInt(code.variableAccesses.size()); - for (VariableAccess access : code.variableAccesses) { - writeString(PythonUtils.toTruffleStringUncached(access.name())); - writeInt(access.startLine()); - writeInt(access.startColumn()); - writeInt(access.endLine()); - writeInt(access.endColumn()); - writeInt(access.write() ? 1 : 0); - } } private PCode readCode() { diff --git a/graalpython/com.oracle.graal.python/src/com/oracle/graal/python/compiler/bytecode_dsl/BytecodeDSLCompiler.java b/graalpython/com.oracle.graal.python/src/com/oracle/graal/python/compiler/bytecode_dsl/BytecodeDSLCompiler.java index 4f79c84ad3..b24ceec155 100644 --- a/graalpython/com.oracle.graal.python/src/com/oracle/graal/python/compiler/bytecode_dsl/BytecodeDSLCompiler.java +++ b/graalpython/com.oracle.graal.python/src/com/oracle/graal/python/compiler/bytecode_dsl/BytecodeDSLCompiler.java @@ -71,7 +71,7 @@ public class BytecodeDSLCompiler { * Bump this with every Truffle import change because serialized GraalPy bytecode effectively * depends on Truffle bytecode internals. */ - public static final int BYTECODE_VERSION = 38; + public static final int BYTECODE_VERSION = 39; public static final record BytecodeDSLCompilerResult(PBytecodeDSLRootNode rootNode, BytecodeDSLCodeUnit codeUnit) { } diff --git a/graalpython/com.oracle.graal.python/src/com/oracle/graal/python/compiler/bytecode_dsl/RootNodeCompiler.java b/graalpython/com.oracle.graal.python/src/com/oracle/graal/python/compiler/bytecode_dsl/RootNodeCompiler.java index f913815f89..05248a1c79 100644 --- a/graalpython/com.oracle.graal.python/src/com/oracle/graal/python/compiler/bytecode_dsl/RootNodeCompiler.java +++ b/graalpython/com.oracle.graal.python/src/com/oracle/graal/python/compiler/bytecode_dsl/RootNodeCompiler.java @@ -81,7 +81,6 @@ import java.util.HashMap; import java.util.HashSet; import java.util.List; -import java.util.LinkedHashSet; import java.util.Map; import java.util.Set; import java.util.function.BiConsumer; @@ -106,7 +105,6 @@ import com.oracle.graal.python.lib.PyObjectRichCompareBool; import com.oracle.graal.python.nodes.StringLiterals; import com.oracle.graal.python.nodes.bytecode_dsl.BytecodeDSLCodeUnit; -import com.oracle.graal.python.nodes.bytecode_dsl.BytecodeDSLCodeUnit.VariableAccess; import com.oracle.graal.python.nodes.bytecode_dsl.PBytecodeDSLRootNode; import com.oracle.graal.python.nodes.bytecode_dsl.PBytecodeDSLRootNodeGen; import com.oracle.graal.python.nodes.bytecode_dsl.PBytecodeDSLRootNodeGen.Builder; @@ -217,7 +215,6 @@ public final class RootNodeCompiler implements BaseBytecodeDSLVisitor variableAccesses = new LinkedHashSet<>(); private final HashMap varnames; private final HashMap cellvars; private final HashMap freevars; @@ -260,6 +257,8 @@ public final class RootNodeCompiler implements BaseBytecodeDSLVisitor statementLocations = new ArrayDeque<>(); + /** Source ranges of the currently open source sections, innermost first. */ + private final Deque openSourceRanges = new ArrayDeque<>(); private SourceRange currentLocation; BytecodeLocal yieldFromGenerator; private int lastTracedLine; @@ -596,7 +595,6 @@ private BytecodeDSLCompilerResult compileRootNode(String name, ArgumentInfo argu yieldFromGenerator != null ? yieldFromGenerator.getLocalIndex() : -1, instrumentationDataLocal.getLocalIndex(), maxProfileCEventStackSize, - List.copyOf(variableAccesses), new BytecodeSupplier(nodes)); ctx.codeUnits.put(key, codeUnit); } @@ -770,6 +768,7 @@ public BytecodeDSLCompilerResult compile() { public void reset() { statementLocations.clear(); + openSourceRanges.clear(); currentLocation = null; currentSaveExceptionLocal = null; prevSaveExceptionLocal = null; @@ -944,7 +943,8 @@ private static SourceRange getRootSourceRange(SSTNode node) { } } - private static void beginSourceSectionInner(Builder b, SourceRange sourceRange) { + private void beginSourceSectionInner(Builder b, SourceRange sourceRange) { + openSourceRanges.push(sourceRange); if (sourceRange.startLine >= 1 && sourceRange != SourceRange.ARTIFICIAL_RANGE) { if (sourceRange.startColumn >= 0 && sourceRange.endLine >= sourceRange.startLine && sourceRange.endColumn >= 0) { int startColumn = sourceRange.startColumn + 1; @@ -969,6 +969,7 @@ private static void beginSourceSectionInner(Builder b, SourceRange sourceRange) void endSourceSection(Builder b) { b.endSourceSection(); + openSourceRanges.pop(); } void beginStatement(SSTNode node, Builder b) { @@ -1918,51 +1919,72 @@ private SourceRange variableSourceRange(SourceRange range) { } } - private boolean recordVariableAccess(ExprTy.Name node, boolean write) { - SourceRange range = variableSourceRange(node); - if (node.id.startsWith(".") || range == SourceRange.ARTIFICIAL_RANGE || range.startLine < 1 || range.startColumn < 0 || - range.endLine < range.startLine || range.endColumn < 1 || !ctx.source.hasCharacters()) { - return false; + /** + * Returns the identifier range of a source-level variable access in Truffle (UTF-16) columns, + * or {@code null} if the access has no usable source token, e.g. compiler-generated names and + * AST-created code without valid positions. Such accesses are not tagged. + */ + private SourceRange variableAccessRange(ExprTy.Name node) { + if (node.id.startsWith(".")) { + return null; } - try { - ctx.source.createSection(range.startLine, range.startColumn + 1, range.endLine, range.endColumn); - } catch (IllegalArgumentException e) { - // AST-created code and incomplete source ranges need not identify a source token. - return false; + SourceRange range = variableSourceRange(node); + if (range == SourceRange.ARTIFICIAL_RANGE || range.startLine < 1 || range.startColumn < 0 || range.endLine < range.startLine || + range.endLine == range.startLine && range.endColumn <= range.startColumn) { + return null; } - variableAccesses.add(new VariableAccess(maybeMangle(node.id), range.startLine, range.startColumn, range.endLine, range.endColumn, write)); - return true; + return range; } private boolean beginVariableAccess(ExprTy.Name node, boolean write, Builder b) { - if (!recordVariableAccess(node, write)) { + SourceRange range = variableAccessRange(node); + if (range == null) { return false; } - beginVariableTag(variableSourceRange(node), write, b); + beginVariableTag(write, b, range); return true; } private boolean beginDirectVariableWrite(ExprTy target, BytecodeLocal local, Builder b) { - if (!isTemporaryLocal(local) && target instanceof ExprTy.Name name && recordVariableAccess(name, true)) { - beginVariableTag(variableSourceRange(target), true, b); - return true; + if (!isTemporaryLocal(local) && target instanceof ExprTy.Name name) { + return beginVariableAccess(name, true, b); } return false; } - private void beginVariableTag(SourceRange accessRange, boolean write, Builder b) { - beginSourceSection(accessRange, b); - // Bytecode DSL stack walks select the innermost tag. Keep its program location at the - // statement (or expression root), while the outer source layer identifies the access. + /** + * Opens a variable tag. The names are not recorded here: instrumentation resolves them lazily + * from the operands of the tagged instructions (see {@code VariableAccessDescriptor}). + *

+ * Layout, from outermost: one identifier range per variable operand of the tagged store or + * load (in reverse operand order), the enclosing statement, the tag, and then the source range + * that was innermost before the tag. The tag's own source sections are therefore, from most to + * least concrete, the statement followed by the identifiers in operand order. Bytecode DSL + * stack walks select the innermost tag, so debugger caller locations stay at the statement. + * The tagged instructions keep their previous innermost location, so exception locations, + * tracebacks and line tables are unaffected by these layers. + */ + private void beginVariableTag(boolean write, Builder b, SourceRange... accessRanges) { + SourceRange enclosing = openSourceRanges.isEmpty() ? SourceRange.ARTIFICIAL_RANGE : openSourceRanges.peek(); + for (int i = accessRanges.length - 1; i >= 0; i--) { + beginSourceSectionInner(b, accessRanges[i]); + } SourceRange statement = statementLocations.peek(); - beginSourceSection(variableSourceRange(statement != null ? statement : getRootSourceRange(startNode)), b); + beginSourceSectionInner(b, variableSourceRange(statement != null ? statement : getRootSourceRange(startNode))); b.beginTag(write ? WriteVariableTag.class : ReadVariableTag.class); + beginSourceSectionInner(b, enclosing); } - private void endVariableTag(boolean write, Builder b) { - b.endTag(write ? WriteVariableTag.class : ReadVariableTag.class); - endSourceSection(b); + private void endVariableTag(boolean write, int accessCount, Builder b) { endSourceSection(b); + b.endTag(write ? WriteVariableTag.class : ReadVariableTag.class); + for (int i = 0; i <= accessCount; i++) { + endSourceSection(b); + } + } + + private void endVariableTag(boolean write, Builder b) { + endVariableTag(write, 1, b); } private void endDirectVariableWrite(boolean tagged, Builder b) { @@ -3877,21 +3899,19 @@ private void visitIterableAssign(ExprTy[] nodes) { if (nodes.length == 2 && containsNoStarred(nodes)) { BytecodeLocal target1 = allowFastLocalLookup ? beginTemporaryLocalOrGetLocal(nodes[0], b) : beginTemporaryLocal(); BytecodeLocal target2 = allowFastLocalLookup ? beginTemporaryLocalOrGetLocal(nodes[1], b) : beginTemporaryLocal(); - boolean tagged = false; - if (!isTemporaryLocal(target1)) { - tagged |= recordVariableAccess((ExprTy.Name) nodes[0], true); - } - if (!isTemporaryLocal(target2)) { - tagged |= recordVariableAccess((ExprTy.Name) nodes[1], true); - } + // One instruction stores both targets; give each operand its own identifier + // layer. Temporary targets are tagged when they are stored to the real target. + SourceRange range1 = isTemporaryLocal(target1) ? null : variableAccessRange((ExprTy.Name) nodes[0]); + SourceRange range2 = isTemporaryLocal(target2) ? null : variableAccessRange((ExprTy.Name) nodes[1]); + boolean tagged = range1 != null || range2 != null; if (tagged) { - beginVariableTag(variableSourceRange(nodes[0]).withEnd(variableSourceRange(nodes[1])), true, b); + beginVariableTag(true, b, range1 != null ? range1 : SourceRange.ARTIFICIAL_RANGE, range2 != null ? range2 : SourceRange.ARTIFICIAL_RANGE); } b.beginUnpackToLocals2(target1, target2); generateValue.run(); b.endUnpackToLocals2(); if (tagged) { - endVariableTag(true, b); + endVariableTag(true, 2, b); } storeTemporaryLocalToTarget(target1, nodes[0], b, allowFastLocalLookup); storeTemporaryLocalToTarget(target2, nodes[1], b, allowFastLocalLookup); @@ -4051,12 +4071,11 @@ public Void visit(ExprTy.Name node) { beginStoreLocal(node.id, b); beginAugAssign(); if (tagged) { - recordVariableAccess(node, false); - b.beginTag(ReadVariableTag.class); + beginVariableTag(false, b, variableAccessRange(node)); } emitReadLocal(node.id, b); if (tagged) { - b.endTag(ReadVariableTag.class); + endVariableTag(false, b); } value.accept(StatementCompiler.this); endAugAssign(); @@ -4892,7 +4911,7 @@ public void endWrapWithDecorators(ExprTy[] decorators) { b.endTraceLineWithArgument(decorators[decorators.length - 1 - i].getSourceRange().startLine); b.endInstrumentCall(); endCallUnaryMethod(); - b.endSourceSection(); + endSourceSection(b); } } diff --git a/graalpython/com.oracle.graal.python/src/com/oracle/graal/python/nodes/bytecode_dsl/BytecodeDSLCodeUnit.java b/graalpython/com.oracle.graal.python/src/com/oracle/graal/python/nodes/bytecode_dsl/BytecodeDSLCodeUnit.java index 0911239db0..6796ad97e5 100644 --- a/graalpython/com.oracle.graal.python/src/com/oracle/graal/python/nodes/bytecode_dsl/BytecodeDSLCodeUnit.java +++ b/graalpython/com.oracle.graal.python/src/com/oracle/graal/python/nodes/bytecode_dsl/BytecodeDSLCodeUnit.java @@ -79,21 +79,6 @@ public final class BytecodeDSLCodeUnit { @CompilationFinal(dimensions = 1) public final TruffleString[] freevars; @CompilationFinal(dimensions = 1) public final Object[] constants; - /** Source-level variable accesses, independent of bytecode indices and live frames. */ - public final List variableAccesses; - - public record VariableAccess(String name, int startLine, int startColumn, int endLine, int endColumn, boolean write) { - public boolean isWithin(SourceSection section) { - return section.isAvailable() && - (startLine > section.getStartLine() || startLine == section.getStartLine() && startColumn + 1 >= section.getStartColumn()) && - (endLine < section.getEndLine() || endLine == section.getEndLine() && endColumn <= section.getEndColumn()); - } - - public SourceSection getSourceSection(Source source) { - return source.createSection(startLine, startColumn + 1, endLine, endColumn); - } - } - public final int startLine; public final int startColumn; public final int endLine; @@ -108,8 +93,7 @@ public SourceSection getSourceSection(Source source) { public BytecodeDSLCodeUnit(TruffleString name, TruffleString qualname, int argCount, int kwOnlyArgCount, int positionalOnlyArgCount, int flags, TruffleString[] names, TruffleString[] varnames, TruffleString[] cellvars, TruffleString[] freevars, Object[] constants, int startLine, int startColumn, int endLine, int endColumn, - int classcellIndex, int selfIndex, int yieldFromGeneratorIndex, int instrumentationDataIndex, int maxProfileCEventStackSize, List variableAccesses, - BytecodeSupplier supplier) { + int classcellIndex, int selfIndex, int yieldFromGeneratorIndex, int instrumentationDataIndex, int maxProfileCEventStackSize, BytecodeSupplier supplier) { assert !PythonOptions.detailedAssertionsEnabled() || isInterned(name); this.name = name; assert !PythonOptions.detailedAssertionsEnabled() || qualname == null || isInterned(qualname); @@ -137,7 +121,6 @@ public BytecodeDSLCodeUnit(TruffleString name, TruffleString qualname, int argCo this.yieldFromGeneratorIndex = yieldFromGeneratorIndex; this.instrumentationDataIndex = instrumentationDataIndex; this.maxProfileCEventStackSize = maxProfileCEventStackSize; - this.variableAccesses = List.copyOf(variableAccesses); } public abstract static class BytecodeSupplier { @@ -149,8 +132,7 @@ public abstract static class BytecodeSupplier { public BytecodeDSLCodeUnit withFlags(int flags) { return new BytecodeDSLCodeUnit(name, qualname, argCount, kwOnlyArgCount, positionalOnlyArgCount, flags, names, varnames, cellvars, freevars, constants, - startLine, startColumn, endLine, endColumn, classcellIndex, selfIndex, yieldFromGeneratorIndex, instrumentationDataIndex, maxProfileCEventStackSize, variableAccesses, - supplier); + startLine, startColumn, endLine, endColumn, classcellIndex, selfIndex, yieldFromGeneratorIndex, instrumentationDataIndex, maxProfileCEventStackSize, supplier); } @TruffleBoundary diff --git a/graalpython/com.oracle.graal.python/src/com/oracle/graal/python/nodes/bytecode_dsl/PBytecodeDSLRootNode.java b/graalpython/com.oracle.graal.python/src/com/oracle/graal/python/nodes/bytecode_dsl/PBytecodeDSLRootNode.java index fc30151647..042cfbc55b 100644 --- a/graalpython/com.oracle.graal.python/src/com/oracle/graal/python/nodes/bytecode_dsl/PBytecodeDSLRootNode.java +++ b/graalpython/com.oracle.graal.python/src/com/oracle/graal/python/nodes/bytecode_dsl/PBytecodeDSLRootNode.java @@ -64,7 +64,6 @@ import java.io.DataOutputStream; import java.io.IOException; import java.math.BigInteger; -import java.util.List; import java.util.HashMap; import java.util.Iterator; import java.util.Map; @@ -294,8 +293,6 @@ import com.oracle.truffle.api.bytecode.BytecodeLocation; import com.oracle.truffle.api.bytecode.BytecodeNode; import com.oracle.truffle.api.bytecode.TagTreeNode; -import com.oracle.truffle.api.instrumentation.StandardTags.ReadVariableTag; -import com.oracle.truffle.api.instrumentation.StandardTags.WriteVariableTag; import com.oracle.truffle.api.bytecode.BytecodeRootNode; import com.oracle.truffle.api.bytecode.ConstantOperand; import com.oracle.truffle.api.bytecode.ContinuationResult; @@ -520,7 +517,7 @@ public byte[] createSerializedBytecode(PythonLanguage rootLanguage) { BytecodeDSLCodeUnit code = new BytecodeDSLCodeUnit(name, name, 0, 0, 0, 0, PythonUtils.EMPTY_TRUFFLESTRING_ARRAY, PythonUtils.EMPTY_TRUFFLESTRING_ARRAY, PythonUtils.EMPTY_TRUFFLESTRING_ARRAY, PythonUtils.EMPTY_TRUFFLESTRING_ARRAY, - PythonUtils.EMPTY_OBJECT_ARRAY, 1, 0, 1, 0, -1, -1, -1, 0, 0, List.of(), supplier); + PythonUtils.EMPTY_OBJECT_ARRAY, 1, 0, 1, 0, -1, -1, -1, 0, 0, supplier); PBytecodeDSLRootNode root = code.createRootNode(language, false); // Preserve the empty co_code of these placeholder code objects. root.setCode(PythonUtils.EMPTY_BYTE_ARRAY); @@ -551,12 +548,7 @@ public Object getInstrumentationNodeObject(TagTreeNode node) { if (co == null) { return null; } - if (node.hasTag(ReadVariableTag.class)) { - return VariableAccessDescriptor.create(ReadVariableTag.NAME, false, node.getSourceSections(), co.variableAccesses); - } else if (node.hasTag(WriteVariableTag.class)) { - return VariableAccessDescriptor.create(WriteVariableTag.NAME, true, node.getSourceSections(), co.variableAccesses); - } - return null; + return VariableAccessDescriptor.create(node, co); } public final PythonLanguage getLanguage() { diff --git a/graalpython/com.oracle.graal.python/src/com/oracle/graal/python/nodes/instrumentation/VariableAccessDescriptor.java b/graalpython/com.oracle.graal.python/src/com/oracle/graal/python/nodes/instrumentation/VariableAccessDescriptor.java index 780a4561d0..06d93a9cc2 100644 --- a/graalpython/com.oracle.graal.python/src/com/oracle/graal/python/nodes/instrumentation/VariableAccessDescriptor.java +++ b/graalpython/com.oracle.graal.python/src/com/oracle/graal/python/nodes/instrumentation/VariableAccessDescriptor.java @@ -44,8 +44,26 @@ import java.util.ArrayList; import java.util.List; -import com.oracle.graal.python.nodes.bytecode_dsl.BytecodeDSLCodeUnit.VariableAccess; +import com.oracle.graal.python.nodes.bytecode_dsl.BytecodeDSLCodeUnit; +import com.oracle.graal.python.nodes.bytecode_dsl.PBytecodeDSLRootNode.ForIterate; +import com.oracle.graal.python.nodes.bytecode_dsl.PBytecodeDSLRootNode.LoadCell; +import com.oracle.graal.python.nodes.bytecode_dsl.PBytecodeDSLRootNode.LoadFromDictOrCell; +import com.oracle.graal.python.nodes.bytecode_dsl.PBytecodeDSLRootNode.LoadFromDictOrGlobals; +import com.oracle.graal.python.nodes.bytecode_dsl.PBytecodeDSLRootNode.ReadName; +import com.oracle.graal.python.nodes.bytecode_dsl.PBytecodeDSLRootNode.StoreCell; +import com.oracle.graal.python.nodes.bytecode_dsl.PBytecodeDSLRootNode.UnpackToLocals2; +import com.oracle.graal.python.nodes.bytecode_dsl.PBytecodeDSLRootNode.WriteGlobal; +import com.oracle.graal.python.nodes.bytecode_dsl.PBytecodeDSLRootNode.WriteName; +import com.oracle.graal.python.nodes.frame.ReadGlobalOrBuiltinNode; import com.oracle.truffle.api.CompilerDirectives.TruffleBoundary; +import com.oracle.truffle.api.bytecode.BytecodeNode; +import com.oracle.truffle.api.bytecode.Instruction; +import com.oracle.truffle.api.bytecode.Instruction.Argument; +import com.oracle.truffle.api.bytecode.LocalAccessor; +import com.oracle.truffle.api.bytecode.TagTree; +import com.oracle.truffle.api.bytecode.TagTreeNode; +import com.oracle.truffle.api.instrumentation.StandardTags.ReadVariableTag; +import com.oracle.truffle.api.instrumentation.StandardTags.WriteVariableTag; import com.oracle.truffle.api.interop.InteropLibrary; import com.oracle.truffle.api.interop.InvalidArrayIndexException; import com.oracle.truffle.api.interop.TruffleObject; @@ -54,8 +72,18 @@ import com.oracle.truffle.api.library.ExportLibrary; import com.oracle.truffle.api.library.ExportMessage; import com.oracle.truffle.api.source.SourceSection; +import com.oracle.truffle.api.strings.TruffleString; -/** Immutable compiler metadata; safe to query without a language context or the GIL. */ +/** + * Node object of a {@link ReadVariableTag} or {@link WriteVariableTag} region. Nothing is recorded + * for it at compile time: the names are resolved on demand from the operands of the load or store + * instruction in the tag's bytecode range, against the names the code unit already holds + * ({@code varnames}, {@code cellvars}, {@code freevars} and name constants). The identifier + * locations are the tag's source layers emitted by the compiler: the enclosing statement first, + * then one identifier range per variable operand. + *

+ * Instances are immutable and safe to create and query without a language context or the GIL. + */ @ExportLibrary(InteropLibrary.class) public final class VariableAccessDescriptor implements TruffleObject { private final String property; @@ -67,38 +95,176 @@ private VariableAccessDescriptor(String property, Object names) { } @TruffleBoundary - public static Object create(String property, boolean write, SourceSection[] sections, List accesses) { - SourceSection section = null; - for (SourceSection candidate : sections) { - if (candidate == null || !candidate.isAvailable()) { + public static Object create(TagTreeNode node, BytecodeDSLCodeUnit code) { + boolean write; + if (node.hasTag(WriteVariableTag.class)) { + write = true; + } else if (node.hasTag(ReadVariableTag.class)) { + write = false; + } else { + return null; + } + BytecodeNode bytecode = node.getBytecodeNode(); + if (bytecode == null) { + return null; + } + List nested = new ArrayList<>(); + collectNestedVariableTags(node, nested); + + // Operands are emitted before the operation that consumes them, so the access is the last + // load or store of this region outside nested variable tags: a custom name operation, or + // a built-in instruction with a local argument (load.local/store.local). Instructions are + // identified by operation class and arguments, never by their debug names. Cell stores + // receive the cell from the region's first built-in local load. + Instruction access = null; + Instruction firstLocalAccess = null; + int returnIndex = node.getReturnBytecodeIndex(); + for (int bci = node.getEnterBytecodeIndex(); bci <= returnIndex;) { + TagTree skip = enclosing(nested, bci); + if (skip != null) { + bci = bytecode.getInstruction(skip.getReturnBytecodeIndex()).getNextBytecodeIndex(); continue; } - for (VariableAccess access : accesses) { - if (access.write() == write && access.isWithin(candidate) && (section == null || isWithin(candidate, section))) { - section = candidate; - break; + Instruction instruction = bytecode.getInstruction(bci); + Class operation = instruction.getDescriptor().getOperationClass(); + if (operation == null) { + if (localSlot(instruction) >= 0) { + if (firstLocalAccess == null) { + firstLocalAccess = instruction; + } + access = instruction; } + } else if (isAccessOperation(operation, write)) { + access = instruction; } + bci = instruction.getNextBytecodeIndex(); } - if (section == null) { + if (access == null) { return null; } - List names = new ArrayList<>(); - for (VariableAccess access : accesses) { - if (access.write() == write && access.isWithin(section)) { - names.add(new VariableName(access.name(), access.getSourceSection(section.getSource()))); + + String[] accessNames = resolveNames(code, bytecode, access, firstLocalAccess); + SourceSection[] sections = node.getSourceSections(); + List result = new ArrayList<>(accessNames.length); + for (int i = 0; i < accessNames.length; i++) { + if (accessNames[i] != null) { + // Layer 0 is the statement; identifier layers follow in operand order. + SourceSection section = sections != null && i + 1 < sections.length ? sections[i + 1] : null; + result.add(new VariableName(accessNames[i], section)); } } - if (names.isEmpty()) { + if (result.isEmpty()) { return null; } - Object value = names.size() == 1 ? names.get(0) : new NameArray(names.toArray()); - return new VariableAccessDescriptor(property, value); + Object value = result.size() == 1 ? result.get(0) : new NameArray(result.toArray()); + return new VariableAccessDescriptor(write ? WriteVariableTag.NAME : ReadVariableTag.NAME, value); } - private static boolean isWithin(SourceSection candidate, SourceSection outer) { - return (candidate.getStartLine() > outer.getStartLine() || candidate.getStartLine() == outer.getStartLine() && candidate.getStartColumn() >= outer.getStartColumn()) && - (candidate.getEndLine() < outer.getEndLine() || candidate.getEndLine() == outer.getEndLine() && candidate.getEndColumn() <= outer.getEndColumn()); + private static void collectNestedVariableTags(TagTree tree, List result) { + for (TagTree child : tree.getTreeChildren()) { + if (child.hasTag(ReadVariableTag.class) || child.hasTag(WriteVariableTag.class)) { + result.add(child); + } else { + collectNestedVariableTags(child, result); + } + } + } + + private static TagTree enclosing(List trees, int bci) { + for (TagTree tree : trees) { + if (bci >= tree.getEnterBytecodeIndex() && bci <= tree.getReturnBytecodeIndex()) { + return tree; + } + } + return null; + } + + private static boolean isAccessOperation(Class operation, boolean write) { + if (write) { + return operation == StoreCell.class || operation == WriteGlobal.class || operation == WriteName.class || operation == ForIterate.class || + operation == UnpackToLocals2.class; + } else { + return operation == LoadCell.class || operation == LoadFromDictOrCell.class || operation == ReadGlobalOrBuiltinNode.class || + operation == LoadFromDictOrGlobals.class || operation == ReadName.class; + } + } + + /** The frame slot of a built-in local load or store, or -1. */ + private static int localSlot(Instruction instruction) { + for (Argument argument : instruction.getArguments()) { + // Python disables block scoping, so local offsets and local indices coincide. + switch (argument.getKind()) { + case LOCAL_OFFSET: + return argument.asLocalOffset(); + case LOCAL_INDEX: + return argument.asLocalIndex(); + default: + break; + } + } + return -1; + } + + /** One entry per variable operand of the access; {@code null} for compiler temporaries. */ + private static String[] resolveNames(BytecodeDSLCodeUnit code, BytecodeNode bytecode, Instruction access, Instruction firstLocalAccess) { + Class operation = access.getDescriptor().getOperationClass(); + if (operation == null) { + return new String[]{slotName(code, localSlot(access))}; + } else if (operation == StoreCell.class) { + return firstLocalAccess == null ? new String[0] : new String[]{slotName(code, localSlot(firstLocalAccess))}; + } else if (operation == LoadCell.class || operation == LoadFromDictOrCell.class) { + for (Argument argument : access.getArguments()) { + Integer index = intArgument(argument); + if (index != null) { + return new String[]{cellOrFreeName(code, index)}; + } + } + return new String[0]; + } + // Global and name operations carry the name; ForIterate and UnpackToLocals2 take local + // accessors, whose info is the varnames index. + List names = new ArrayList<>(2); + for (Argument argument : access.getArguments()) { + if (argument.getKind() != Argument.Kind.CONSTANT) { + continue; + } + Object constant = argument.asConstant(); + if (constant instanceof TruffleString name) { + return new String[]{name.toJavaStringUncached()}; + } else if (constant instanceof LocalAccessor local) { + names.add(local.getLocalInfo(bytecode) instanceof Integer index && index < code.varnames.length ? code.varnames[index].toJavaStringUncached() : null); + } + } + return names.toArray(new String[0]); + } + + private static Integer intArgument(Argument argument) { + return switch (argument.getKind()) { + case INTEGER -> argument.asInteger(); + case CONSTANT -> argument.asConstant() instanceof Integer i ? i : null; + default -> null; + }; + } + + /** Frame slot layout: varnames, then cellvars, then freevars, then compiler temporaries. */ + private static String slotName(BytecodeDSLCodeUnit code, int slot) { + if (slot < 0) { + return null; + } else if (slot < code.varnames.length) { + return code.varnames[slot].toJavaStringUncached(); + } + return cellOrFreeName(code, slot - code.varnames.length); + } + + private static String cellOrFreeName(BytecodeDSLCodeUnit code, int index) { + if (index < 0) { + return null; + } else if (index < code.cellvars.length) { + return code.cellvars[index].toJavaStringUncached(); + } else if (index - code.cellvars.length < code.freevars.length) { + return code.freevars[index - code.cellvars.length].toJavaStringUncached(); + } + return null; } @ExportMessage From 19eda1b2dcbb84c593cc083a4f2fb76f2087e718 Mon Sep 17 00:00:00 2001 From: Sam Gammon Date: Tue, 17 Nov 2026 16:01:27 -0800 Subject: [PATCH 2/2] Pin Graal to elide-25.6.x with the 25.5 forward-port (elide-dev/graal#242) The lazy variable-access resolution needs InstructionDescriptor.getOperationClass() (elide-dev/graal#241), which reaches 25.6 with #242. Co-Authored-By: Claude Opus 5.5 --- mx.graalpython/suite.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mx.graalpython/suite.py b/mx.graalpython/suite.py index fd7f1ac893..719ca98944 100644 --- a/mx.graalpython/suite.py +++ b/mx.graalpython/suite.py @@ -53,7 +53,7 @@ }, { "name": "tools", - "version": "258e9f44a453191483afc6c087679ac738c5e26c", + "version": "21dea4deb60d8aa9b07cbd3cc77b1581b61b6fb2", "subdir": True, "urls": [ {"url": "https://github.com/elide-dev/graal", "kind": "git"}, @@ -61,7 +61,7 @@ }, { "name": "regex", - "version": "258e9f44a453191483afc6c087679ac738c5e26c", + "version": "21dea4deb60d8aa9b07cbd3cc77b1581b61b6fb2", "subdir": True, "urls": [ {"url": "https://github.com/elide-dev/graal", "kind": "git"},