Repository navigation
Resolve variable-access names lazily from tagged bytecode - #12
Conversation
Drop the per-code-unit variable-access side table added in #11. It duplicated names the code unit already holds and locations the Bytecode DSL already records: every name occurrence was copied into the code unit, marshaled (UTF-8 name plus five ints), re-interned on load, and kept on the heap whether or not a tool was attached. The ReadVariableTag/WriteVariableTag node object now resolves names on demand from the tagged region's bytecode (TagTreeNode#getBytecodeNode and the enter/return bytecode indices). The access is the last matching load or store outside nested variable tags. Its operands identify the name: frame slots map to varnames, cellvars and freevars; LocalAccessor operands map through the locals table info; cell instructions carry an index; and global/name instructions carry the name constant. Identifier locations come from the tag's own source layers. Fused two-target unpacking now gets one identifier layer per operand, so per-name ranges are preserved. Variable tags no longer overwrite the compiler's currentLocation, which is used for syntax error positions and loop-header trace lines. Bump the bytecode cache version to 37 for the marshal format change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 3fa9e56)
| } | ||
| Instruction instruction = bytecode.getInstruction(bci); | ||
| String name = baseName(instruction); | ||
| if (firstLocalLoad == null && name.equals("load.local")) { |
There was a problem hiding this comment.
I would not use an instruction name anywhere. these are purely for debugging and can change at any point.
I think the best way is to walk instructions and their arguments and stop at the first that has a Instruction.Argument.Kind#LOCAL_INDEX
There was a problem hiding this comment.
agreed, dropping the names. one wrinkle with "first LOCAL_INDEX": we run with block scoping off so load/store.local only carry local_offset, and store values can contain untagged local loads (e.g. self/class for super()), so i'll take the last local-carrying instruction outside nested variable tags
| case "c.StoreCell": | ||
| return firstLocalLoad == null ? new String[0] : resolveNames(code, bytecode, firstLocalLoad, "load.local", null); | ||
| case "c.LoadCell": | ||
| case "c.LoadFromDictOrCell": |
There was a problem hiding this comment.
Instead of resolving names from custom instructions I am about to add:
oracle/graal#15030
should clean this up.
Names will change in the future.
There was a problem hiding this comment.
nice, that's the piece i was missing. i'll backport 15030 to our fork and switch the custom ops over to getOperationClass()
Instruction names are debugging aids and can change at any time. Use InstructionDescriptor#getOperationClass() (GR-81626, backported in elide-dev/graal#241) to recognize the custom name operations, and recognize built-in local loads/stores by their local argument. The access is still the last match outside nested variable tags: store values can contain untagged local loads, such as the self/__class__ loads that super() generates. Pin the Graal imports to the backport commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ev/graal#241) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#11 made the enclosing statement the innermost source section of every tagged variable access. Source sections exist whether or not tags are instrumented, so exception locations, traceback carets and line tables moved to the statement. This broke the tagged CPython tests test_traceback (test_caret_in_type_annotation, test_byte_offset_with_emojis_term_highlight) and test_compile.test_lineno_procedure_call. Open the statement and identifier layers outside the tag, and re-open the previously innermost range inside it. The tag's own location stays at the statement, which keeps debugger caller locations stable (PythonDebugTest), and the tagged instructions keep their previous location. The augmented-assignment read tag now uses the same layout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#11 moved the Graal imports to a revision that expects labsjdk with JVMCI 25.5 (and mx 7.88.0), but ci/graal/common.json still pinned JVMCI 25.4. On that JDK Truffle falls back to the non-optimizing runtime ("Graal expected a JVMCI release version of 25.5, but got 25.4"), so every junit test fails on the -Dpolyglot.engine.CompilationFailureAction option the gate sets. This went unnoticed while the JDK 21 bootstrap download failed before any test ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The style gate's CodeFormatCheck with mx 7.88.0 indents the sealed class's permits continuation line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…5.6) (#14) * Resolve variable-access names lazily from tagged bytecode (#12) * Resolve variable-access names lazily from tagged bytecode Drop the per-code-unit variable-access side table added in #11. It duplicated names the code unit already holds and locations the Bytecode DSL already records: every name occurrence was copied into the code unit, marshaled (UTF-8 name plus five ints), re-interned on load, and kept on the heap whether or not a tool was attached. The ReadVariableTag/WriteVariableTag node object now resolves names on demand from the tagged region's bytecode (TagTreeNode#getBytecodeNode and the enter/return bytecode indices). The access is the last matching load or store outside nested variable tags. Its operands identify the name: frame slots map to varnames, cellvars and freevars; LocalAccessor operands map through the locals table info; cell instructions carry an index; and global/name instructions carry the name constant. Identifier locations come from the tag's own source layers. Fused two-target unpacking now gets one identifier layer per operand, so per-name ranges are preserved. Variable tags no longer overwrite the compiler's currentLocation, which is used for syntax error positions and loop-header trace lines. Bump the bytecode cache version to 39 for the marshal format change (37 on elide-25.5.x; 25.6 was already at 38). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Fix JDK 21 download URLs for github jobs Ported from elide-25.5.x; the 25.6-only placeholder code unit in PBytecodeDSLRootNode drops the argument too. (cherry picked from commit 3fa9e56) * Identify variable accesses by operation class, not instruction names Instruction names are debugging aids and can change at any time. Use InstructionDescriptor#getOperationClass() (GR-81626, backported in elide-dev/graal#241) to recognize the custom name operations, and recognize built-in local loads/stores by their local argument. The access is still the last match outside nested variable tags: store values can contain untagged local loads, such as the self/__class__ loads that super() generates. Pin the Graal imports to the backport commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Pin Graal imports to the merged getOperationClass() backport (elide-dev/graal#241) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Keep instruction locations unchanged inside variable tags tagged variable access. Source sections exist whether or not tags are instrumented, so exception locations, traceback carets and line tables moved to the statement. This broke the tagged CPython tests test_traceback (test_caret_in_type_annotation, test_byte_offset_with_emojis_term_highlight) and test_compile.test_lineno_procedure_call. Open the statement and identifier layers outside the tag, and re-open the previously innermost range inside it. The tag's own location stays at the statement, which keeps debugger caller locations stable (PythonDebugTest), and the tagged instructions keep their previous location. The augmented-assignment read tag now uses the same layout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: sync ci/graal/common.json with the pinned Graal revision JVMCI 25.5 (and mx 7.88.0), but ci/graal/common.json still pinned JVMCI 25.4. On that JDK Truffle falls back to the non-optimizing runtime ("Graal expected a JVMCI release version of 25.5, but got 25.4"), so every junit test fails on the -Dpolyglot.engine.CompilationFailureAction option the gate sets. This went unnoticed while the JDK 21 bootstrap download failed before any test ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Apply mx 7.88 Eclipse formatting to TpSlotDescrGet The style gate's CodeFormatCheck with mx 7.88.0 indents the sealed class's permits continuation line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Michael Simacek <michael.simacek@oracle.com> Ported from elide-25.5.x; the 25.6-only placeholder code unit in PBytecodeDSLRootNode drops the argument too. (cherry picked from commit 09e64c7) * Pin Graal to elide-25.6.x with the 25.5 forward-port (elide-dev/graal#242) The lazy variable-access resolution needs InstructionDescriptor.getOperationClass() (elide-dev/graal#241), which reaches 25.6 with oracle#242. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Build GraalJS and GraalPy from their elide-25.6.x branches (GraalPy with elide-dev/graalpython#12 ported), the Shenandoah GC libraries from labs-openjdk elide-25.6.x (jvmci-25.6-b27) with the elide.28 safepoint fix, and the Linux legs and sanitizer add-ons with toolchain 2026.11.7. Ship CE only for now: Oracle's only 25.6 EA, jdk-25i6-25.0.4.1.1-ea.02, is built on jvmci-25.5-b24 while the 25.6 line is on jvmci-25.6-b27, and build-ee.py refuses a JVMCI mismatch. BUILD_EE turns the EE steps back on once a 25.6 EA on the fork's JVMCI exists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Follow-up to #11, based on review feedback: the per-code-unit variable-access side table was wasteful. It duplicated names the code unit already holds and locations the Bytecode DSL already records.
What it cost
co_varnames/co_names. A CPython-AST estimate over 155 stdlib modules puts this at about 3.5 MB, roughly 64% of their CPython marshal size.List<VariableAccess>was kept on every code unit, whether or not a tool was attached. After unmarshaling, each occurrence also held its ownStringcopy.SourceSection, and aLinkedHashSetinsert. Each import from cache re-interned every name.Change
BytecodeDSLCodeUnit.variableAccessesand its marshal fields, and bumpBYTECODE_VERSIONto 37.VariableAccessDescriptorresolves names on demand from the tagged region:TagTreeNode#getBytecodeNode()plusgetEnterBytecodeIndex()/getReturnBytecodeIndex(). The access is the last matching load or store outside nested variable tags. Its operands identify the name:load.local/store.locallocal offsets map to frame slots (varnames, then cellvars, then freevars), likeBytecodeDSLFrameInfo.getVariableName.ForIterate/UnpackToLocals2LocalAccessoroperands map through the locals table info (the varnames index).LoadCell/LoadFromDictOrCellcarry the cell/free index.StoreCelluses the cell local it loads.ReadGlobal/WriteGlobal/ReadName/WriteName/LoadFromDictOrGlobalscarry the name constant.a, b = …unpacking now gets one identifier layer per operand, so per-name ranges are kept without a side table.currentLocation, which is used for syntax error positions and loop-header trace lines. Adopt SDK .23 variable-access instrumentation metadata #11 had changed that by accident.The SDK hook is unchanged. Upstream merged it as oracle/graal#15029 (GR-81544,
@since 25.6).Validation
Linux/amd64, labsjdk-ce 25.0.4.1 (jvmci-25.5-b24), against graal
fe7f970d99b(the.23pin):mx python-jvm: passed.mx unittest VariableAccessTests LexicalScopeTests PythonDebugTest: 21 passed. NewscopesAndNestedAccessescovers module and class name stores, explicit globals with augmented assignment, free-variable reads, cell stores and nonlocal augmented assignment, and a walrus store nested inside another store. The serialization test now checks that original and unmarshaled code report the same names and positions, instead of comparing the removed list.mx graalpytest test_marshal: 18 passed.mx checkstyle --primary: passed. I couldn't run the Eclipse formatter because no Eclipse install is available on this host..pyc, gives the same result.Update after review: instructions are no longer identified by name, since names are only for debugging. Custom operations are matched by
InstructionDescriptor#getOperationClass()(oracle/graal#15030 / GR-81626, backported in elide-dev/graal#241). Built-in local loads and stores are matched by their local argument. The access is still the last match outside nested variable tags, because store values can contain untagged local loads (for example, theself/__class__loadssuper()generates). A new regression test covers that case.Graal imports are pinned to
67e12a237ab, the merge commit of elide-dev/graal#241. Its tree is identical to the backport head these results were validated on. Re-validated against that commit: unit tests 21/21,test_marshal18/18, checkstyle clean.🤖 Generated with Claude Code