Skip to content

Adopt SDK .23 variable-access instrumentation metadata - #11

Merged
Sam Gammon (sgammon) merged 1 commit into
elide-25.5.xfrom
feat/variable-access-metadata
Nov 3, 2026
Merged

Sam Gammon (sgammon) merged 1 commit into
elide-25.5.xfrom
feat/variable-access-metadata

Conversation

@sgammon

@sgammon Sam Gammon (sgammon) commented Nov 1, 2026 •

Copy link
Copy Markdown
Member

Parsing inner = outer now exposes a write of inner and a read of outer, including the exact identifier locations, through Truffle's standard variable-tag node objects before the function executes. This adopts the Bytecode DSL metadata hook shipped in SDK 25.5.5-elide.23 (elide-dev/graal#177).

Implementation

  • Pin Graal imports to the fork's immutable .23 revision fe7f970d99b666853ebe7abd14f14328aec29118.
  • Emit variable tags for name expressions, assignment and augmented-assignment targets, optimized two-local unpacking, loops, and comprehensions. Compiler records preserve normalized and private-mangled Python names and UTF-16 source ranges.
  • Supply immutable, read-only interop name/location descriptors, including arrays for writes to multiple variables. Metadata queries work without an entered Python context and support concurrent readers.
  • Keep tag program locations at the enclosing statement/expression root so debugger caller locations remain stable; the name descriptor supplies the exact identifier location.
  • Serialize the access metadata with code units and bump the bytecode cache version from 35 to 36, invalidating older caches.
  • Update the generator debugger test for the SDK's intentional once-per-statement breakpoint behavior (Graal PR Expose the Python truffle interpreter in a Maven package oracle/graalpython#96): resuming a yield no longer hits its BEFORE breakpoint twice.

Coverage is documented in docs/user/Tooling.md. This is compiler-emitted access metadata; parameter declarations, import/definition bindings, deletion, optimized-away expressions and names without valid source ranges are outside the initial coverage. Locations describe access sites, not resolved declarations or types.

Validation

On macOS/AArch64, using --java-home=lookup:default --suitemodel nested:

  • mx python-jvm: passed against the .23 SDK checkout.
  • mx unittest VariableAccessTests LexicalScopeTests PythonDebugTest: 20 passed.
  • mx graalpytest test_marshal: 18 passed.
  • New tests cover all four instrumentation-option combinations; poison imports/decorators/defaults/annotations without execution; same-URI edits; closures and globals/nonlocals; normalized/private names; non-BMP identifiers and CRLF; optimized unpacking and loop execution; marshal round trips; concurrent queries without an entered context; and read-only interop contracts.
  • Eclipse formatting, changed-file Checkstyle, copyright and whitespace checks passed.
  • Pylint 3.3.9 passed for the changed suite.py. The repository-wide run stops on existing W1514/R1732 warnings in unchanged mx.graalpython/live_heap_tracker.py.

Native Image metadata behavior, downstream Elide integration, and performance/retained-metadata costs remain separate qualifications. This patch is newer than the Python revision published in .23; consuming it requires local overrides or a subsequent release pin.

Patch tracking: #10; related runtime stack: elide-dev/graal#3.

GitHub CI: the style job failed before checking out or testing the code, while downloading Oracle JDK 21 (wget exit 8). This is the same CI bootstrap issue previously recorded for #9.

This was referenced Nov 1, 2026
@sgammon
Sam Gammon (sgammon) merged commit 57c2c23 into elide-25.5.x Nov 3, 2026
11 of 16 checks passed
Sam Gammon (sgammon) added a commit that referenced this pull request Nov 16, 2026
#11 made the enclosing statement the innermost source section of every
tagged variable access. Source sections exist whether or not tags are
instrumented, so exception locations, traceback carets and line tables
moved to the statement. This broke the tagged CPython tests
test_traceback (test_caret_in_type_annotation,
test_byte_offset_with_emojis_term_highlight) and
test_compile.test_lineno_procedure_call.

Open the statement and identifier layers outside the tag, and re-open the
previously innermost range inside it. The tag's own location stays at
the statement, which keeps debugger caller locations stable
(PythonDebugTest), and the tagged instructions keep their previous
location. The augmented-assignment read tag now uses the same layout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sam Gammon (sgammon) added a commit that referenced this pull request Nov 17, 2026
#11 moved the Graal imports to a revision that expects labsjdk with
JVMCI 25.5 (and mx 7.88.0), but ci/graal/common.json still pinned JVMCI
25.4. On that JDK Truffle falls back to the non-optimizing runtime
("Graal expected a JVMCI release version of 25.5, but got 25.4"), so
every junit test fails on the -Dpolyglot.engine.CompilationFailureAction
option the gate sets. This went unnoticed while the JDK 21 bootstrap
download failed before any test ran.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sam Gammon (sgammon) added a commit that referenced this pull request Nov 17, 2026
* Resolve variable-access names lazily from tagged bytecode

Drop the per-code-unit variable-access side table added in #11. It
duplicated names the code unit already holds and locations the Bytecode
DSL already records: every name occurrence was copied into the code unit,
marshaled (UTF-8 name plus five ints), re-interned on load, and kept on
the heap whether or not a tool was attached.

The ReadVariableTag/WriteVariableTag node object now resolves names on
demand from the tagged region's bytecode (TagTreeNode#getBytecodeNode and
the enter/return bytecode indices). The access is the last matching load
or store outside nested variable tags. Its operands identify the name:
frame slots map to varnames, cellvars and freevars; LocalAccessor operands
map through the locals table info; cell instructions carry an index; and
global/name instructions carry the name constant. Identifier locations
come from the tag's own source layers. Fused two-target unpacking now gets
one identifier layer per operand, so per-name ranges are preserved.

Variable tags no longer overwrite the compiler's currentLocation, which
is used for syntax error positions and loop-header trace lines.

Bump the bytecode cache version to 37 for the marshal format change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Fix JDK 21 download URLs for github jobs

(cherry picked from commit 3fa9e56)

* Identify variable accesses by operation class, not instruction names

Instruction names are debugging aids and can change at any time. Use
InstructionDescriptor#getOperationClass() (GR-81626, backported in
elide-dev/graal#241) to recognize the custom name operations, and
recognize built-in local loads/stores by their local argument. The
access is still the last match outside nested variable tags: store values
can contain untagged local loads, such as the self/__class__ loads that
super() generates.

Pin the Graal imports to the backport commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Pin Graal imports to the merged getOperationClass() backport (elide-dev/graal#241)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep instruction locations unchanged inside variable tags

#11 made the enclosing statement the innermost source section of every
tagged variable access. Source sections exist whether or not tags are
instrumented, so exception locations, traceback carets and line tables
moved to the statement. This broke the tagged CPython tests
test_traceback (test_caret_in_type_annotation,
test_byte_offset_with_emojis_term_highlight) and
test_compile.test_lineno_procedure_call.

Open the statement and identifier layers outside the tag, and re-open the
previously innermost range inside it. The tag's own location stays at
the statement, which keeps debugger caller locations stable
(PythonDebugTest), and the tagged instructions keep their previous
location. The augmented-assignment read tag now uses the same layout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: sync ci/graal/common.json with the pinned Graal revision

#11 moved the Graal imports to a revision that expects labsjdk with
JVMCI 25.5 (and mx 7.88.0), but ci/graal/common.json still pinned JVMCI
25.4. On that JDK Truffle falls back to the non-optimizing runtime
("Graal expected a JVMCI release version of 25.5, but got 25.4"), so
every junit test fails on the -Dpolyglot.engine.CompilationFailureAction
option the gate sets. This went unnoticed while the JDK 21 bootstrap
download failed before any test ran.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Apply mx 7.88 Eclipse formatting to TpSlotDescrGet

The style gate's CodeFormatCheck with mx 7.88.0 indents the sealed
class's permits continuation line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Michael Simacek <michael.simacek@oracle.com>
Sam Gammon (sgammon) added a commit that referenced this pull request Nov 18, 2026
…5.6) (#14)

* Resolve variable-access names lazily from tagged bytecode (#12)

* Resolve variable-access names lazily from tagged bytecode

Drop the per-code-unit variable-access side table added in #11. It
duplicated names the code unit already holds and locations the Bytecode
DSL already records: every name occurrence was copied into the code unit,
marshaled (UTF-8 name plus five ints), re-interned on load, and kept on
the heap whether or not a tool was attached.

The ReadVariableTag/WriteVariableTag node object now resolves names on
demand from the tagged region's bytecode (TagTreeNode#getBytecodeNode and
the enter/return bytecode indices). The access is the last matching load
or store outside nested variable tags. Its operands identify the name:
frame slots map to varnames, cellvars and freevars; LocalAccessor operands
map through the locals table info; cell instructions carry an index; and
global/name instructions carry the name constant. Identifier locations
come from the tag's own source layers. Fused two-target unpacking now gets
one identifier layer per operand, so per-name ranges are preserved.

Variable tags no longer overwrite the compiler's currentLocation, which
is used for syntax error positions and loop-header trace lines.

Bump the bytecode cache version to 39 for the marshal format change (37 on elide-25.5.x; 25.6 was already at 38).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Fix JDK 21 download URLs for github jobs

Ported from elide-25.5.x; the 25.6-only placeholder code unit in PBytecodeDSLRootNode drops the argument too.

(cherry picked from commit 3fa9e56)

* Identify variable accesses by operation class, not instruction names

Instruction names are debugging aids and can change at any time. Use
InstructionDescriptor#getOperationClass() (GR-81626, backported in
elide-dev/graal#241) to recognize the custom name operations, and
recognize built-in local loads/stores by their local argument. The
access is still the last match outside nested variable tags: store values
can contain untagged local loads, such as the self/__class__ loads that
super() generates.

Pin the Graal imports to the backport commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Pin Graal imports to the merged getOperationClass() backport (elide-dev/graal#241)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep instruction locations unchanged inside variable tags

tagged variable access. Source sections exist whether or not tags are
instrumented, so exception locations, traceback carets and line tables
moved to the statement. This broke the tagged CPython tests
test_traceback (test_caret_in_type_annotation,
test_byte_offset_with_emojis_term_highlight) and
test_compile.test_lineno_procedure_call.

Open the statement and identifier layers outside the tag, and re-open the
previously innermost range inside it. The tag's own location stays at
the statement, which keeps debugger caller locations stable
(PythonDebugTest), and the tagged instructions keep their previous
location. The augmented-assignment read tag now uses the same layout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: sync ci/graal/common.json with the pinned Graal revision

JVMCI 25.5 (and mx 7.88.0), but ci/graal/common.json still pinned JVMCI
25.4. On that JDK Truffle falls back to the non-optimizing runtime
("Graal expected a JVMCI release version of 25.5, but got 25.4"), so
every junit test fails on the -Dpolyglot.engine.CompilationFailureAction
option the gate sets. This went unnoticed while the JDK 21 bootstrap
download failed before any test ran.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Apply mx 7.88 Eclipse formatting to TpSlotDescrGet

The style gate's CodeFormatCheck with mx 7.88.0 indents the sealed
class's permits continuation line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Michael Simacek <michael.simacek@oracle.com>
Ported from elide-25.5.x; the 25.6-only placeholder code unit in PBytecodeDSLRootNode drops the argument too.

(cherry picked from commit 09e64c7)

* Pin Graal to elide-25.6.x with the 25.5 forward-port (elide-dev/graal#242)

The lazy variable-access resolution needs InstructionDescriptor.getOperationClass()
(elide-dev/graal#241), which reaches 25.6 with oracle#242.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant