Skip to content

feat(mcp): group sessions by ChatGPT, Codex and VS Code conversation ids - #1193

Open
lucasheriques wants to merge 11 commits into
mainfrom
lucas/mcp-client-meta-session
Open

lucasheriques wants to merge 11 commits into
mainfrom
lucas/mcp-client-meta-session

Conversation

@lucasheriques

@lucasheriques lucasheriques commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Servers that need tool schemas and results left unchanged turn conversation IDs off. On a stateless server that builds a fresh instance per request, every tool call then gets its own $session_id (PostHog/posthog#119784).

Some clients already name the conversation in each tool call's _meta. sdk-specs main defines which keys the SDKs read, from which clients, and in what order: "Session identity", from PostHog/sdk-specs#175 and #176. This PR implements those rules exactly. It is the twin of PostHog/posthog-js#5577.

Changes

  • _client_conversation.CLIENT_CONVERSATION_ROWS mirrors the spec table:

    • ChatGPT openai/session: 1 to 128 printable ASCII characters. Handle-free.
    • Codex sessionId: a UUID. Keeps the handle as a fallback.
    • VS Code vscode.conversationId: a UUID. Keeps the handle as a fallback.

    UUIDs are lowercased and the nil UUID is rejected.

  • Recognition. A key is read only from a request whose User-Agent or client name matches its row. The client name comes from the request's _meta clientInfo first, then the name the adapter resolved.

  • The client's id is session source 1 on tool calls, ahead of the handle and the transport. tools/list, resource requests, and initialize no longer read it. A handle echoed alongside it is stripped but ignored.

  • Handle-free (ChatGPT). Its client-facing tools/list has no conversation_id or _mcp_instructions, and it is minted no handle. The SDK's internal tool-cache pass still injects, so validation stays the same for every client.

  • Connection activity. On a one-client connection (stdio), a client-grouped call counts as activity, so the session doesn't rotate early.

  • $mcp_session_source on every instrument() event. The spec addition for it is docs(mcp-analytics): record which source grouped each session as $mcp_session_source sdk-specs#185.

  • PostHogMCP. prepare_tool_list and prepare_tool_call take headers, and prepare_tool_call also takes client_info. capture_tool_call takes request_meta. A ChatGPT prepare_tool_list still records conversation_id ownership.

  • On mcp 2.x low-level servers, the SDK's internal tools/list dispatch (Mcp-Param header validation) emits no event. That part is unchanged from the earlier version of this PR.

Not verified against real clients

  • Does openai/session stay the same for a whole ChatGPT conversation? Not verified. I have no ChatGPT connector session to log from. One way to confirm is to log _meta across a multi-turn chat.
  • Do the Codex versions in use today send sessionId? Partly verified, from source only. Codex adds it to tool-call _meta starting at rust-v0.156.0 (2026-09-22). Older installs fall back to the handle.

How I tested

  • pytest posthog/test/mcp: 745 passed on mcp 1.30.0 and 603 passed on mcp 2.3.0.
    • test_client_conversation_session.py covers every spec scenario and vector, and recognition examples.
    • It covers the manual API and connection rotation.
    • It covers real HTTP fleets over the official, FastMCP, and low-level server kinds: listings, and ChatGPT calls across pods.
  • ruff, mypy (CI mode with baseline), and the public API snapshot pass.
  • Pre-PR gate: two adversarial reviewers (Opus and Sonnet), on both mcp majors.
    • Fixed: counting client-grouped calls as activity also ran on shared stateless HTTP pods. There it kept the instance session alive and merged two unrelated clients. It now applies only to a one-client connection.
    • Fixed: a ChatGPT prepare_tool_list skipped recording ownership, so prepare_tool_call passed an echoed handle to the handler.
    • Both fixes have tests that fail on the previous commit.
    • Dismissed: the inactivity clock is reset by any request, not only tool calls, and is measured from arrival with >. It predates this PR and matches posthog-js, so it's a follow-up.
    • Dismissed: on mcp 1.x, a ChatGPT listing populates the tool cache without the analytics fields. A later client on that instance then gets the handle only as a text block, not mirrored into structuredContent. Mirroring is a MAY and the handle is still delivered.
    • Dismissed: the manual capture methods set $mcp_session_source only for a recognized client id (spec MAY, follow-up).

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Built with Claude Code (pstack poteto-mode) at Lucas's direction. An earlier version ranked openai/session below the transport session, with no recognition rules. Lucas decided the merged sdk-specs rules win, so this version implements them as written.

Read ChatGPT's `_meta["openai/session"]` conversation id as a session source
for calls with no conversation handle and no transport session, and stamp
`$mcp_session_source` (conversation, transport, client_metadata, generated)
on every event whose session the SDK resolves.

Why: on per-request stateless servers with `enable_conversation_id=False`
(common for ChatGPT/Claude directory submissions) every call got its own
`$session_id` (PostHog/posthog#119784). ChatGPT already sends a
per-conversation id, so grouping needs no schema or result change. Order is
handle, transport (token or mcp id), client id, generated, so it only groups
calls that fragment today. A call carrying it mints no handle and gets no
prompt-back. Same rule in `PostHogMCP.prepare_tool_call` via `request_meta`,
after an explicit `session_id`. Twin of posthog-js 3eb19d9 and ADR-0014; the
derivation `ses_35e4cc45ea3e5e1282c3f091de632132` is a cross-SDK contract.

Internals: `resolve_session_id_with_source` / `prepare_request` return a
`ResolvedSession(session_id, source)`, which the record_* functions take, so
the source reaches every event. `token`/`mcp` map to `transport` at the
property boundary. `handshake_session_id` becomes `handshake_session`. A
client-conversation session never stands in for an unobserved `initialize`,
like a conversation one.

Tested: `pytest posthog/test/mcp` -> 699 passed, 10 skipped on mcp 1.30.0 and
557 passed, 42 skipped on mcp 2.3.0 (incl. new parameterized
test_client_conversation_session.py plus a v1 FastMCP and a v2 wire-level
fresh-instance test); `ruff format --check`, `ruff check`, `mypy`,
`make public_api_check` clean.
Pre-PR gate findings on the TypeScript twin (posthog-js 26d720abd), applied
here:
- tools/list, resources/list|read and initialize resolved sessions without
  the client conversation id, which the spec's session order covers for every
  event. Each call site now passes it (`params.meta` on request handlers,
  `_meta` of the initialize params through `handshake_session`). Still
  per-request, never written to shared state, and it still does not stand in
  for an unobserved initialize.
- Custom events from `McpAnalytics.capture()` now carry `$mcp_session_source`,
  the instance's own source (`generated`, or `transport` for a token or mcp
  session id).
- `prepare_tool_call(session_id="")` already treated an empty id as absent;
  a regression case now pins it.

Tested: `pytest posthog/test/mcp` -> 714 passed, 11 skipped on mcp 1.30.0 and
570 passed, 45 skipped on mcp 2.3.0 (fastmcp uninstalled); new parameterized
cases failed before the fix (12 on mcp 2.3.0); `ruff format --check`,
`ruff check`, `mypy`, `make public_api_check` clean.
…ernal tools/list

What
- ToolCallLifecycle.prime_session now passes the request's client conversation to
  resolve_session_id, so a call carrying only openai/session returns at the
  client_metadata step before the memory branch. It no longer refreshes
  data.last_activity or rotates data.session_id, so ChatGPT calls cannot keep the
  instance's generated session alive past its inactivity window.
- The v2 tools/list wrapper skips analytics when the SDK calls the handler itself.
  On the 2026-07-28 streamable-HTTP path, mcp 2.3.0 runs tools/list during every
  tools/call (Mcp-Param-* validation) on a server without get_tool_input_schema
  (the low-level Server; MCPServer provides it). That call has no _meta, resolved
  to a fresh generated session, and emitted a phantom $mcp_tools_list. It is
  detected by the shared HTTP request's required Mcp-Method header naming a
  different method than the handler's own.

Why
Pre-PR gate findings on the client-metadata session work; the spec limits
events to requests the client sent.

Tested
- pytest posthog/test/mcp: mcp 1.30.0 716 passed, 11 skipped; mcp 2.3.0 576
  passed, 45 skipped (fastmcp uninstalled)
- ruff format --check, ruff check, mypy | mypy-baseline filter, make
  public_api_check: clean
- New tests failed before the fixes (instance state moved; phantom event emitted).
  A real tools/list still emits on both MCPServer and low-level Server.

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed 208707d: 1 Must fix.

See this review and its findings in PostHog Review. Comment @posthog review --deep for a Deep review. A Deep review takes longer and looks deeper than a Standard review.

Comment thread posthog/mcp/_instrumentation.py Resolved
@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

posthog-python Compliance Report

Date: 2026-11-16T22:51:05.440901+00:00
Duration: 253247ms

✅ All Tests Passed!

271/271 tests passed


Capture_V1 Tests

✅ 95/95 tests passed

View Details
Test Status Duration
Endpoint And Method.Targets V1 Endpoint ✅ 516ms
Endpoint And Method.Does Not Use Legacy Endpoints ✅ 508ms
Required Headers.Has Authorization Bearer Header ✅ 509ms
Required Headers.Has Content Type Json ✅ 511ms
Required Headers.Has Posthog Sdk Info Format ✅ 508ms
Required Headers.Has Posthog Attempt Header ✅ 508ms
Required Headers.Has Posthog Request Id ✅ 508ms
Required Headers.Has Posthog Request Timestamp ✅ 508ms
Required Headers.Has User Agent ✅ 508ms
Body Format.Body Has Created At And Batch ✅ 508ms
Body Format.No Api Key In Body ✅ 510ms
Body Format.No Sent At In Body ✅ 510ms
Event Format.Event Has Required Root Fields ✅ 512ms
Event Format.Event Uuid Is Valid ✅ 508ms
Event Format.Event Timestamp Is Rfc3339 ✅ 508ms
Event Format.Non Utc Event Timestamp Is Converted To Utc ✅ 520ms
Event Format.Distinct Id Is String ✅ 508ms
Event Format.Distinct Id At Root Not Properties ✅ 508ms
Event Format.Custom Properties Preserved ✅ 508ms
Event Format.Set Properties Preserved ✅ 508ms
Event Format.Set Once Properties Preserved ✅ 509ms
Event Format.Groups Properties Preserved ✅ 509ms
Event Format.Sdk Generates Uuid If Not Provided ✅ 509ms
Event Format.Event Has Required Root Fields Batch ✅ 511ms
Event Format.Event Uuid Is Valid Batch ✅ 511ms
Event Format.Event Timestamp Is Rfc3339 Batch ✅ 512ms
Event Format.Distinct Id Is String Batch ✅ 511ms
Event Format.Distinct Id At Root Not Properties Batch ✅ 511ms
Event Format.Custom Properties Preserved Batch ✅ 511ms
Event Format.Set Properties Preserved Batch ✅ 512ms
Event Format.Set Once Properties Preserved Batch ✅ 511ms
Event Format.Groups Properties Preserved Batch ✅ 511ms
Event Format.Sdk Generates Uuid If Not Provided Batch ✅ 511ms
Batch Behavior.Multiple Events In Single Batch ✅ 516ms
Batch Behavior.Batch Envelope Smoke ✅ 514ms
Batch Behavior.Flush With No Events Sends Nothing ✅ 505ms
Batch Behavior.Flush At Triggers Batch ✅ 1010ms
Batch Behavior.Created At Reflects Batch Creation Time ✅ 509ms
Deduplication.Generates Unique Uuids ✅ 514ms
Deduplication.Different Events Same Content Different Uuids ✅ 511ms
Deduplication.Preserves Uuid On Retry ✅ 6517ms
Deduplication.Preserves Timestamp On Retry ✅ 6512ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry ✅ 6515ms
Deduplication.No Duplicate Events In Batch ✅ 513ms
Header Behavior On Retry.Attempt Header Starts At One ✅ 508ms
Header Behavior On Retry.Attempt Header Increments On Retry ✅ 13516ms
Header Behavior On Retry.Request Id Preserved On Retry ✅ 6513ms
Header Behavior On Retry.Different Requests Have Different Request Ids ✅ 3016ms
Header Behavior On Retry.Request Timestamp Changes On Retry ✅ 6513ms
Response Format Validation.Success Response Has Uuid Keyed Results ✅ 508ms
Response Format Validation.Success Response Has Ok For Each Event ✅ 511ms
Response Format Validation.Success No Retry After When All Ok ✅ 511ms
Response Format Validation.Success Retry After Present When Retry Events ✅ 1513ms
Response Format Validation.Success No Retry After When Drop Only ✅ 510ms
Response Format Validation.Response Echoes Request Id ✅ 508ms
Retry Behavior.Retries On 408 ✅ 6511ms
Retry Behavior.Retries On 500 ✅ 6514ms
Retry Behavior.Retries On 503 ✅ 8515ms
Retry Behavior.Retries On 504 ✅ 6513ms
Retry Behavior.Retries On 404 ✅ 6513ms
Retry Behavior.Retries On 429 ✅ 6517ms
Retry Behavior.Retries On 502 ✅ 6513ms
Retry Behavior.Retryable Errors Have Retry After ✅ 3515ms
Retry Behavior.Respects Retry After On Retryable Error ✅ 11521ms
Retry Behavior.Does Not Retry On 400 ✅ 2510ms
Retry Behavior.Does Not Retry On 401 ✅ 2513ms
Retry Behavior.Does Not Retry On 402 ✅ 2511ms
Retry Behavior.Does Not Retry On 413 ✅ 2511ms
Retry Behavior.Does Not Retry On 415 ✅ 2512ms
Retry Behavior.Non Retryable Errors Have No Retry After ✅ 2511ms
Retry Behavior.Implements Backoff ✅ 19531ms
Partial Batch Handling.Handles 200 Full Success ✅ 2510ms
Partial Batch Handling.Handles 200 With All Ok ✅ 3514ms
Partial Batch Handling.Does Not Retry Dropped Events ✅ 3511ms
Partial Batch Handling.Does Not Retry Warning Events ✅ 3512ms
Partial Batch Handling.Prunes Ok Events On Partial Retry ✅ 6514ms
Partial Batch Handling.Prunes Dropped Events On Partial Retry ✅ 6515ms
Partial Batch Handling.Retries Only Retry Events From Partial ✅ 6516ms
Partial Batch Handling.Partial Retry Preserves Uuids ✅ 6514ms
Partial Batch Handling.Partial Retry Attempt Header Increments ✅ 6515ms
Partial Batch Handling.Partial Retry Request Id Preserved ✅ 6515ms
Partial Batch Handling.Respects Retry After On Partial ✅ 8519ms
Partial Batch Handling.Unknown Result Treated As Terminal ✅ 3510ms
Partial Batch Handling.Mixed Ok Drop Warning No Retry ✅ 3514ms
Compression.Sends Gzip Content Encoding ✅ 509ms
Compression.No Content Encoding When Disabled ✅ 508ms
Compression.Compressed Body Is Decompressible ✅ 508ms
Error Handling.Does Not Retry On Unknown 4Xx ✅ 2511ms
Event Options.Cookieless Mode Override ✅ 509ms
Event Options.Disable Skew Correction Override ✅ 508ms
Event Options.Process Person Profile Override ✅ 508ms
Event Options.Product Tour Id Override ✅ 508ms
Event Options.Unset Options Omitted ✅ 509ms
Event Options.Options Override In Batch ✅ 510ms
Geoip And Historical Migration.Historical Migration Absent By Default ✅ 509ms

Capture_Ai Tests

✅ 5/5 tests passed

View Details
Test Status Duration
Routing.Capture Ai Posts To Ai Endpoint ✅ 508ms
Routing.Capture Does Not Reroute Ai Named Events ✅ 508ms
Identity.Capture Ai Event Has Uuid ✅ 507ms
Identity.Capture Ai Keeps Supplied Uuid ✅ 508ms
Timestamp Format.Non Utc Event Timestamp Is Converted To Utc ✅ 508ms

Feature_Flags Tests

✅ 17/17 tests passed

View Details
Test Status Duration
Request Payload.Request With Person Properties Device Id ✅ 9ms
Request Payload.Flags Request Uses V4 Query Param ✅ 6ms
Request Payload.Flags Request Hits Flags Path Not Decide ✅ 7ms
Request Payload.Flags Request Omits Authorization Header ✅ 6ms
Request Payload.Token In Flags Body Matches Init ✅ 7ms
Request Payload.Groups Round Trip ✅ 6ms
Request Payload.Groups Default To Empty Object ✅ 6ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False ✅ 7ms
Request Payload.Disable Geoip Omitted Defaults To False ✅ 6ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key ✅ 6ms
Request Lifecycle.No Flags Request On Init Alone ✅ 3ms
Request Lifecycle.No Flags Request On Normal Capture ✅ 506ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests ✅ 11ms
Request Lifecycle.Mock Response Value Is Returned To Caller ✅ 7ms
Retry Behavior.Retries Flags On 502 ✅ 310ms
Retry Behavior.Retries Flags On 504 ✅ 309ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event ✅ 508ms

Feature_Flags_V4 Tests

✅ 150/150 tests passed

View Details
Test Status Duration
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.Complete Split] ✅ 9ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.Partial Split Missing Config Version] ✅ 6ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.Partial Split Missing Rule Type] ✅ 7ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.Partial Split Missing Rule Id] ✅ 6ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.Partial Split Missing Variant Key] ✅ 7ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.Split Wrong Type Config Version] ✅ 6ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.Split Wrong Type Rule Type] ✅ 7ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.Split Wrong Type Rule Id] ✅ 7ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.Split Wrong Type Experiment Id] ✅ 6ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.Split Extra Experiment Id] ✅ 7ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.Split Wrong Type Variant Key] ✅ 6ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.Wrong Rule Type Literal] ✅ 7ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.Unknown Fields] ✅ 7ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.Old Server Variant] ✅ 6ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.Old Server Boolean] ✅ 6ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.Old Server Null Variant] ✅ 7ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.Old Server False] ✅ 7ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.V4 Value Takes Precedence] ✅ 6ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.Missing Flag] ✅ 6ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.Targeting Match Targeted Release] ✅ 7ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.Targeting Match Percentage Rollout] ✅ 6ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.Rollout Miss Percentage Rollout] ✅ 7ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.Rollout Miss Experiment] ✅ 6ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.Experiment Paused] ✅ 7ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.Holdout] ✅ 6ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.Holdout Variant Key] ✅ 7ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.No Rule Match] ✅ 6ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.Missing Group Key] ✅ 6ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.Dependency Error] ✅ 7ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.Error] ✅ 6ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.Error With Value] ✅ 7ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.Malformed Sibling Isolation] ✅ 6ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.Partial Split String Value] ✅ 7ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.V1 At V4] ✅ 6ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.V1 Failed] ✅ 7ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.Integral Json Number Context] ✅ 7ms
Legacy Getters.Legacy Getter Reads Reader Fixture[Readers.Flag Key Seed] ✅ 6ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.String.Target] ✅ 7ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.String.Rollout Hit] ✅ 6ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.String.Miss Return Default] ✅ 7ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.String.Miss Return Default Null] ✅ 6ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.String.Miss Continue] ✅ 6ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.String.No Match Default] ✅ 7ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.String.No Match Null] ✅ 6ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.String.Order First Match] ✅ 7ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.String.Order Targeting Miss] ✅ 6ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.Number.Target] ✅ 7ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.Number.Rollout Hit] ✅ 7ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.Number.Miss Return Default] ✅ 6ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.Number.Miss Return Default Null] ✅ 7ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.Number.Miss Continue] ✅ 6ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.Number.No Match Default] ✅ 6ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.Number.No Match Null] ✅ 19ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.Number.Order First Match] ✅ 6ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.Number.Order Targeting Miss] ✅ 6ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.Object.Target] ✅ 7ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.Object.Rollout Hit] ✅ 6ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.Object.Miss Return Default] ✅ 7ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.Object.Miss Return Default Null] ✅ 6ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.Object.Miss Continue] ✅ 7ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.Object.No Match Default] ✅ 6ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.Object.No Match Null] ✅ 7ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.Object.Order First Match] ✅ 6ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.Object.Order Targeting Miss] ✅ 6ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.String.Unicode] ✅ 7ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.String.Looks Boolean] ✅ 6ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.Number.Zero] ✅ 7ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.Number.Negative Decimal] ✅ 6ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.Number.Short Decimal] ✅ 7ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.Number.Exponent] ✅ 6ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.Number.Max Safe Integer] ✅ 7ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.Number.Min Safe Integer] ✅ 6ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.Object.Empty] ✅ 6ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.Object.Nested] ✅ 6ms
Legacy Getters.Legacy Getter Reads Typed Value[V2 Value.Object.Max Depth] ✅ 7ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.Complete Split] ✅ 5ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.Partial Split Missing Config Version] ✅ 5ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.Partial Split Missing Rule Type] ✅ 5ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.Partial Split Missing Rule Id] ✅ 5ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.Partial Split Missing Variant Key] ✅ 4ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.Split Wrong Type Config Version] ✅ 6ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.Split Wrong Type Rule Type] ✅ 5ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.Split Wrong Type Rule Id] ✅ 5ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.Split Wrong Type Experiment Id] ✅ 4ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.Split Extra Experiment Id] ✅ 5ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.Split Wrong Type Variant Key] ✅ 5ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.Wrong Rule Type Literal] ✅ 5ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.Unknown Fields] ✅ 5ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.Old Server Variant] ✅ 5ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.Old Server Boolean] ✅ 5ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.Old Server Null Variant] ✅ 5ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.Old Server False] ✅ 4ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.V4 Value Takes Precedence] ✅ 5ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.Missing Flag] ✅ 5ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.Targeting Match Targeted Release] ✅ 5ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.Targeting Match Percentage Rollout] ✅ 5ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.Rollout Miss Percentage Rollout] ✅ 5ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.Rollout Miss Experiment] ✅ 5ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.Experiment Paused] ✅ 5ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.Holdout] ✅ 4ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.Holdout Variant Key] ✅ 5ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.No Rule Match] ✅ 5ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.Missing Group Key] ✅ 5ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.Dependency Error] ✅ 5ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.Error] ✅ 5ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.Error With Value] ✅ 5ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.Malformed Sibling Isolation] ✅ 5ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.Partial Split String Value] ✅ 4ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.V1 At V4] ✅ 5ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.V1 Failed] ✅ 5ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.Integral Json Number Context] ✅ 5ms
Legacy Payloads.Legacy Payload Reads Reader Fixture[Readers.Flag Key Seed] ✅ 5ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.String.Target] ✅ 4ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.String.Rollout Hit] ✅ 5ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.String.Miss Return Default] ✅ 5ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.String.Miss Return Default Null] ✅ 4ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.String.Miss Continue] ✅ 5ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.String.No Match Default] ✅ 5ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.String.No Match Null] ✅ 5ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.String.Order First Match] ✅ 5ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.String.Order Targeting Miss] ✅ 4ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.Number.Target] ✅ 5ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.Number.Rollout Hit] ✅ 6ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.Number.Miss Return Default] ✅ 5ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.Number.Miss Return Default Null] ✅ 4ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.Number.Miss Continue] ✅ 5ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.Number.No Match Default] ✅ 5ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.Number.No Match Null] ✅ 5ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.Number.Order First Match] ✅ 4ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.Number.Order Targeting Miss] ✅ 5ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.Object.Target] ✅ 5ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.Object.Rollout Hit] ✅ 4ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.Object.Miss Return Default] ✅ 5ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.Object.Miss Return Default Null] ✅ 5ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.Object.Miss Continue] ✅ 4ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.Object.No Match Default] ✅ 4ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.Object.No Match Null] ✅ 5ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.Object.Order First Match] ✅ 5ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.Object.Order Targeting Miss] ✅ 4ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.String.Unicode] ✅ 5ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.String.Looks Boolean] ✅ 5ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.Number.Zero] ✅ 5ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.Number.Negative Decimal] ✅ 4ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.Number.Short Decimal] ✅ 5ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.Number.Exponent] ✅ 5ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.Number.Max Safe Integer] ✅ 5ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.Number.Min Safe Integer] ✅ 5ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.Object.Empty] ✅ 5ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.Object.Nested] ✅ 4ms
Legacy Payloads.Legacy Payload Is The Typed Value[V2 Value.Object.Max Depth] ✅ 5ms

Feature_Flags_Local_Evaluation Tests

✅ 4/4 tests passed

View Details
Test Status Duration
Versioned Boolean Matching.Matching Version Missing ✅ 46ms
Versioned Boolean Matching.Matching Version 1 ✅ 44ms
Versioned Boolean Matching.Matching Version 2 ✅ 45ms
Versioned Boolean Matching.Version Only Reload 1 2 1 2 Missing ✅ 24ms

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

SDK performance: posthog-python

2 changes worth reviewing: 2 observed regressions.

Flags require at least a 5% change in the median and all six pairs agreeing on direction. This is a noise filter, not proof of a performance change or a CI gate.

Test Measurement Before PR With PR Change
Capture and deliver events (4,096-byte payload, burst) Process CPU time per operation 93.11 µs/op 104.48 µs/op 12.2% more time
Capture and deliver events (4,096-byte payload, burst) Time per operation 100 µs/op 109.77 µs/op 9.8% more time
View all results
Test Measurement Before PR With PR Change Assessment
Capture and deliver events (32-byte payload, burst)
Shard: capture
Managed heap after settling — — — unavailable; base:​ unsupported; head:​ unsupported; base:​ CPython has no uninstrumented total managed-heap byte counter; tracing would distort this RSS pass.​; head:​ CPython has no uninstrumented total managed-heap byte counter; tracing would distort this RSS pass.​
Capture and deliver events (4,096-byte payload, burst)
Shard: capture
Managed heap after settling — — — unavailable; base:​ unsupported; head:​ unsupported; base:​ CPython has no uninstrumented total managed-heap byte counter; tracing would distort this RSS pass.​; head:​ CPython has no uninstrumented total managed-heap byte counter; tracing would distort this RSS pass.​
Capture and deliver events (32-byte payload, steady)
Shard: capture
Managed heap after settling — — — unavailable; base:​ unsupported; head:​ unsupported; base:​ CPython has no uninstrumented total managed-heap byte counter; tracing would distort this RSS pass.​; head:​ CPython has no uninstrumented total managed-heap byte counter; tracing would distort this RSS pass.​
Capture and deliver events (4,096-byte payload, steady)
Shard: capture
Managed heap after settling — — — unavailable; base:​ unsupported; head:​ unsupported; base:​ CPython has no uninstrumented total managed-heap byte counter; tracing would distort this RSS pass.​; head:​ CPython has no uninstrumented total managed-heap byte counter; tracing would distort this RSS pass.​
Queue an event (32-byte payload)
Shard: capture
Allocated memory per operation — — — unavailable; base:​ unsupported; head:​ unsupported; base:​ Background preparation has no public completion boundary before flush; see capture.​drain for full-pipeline allocations.​; head:​ Background preparation has no public completion boundary before flush; see capture.​drain for full-pipeline allocations.​
Queue an event (32-byte payload)
Shard: capture
Allocations per operation — — — unavailable; base:​ unsupported; head:​ unsupported; base:​ Background preparation has no public completion boundary before flush; see capture.​drain for full-pipeline allocations.​; head:​ Background preparation has no public completion boundary before flush; see capture.​drain for full-pipeline allocations.​
Queue an event (4,096-byte payload)
Shard: capture
Allocated memory per operation — — — unavailable; base:​ unsupported; head:​ unsupported; base:​ Background preparation has no public completion boundary before flush; see capture.​drain for full-pipeline allocations.​; head:​ Background preparation has no public completion boundary before flush; see capture.​drain for full-pipeline allocations.​
Queue an event (4,096-byte payload)
Shard: capture
Allocations per operation — — — unavailable; base:​ unsupported; head:​ unsupported; base:​ Background preparation has no public completion boundary before flush; see capture.​drain for full-pipeline allocations.​; head:​ Background preparation has no public completion boundary before flush; see capture.​drain for full-pipeline allocations.​
Evaluate 1,000 flags under concurrent load
Shard: flags-memory
Managed heap after settling — — — unavailable; base:​ unsupported; head:​ unsupported; base:​ CPython has no uninstrumented total managed-heap byte counter; tracing would distort this RSS pass.​; head:​ CPython has no uninstrumented total managed-heap byte counter; tracing would distort this RSS pass.​
Refresh 1,000 flag definitions
Shard: flags-memory
Managed heap after settling — — — unavailable; base:​ unsupported; head:​ unsupported; base:​ CPython has no uninstrumented total managed-heap byte counter; tracing would distort this RSS pass.​; head:​ CPython has no uninstrumented total managed-heap byte counter; tracing would distort this RSS pass.​
Capture and deliver events (32-byte payload, burst)
Shard: capture
Allocated memory per operation 115.72 KiB/op 115.72 KiB/op <0.1% more memory Not flagged
Capture and deliver events (32-byte payload, burst)
Shard: capture
Allocations per operation 1,107.6 count/op 1,107.61 count/op <0.1% more Not flagged
Capture and deliver events (32-byte payload, burst)
Shard: capture
Time per operation 71 µs/op 73.54 µs/op 3.6% more time Not flagged
Capture and deliver events (32-byte payload, burst)
Shard: capture
Idle process memory (RSS) 36.35 MiB 36.36 MiB <0.1% more memory Not flagged
Capture and deliver events (32-byte payload, burst)
Shard: capture
Observed peak process memory (RSS) 38.08 MiB 38.09 MiB <0.1% more memory Not flagged
Capture and deliver events (32-byte payload, burst)
Shard: capture
Process memory after settling (RSS) 38.08 MiB 38.09 MiB <0.1% more memory Not flagged
Capture and deliver events (32-byte payload, burst)
Shard: capture
Process CPU time per operation 71.56 µs/op 74.63 µs/op 4.3% more time Not flagged
Capture and deliver events (32-byte payload, burst)
Shard: capture
Lifetime peak process memory (RSS) 40.36 MiB 40.36 MiB No observed change Not flagged
Capture and deliver events (4,096-byte payload, burst)
Shard: capture
Allocated memory per operation 168.28 KiB/op 168.28 KiB/op <0.1% less memory Not flagged
Capture and deliver events (4,096-byte payload, burst)
Shard: capture
Allocations per operation 1,100.61 count/op 1,100.59 count/op <0.1% fewer Not flagged
Capture and deliver events (4,096-byte payload, burst)
Shard: capture
Time per operation 100 µs/op 109.77 µs/op 9.8% more time Worth reviewing: observed regression
Capture and deliver events (4,096-byte payload, burst)
Shard: capture
Idle process memory (RSS) 41.2 MiB 41.24 MiB <0.1% more memory Not flagged
Capture and deliver events (4,096-byte payload, burst)
Shard: capture
Observed peak process memory (RSS) 46.27 MiB 46.32 MiB <0.1% more memory Not flagged
Capture and deliver events (4,096-byte payload, burst)
Shard: capture
Process memory after settling (RSS) 46.27 MiB 46.32 MiB <0.1% more memory Not flagged
Capture and deliver events (4,096-byte payload, burst)
Shard: capture
Process CPU time per operation 93.11 µs/op 104.48 µs/op 12.2% more time Worth reviewing: observed regression
Capture and deliver events (4,096-byte payload, burst)
Shard: capture
Lifetime peak process memory (RSS) 51.07 MiB 51.16 MiB 0.2% more memory Not flagged
Capture and deliver events (32-byte payload, steady)
Shard: capture
Allocated memory per operation 119.86 KiB/op 119.87 KiB/op <0.1% more memory Not flagged
Capture and deliver events (32-byte payload, steady)
Shard: capture
Allocations per operation 1,147.58 count/op 1,147.59 count/op <0.1% more Not flagged
Capture and deliver events (32-byte payload, steady)
Shard: capture
Time per operation 95.75 µs/op 87.46 µs/op 8.7% less time Not flagged
Capture and deliver events (32-byte payload, steady)
Shard: capture
Idle process memory (RSS) 36.18 MiB 36.29 MiB 0.3% more memory Not flagged
Capture and deliver events (32-byte payload, steady)
Shard: capture
Observed peak process memory (RSS) 36.27 MiB 36.38 MiB 0.3% more memory Not flagged
Capture and deliver events (32-byte payload, steady)
Shard: capture
Process memory after settling (RSS) 36.27 MiB 36.38 MiB 0.3% more memory Not flagged
Capture and deliver events (32-byte payload, steady)
Shard: capture
Process CPU time per operation 94.11 µs/op 83.88 µs/op 10.9% less time Not flagged
Capture and deliver events (32-byte payload, steady)
Shard: capture
Lifetime peak process memory (RSS) 40.36 MiB 40.36 MiB No observed change Not flagged
Capture and deliver events (4,096-byte payload, steady)
Shard: capture
Allocated memory per operation 172.76 KiB/op 172.75 KiB/op <0.1% less memory Not flagged
Capture and deliver events (4,096-byte payload, steady)
Shard: capture
Allocations per operation 1,140.51 count/op 1,140.5 count/op <0.1% fewer Not flagged
Capture and deliver events (4,096-byte payload, steady)
Shard: capture
Time per operation 111.1 µs/op 114.06 µs/op 2.7% more time Not flagged
Capture and deliver events (4,096-byte payload, steady)
Shard: capture
Idle process memory (RSS) 41.04 MiB 41.1 MiB 0.1% more memory Not flagged
Capture and deliver events (4,096-byte payload, steady)
Shard: capture
Observed peak process memory (RSS) 41.04 MiB 41.1 MiB 0.1% more memory Not flagged
Capture and deliver events (4,096-byte payload, steady)
Shard: capture
Process memory after settling (RSS) 41.04 MiB 41.09 MiB 0.1% more memory Not flagged
Capture and deliver events (4,096-byte payload, steady)
Shard: capture
Process CPU time per operation 103.24 µs/op 106.89 µs/op 3.5% more time Not flagged
Capture and deliver events (4,096-byte payload, steady)
Shard: capture
Lifetime peak process memory (RSS) 41.4 MiB 41.5 MiB 0.2% more memory Not flagged
Queue an event (32-byte payload)
Shard: capture
Caller-thread CPU time per operation 47.71 µs/op 50.11 µs/op 5% more time Not flagged
Queue an event (32-byte payload)
Shard: capture
Time per operation 50.24 µs/op 50.45 µs/op 0.4% more time Not flagged
Queue an event (4,096-byte payload)
Shard: capture
Caller-thread CPU time per operation 49.7 µs/op 46.9 µs/op 5.6% less time Not flagged
Queue an event (4,096-byte payload)
Shard: capture
Time per operation 52.22 µs/op 48 µs/op 8.1% less time Not flagged
Evaluate all 10 flags
Shard: flags-scaling
Allocated memory per operation 226.56 KiB/op 226.56 KiB/op No observed change Not flagged
Evaluate all 10 flags
Shard: flags-scaling
Allocations per operation 2,240.44 count/op 2,240.44 count/op No observed change Not flagged
Evaluate all 10 flags
Shard: flags-scaling
Time per operation 84.4 µs/op 83.76 µs/op 0.8% less time Not flagged
Evaluate all 10 flags
Shard: flags-scaling
Process CPU time per operation 84.41 µs/op 83.76 µs/op 0.8% less time Not flagged
Evaluate all 100 flags
Shard: flags-scaling
Allocated memory per operation 2.17 MiB/op 2.17 MiB/op No observed change Not flagged
Evaluate all 100 flags
Shard: flags-scaling
Allocations per operation 22,014.88 count/op 22,014.88 count/op No observed change Not flagged
Evaluate all 100 flags
Shard: flags-scaling
Time per operation 783.79 µs/op 787.8 µs/op 0.5% more time Not flagged
Evaluate all 100 flags
Shard: flags-scaling
Process CPU time per operation 783.82 µs/op 787.55 µs/op 0.5% more time Not flagged
Evaluate all 1,000 flags
Shard: flags-scaling
Allocated memory per operation 21.69 MiB/op 21.69 MiB/op No observed change Not flagged
Evaluate all 1,000 flags
Shard: flags-scaling
Allocations per operation 219,656.62 count/op 219,656.62 count/op No observed change Not flagged
Evaluate all 1,000 flags
Shard: flags-scaling
Time per operation 7.84 ms/op 7.82 ms/op 0.2% less time Not flagged
Evaluate all 1,000 flags
Shard: flags-scaling
Process CPU time per operation 7.84 ms/op 7.82 ms/op 0.2% less time Not flagged
Evaluate 1,000 flags under concurrent load
Shard: flags-memory
Idle process memory (RSS) 42.9 MiB 42.98 MiB 0.2% more memory Not flagged
Evaluate 1,000 flags under concurrent load
Shard: flags-memory
Observed peak process memory (RSS) 43.27 MiB 43.3 MiB <0.1% more memory Not flagged
Evaluate 1,000 flags under concurrent load
Shard: flags-memory
Process memory after settling (RSS) 43.27 MiB 43.3 MiB <0.1% more memory Not flagged
Evaluate 1,000 flags under concurrent load
Shard: flags-memory
Lifetime peak process memory (RSS) 43.14 MiB 43.16 MiB <0.1% more memory Not flagged
Refresh 1,000 flag definitions
Shard: flags-memory
Idle process memory (RSS) 42.08 MiB 42.05 MiB <0.1% less memory Not flagged
Refresh 1,000 flag definitions
Shard: flags-memory
Observed peak process memory (RSS) 42.15 MiB 42.12 MiB <0.1% less memory Not flagged
Refresh 1,000 flag definitions
Shard: flags-memory
Process memory after settling (RSS) 41.5 MiB 42.06 MiB 1.4% more memory Not flagged
Refresh 1,000 flag definitions
Shard: flags-memory
Lifetime peak process memory (RSS) 42.5 MiB 42.45 MiB 0.1% less memory Not flagged
Evaluate one boolean flag
Shard: flags-single
Allocated memory per operation 16.63 KiB/op 16.63 KiB/op No observed change Not flagged
Evaluate one boolean flag
Shard: flags-single
Allocations per operation 164.77 count/op 164.77 count/op No observed change Not flagged
Evaluate one boolean flag
Shard: flags-single
Time per operation 10.6 µs/op 10.6 µs/op <0.1% more time Not flagged
Evaluate one boolean flag
Shard: flags-single
Process CPU time per operation 10.6 µs/op 10.6 µs/op <0.1% more time Not flagged
Evaluate a cohort-filtered flag
Shard: flags-single
Allocated memory per operation 25.49 KiB/op 25.49 KiB/op No observed change Not flagged
Evaluate a cohort-filtered flag
Shard: flags-single
Allocations per operation 241.28 count/op 241.28 count/op No observed change Not flagged
Evaluate a cohort-filtered flag
Shard: flags-single
Time per operation 14.12 µs/op 14.11 µs/op <0.1% less time Not flagged
Evaluate a cohort-filtered flag
Shard: flags-single
Process CPU time per operation 14.12 µs/op 14.11 µs/op <0.1% less time Not flagged
Evaluate a flag (early match)
Shard: flags-single
Allocated memory per operation 40.7 KiB/op 40.7 KiB/op No observed change Not flagged
Evaluate a flag (early match)
Shard: flags-single
Allocations per operation 390.27 count/op 390.27 count/op No observed change Not flagged
Evaluate a flag (early match)
Shard: flags-single
Time per operation 19.94 µs/op 19.87 µs/op 0.3% less time Not flagged
Evaluate a flag (early match)
Shard: flags-single
Process CPU time per operation 19.94 µs/op 19.87 µs/op 0.3% less time Not flagged
Evaluate a flag (late match)
Shard: flags-single
Allocated memory per operation 61.48 KiB/op 61.48 KiB/op No observed change Not flagged
Evaluate a flag (late match)
Shard: flags-single
Allocations per operation 586.27 count/op 586.27 count/op No observed change Not flagged
Evaluate a flag (late match)
Shard: flags-single
Time per operation 28.16 µs/op 28.3 µs/op 0.5% more time Not flagged
Evaluate a flag (late match)
Shard: flags-single
Process CPU time per operation 28.16 µs/op 28.3 µs/op 0.5% more time Not flagged
Evaluate a flag (no match)
Shard: flags-single
Allocated memory per operation 60.11 KiB/op 60.11 KiB/op No observed change Not flagged
Evaluate a flag (no match)
Shard: flags-single
Allocations per operation 569.77 count/op 569.77 count/op No observed change Not flagged
Evaluate a flag (no match)
Shard: flags-single
Time per operation 26.8 µs/op 26.76 µs/op 0.2% less time Not flagged
Evaluate a flag (no match)
Shard: flags-single
Process CPU time per operation 26.8 µs/op 26.76 µs/op 0.2% less time Not flagged
Evaluate a property-filtered flag
Shard: flags-single
Allocated memory per operation 19.93 KiB/op 19.93 KiB/op No observed change Not flagged
Evaluate a property-filtered flag
Shard: flags-single
Allocations per operation 194.28 count/op 194.28 count/op No observed change Not flagged
Evaluate a property-filtered flag
Shard: flags-single
Time per operation 11.77 µs/op 11.7 µs/op 0.6% less time Not flagged
Evaluate a property-filtered flag
Shard: flags-single
Process CPU time per operation 11.76 µs/op 11.7 µs/op 0.6% less time Not flagged
Evaluate a regex-filtered flag
Shard: flags-single
Allocated memory per operation 21.56 KiB/op 21.56 KiB/op No observed change Not flagged
Evaluate a regex-filtered flag
Shard: flags-single
Allocations per operation 199.78 count/op 199.78 count/op No observed change Not flagged
Evaluate a regex-filtered flag
Shard: flags-single
Time per operation 12.46 µs/op 12.38 µs/op 0.6% less time Not flagged
Evaluate a regex-filtered flag
Shard: flags-single
Process CPU time per operation 12.46 µs/op 12.38 µs/op 0.6% less time Not flagged
Evaluate a rollout flag
Shard: flags-single
Allocated memory per operation 15.79 KiB/op 15.79 KiB/op No observed change Not flagged
Evaluate a rollout flag
Shard: flags-single
Allocations per operation 154.77 count/op 154.77 count/op No observed change Not flagged
Evaluate a rollout flag
Shard: flags-single
Time per operation 10.01 µs/op 10.01 µs/op <0.1% less time Not flagged
Evaluate a rollout flag
Shard: flags-single
Process CPU time per operation 10.01 µs/op 10.01 µs/op <0.1% less time Not flagged
Evaluate a multivariate flag
Shard: flags-single
Allocated memory per operation 16.79 KiB/op 16.79 KiB/op No observed change Not flagged
Evaluate a multivariate flag
Shard: flags-single
Allocations per operation 168.77 count/op 168.77 count/op No observed change Not flagged
Evaluate a multivariate flag
Shard: flags-single
Time per operation 10.93 µs/op 10.95 µs/op 0.2% more time Not flagged
Evaluate a multivariate flag
Shard: flags-single
Process CPU time per operation 10.92 µs/op 10.95 µs/op 0.2% more time Not flagged

Values are medians of paired measurements. Not flagged does not mean unchanged. Unavailable means a measurement could not be provided, not zero. Estimated allocation measurements retain their instrumentation uncertainty. RSS measures the whole process, not just SDK allocations; observed peaks are sampled checkpoints. Paired ranges in the diagnostics describe variability, not confidence intervals.

Each shard uses its own runner; compare Before PR against With PR within a row, not across shards.

Compared f22511891225fb15a4e6a0f889e723e9b5d0ebe4 → be1d543c1909f6b9d21e1aa449258cbc60dc8af1.

Full report and diagnostics

102 metric rows; 102 rows shown in all results. Omitted or shortened data remain in comparison.json, including per-runner environments, resource limits, measurement methods, samples and checkpoints.

@dustinbyrne dustinbyrne left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reproduced a custom-event attribution regression: after a token-bearing request, a metadata-only caller’s in-tool capture can inherit the previous caller’s session and person. The same public-entry test passes on base and fails on this head.

The metadata policy also needs reconciliation with the current canonical MCP specification; the linked spec proposal remains open. Details and executed evidence are inline.

AI-assisted review.

Comment thread posthog/mcp/_instrumentation.py Resolved
Comment thread posthog/mcp/session.py Resolved
…x, VS Code)

Rework the openai/session change to the merged sdk-specs rules
(PostHog/sdk-specs 2026-11-04-mcp-client-conversation-ids and
-codex-vscode-conversation-ids), matching posthog-js#5577:

- _client_conversation.CLIENT_CONVERSATION_ROWS mirrors the spec table:
  openai/session (1-128 printable ASCII, handle-free), Codex sessionId and
  VS Code vscode.conversationId (UUID, lowercased, non-nil, handle kept).
- A key is read only from a request its row recognizes by User-Agent or
  client name (_meta clientInfo first, then the adapter's resolved name).
- The client id is session source 1, ahead of the handle and transport, on
  tool calls only (tools/list, resources and initialize no longer read it).
  A handle echoed with it is stripped but ignored.
- A handle-free client's client-facing listing has no conversation_id or
  _mcp_instructions, and it is minted no handle. The SDK's internal cache
  pass still injects, so validation stays client-agnostic.
- Client- and handle-grouped calls count as connection activity, so a
  stdio session does not rotate early (gate finding from the JS port).
- PostHogMCP: prepare_tool_list/prepare_tool_call take headers
  (+client_info), capture_tool_call takes request_meta.

Tested: pytest posthog/test/mcp 743 passed on mcp 1.30.0, 601 passed on
mcp 2.3.0; the new tests cover every spec scenario and vector, including
real HTTP fleets over official, fastmcp and lowlevel servers. ruff, mypy
(CI mode with baseline) and the public API snapshot pass.

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed 02a4815: 1 Must fix, 1 Should fix. This pull request is large, so the review ran in 4 parts with less depth than usual.

Comment thread posthog/mcp/_instrument_fastmcp.py Resolved
Comment thread posthog/mcp/posthog_mcp.py Resolved
Comment thread posthog/mcp/_instrumentation.py Resolved
@veria-ai

veria-ai Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

PR overview

The PR groups MCP sessions using ChatGPT, Codex, and VS Code conversation IDs and defers custom events while conversation handles are being created.

One issue has been addressed, and one memory-exhaustion concern remains open. Calls that create conversation handles retain all custom events until the tool finishes, bypassing the client's bounded queue during that period. Bulk or streaming tools that capture events per client-supplied item can therefore cause unbounded SDK memory growth.

Open issues (1)

Fixed/addressed: 1 · PR risk: 5/10

Reviews

- Count a client-grouped call as connection activity only on a one-client
  connection (no HTTP request, no token, no transport id). On a shared
  stateless HTTP pod, ChatGPT calls kept the instance session alive and
  merged two unrelated clients into one $session_id.
- PostHogMCP.prepare_tool_list for a handle-free client still records
  conversation_id ownership, so prepare_tool_call strips an echoed handle
  like the automatic path does.

Tested: both regressions have tests that fail on 02a4815 and pass now;
pytest posthog/test/mcp 745 passed on mcp 1.30.0 and 603 on mcp 2.3.0;
ruff, mypy (CI mode) and the public API check pass.
@lucasheriques lucasheriques changed the title feat(mcp): group stateless ChatGPT calls by openai/session feat(mcp): group sessions by ChatGPT, Codex and VS Code conversation ids Nov 5, 2026

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed 3cbcd29: 1 Must fix, 2 Should fix. This pull request is large, so the review ran in 4 parts with less depth than usual.

This is review round 3 on this pull request. When findings keep coming back in the same code, the approach is usually the problem. Before the next fix, check whether a different design would remove the whole class of findings.

Comment thread posthog/mcp/_instrumentation.py Resolved
Comment thread posthog/mcp/posthog_mcp.py Resolved
Comment thread posthog/mcp/_instrumentation.py Resolved

@dustinbyrne dustinbyrne left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The earlier recognition, bounds and precedence feedback is addressed. The custom-event attribution regression remains and now also reproduces when the second caller supplies its own token.

I also reproduced session/identity merging for handle-free HTTP calls without usable metadata, output-validation failures after mixed-client FastMCP listings, and schema-invalid results after mixed manual listings. The corresponding base controls pass; details are inline.

AI-assisted review.

Comment thread posthog/mcp/_instrumentation.py Resolved
Comment thread posthog/mcp/_instrumentation.py Resolved
Comment thread posthog/mcp/_instrument_fastmcp.py Resolved
Comment thread posthog/mcp/posthog_mcp.py Resolved
@dustinbyrne
dustinbyrne requested a review from a team November 5, 2026 22:28
Resolve conflicts with main's bounded intent scans, unknown-tool event and
new_session_id: keep lists_conversation_id and the client-conversation imports,
and thread the resolved session through record_unknown_tool so $mcp_unknown_tool
carries $mcp_session_source like every other event. Code matches the merged
spec plus sdk-specs#185.

Tested: uv run pytest posthog/test/mcp (776 passed, 11 skipped); ruff format and
check, mypy-baseline filter, public API check, import under -W error all clean.
Full suite: 6326 passed, 12 failed outside posthog/mcp (langchain, metrics autocapture fork).

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed 08774e9: 1 Must fix. This pull request is large, so the review ran in 4 parts with less depth than usual.

This is review round 4 on this pull request. When findings keep coming back in the same code, the approach is usually the problem. Before the next fix, check whether a different design would remove the whole class of findings.

Comment thread posthog/mcp/_instrument_v2.py Resolved
@lucasheriques lucasheriques added the stamphog Request stamphog review label Nov 9, 2026

@stamphog stamphog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not approved — this change needs a human reviewer.

Re-adding the stamphog label gives the same result unless the changed files change.

Stamphog can't auto-review this pull request because two gates refused it. The deny-list gate flagged it for matching the auth and public_api categories, which fits the session and identity changes in posthog/mcp/session.py, posthog/mcp/_instrumentation.py and posthog/mcp/posthog_mcp.py, and the signature changes recorded in references/public_api_snapshot.txt. The tier gate classified it as T2-never because it is a cross-cutting feature spanning 22 files and 1508 lines. The size gate passed, so size is not the blocker.

Please ask a human reviewer to look at it. Splitting it, for example into the client-conversation parsing module, the session resolution changes, and the PostHogMCP API changes, may make that review easier, but the sensitive areas will still need a human.

  • posthog[bot] reviewed the current head.
Gate mechanics and policy version
Gate Result
prerequisites ✓ all clear
deny-list ✗ matches: auth, public_api
size ✓ 722L, 13F substantive, 1508L/22F incl. docs/generated/snapshots — within ceiling
tier ✗ classified as T2-never: T2-never (1508L, 22F, cross-cutting, feat)
stamphog 2.7.0 .stamphog/policy.yml @ unknown · reviewed head 08774e9

@stamphog stamphog Bot removed the stamphog Request stamphog review label Nov 9, 2026

@dustinbyrne dustinbyrne left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Complete read-only follow-up against current base 133d7d12c3f15b04287b6d8389b0412fb7dfe6de.

The recognition table, validation, namespaced derivation and metadata precedence now match the canonical MCP contract, including automatic session-source reporting. All four required regressions from my prior review still reproduce against this updated base: custom capture inherits the previous caller's identity; missing/empty handle-free HTTP metadata merges sessions and person properties; mixed-client official FastMCP listings turn successful strict output into errors; and manual listing/preparation returns a schema-incompatible result.

I independently reproduced the existing MCP 2.x internal-list ownership report: base passes all four cases; head fails fresh raw HTTP/no-resolver with JSON-RPC -32603, while warm-listing and resolver controls pass. The adversarial pass also identified a tracked public-import removal; the old import/string-return smoke test passes on base and raises ImportError on head.

These are five P1 corrections plus one P2 compatibility correction, so I cannot recommend approval yet. Current CI is green (26 successes, one neutral), but it does not cover these sequences. The exact regressions pass on the current base and fail on this head. Full local suites also have nine identical standalone FastMCP dependency/import failures on both revisions; those are environmental, not additional PR regressions. No live PostHog delivery, provider integration, deployed profile mutation or UI verification was performed.

AI-assisted review.

Comment thread posthog/mcp/_instrumentation.py Resolved
Comment thread posthog/mcp/_instrumentation.py Resolved
Comment thread posthog/mcp/_instrument_fastmcp.py Resolved
Comment thread posthog/mcp/posthog_mcp.py Resolved
Comment thread posthog/mcp/_instrument_v2.py Resolved
Comment thread posthog/mcp/asgi.py Resolved
@dustinbyrne
dustinbyrne requested a review from a team November 9, 2026 21:17
…s consistent

Address the 11-09 review of #1193, checked against sdk-specs main
(openspec/specs/mcp-analytics/spec.md):

- Priming no longer short-circuits on a client conversation id, so an in-tool
  McpAnalytics.capture() never reads the previous caller's session or identity.
  Over HTTP such a call, and a handle-free (ChatGPT) call without a usable id,
  primes a session reserved for that one request ("request" internally,
  "generated" on the wire). The metadata-derived id is never stored on shared
  state, and a one-client connection (stdio, SSE) keeps the process session.
- Two handle-free HTTP calls without openai/session no longer share the process
  session and its cached person properties (spec rule 4).
- A listing that declares no _mcp_instructions turns mirroring off, because
  mcp 1.x validates the next result against the latest listing. Ordinary list,
  ChatGPT list, ordinary call no longer fails an extra="forbid" result.
- PostHogMCP records real ownership on a suppressed listing, so a later
  ordinary listing declares _mcp_instructions whenever its results carry it.
- MCP 2.x internal tools/list dispatch still prepares schemas and caches
  ownership. It only skips the $mcp_tools_list event.
- A finished call grouped by a client id refreshes stdio connection activity.
  Rotation happens on arrival only. Virtual tools now prime on arrival too.
- Restore posthog.mcp.asgi.handshake_session_id as an async str wrapper.

Tested, one red-first test per finding:
- pytest posthog/test/mcp: mcp 1.30 791 passed / 11 skipped; mcp 2.3 +
  fastmcp 4.1.8 677 passed / 28 skipped; mcp 2.3 bare 626 passed / 60 skipped.
- ruff format --check, ruff check, mypy | mypy-baseline filter, public API
  check, python -W error -c "import posthog": clean.

Reviewer notes: rule 4 is applied only where a handle used to isolate the call.
The stateless-route-without-middleware fallback for other clients is unchanged
(test_a_route_without_the_middleware_keeps_the_fallback). Separately, an
ordinary stdio call longer than 30 minutes is stamped with a rotated session at
completion; that predates this PR and is left for its own fix.
@lucasheriques lucasheriques added the stamphog Request stamphog review label Nov 12, 2026

@stamphog stamphog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not approved — this change needs a human reviewer.

Re-adding the stamphog label gives the same result unless the changed files change.

stamphog can't approve this pull request automatically. The deny-list gate refused it because it touches sensitive areas, matching both auth and public_api. This shows up in the session and identity changes under posthog/mcp/ and in the references/public_api_snapshot.txt update. The tier gate also classified it as T2-never: a cross-cutting feature spanning 1875 lines across 23 files, which is never eligible for automated approval.

Please ask a human reviewer to take this one. It would also help to split it up, for example the client-conversation recognition module, the PostHogMCP signature changes, and the $mcp_session_source event property as separate pull requests.

Gate mechanics and policy version
Gate Result
prerequisites ✓ all clear
deny-list ✗ matches: auth, public_api
size ✓ 813L, 13F substantive, 1875L/23F incl. docs/generated/snapshots — within ceiling
tier ✗ classified as T2-never: T2-never (1875L, 23F, cross-cutting, feat)
stamphog 2.8.0 .stamphog/policy.yml @ unknown · reviewed head 5417daa

@stamphog stamphog Bot removed the stamphog Request stamphog review label Nov 12, 2026

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed 5417daa: no findings. This pull request is large, so the review ran in 4 parts with less depth than usual.

@dustinbyrne dustinbyrne left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The six previously reported exact regressions now pass against the updated base. Recognition, derivation and precedence also match the canonical contract.

One overlapping-request isolation regression remains: if Bob finishes while Alice’s async tool is waiting, Alice’s custom capture can use Bob’s session, identity and person properties. The same test passes on base and fails only for overlap on head under both official MCP majors; awaited controls pass. Consider keeping fallback attribution request-local before approval.

Current CI is green but omits this overlap. Validation establishes emitted payloads, not live PostHog delivery or deployed profile changes.

AI-assisted review.

Comment thread posthog/mcp/session.py
Comment on lines +203 to +205
data.session_id = request_session_id
data.session_source = "request"
return ResolvedSession(request_session_id, "request")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Keep custom capture's fallback isolated across overlapping HTTP calls

IMPORTANT · REQUIRED · Introduced/exposed by this PR relative to the merge base.
Location: posthog/mcp/session.py:203–205, RIGHT. Related: _instrumentation.py:782–790, __init__.py:164–174, _capture.py:31–32.

On a shared stateless HTTP server, Alice can emit a custom event as Bob when Bob finishes while Alice's async tool is awaiting work. The event contains Alice's payload, Bob's distinct_id, Bob's request $session_id, and Bob's cached $set. Expected: Alice's custom event must not adopt another caller's attribution; an isolated anonymous fallback is sufficient.

A supported public caller is enough:

@server.tool()
async def probe(marker: str) -> str:
    if marker == "alice":
        entered.set()
        await resume.wait()  # another HTTP request can complete during async work
        await analytics.capture("inside_tool", {"marker": "alice"})
    return marker

Trigger: default-enabled conversation anchoring; ChatGPT-recognized User-Agent; no token/transport session; an application identify callback resolving the request user; Bob has no usable client conversation ID. Alice may have valid openai/session metadata or none. Alice starts, Bob completes, then Alice resumes. The test uses valid HTTP/MCP requests and no private-state mutation; asynchronous overlap is supported by both official HTTP adapters.

The new request fallback is written to server-wide data.session_id. Bob's automatic completion identifies that fallback. When Alice resumes, McpAnalytics.capture() reads Bob's shared value, and capture_event() looks up Bob's cached actor. Locking individual resolver updates does not protect the lifetime of an awaiting tool invocation.

For both tested official MCP versions, the overlapping head case emits:

{
  "event": "inside_tool",
  "distinct_id": "bob",
  "properties": {
    "marker": "alice",
    "$session_id": "<same session as Bob's automatic event>",
    "$set": {"owner": "bob"}
  }
}

Base/head causality: on base 7eabca81234311dfd86e516c4354c5322f73bfc5, Bob's automatic event identifies a minted-handle session and leaves the custom-event fallback anonymous. On head 5417daa9361862cdc1934abf7e14ffaed9ecc414, handle suppression replaces that path with an identified request fallback exposed through shared state. Existing token-backed races are not the evidence for this finding. This is a newly identified case against the merge base, not a claim that it first appeared in the latest fix commit.

The canonical Session identity requirement forbids grouping different clients, and Person identity uses the request's actor or its session. The priming contract in _instrumentation.py:408–419 explicitly protects in-tool capture from another caller's identity. The instance-source documentation does not justify attaching Alice's payload to Bob's identity.

Executed regression: Python 3.11.13, pytest 8.4.1, pytest-asyncio 1.0.0, httpx 0.28.1 and Pydantic 2.12.5; identical base/head inputs/dependencies within each lane.

Official entry path Base Head
MCP 1.28.1 FastMCP, legacy stateless HTTP 4 passed 2 failed, 2 passed
MCP 2.3.0 MCPServer, modern stateless HTTP 4 passed 2 failed, 2 passed

Only the two overlapping variants fail on head, at custom["distinct_id"] != "bob"; awaited controls pass. Printed payloads independently show the wrong session and person properties. The six previously reported exact regressions pass on this head and are not being reopened here.

Consider retaining custom-capture fallback attribution in request/task-local context, bound for the invocation and reset in finally, rather than reading whichever request last updated the server-wide value. Preserve outside-request behavior and the passing sequential/token controls; add this overlap coverage for both majors. No public capture argument, global metadata-derived ID, or serialization of unrelated requests is needed.

Impact limits: demonstrated wrong event/person association within the receiving project, not an authentication bypass, response leak, cross-project disclosure, wrong automatic-event attribution, or executed backend profile mutation. The test uses the real inbound adapters and event-shaping pipeline with a FakeClient sink; outbound PostHog HTTP delivery and live client metadata frequency were not verified.

Executed regression harness

Save as test_concurrent_custom_capture.py in the SDK test environment and run pytest -q -s --asyncio-mode=auto test_concurrent_custom_capture.py. Run each revision under the same MCP major/version. The header-based identity callback is a test fixture for an application's request-user resolver, not a credential.

"""Overlapping public HTTP calls must not attribute Alice's custom event to Bob."""
import asyncio
import json
import importlib.metadata
import httpx
import pytest
MCP_MAJOR = int(importlib.metadata.version('mcp').split('.')[0])
if MCP_MAJOR < 2:
    from mcp.server.fastmcp import FastMCP
    from mcp.server.transport_security import TransportSecuritySettings
else:
    from mcp.server.mcpserver import MCPServer
    from posthog.test.mcp._helpers_v2 import modern_headers, modern_meta
from posthog.mcp import instrument, MCPAnalyticsOptions, UserIdentity, get_request_headers
from posthog.test.mcp._helpers import FakeClient

HEADERS = {'content-type': 'application/json', 'accept': 'application/json, text/event-stream', 'mcp-protocol-version': '2025-06-18', 'user-agent': 'openai-mcp/1.0.0 (ChatGPT)'}

@pytest.mark.parametrize('overlap', [False, True], ids=['awaited-control', 'overlapping'])
@pytest.mark.parametrize('alice_meta', [{}, {'openai/session': 'alice-chat'}], ids=['both-missing-id', 'alice-valid-id'])
async def test_custom_event_does_not_inherit_another_inflight_call(alice_meta, overlap):
    entered = asyncio.Event()
    resume = asyncio.Event()
    client = FakeClient()
    server = (FastMCP('concurrent-review', stateless_http=True, json_response=True, transport_security=TransportSecuritySettings(enable_dns_rebinding_protection=False)) if MCP_MAJOR < 2 else MCPServer('concurrent-review'))
    @server.tool()
    async def probe(marker: str) -> str:
        if marker == 'alice':
            entered.set()
            if overlap:
                await resume.wait()
            await analytics.capture('inside_tool', {'marker': 'alice'})
        return marker
    def identify(request, extra):
        user = get_request_headers(extra['ctx']).get('x-review-user')
        return UserIdentity(distinct_id=user, properties={'owner': user}) if user else None
    analytics = instrument(server, client, MCPAnalyticsOptions(context=False, capture_model=False, send_initialize_event=False, identify=identify))
    app = server.streamable_http_app() if MCP_MAJOR < 2 else server.streamable_http_app(json_response=True, stateless_http=True)
    async with server.session_manager.run():
        async with httpx.AsyncClient(transport=httpx.ASGITransport(app=app), base_url='http://127.0.0.1:8000') as http:
            async def call(who, meta, request_id):
                protocol_headers = HEADERS if MCP_MAJOR < 2 else {**modern_headers('tools/call', 'probe'), 'user-agent': HEADERS['user-agent']}
                request_meta = meta if MCP_MAJOR < 2 else {**modern_meta(), **meta}
                response = await http.post('/mcp', headers={**protocol_headers, 'x-review-user': who}, json={'jsonrpc': '2.0', 'id': request_id, 'method': 'tools/call', 'params': {'name': 'probe', 'arguments': {'marker': who}, '_meta': request_meta}})
                assert response.status_code == 200, response.text
                body = response.json()
                assert 'error' not in body and not body['result'].get('isError'), body
                await analytics.flush()
                return body
            if overlap:
                alice_task = asyncio.create_task(call('alice', alice_meta, 1))
                await asyncio.wait_for(entered.wait(), 5)
                try:
                    await call('bob', {}, 2)
                finally:
                    resume.set()
                await asyncio.wait_for(alice_task, 5)
            else:
                await call('alice', alice_meta, 1)
                await call('bob', {}, 2)
    custom = next(e for e in client.events if e['event'] == 'inside_tool')
    bob = next(e for e in client.events if e['event'] == '$mcp_tool_call' and e['distinct_id'] == 'bob')
    print(json.dumps({'mcp_version': importlib.metadata.version('mcp'), 'overlap': overlap, 'alice_meta': alice_meta, 'alice_custom': custom, 'bob_automatic': bob}, default=str))
    assert custom['distinct_id'] != 'bob', 'Alice custom event uses Bob identity'
    assert custom['properties'].get('$set', {}).get('owner') != 'bob', 'Alice payload updates Bob properties'
    assert custom['properties']['$session_id'] != bob['properties']['$session_id'], 'Alice custom event uses Bob request session'

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 4b103b5. I followed your suggestion: the session is request-local, bound for the invocation and reset in finally.

  • Binding. Each adapter (FastMCP, lowlevel, and both v2 paths) runs the tool body inside lifecycle.in_tool(). It binds the call's session to a ContextVar keyed to the server's tracking data and resets it when the body returns. McpAnalytics.capture() reads it, and falls back to data.session_id outside a tool body or on another server.
  • Which session. The call's client conversation id or handle if it has one (an echoed handle, or one minted for this call, as posthog-js does), otherwise its primed transport session.
  • on_feedback. The host handler runs bound to the $mcp_feedback session.
  • Merge. main is merged in (c8df7b0). The only conflict was posthog_mcp.py, where the client-id logic moved into the new _prepare_conversation helper.

Tests in test_client_conversation_session.py:

  • Your overlap scenario (Alice primes, Bob completes, Alice captures), with no id, a client id, an echoed handle and a minted handle. Alice's event matches her own $mcp_tool_call session.
  • An on_feedback overlap.
  • Captures outside a tool: on the same server after a call, and on another server inside one.

Each row fails when the matching part of the binding is removed. posthog/test/mcp: 884 pass. Not run here: the 2.x adapters (only mcp 1.x is installed locally). CI covers both majors.

@dustinbyrne
dustinbyrne requested a review from a team November 12, 2026 21:05
…-session

# Conflicts:
#	posthog/mcp/posthog_mcp.py
A custom event captured inside a tool read the server-wide data.session_id.
On a shared stateless HTTP server, another request finishing while the tool
awaited moved that slot, so the event took the other caller's session and
identity (Dustin's P1 on this PR).

Each adapter now runs the tool body inside lifecycle.in_tool(), which binds
the call's session to a ContextVar keyed to its server and resets it when the
body returns. The session is the call's client conversation id or handle
(echoed, or minted for this call, as posthog-js does) when it has one, else
its primed transport session. The host's on_feedback handler runs bound to
the $mcp_feedback session. Rules 1 and 2 move into resolve_anchored_session
so the resolver and the binding share them.

Tested: overlap rows (no id, client id, echoed and minted handle), an
on_feedback overlap, and outside-a-tool rows (same server after a call,
another server inside one) fail before and pass after; dropping each part
of the binding turns its rows red. pytest posthog/test/mcp: 884 passed.
ruff, public API snapshot clean; mypy unchanged (5 errors in untouched files).

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed 4b103b5: 1 Should fix. This pull request is large, so the review ran in 4 parts with less depth than usual.

This is review round 5 on this pull request. When findings keep coming back in the same code, the approach is usually the problem. Before the next fix, check whether a different design would remove the whole class of findings.

Comment thread posthog/mcp/_instrumentation.py Resolved
…ll's outcome

A handle minted for a call groups it only once its result delivers it. The
in-tool binding put custom events on that handle's session right away, but a
call that raised, or whose result could not carry the handle, records its
$mcp_tool_call under the fallback session. The two then split (reviewhog's
P2 on this PR).

While a call runs on a handle minted for it, in_tool buffers its custom events.
When the call records its outcome (result, error, unknown tool), they are sent
with that event's session, without the call waiting on them or failing because
of them. Each adapter settles whatever is left in a finally with the primed
session, so a cancelled call, or one that records nothing, loses none. They
keep the duration they had when captured. Work the tool started that captures
later goes straight to the settled session. Echoed handles and client
conversation ids are unchanged.

Tested: a parameterized test crosses the outcome (delivered, undelivered,
raised) with when the event is captured (in the tool, after the call settles);
5 of 6 rows fail before. test_minted_handle_custom_events.py runs cancellation
(in the tool, and while the call records), an empty tool name, an unbuildable
event and the duration through the real mcp 1.x adapters; all fail without the
fix. pytest posthog/test/mcp: 895 passed. ruff, public API snapshot clean;
mypy unchanged. The mcp 2.x adapter gets the same finally; CI runs it.

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed be1d543: 2 Should fix. This pull request is large, so the review ran in 4 parts with less depth than usual.

This is review round 6 on this pull request. When findings keep coming back in the same code, the approach is usually the problem. Before the next fix, check whether a different design would remove the whole class of findings.

Comment thread posthog/mcp/__init__.py
pending = unsettled_custom_events(data)
if pending is not None:
# Sent once the call settles; its duration is still the moment it was captured.
pending.events.append({**event_input, "duration": 0})

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 · Snapshot custom properties before queuing events

When a tool reuses a properties dictionary across awaited capture() calls, this shallow copy queues the same dictionary for every event. Updating or clearing it before the tool finishes rewrites or erases previously captured values: two progress events both lose their steps after properties.clear(). Snapshot the properties when queuing each event.

Comment thread posthog/mcp/__init__.py
pending = unsettled_custom_events(data)
if pending is not None:
# Sent once the call settles; its duration is still the moment it was captured.
pending.events.append({**event_input, "duration": 0})

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 · Bound the deferred custom-event buffer

During a long-running tool call that mints a handle, each awaited capture() retains the full properties object in pending.events until completion. This bypasses the client's bounded queue and payload truncation, so repeated progress events grow memory without limit; settlement then creates one task per event. Bound the buffer by count and bytes with explicit overflow handling.

Comment thread posthog/mcp/__init__.py
pending = unsettled_custom_events(data)
if pending is not None:
# Sent once the call settles; its duration is still the moment it was captured.
pending.events.append({**event_input, "duration": 0})

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Medium: Unbounded deferred-event buffer

Calls that mint a conversation handle now accumulate every full custom event until the tool finishes, before the sink or the client's bounded queue can drop anything. For bulk or streaming tools that capture per client-supplied item, this allows unbounded SDK memory growth despite max_queue_size; cap the deferred event count and retained bytes, dropping excess events before appending.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants