Skip to content

feat(twilio): a relay session whose playback is stuck is connected again and says what never played - #1500

Open
yuyixg wants to merge 2 commits into
SciSharp:masterfrom
yuyixg:fix/relay-reconnect-when-playback-stuck
Open

yuyixg wants to merge 2 commits into
SciSharp:masterfrom
yuyixg:fix/relay-reconnect-when-playback-stuck

Conversation

@yuyixg

@yuyixg yuyixg commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Why

On some calls Twilio ConversationRelay said the agent started speaking (agentSpeaking on) and never said it stopped. After that:

  • nothing sent played, neither the lines queued behind it nor the idle prompts;
  • none of the caller's words came through as prompts (only clientSpeaking arrived);
  • Twilio sent no error message and logged no 6410x error;
  • the call stayed silent until the caller hung up.

Seen on 2 of 15 calls with a full closing on an RSP dev line in 40 hours. A healthy call with the same lines, the same timing and the caller talking over the first line played normally, so nothing we send tells the two apart.

What changed

All of it is off unless StuckPlaybackSeconds is set, globally or per agent. It needs Twilio's speaker events (the events attribute set to speaker-events).

Detection

  • The session reads info frames (agentSpeaking on/off); a cut-in counts as the playback stopping.
  • It tracks when everything said should have finished playing, using the existing EstimateSpeech, which runs on the slow side.
  • The playback is stuck when Twilio still says the agent is speaking StuckPlaybackSeconds past that moment, nothing was cut into, and no turn is running.

Reconnect

  • The session keeps a ConversationRelayResume and drops its socket. The resume holds the lines said since the playback last stopped, each with whether it went out uninterruptible, plus the flow's quiet wait and goodbye on silence. Fillers and idle prompts are left out.
  • It does this only while reconnects < MaxReconnects. The session reads its count from a new reconnects Parameter on every <ConversationRelay>.
  • Why it doesn't send end: Twilio acts on end only once what is queued has played. On dev, every session ended with a goodbye and end sent together was reported ended 5.4–6.0 s later, which is the goodbye's length. A stuck playback never finishes, so end would never take effect. A dropped socket ends the session at once, and Twilio's callback finds it failed, the same as when an instance holding a socket goes away. Dev already recovers from that case (calls a2f3dffa, 469eeb86).
  • The resume is kept in ConversationRelayResumes. It goes to Redis when there is one, because the callback may run on another instance; otherwise it stays on this instance for 5 minutes.
  • TwilioConversationRelayController takes the resume for the conversation, once, whatever status Twilio gives the dropped session. It answers with a new <ConversationRelay> that carries the resume as a resume Parameter, and tells the hooks nothing, since the call goes on.
    • Not for a session that ended with a handoff (hang-up or transfer).
    • Not past MaxReconnects.

The new session

  • On setup it says those lines again, each with its interruptible flag. They are already in the history, so they aren't kept again.
  • It takes back the quiet wait and the goodbye on silence.

While watching

  • No idle prompt is said while Twilio still plays. It would only queue behind a stuck playback and keep putting off detection until the idle prompts ran out and hung up.
  • The caller's silence starts no sooner than agentSpeaking off.
  • After off or a cut-in, the playback estimate restarts from then.

Trying it on a call

  • A negative StuckPlaybackSeconds treats a playback still going that long before its expected end as stuck. No real stuck playback can be made on demand, so this is how to exercise the reconnect on a test call.

Limits

  • Detection is late: it fires StuckPlaybackSeconds after the estimated end of everything queued, because Twilio reports one on/off for lines played back to back. For a three-line closing that is about 35 + 8 s.
  • Lines may repeat: every line that went out since the last off is said again, even if the first of them actually played.
  • Size limits unknown: Twilio documents no limit for <Parameter> values. The resume is usually under 700 characters.
  • Not verified on a real stuck call.

Tests

The tests are in onebrain, which runs this plugin's relay session and its action callback over a fake socket (UnitTest/TwilioHooks/ConversationRelayWatchdogTests.StuckPlayback.cs). All 107 relay tests pass:

  • a stuck playback drops the socket, keeps the unheard line and sends no end;
  • a playback Twilio says finished, one the caller cut into, and a call without speaker events are not taken as stuck;
  • only lines since the last off are kept, each with its interruptible flag, and fillers are left out;
  • a negative setting fires while Twilio still plays;
  • no idle prompt is said while Twilio plays, and silence counts from off;
  • a call already connected again MaxReconnects times is not connected again;
  • a resumed session says its lines as they went out and waits quietly again;
  • the callback connects the call again with what was kept, once; it doesn't turn a hang-up, or a call past its reconnects, into a reconnect;
  • ReturnConversationRelay carries reconnects and resume.

🤖 Generated with Claude Code

…ain and says what never played

On some calls Twilio said the agent started speaking and never that it stopped. Nothing sent after that played, none of
the caller's words came through, and Twilio reported no error, until the caller hung up (2 of 15 calls with a full
closing on an RSP dev line in 40 hours). The session now reads Twilio's speaker events (the events attribute set to
speaker-events) and can recover:

- ConversationRelaySetting.StuckPlaybackSeconds, also per agent: how long past the moment everything said should have
  finished playing (estimated from its length, on the slow side) Twilio may still say the agent is speaking, with nothing
  cut into and no turn running, before the playback is taken as stuck. 0, the default, turns it off.
- Stuck, the session ends with handoffData {"action":"reconnect","resume":{...}}: the lines said since Twilio last said
  the playback stopped (fillers and idle prompts left out), each with whether it went out to play to the end, and the
  flow's quiet wait and goodbye on silence. Only while the call may be connected again (MaxReconnects): the session reads
  its count from a new "reconnects" Parameter.
- The action callback answers it with a new <ConversationRelay> carrying the resume as a "resume" Parameter, and tells
  the hooks nothing, as the call goes on. A call that can't be connected again ends as the agent's.
- The new session says those lines again, each as it went out (they are in the history already, so not kept again),
  and takes the quiet wait and goodbye on silence back.
- While it watches, no idle prompt is said while Twilio still plays (it would only queue behind a stuck playback and put
  off finding it), the caller's silence starts no sooner than Twilio says the playback ended, and what goes out after
  "agentSpeaking off" or a cut-in plays from then, not after what the estimate still had to come.

Tests are in onebrain (UnitTest/TwilioHooks/ConversationRelayWatchdogTests.StuckPlayback.cs), which runs this plugin's
relay session over a fake socket: 71 of 71 relay tests pass, 10 of them new.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@qodo-code-review

Copy link
Copy Markdown
Contributor

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

…g the session

Twilio acts on an "end" message only once what is queued has played: on dev, every session we ended with a goodbye and
"end" sent together was reported ended 5.4 to 6.0 s later, the goodbye's length (9 calls). A stuck playback never
finishes, so the "end" the stuck session sent would never have been acted on and the call would never have been
connected again.

- The stuck session now keeps what to say again (ConversationRelayResumes: in Redis when there is one, as the callback may
  run on another instance; on this instance otherwise, for 5 minutes) and drops its socket. Twilio ends the session at
  once and its callback finds it failed, as when an instance holding a socket goes away, which dev already recovers from
  (calls a2f3dffa, 469eeb86).
- The action callback takes what was kept for the conversation, once, whatever status Twilio gives the dropped session,
  and connects the call again with it. Not for a session that ended with a handoff (a hang-up or a transfer), nor past
  MaxReconnects.
- The "reconnect" handoff and its handoffData are gone.
- A negative StuckPlaybackSeconds takes a playback still going that long before its expected end as stuck, so the
  reconnect can be tried on a test call: no real stuck playback can be made on demand.

Tests (onebrain UnitTest/TwilioHooks/ConversationRelayWatchdogTests.StuckPlayback.cs): the stuck session drops the socket,
keeps the unheard lines and sends no "end"; the callback connects the call again with what was kept, once; a hang-up or
a call past its reconnects isn't connected again; a negative setting fires while Twilio still plays. 107 of 107 relay
tests pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yileicn
yileicn requested a balanced review from Copilot November 18, 2026 07:46

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants