Our voice agent only spoke once per call

A caller would connect, ask a question, and hear the answer. Then they would ask a second question, watch the transcript update, watch the audio-level meter move, and hear nothing at all. The session stayed connected. No error was raised anywhere. From the outside it looked like the agent had decided to stop talking.

The events kept arriving, which is what made it confusing. The client was receiving volume updates for audio it could not hear, so the obvious suspects — a dead socket, a failed synthesis call, a muted track — were all clearly innocent.

The first bug: a producer per interruption

Barge-in works by discarding the reply in progress the moment the caller speaks. Our implementation discarded rather more than that: it closed the mediasoup producer carrying the agent's audio, and opened a fresh one for the next reply.

On the wire that is legitimate. The problem was on the client. The SDK never subscribed to producer-closed, so it never removed the consumer for the producer that had just gone away. Dead tracks accumulated at the head of the MediaStream, and an <audio> element bound to a MediaStream plays the first audio track it finds — which by then was one that would never carry samples again.

One session had five producers in it. The browser was faithfully playing the first.

The fix was to stop destroying the producer at all. An interruption should stop the injection loop, not tear down the transport underneath it — the producer now lives for the whole session, and the client has one track that never goes stale. The SDK also handles producer-closed now, because a client that ignores a lifecycle event will find another way to be wrong about it later.

The second bug: a frozen RTP clock

With the producer surviving, audio came back — and immediately exposed a bug that the first one had been hiding. Synthesized frames are hand-built RTP packets, and the timestamp advanced only while frames were being sent. Between two utterances, the wall clock moved and the RTP clock did not.

A receiver reads that as audio it should have played several seconds ago. Its jitter buffer does what it is designed to do with late packets: drops them. Silence, with no error, on a stream that is technically healthy.

const nowMs = Date.now();
if (inj.lastFrameAtMs > 0 && nowMs - inj.lastFrameAtMs > FRAME_DURATION_MS * 3) {
  // A gap in wall-clock time is a gap in the RTP timeline too. Advance the
  // clock across it and mark the next packet as the start of a talkspurt,
  // or the receiver treats these frames as arriving late and drops them.
  const gapMs = nowMs - inj.lastFrameAtMs;
  inj.timestamp = (inj.timestamp + Math.round((gapMs * DST_RATE) / 1000)) >>> 0;
  inj.markerPending = true;
}

The marker bit matters as much as the timestamp. RFC 3551 defines it as the first packet of a talkspurt after a silence, and it is how a receiver knows to resynchronise its playout rather than assume it has fallen behind. We had never set it, because with a fresh producer per utterance the question had never come up.

What we took from it

The second bug had existed the whole time and was unreachable. Every utterance had arrived on a brand-new producer with a fresh timeline, so the frozen clock never had a gap to be wrong across. Fixing the first bug is what made the second one reachable — and if we had shipped that fix alone, the symptom would have looked identical.

In production since: one producer across 92 interruptions in a single session, and 14 clean RTP resynchronisations on gaps of six to eight seconds.