Version: @livekit/agents 1.5.2 (say path unchanged in 1.5.3 dist)
Setup: TTS with aligned transcripts enabled (cartesia + add_timestamps: true; useTtsAlignedTranscript default true). LLM-generated turns caption in sync (aligned mode working). Scripted turns via session.say(text) do not.
Actual: say() captions stream at the estimated fixed pace (STANDARD_SPEECH_RATE 3.83 hyphens/s), fall progressively behind actual speech, and when playout finishes the TranscriptionSynchronizer flushes the entire remaining text in one visible chunk. On a ~15s scripted greeting this reads as several seconds of trailing captions, then a wall of text at the end.
Why: AgentActivity.ttsTask (the say pipeline, agent_activity.js ~L1753) tees the raw text straight into transcriptionNode and never consults ttsGenData.timedTextsFut — even though performTTSInference runs in that same path and the TTS frames DO carry USERDATA_TIMED_TRANSCRIPT. The aligned swap exists only in the generation path (~L2075).
Expected: say() turns use the same aligned timed-transcript input as generations when the TTS advertises alignedTranscript — scripted openers/goodbyes are exactly the turns where text is known upfront and users notice desync most.
Context: we hit this on a production voice-screening product where the scripted opener is the first thing every candidate sees; the mismatch reads as "captions broken" even though the rest of the call is aligned.
Version: @livekit/agents 1.5.2 (say path unchanged in 1.5.3 dist)
Setup: TTS with aligned transcripts enabled (
cartesia+add_timestamps: true;useTtsAlignedTranscriptdefault true). LLM-generated turns caption in sync (aligned mode working). Scripted turns viasession.say(text)do not.Actual:
say()captions stream at the estimated fixed pace (STANDARD_SPEECH_RATE3.83 hyphens/s), fall progressively behind actual speech, and when playout finishes theTranscriptionSynchronizerflushes the entire remaining text in one visible chunk. On a ~15s scripted greeting this reads as several seconds of trailing captions, then a wall of text at the end.Why:
AgentActivity.ttsTask(the say pipeline,agent_activity.js~L1753) tees the raw text straight intotranscriptionNodeand never consultsttsGenData.timedTextsFut— even thoughperformTTSInferenceruns in that same path and the TTS frames DO carryUSERDATA_TIMED_TRANSCRIPT. The aligned swap exists only in the generation path (~L2075).Expected: say() turns use the same aligned timed-transcript input as generations when the TTS advertises
alignedTranscript— scripted openers/goodbyes are exactly the turns where text is known upfront and users notice desync most.Context: we hit this on a production voice-screening product where the scripted opener is the first thing every candidate sees; the mismatch reads as "captions broken" even though the rest of the call is aligned.