Skip to content
mlmentorship

Streaming automatic speech recognition

Emit transcripts with bounded latency using chunked encoders, monotonic alignment, endpointing, and stability-aware evaluation.

Published · 4 min read ·Specialist ·Advanced

Visual quick review

Visual first · depth when needed

Trace why streaming ASR can emit a token before speech ends while that token remains revisable, and distinguish the waits that determine first-token latency from endpoint finalization latency.

Preparing the visual…

Summary

Streaming ASR is the difference between a voice assistant that feels responsive and one that feels broken: it has to emit transcript while the user is still speaking, with bounded latency. The catch is that the model cannot attend to unlimited future context, so it trades recognition quality against emission latency, compute, and transcript stability. Offline ASR has none of those constraints, which is why a system that scores well offline can still feel terrible live.

Learning objective: trace why a token can appear before speech ends while remaining revisable, and distinguish first-token latency from finalization latency.

Learning objective

How can streaming ASR respond early without committing early?

A streaming transcript changes from provisional to final as audio arrives Audio arrives from left to right as the words play, their, and song, followed by a pause. After a bounded chunk and right-context wait, the recognizer emits the provisional words play the. Once the word song arrives, the recognizer revises the to their and adds song. Only after an endpointing wait during the pause does it mark play their song final. Thus first-token latency ends before speech ends, while finalization latency includes endpointing. AUDIO ARRIVES LEFT TO RIGHT · CONCEPTUAL, NOT TO SCALE /play/ /their/ /song/ pause · endpoint wait CHECKPOINT 1 · EARLY PARTIAL chunk + bounded right context are available play the… PROVISIONAL future audio may revise text CHECKPOINT 2 · LATER PARTIAL play their song the → their REVISED speech stops; endpoint must fire FINAL RESULT play their song FINAL
Read it this way: the first partial result appears once a chunk and its bounded right context are available, so time to first token can end while speech is still arriving. That text remains provisional: later audio changes the to their. Finalization latency ends only after the pause triggers endpointing and remaining processing completes. Measure both milestones and the revision between them. Original schematic informed by He et al. (2019), Shangguan et al. (2020), and Li et al. (2020).

Architecture choices

  • Chunked encoder: processes bounded windows with cached state or limited lookahead.
  • RNN-T / transducer: learns a monotonic alignment and emits tokens incrementally.
  • CTC: a simple monotonic objective, often paired with streaming beam search and a language model.
  • Streaming attention: constrains or chunks encoder-decoder attention so it does not need the whole utterance.

Latency components

End-to-end latency is more than model runtime: audio chunk size, feature extraction, model compute, lookahead, decoding, endpointing, network, and client rendering all add up. Reporting model runtime alone hides the delay the user actually feels.

Endpointing and partial stability

The system has to decide when speech has ended. Aggressive endpointing lowers latency but clips pauses; conservative endpointing feels sluggish. Partial hypotheses may also revise earlier words, so measure flicker or edit overhead alongside final word error rate.

Evaluation

  • Final and streaming word error rate
  • Time to first token and finalization latency
  • Partial stability / revision rate
  • Real-time factor and peak compute
  • Slice quality by accent, noise, device, language, and speech rate

In an interview

  1. Clarify the latency and quality targets.
  2. Choose a monotonic or chunked architecture.
  3. Account for every latency component, not just the model.
  4. Discuss endpointing and partial-transcript stability.
  5. Design fallback, offline rescoring, and slice monitoring.

Common confusions

  • “Streaming means batch size one.” It means bounded future context and incremental output; you can still batch across streams.
  • “WER captures latency.” It says nothing about when words appear or how often they change.
  • “More lookahead is free quality.” Every frame of lookahead is user-visible latency.

Related: automatic speech recognition, RNN-T, and CTC.

Found an error or stale claim? Report a correction or send a private note.