Streaming automatic speech recognition

Summary

Show tentative words before speech ends, but define when they may change and when the transcript becomes final.

Problem

One synthetic trace uses a shared clock, not measured service guarantees:

Event	Time
Speech starts	0 ms
First word ends	300 ms
First displayed partial includes that word	460 ms
Speech ends	1,200 ms
Final text appears	1,550 ms

The first partial is “play the”; later audio changes it to “play their song”. Compute first-text, word-emission and finalization delays without changing their anchors. Separately, a ten-second clip needs two seconds of processing. Does that compute ratio certify the live experience?

Picture: Useful Early Text Can Still Change

Learning objective

Visible partial words remain revisable until the finalization rule permits commitment

Derivation

compare partial, revised and final text, not geometric distances on this schematic clock. First text precedes speech end; finalization includes endpointing and remaining delivery.

Check: Why can play their song be visible before the system is allowed to mark it final?

Read it this way: The partial hypothesis changes as audio arrives. Only the declared finalization rule turns a provisional token sequence into committed output.
Can the first visible I scream be treated as final words?

Calculate Distinct Delays

Measurement	Calculation	Result
First text from speech start	460 - 0	460 ms
This word’s emission from word end	460 - 300	160 ms
Finalization from speech end	1,550 - 1,200	350 ms
Separate clip’s real-time factor	2 / 10	0.2

Real-time factor is processing duration divided by audio duration. It does not bound queueing, network delivery, revision frequency or finalization tails. Measure stable-prefix time separately; an early partial is not necessarily stable.

Endpointing and partial stability

For word-token indices, keep “play” stable and replace range $[1, 2)$ of the first partial with [“their”, “song”]. Advance the version; later mark the unchanged full transcript final. Declare replacement ranges, stable-prefix length, endpoint state and reconnect/reset behavior. Reject stale updates or changes to committed prefixes.

Aggressive endpointing can clip pauses; conservative endpointing delays finalization. Keep tentative state reversible and measure false endpoints alongside delay. Provisional words do not authorize consequential actions. Even final text can require separate semantic, permission and confirmation checks.

Change 1: More Future Context

New task. Additional lookahead helps recognition but threatens the deadline.

Picture and change. Measure actual critical-path delay, quality and cache cost under live load.

Check. Overlap changes incremental delay; model runtime alone is insufficient.

Use when. Context and responsiveness conflict.

Change 2: Pauses Cause Premature Finalization

New task. The endpoint closes an unfinished request.

Picture and change. Adjust endpoint/stability policy and test pauses, rushed speech, noise and interruption.

Check. Lower mean delay can hide worse false-endpoint and task-completion rates.

Use when. Commitment errors dominate.

Evaluation

Group	Required observation
Recognition	Final/streaming WER with fixed normalization
Delay	Named first-text, word, stable-prefix and final anchors, including tails
Stability	Revisions/edit overhead and false endpoints
Resources	Real-time factor, peak compute, queue/cache behavior
Slices	Accent, noise, device, language and speech rate

Failure: Good Final WER Arrives Too Late

Accurate final text can miss a live task’s deadline. Acting on a subsequently revised provisional word creates a different failure.

Transfer Check

Does streaming require batch size one?

Reasoned Answer

No. Independent live streams may be batched while preserving bounded future context, state and output deadlines.

Architecture choices

Route	Runtime contract
Chunked encoder	Bounded windows/lookahead, persistent caches
RNN-T	Time/token decoding with streaming-compatible encoder/state
CTC	Frame objective with compatible front end, encoder and search
Streaming attention	Explicitly constrained/chunked context, not full-utterance dependence

Latency components

Trace chunk buffering, feature extraction, lookahead, compute, decoding, endpointing, queueing, network and client rendering. Overlap matters; measure end-to-end distributions from joint traces, not sums of stage percentiles. Offline processing still has resource and deadline constraints even without live-context restrictions.

PartialUpdate
	utterance_id, stream_epoch, version, token_replace_range
	tokens, stable_prefix_length, final, endpoint_state
	audio_event_time, display_time, model_policy_version

Bind replacements to the utterance/epoch and increasing version. On reconnect, explicitly resume acknowledged state or reset; do not mix caches across utterances. Specify stale/missing updates, bounded queues and safe timeout/fallback behavior. Later offline rescoring needs a distinct result identity and cannot silently rewrite an already consumed final result.

In an interview

Fix latency anchors and quality/stability targets.
Declare context, chunk, cache and decoding policy.
Trace buffering through client display.
Define tentative, stable, endpoint and final states.
Test fallback, rescoring, interruptions and slice monitoring.

Common confusions

Low WER is not a delay bound; extra context changes the actual critical path, not necessarily every frame additively.

Sources

Related: automatic speech recognition, RNN-T, and CTC.