Summary
Audio frames arrive on a clock; transcript labels lack supplied frame alignment. Choose alignment, output history and available audio context separately.
Problem
One second contains 16,000 samples. Use uncentered 400-sample windows, a 160-sample hop and no padding. How many complete frames exist?
Separately, a reference says “turn left” while a hypothesis says “please turn right now”. Compute word error rate, then choose an architecture for captions that must appear before speech ends.
Calculate the Feature Clock
At this sample rate, the window is 25 milliseconds and the hop is ten. The exact full-window count is:
The last frame starts at sample 15,520 and ends before sample 15,920 under half-open indexing. The approximate 100-hops/second rate is not this clip’s frame count. Record padding, centering, normalization and encoder subsampling before interpreting output lengths.
Picture: What Advances When a Label Is Emitted?
Learning objective
RNN-T can emit another token without advancing the audio frame
Derivation
Which edge changes audio time rather than output position?
| Family | Alignment and output history | Streaming condition |
|---|---|---|
| CTC | Frame-label/blank paths summed; no explicit autoregressive label prefix | Streaming-compatible features and encoder |
| RNN-T | Blank advances time; token advances output with prediction-network prefix | Bounded-context encoder, state and decoding |
| Attention encoder-decoder | Prefix plus attended audio; ordinary offline version sees encoded utterance | Changed context/chunk policy and measured delivery |
CTC’s factorization does not prevent its encoder learning linguistic regularities. An output-history network is also not proof of complete language knowledge.
Evaluation
Align the worked text under a fixed normalization/reference policy:
| Reference | Hypothesis | Edit |
|---|---|---|
| - | please | Insertion |
| turn | turn | Match |
| left | right | Substitution |
| - | now | Insertion |
Word error rate divides edits by reference words:
Here , , and , giving . Insertions allow WER above 100%. Specify empty-reference handling; ordinary WER has a zero denominator there. Use CER when character units fit, and report critical entities alongside aggregate errors.
Change 1: Captions Must Arrive During Speech
New task. Deliver stable text before the utterance ends.
Picture and change. Bound future context, preserve streaming state and measure token/finalization delay from named audio events.
Check. CTC or RNN-T loss alone cannot make a full-utterance encoder stream. Chunking an offline model changes quality and boundary behavior.
Use when. Live delivery is required.
Change 2: Rare Names Drive Failures
New task. Domain entities matter more than average WER.
Picture and change. Check acoustic/token coverage and contextual bias/fusion with held-out terms and false-insertion tests.
Check. A stronger LM can prefer plausible words unsupported by audio.
Use when. Domain terms determine usefulness.
Failure: Fluent Text Is Called Acoustically Verified
A better language score can mask wrong words. Report audio grounding, text error, stability, delay and resources separately.
Transfer Check
Can rescoring a fixed n-best list recover a transcript absent from that list?
Reasoned Answer
No. Candidate generation or the search must change; rescoring only reorders existing candidates.
The pipeline
waveform -> features -> encoder -> decoder/LM -> transcript1. Audio features
Log-mel features apply short-time Fourier analysis, mel filtering and logarithms. MFCCs add a discrete cosine transform to log-mel energies. Learned waveform front ends are another option; specify sample rate, feature shape and time stride rather than assume universality. SpecAugment masks time/frequency, optionally warps time; noise mixing and speed perturbation require task-valid labels. Self-supervised wav2vec/HuBERT-style pretraining offers another data path, not guaranteed gains.
2. Acoustic / sequence model
Transformer/Conformer encoders can have different future-context policies. LAS/Whisper exemplify attention-based decoding; inspect the particular model’s context and deployment wrapper. Link CTC and RNN-T for objective derivations. Declare vocabulary, blank/EOS conventions and true post-subsampling lengths.
3. Language model fusion
| Route | What changes |
|---|---|
| Shallow fusion | Add during beam search under a specified score convention |
| Deep/cold fusion | Integrate LM hidden state into decoding |
| Rescoring | Reorder an existing n-best list/lattice; missing candidates stay missing |
External LMs are optional; compare entity gains, false insertions, costs and acoustic support.
Historical Decomposition
Hybrid systems combine lexicon/pronunciation, HMM state transitions, GMM/DNN acoustic estimates and language scores, often using alignment training. End-to-end objectives reduce some component boundaries, not every lexicon/LM or decoder choice. Neither family wins every data, resource, customization and latency setting.
Delivery Checklist
- State waveform, feature and subsampling units.
- Trace alignment, label history and audio context.
- Specify fusion/search and missing-candidate limits.
- Fix reference normalization, slices, WER/CER and entity checks.
- Measure onset/word-end/utterance-end delay, revision stability, resources and fallback.
Common confusions
Loss factorization, encoder context, linguistic evidence and runtime delivery are distinct contracts.
Related: Connectionist Temporal Classification (CTC), RNN-Transducer (RNN-T), Transformer architecture, Tokenization.