WhisperX and forced alignment, explained
WhisperX exists because of a gap most people only discover after their first transcription project: Whisper produces excellent text but approximate timestamps. Segment times can drift by a second or more, and Whisper does not natively emit per-word timing at all. WhisperX layers a second, smaller model on top — a forced aligner — that takes Whisper's text and pins every word to the audio within tens of milliseconds. If you are building karaoke captions, clickable transcripts, clip extraction, or dubbing workflows, this page is the mechanism.
The 60-second version
Whisper listens to 30-second windows and writes text with rough segment boundaries. Forced alignment is a different, easier task: given audio and its known text, compute exactly when each word (or phoneme) occurs. WhisperX runs Whisper for the words, then a phoneme-recognition model (a wav2vec2 variant) for the timing, and merges the two. The result: word-level timestamps accurate enough to cut audio mid-sentence without clipping a syllable.
Why Whisper's own timestamps drift
Whisper is a sequence-to-sequence transformer: it consumes a mel spectrogram of a 30-second window and generates tokens, some of which are timestamp tokens the model predicts like any other token. Nothing in the architecture forces those predictions to line up with the acoustic evidence. On clean speech they are usually close; after silences, music, or within long fluent passages they wander — famously enough that whole toolchains exist to correct them.
Forced alignment attacks the timing problem directly. The aligner already knows the transcript, so it does not need to decide what was said — only when. It scores each audio frame against the expected phoneme sequence and finds the optimal path through them. Removing the "what" makes the "when" almost trivially accurate.
What WhisperX actually runs, step by step
- VAD pre-segmentation. A voice-activity model chops the audio into speech regions, which kills the classic long-silence hallucination failure and enables batched inference — the reason WhisperX reports far faster-than-realtime throughput on GPUs.
- Whisper pass. Each speech region is transcribed normally. This produces the words.
- Alignment pass. A phoneme model (wav2vec2 fine-tuned per language) force-aligns the words against the audio, emitting start/end times per word.
- Optional diarization. A speaker-diarization model assigns each aligned word to a speaker turn.
The alignment model is per-language — English, Spanish, German, French and a growing set have strong aligners; languages without one fall back to Whisper's native timing. Numbers and currency amounts sometimes misalign because the spoken form ("twenty twenty-six") and written form ("2026") disagree — a known caveat, worth a spot-check when timing matters on figures.
When you genuinely need it
- Karaoke / word-highlight captions — the social-video style lives or dies on per-word timing; segment timing looks broken.
- Interactive transcripts — click a word, seek the player there. Off-by-a-second reads as a bug.
- Clip extraction — cutting on word boundaries without swallowing syllables.
- Dubbing & lip-sync — phoneme-level timing is the whole game.
- Compliance review — citing the exact second a phrase was spoken.
If none of those apply — meeting notes, article drafts, searchable archives — plain Whisper timing is fine, and the extra pipeline is complexity you do not need.
Run it yourself vs. get it from a service
Running WhisperX means a Python environment, a GPU for reasonable speed, model downloads per language, and glue code for output formats — a fun afternoon for an engineer, a wall for everyone else. The pipeline view: pip install whisperx, load model, transcribe, align, (optionally) diarize, serialize to SRT/VTT/JSON. It is genuinely good software and the de-facto standard for research-grade word timing.
If you just want the output, Whipscribe runs word-level timestamps and speaker labels as part of every transcription — upload, get back text with per-word timing, captions and speakers, no environment to maintain. Same science, zero setup. (Our glossary entry on forced alignment covers the concept in two paragraphs if that is all you needed.)