Quantization (of speech models)

Quantization shrinks a neural speech model by storing its weights at lower numeric precision — for example INT8 instead of 32-bit floats — cutting memory use and speeding up inference with a small accuracy cost.

Quantization is why large models run on laptops and phones at all: an INT8 Whisper variant needs a fraction of the memory of the original. The accuracy loss is usually small on clear audio and shows up first on hard audio and rare words.

Related terms