Quantization is storing a model's numbers at lower precision, such as 4-bit or 8-bit instead of 16-bit, to cut memory use and speed up inference. A quantized model is smaller, so it can fit on devices with limited memory and needs less data movement for each step.
It is one of the main ways large language models are made to run on phones and laptops, because memory capacity and bandwidth are often the limiting factors there.
The trade-off is quality. Lower precision can reduce accuracy, and the effect varies by model, task and method. Some models lose little at 8-bit and more at 4-bit or below, and results on one benchmark may not carry over to your use. Hardware support also matters, since a chip may run some formats much better than others.
When choosing a quantized model, compare its quality on tasks that resemble your real use, check which formats the target hardware supports well and keep the original model available so that you can compare outputs.