Quantization stores and computes numbers with fewer bits. A model trained with 16-bit numbers can be converted to 8-bit or 4-bit values, which cuts the memory needed by half or by three quarters. Smaller models load faster, fit on cheaper hardware and often run faster, because less data has to move through memory.
The price is some loss of precision, which can reduce quality. The size of the effect depends on the method, the model and the task. Careful techniques, such as grouping values with shared scaling factors or keeping sensitive layers at higher precision, keep the loss small. Quantization can be applied after training, or the model can be trained with it in mind, which usually gives better results.
Very low precision tends to hurt more on tasks that need accuracy, such as math and code, and on small models.
Test the quantized version on your own tasks and compare it with the original, and choose the lowest precision that gives acceptable quality.