During training, which can take weeks, the system saves checkpoints at intervals. If a machine fails, training resumes from the last one instead of from the start. Checkpoints can also include the optimizer's state, which is needed to continue training, or only the weights, which are enough to run the model.
They are also used to compare progress, to pick the best version and to release models at different stages, such as a base model and an instruction-tuned one.
Files in some older formats, notably those based on Python's pickle, can execute code when loaded, so a downloaded checkpoint from an untrusted source is a security risk.
Keep several checkpoints, record which data and settings produced each one, verify downloaded files and prefer safe formats when loading models from outside.