How Distillation Made Small AI Models Smarter

Small AI Models Are Winning: How Distillation Made "Smaller but Smarter" the New Standard

Google's Gemma 4, released April 2, 2026, scores 89.2% on the AIME 2026 math benchmark at 31 billion parameters — up from 20.8% for Gemma 3's 27-billion-parameter model a year earlier. That's not a small-model curiosity; it's evidence that the "bigger model, better results" assumption that defined the last three years of AI progress no longer holds by default. The mechanism behind the jump is distillation, and new research from Apple has just mapped out exactly when that technique works — and when it quietly stops paying off.

The gap that shouldn't exist, according to old assumptions

For most of the generative AI boom, capability tracked parameter count closely enough that "small model" was shorthand for "compromised model." That's no longer a safe assumption. Google's own model card for Gemma 4 shows the 26B mixture-of-experts variant reaching 88.3% on AIME 2026 while activating only about 3.8–4 billion parameters per token, and the 31B dense model posting an 80.0% score on LiveCodeBench v6 against Gemma 3 27B's 29.1%. Google also moved Gemma 4 to the Apache 2.0 license, a shift from the more restrictive Gemma Terms of Use that governed Gemma 3 and earlier releases, and the family had already passed 150 million downloads before its 12B variant shipped in June 2026.

Microsoft's smaller Phi line tells a similar story at a different scale. According to Phi-4-mini's official Microsoft model card, the 3.8-billion-parameter model scores 67.3% on MMLU (5-shot) and 88.6% on GSM8K — competitive with models several times its size from two years prior — while running in roughly 3GB of VRAM.

What's actually driving it: distillation, not magic

The consistent thread across these releases is knowledge distillation: training a smaller "student" model to replicate the output distribution of a larger "teacher" model, rather than training the student from raw data alone. The technique itself isn't new, but until recently it ran on intuition and trial-and-error rather than a predictive framework — the same state pretraining was in before the Chinchilla scaling laws gave labs a formula for balancing model size against training tokens.

That changed in early 2025, when a team at Apple led by Dan Busbridge published "Distillation Scaling Laws," a study that trained students ranging up to 800 million parameters against teachers of varying size and data budgets to build a predictive formula for distilled model performance. According to the paper, the resulting scaling law predicts a student's final loss from compute budget and its allocation between teacher and student with an error of roughly 1%.

The counterpoint the hype skips

Here's where the "smaller but smarter" narrative needs a real caveat, and it's the part most coverage leaves out: according to Busbridge and coauthors, distillation does not outperform standard supervised training once a lab has enough compute and is only training a single model from scratch. Their findings indicate distillation only becomes the better choice under two specific conditions — when a suitable teacher model already exists for other reasons, or when many student models need to be produced from one teacher, spreading the teacher's training cost across all of them. If neither condition applies, the paper's own conclusion is that plain supervised training remains the more compute-efficient path.

That reframes what's actually happening with releases like Gemma 4 and Phi-4: these aren't proof that distillation beats scale in some general sense. They're proof that Google and Microsoft already have frontier-scale teacher models sitting around — Gemini and larger internal checkpoints, respectively — which makes distilling from them close to free from a marginal-compute standpoint. A startup without an existing frontier model to distill from doesn't get the same deal.

Why this matters for cost, not just benchmarks

The practical pull toward small models isn't only capability — it's economics. Per pricing data compiled by Deploybase, small LLM API access now runs roughly $0.05 to $1.50 per million tokens, and running the same class of model locally can cost as little as $20 a month in server costs. Separately, guidance from Intuz on SLM deployment estimates that a private small-model endpoint serving 10,000 daily queries typically costs $500–$2,000 a month, against $5,000–$50,000 for an equivalent large-model API workload — a roughly 5–20x difference for comparable query volume. Those figures come from vendor and analyst estimates rather than a single audited source, so they're directional rather than exact, but the gap is consistent across multiple independent write-ups on the topic.

That cost gap is why "smaller but smarter" is becoming an operating standard rather than a niche choice: teams are increasingly routing routine, high-volume tasks to a distilled small model and reserving frontier-model calls for the subset of requests that genuinely need broader reasoning — an architecture sometimes called model cascading or task routing, rather than picking one model size for an entire application.

The nuance builders should actually take away

None of this means "small" means "tiny" anymore, either. Gemma 4's lineup now spans from a sub-2GB E2B edge model up through a 31B dense model that competes with systems many times its size, and Google has continued adding variants — a 12B multimodal "Unified" model arrived in June 2026, months after the initial April launch. The boundary between "small" and "large" model is becoming a spectrum of trade-offs rather than a hard cutoff, and mixture-of-experts architectures mean a model can carry tens of billions of total parameters while activating a small fraction per token, blurring the comparison further.

The honest takeaway is narrower than "small models have won": distillation from an existing frontier teacher is now a reliable way to compress a large share of that teacher's capability into a dramatically cheaper model, and companies that already have frontier models are shipping the proof of that regularly. But the underlying math Apple published says this is a benefit of already owning a teacher model, not a universal law that smaller training runs beat bigger ones. For teams evaluating whether to fine-tune, distill, or train from scratch, that distinction determines which approach is actually compute-optimal for their situation — and it's worth checking before assuming the smaller option is automatically the smarter one.