Synthetic Data and Model Collapse: What the Research Actually Shows

Synthetic data does work — but only under one specific condition that most coverage skips: it has to be added to the real data a model already has, never used to replace it. Peer-reviewed research from 2024–2025 shows that "model collapse," the degenerative spiral widely blamed on synthetic training data, is a real and reproducible phenomenon under one training pattern (repeatedly replacing data with each generation's own synthetic output) and something closer to a manageable engineering constraint under another (accumulating synthetic data alongside a fixed, non-shrinking pool of real data).

The panic had a real paper behind it

The "AI is running out of data" narrative and the "AI trained on AI will go insane" narrative are actually two separate claims that got fused together in public discussion, and it's worth separating them again. The data-exhaustion claim traces to Epoch AI's widely cited projection that the stock of high-quality public text online — estimated at roughly 300 trillion tokens — would be effectively exhausted by frontier labs' training needs around 2026, given that models like GPT-4 were already trained on 6-13 trillion tokens and each new generation demands more. The collapse claim comes from a different source: a 2024 Nature paper by Shumailov et al., "AI models collapse when trained on recursively generated data," which showed that when a model is trained iteratively on its own generated output rather than on real data, it progressively loses the rare, low-probability "tail" events in the original data distribution — after enough generations, outputs converge toward a narrow, repetitive mean and stop representing the true variety of the original data. Those two findings together produced the specific fear GetCoreTech readers have likely heard: that the industry is being forced toward synthetic data by a genuine data shortage, and that the fix itself is a trap.

The finding that actually resolves the argument

The critical piece of follow-up research came from Gerstgrasser et al., presented at COLM 2024, and it isolates exactly what causes collapse versus what doesn't. According to that paper, the deciding factor isn't whether synthetic data is used at all — it's whether each new generation of training data replaces the previous real data or accumulates alongside it. When synthetic data replaces prior data at each iteration, test error grows and collapse follows, matching Shumailov's original result. When synthetic data instead accumulates on top of a real-data anchor that's never removed, the paper reports that test error stays bounded across iterations instead of climbing, and this held across transformer language models, variational autoencoders, and diffusion models in their tests. That single distinction, replace versus accumulate, is doing more explanatory work than most consumer coverage of "model collapse" acknowledges. A subsequent NeurIPS workshop paper by Kazdan et al., "Accumulating data avoids model collapse," reinforced the same conclusion, though it also noted a real trade-off: accumulating too much synthetic data relative to real data can still produce a model that underperforms one trained purely on real data, even without full collapse. Accumulation prevents catastrophe; it doesn't make synthetic data a free substitute for real data.

A 2025 paper pushed back on the panic itself

Separately from the mitigation research, a 2025 position paper by Schaeffer, Kazdan, Arulandu, and Koyejo — "Position: Model Collapse Does Not Mean What You Think" — argued the public narrative around collapse had outrun the evidence in a different way. The authors hand-annotated 28 prior publications and found the term "model collapse" had been used to describe at least eight distinct, sometimes conflicting phenomena. Their central claim is that many of the most alarming collapse results came from experimental setups that don't match how frontier labs actually train models — specifically, setups that assume all real data is deleted and never reused after each generation, which isn't standard industry practice. Their conclusion isn't that collapse is fake; it's that the specific catastrophic version of it requires assumptions realistic training pipelines don't make, while a narrower, real risk — the loss of distributional tails and reduced output diversity — deserves more attention than it's getting relative to the more dramatic version of the story.

What "using synthetic data correctly" looks like in production already

This isn't a hypothetical distinction labs are still working out — it's already how major synthetic-data-heavy models get built. Microsoft has stated that Phi-4 was trained on roughly 400 billion tokens of synthetic data, generated and filtered specifically as a curated supplement rather than a wholesale replacement for real-world text, following the same "textbook-quality" data philosophy Microsoft used for earlier Phi models. A separate 2025 systematic study on large-scale pretraining, examining single-round synthetic data use rather than the recursive multi-generation setups that produce collapse in lab experiments, reported that using rephrased synthetic data during pretraining did not reproduce the degradation patterns predicted by earlier collapse literature — consistent with the idea that one clean, well-filtered generation of synthetic data behaves very differently from an uncontrolled multi-generation feedback loop.

The part of the story that's genuinely still unresolved

None of this fully closes the loop on the internet-scale version of the problem. As more AI-generated content gets published and subsequently scraped into future training sets by different companies with no coordination and no shared "real data anchor," the tidy replace-versus-accumulate distinction gets harder to enforce, because no single lab controls what fraction of the open web is real versus synthetic anymore. Researchers tracking this note that AI-generated text already makes up a measurable and growing share of newly published web content, and the accumulation research that resolves collapse in controlled lab settings assumes a level of control over the training pipeline that doesn't exist when the "pipeline" is the public internet.

The actual answer to the headline question

So: does synthetic data work, or does it just mask a real shortage? Both, but not in tension with each other the way the framing implies. It works, with strong published evidence, as a tool for expanding a fixed real-data foundation — provided the real data anchor is preserved rather than discarded, and provided teams are watching for the narrower, real risk of eroding tail diversity rather than assuming any use of synthetic data is inherently dangerous. It does not work as a way to make a genuine data shortage disappear entirely, because the mitigation research that prevents collapse explicitly depends on real data still being present in the mix, not phased out. Teams evaluating synthetic data for their own training or fine-tuning pipelines have a concrete, checkable question to ask before committing to a vendor or an internal pipeline: is real data being preserved as a permanent anchor across iterations, or is it being quietly phased out in favor of synthetic volume? The published research says that answer, more than the mere presence of synthetic data, is what determines the outcome.

FAQ

Q: What is AI model collapse, in simple terms? A: Model collapse refers to progressive performance degradation that occurs when a generative model is repeatedly trained on data generated by earlier versions of itself (or similar models), rather than on real-world data, especially when that real data is discarded rather than retained across training generations. Q: Is model collapse an actual proven phenomenon, or is it overstated? A: Both, depending on the setup. A 2024 Nature paper demonstrated it empirically under a "replacement" training pattern, and this has been reproduced across model types. A 2025 position paper argues the popular narrative overstates the risk by drawing on unrealistic experimental assumptions, while agreeing that a narrower risk — the erosion of rare, low-probability examples in the data — is real and under-addressed. Q: What's the difference between "replacing" and "accumulating" data, and why does it matter so much? A: In a replacement setup, each new model generation trains only on the previous generation's synthetic output, and real data is discarded. In an accumulation setup, each generation trains on all real data plus all synthetic data generated so far, so the real-data anchor never shrinks. Research presented at COLM 2024 found that accumulation keeps test error bounded, while replacement causes it to grow without limit. Q: Does using synthetic data ever hurt a model even without causing full collapse? A: Yes. Research on the accumulation approach found that using too much synthetic data relative to real data can still produce a model that underperforms one trained on real data alone, even without the model actually collapsing — accumulation prevents catastrophe, but it isn't a guarantee of equal or better quality. Q: Are companies actually using large amounts of synthetic data in production models today? A: Yes. Microsoft has stated that its Phi-4 model was trained on roughly 400 billion tokens of synthetic data as part of a deliberately curated data strategy, and synthetic data generation is now a standard offering across major cloud AI platforms.