Why feeding AI its own output could lead to a digital collapse

Why feeding AI its own output could lead to a digital collapse
Image: Intellect (generated)

In short: As generative models increasingly scrape the web, they are beginning to train on the outputs of their predecessors. This recursive loop creates a degenerative process known as model collapse, where AI loses its grasp on reality, eventually forgetting the nuances of the human world.

Model collapse is a degenerative process where generative models, such as LLMs, lose the ability to produce diverse, high-quality output because they are trained on uncurated synthetic data. This phenomenon, sometimes colloquially called "AI inbreeding" or "Habsburg AI," occurs when a model's training set is polluted by the generations that came before it. The collapse typically unfolds in two stages: early model collapse, where the model begins losing information about the "tails" or minority data of a distribution, and late model collapse, where the model's output converges to a point estimate with very little variance, losing significant performance and confusing fundamental concepts.

The mechanism behind this decay involves three compounding errors. First, statistical approximation error arises from finite sampling, where information is lost during resampling. Second, functional expressivity error occurs because neural networks have limited capacity to approximate complex distributions, potentially assigning zero likelihood to valid data. Finally, functional approximation error stems from the limitations of learning procedures, such as the structural bias of stochastic gradient descent. In complex models, these errors often cascade, leading to a rapid decline in lexical, syntactic, and semantic diversity.

The long-term implications remain a subject of intense debate. Some researchers warn that the proliferation of "AI slop" on the internet poses a fundamental threat to future development. However, others argue that if synthetic data accumulates alongside human-generated data, the collapse might be avoided. The stakes are high: a 2024 study demonstrated that after just nine generations of training on its own output, a model produced nonsensical, repetitive strings of text. As the internet becomes increasingly populated by AI-generated content, preserving access to original, human-produced data becomes crucial for the survival of intelligent systems.

Opens the Intellect+ feed with this card first. Swipe up for the next one.

Sources

To cite this card, link to https://intellectplus.co.uk/app/technology-model-collapse.

How we source, check and correct cards: Editorial standards.

General information, not professional advice. Spotted an error? Flag the card or see corrections and complaints.

Intellect Plus is a visual educational discovery platform for exploring fascinating facts and ideas. Brain Food, Not Brain Rot. What is Intellect Plus?

Related

Finding something worth knowing…