AI Without The Hype · 06 / 08
How AI Learns to Create Images It Has Never Seen Before
Generating a convincing image is one problem solved two opposite ways, a tidy map you sample from and a forgery contest where quality emerges from competition.
Key takeaways
- Image generation is one problem, learning the pattern behind real data well enough to make new examples, not copying.
- Two early methods solved it in opposite ways, a stable mapmaker and a sharp but temperamental forgery contest.
- The business value is manufacturing data you need but do not have, for stress testing, fraud defence, and privacy.
The one problem underneath all image generation
Strip away the jargon and every image generator is chasing the same thing. It wants to learn the pattern that produced a set of real images, the typical shapes, textures, and arrangements, so precisely that it can produce a new image that fits the pattern but was never in the training set. The catch is that this pattern, for something as rich as a photograph, is impossibly complicated to write down directly. So instead of writing it down, these models learn to imitate it.
One line separates a real generator from a cheat, and it matters. Memorising is not generating. A model that simply stores and replays its training pictures has learned nothing about how images are built. It has overfit, in the language of an earlier chapter. The honest test is whether it can produce something new and plausible that it has never seen. Think of a risk team that wants realistic but entirely fictional transaction records for stress testing. Copying real customer records is a privacy problem and gives no new variety. What they actually want is to learn the shape of real transactions well enough to invent fresh ones. That is the generative problem exactly.
In plain terms: learning the pattern
Every generative model tries to learn the hidden distribution behind a pile of examples, meaning the full sense of what is typical and what is rare. Get that right and creating something new is easy, you simply draw a fresh sample that fits. Get it wrong, or merely memorise the examples, and the model can only repeat what it was shown. Creating truly new work is the whole point.
The first way, draw a map and sample from it
The first approach turns the data into a navigable map. Imagine filing every training image at a location based on its features, so similar images sit near each other. A naive version files each image at a single pinpoint, which is useless for creating, because the space between the pins is empty and picking a random spot gives you noise. The clever fix is to file each image not as a pin but as a soft, fuzzy region. Do that for everything and the regions overlap into one continuous, populated map with no dead gaps. Now creating is simple. Pick any point in a populated area, and a decoder turns that point back into a fresh image.
This mapmaker approach, known as a variational autoencoder, is stable and predictable to train, and its map is genuinely useful. You can glide from one kind of image to another and watch it change smoothly, because nearby points mean similar things. Its one weakness is sharpness. Because each spot on the map stands for a small neighbourhood of possibilities, the decoder tends to produce a slight average of them, and outputs can look a touch blurry. The foundations of this method were set out by Kingma and Welling in 2013.
The second way, stage a forgery contest
The second approach throws out the map entirely and sets up a rivalry. Two networks are trained against each other. One, the forger, starts from random noise and tries to produce a convincing fake image. The other, the detective, is shown a mix of real images and the forger's fakes and has to call each one real or fake. After every round, both get feedback. The forger learns which of its tells got caught. The detective learns which fakes slipped past. Neither was ever told what a good image looks like. Quality emerges purely from the competition, the way a counterfeiter and an authentication team would sharpen each other over time.
This is the generative adversarial network, and when it works, the results are strikingly sharp. The endgame is a standoff. The forger becomes so good that the detective can do no better than a coin toss, right about half the time, because the fakes are genuinely indistinguishable from real images. The core idea was introduced by Goodfellow and colleagues in 2014 and set off a decade of rapid progress in synthetic imagery.
Why the contest is famously hard to run
The rivalry that makes this method powerful also makes it temperamental, and the failures are worth recognising. First, the forger can get lazy. If it stumbles on a single fake that reliably fools the detective, it may just produce that one thing over and over, so a system that should make ten different digits makes only one. Second, the detective can become too good too soon. If it starts catching every fake with total confidence, the forger stops getting any useful hint about how to improve and stalls. Third, because the two are pulling in opposite directions, their scores can swing back and forth forever without ever settling. Reading a falling score as progress is a classic trap here. Holding that balance is the entire craft of training one of these systems.
Same goal, two machines. Slide between them and compare what each is good and bad at. If scripting is off, the table below tells the same story.
| What you care about | The mapmaker | The forgery contest |
|---|---|---|
| Training stability | Steady and predictable | Temperamental, needs care |
| Sharpness of output | Slightly blurry | Crisp and realistic |
| Variety across results | Broad and reliable | Can collapse to a few looks |
| A navigable map of the data | Yes, smooth to explore | No, harder to steer |
Why a business reader should care
The reason these models matter outside a research lab is blunt. They manufacture data you need but do not have. A credit risk team can generate synthetic transaction records to stress-test a model against a scenario that has never actually happened, with full control over the volume and the conditions, and without exposing one real customer. A fraud team can generate plausible new fraud patterns, not copies of old ones, and train its detector to catch attacks it has not met yet. And because a well-built generator captures the statistics of real data without storing any real record, institutions can share realistic synthetic datasets where sharing the real thing would be impossible. The common thread runs through all of it, the data you need does not exist in the form you need it, and a generator makes it.
Why this matters even if you never touch the code
These two methods were the first generation, and the useful lesson is not which one won. It is that the same goal admitted two completely different designs with different strengths, a stable, navigable map against a sharp, hard-to-tame contest, and the right choice follows the shape of the problem rather than the fashion of the moment. That is the same judgement call that runs through the whole of this series on matching the method to the problem. These early models also set the benchmark, and exposed the failure modes, that the next generation of image AI was built to beat. That is where the series turns next.
Frequently asked questions
How does AI generate an image of something that does not exist?
It learns the underlying pattern behind a large pile of real images, then produces a new example that fits the pattern. It is not copying. A model that memorised its training pictures has not learned to create, it has only learned to repeat. The real test is whether it can make new, plausible images it has never seen before.
What is the difference between a VAE and a GAN?
They solve the same problem two opposite ways. A VAE builds a tidy, navigable map of the data and generates by sampling a point on it, which is stable but slightly blurry. A GAN stages a contest between a forger and a detective, and quality emerges from the competition, which is sharp but harder to train. One is a mapmaker, the other is a rivalry.
Why are GANs so hard to train?
Because two networks are pulling against each other, so there is no single score that simply falls as training improves. Three problems recur. The forger can find one fake it repeats forever, the detective can get so good so fast that the forger stops learning, and the two can oscillate without ever settling. Holding that balance is the whole craft.
What are generative models actually used for in business?
They create data you need but do not have. Teams generate synthetic transaction records to stress-test risk models against scenarios that never happened, invent plausible new fraud patterns to harden fraud detectors, and share synthetic datasets that keep the statistics of real data without exposing a single real customer. The common thread is data that does not exist in the form you need.
Which is better, a VAE or a GAN?
Neither is better overall. A VAE wins when stability, a structured map of the data, and reliable variety matter more than crispness. A GAN wins when sharp, realistic output is the priority and the team can handle unstable training. As with earlier choices in AI, the right pick follows the shape of the problem, not fashion.