How AI Composes Music: From Training Data to a Finished Track

The phrase “the AI wrote a song” hides a pipeline with several distinct stages. Understanding them makes it much easier to judge what a system can and cannot do — and to hear where a track came from.

Stage one: listening at scale

Every generative music model begins with a corpus of recordings. During training, the model is shown excerpts and asked to predict what comes next, over and over, adjusting its internal parameters each time it is wrong. No labels are required for this: the audio is its own answer key.

What the model acquires is not a memory of the songs. It is a compressed sense of regularity — which timbres follow which, how rhythm tends to sit against harmony, how a phrase usually resolves. This is also why training data debates matter: the statistical fingerprint of a corpus can surface in output even when no single track is reproduced.

Stage two: turning sound into something predictable

Raw audio is a very long sequence of numbers, and predicting it directly is expensive. Most systems solve this by compressing audio into discrete tokens — a codebook of short sound units — and then learning to predict those tokens instead. The model composes in this compressed space, and a decoder turns the tokens back into sound.

The practical consequence is subtle but audible: compression smooths detail. Cymbals and breath often lose their edge first, which is one reason AI tracks can sound slightly polished and slightly airless at the same time.

Stage three: conditioning

A model that only predicts the next token would produce generic music. Conditioning is how a direction is imposed: a text description, a genre label, a reference audio clip, a tempo, a key. The prompt is not an instruction the system understands the way a person would; it is a vector that steers probability.

This is why prompt wording matters more than it seems. “Warm analog jazz trio, brushed drums, upright bass, late night” and “jazz” do not point at the same region of the model’s learned space.

Stage four: sampling, and why streams differ

Generation is stochastic. At each step the model produces a distribution over possible next tokens, and a sampler picks one. A low temperature makes conservative choices; a higher one allows surprise. Identical prompts therefore produce different results — which is precisely what makes a generative radio stream able to run for hours without looping.

Stage five: the human decisions

This is the stage marketing usually omits. Someone chooses the prompt, the seed, the length, and which of many generated candidates is worth keeping. Someone decides whether the bass is too loud, whether the transition between pieces works, whether the station’s overall texture holds together at three in the morning.

That work is curation, and it is where most of the craft now sits. The model is prolific and indifferent; the human is the one with taste and a deadline.

What this means when you listen

Knowing the pipeline gives you better ears. A track that develops structurally probably involved symbolic generation or deliberate editing. A track that holds a mood beautifully but never arrives anywhere is closer to raw audio generation. A mix where the low end is soft and the highs are glassy has likely passed through a compressed token space.

None of these are defects. They are signatures — the same way tape hiss or a particular reverb tells you something about how a record was made.