Why the same prompt gives a different answer every time
We asked one AI model the same question ten times at temperature 0, its least random setting, and got six different answers.
On this page
Ask an AI model to name a sourdough bakery and you might get “The Daily Crumb”. Ask again, word for word, and you get “The Daily Loaf”. Nothing broke. The model was built to do that, and most of the time it helps: a second answer is a second idea.
The trouble starts when you need the same answer twice. Think of a product description that has to match last week’s, a data-extraction step in a workflow, or a picture you want to tweak without losing it. AI answers vary because the model picks each word by weighted chance. You can turn that chance down, but on the biggest cloud models you can no longer switch it off. This post explains where the randomness comes from, shows what happened when we ran the same prompts dozens of times, and ends with what to do when you need a repeatable result.
The model picks every word with weighted dice
A language model writes one small chunk of text at a time, called a token; our token explainer covers them. At each step it does not settle on one next word. It scores every possible next token, turns the scores into odds, and then draws one. The draw is the random part.
We asked Qwen3.8 27B, an open model, to name a sourdough bakery and read those odds straight off the API. After “The Daily”, it gave “Loaf” a 57% chance, a word starting “Cr” (Crumb or Crust) 33%, and “Rise” 7%. A run that draws “Cr” ends with a different name, and every word after that is chosen against a different text.
Temperature is the setting that reshapes those odds before the draw. Below 1, the favourite gets stronger: at 0.5, Loaf climbs to 74% and Rise nearly disappears. At 0, the model always takes the top choice. Above 1, the odds flatten and long shots get a real chance. Two related settings, top-k and top-p, trim the list instead. Top-k keeps only the k likeliest words. Top-p keeps the smallest group of words whose odds add up to p.
Why not always take the top word? Because the result reads badly. In 2019, Ari Holtzman and colleagues showed that choosing the most likely text produces writing that is “bland and strangely repetitive”, and proposed top-p sampling, which they called nucleus sampling, as the fix. That is why chat apps sample by default. Ollama, a popular tool for running models locally, ships with temperature 0.8, top-k 40 and top-p 0.9. DeepSeek’s API docs suggest 0.0 for coding and maths, 1.3 for conversation and translation, and 1.5 for creative writing.
Temperature 0 makes short answers repeat
To see how much this matters, we sent the bakery prompt to Qwen3.8 27B ten times at each of three settings. Every call went to the same host, DeepInfra, through OpenRouter, with the model’s thinking mode off.
At temperature 1, the ten runs produced five different names. At temperature 0, all ten said “The Daily Loaf”, as the hero at the top of this page shows. The third setting is the interesting one: temperature 1 with a fixed seed of 42. A seed fixes the random numbers behind the draw, so in theory every run should match. Nine did. One said “The Crumb”.
That miss matches the documentation. When OpenAI added a seed setting in 2023, its announcement said the system would make “a best effort to sample deterministically” and that “determinism is not guaranteed”. It added a system_fingerprint field so developers can spot when the serving setup has changed, which it said may happen a few times a year.
We also asked Claude Haiku 5.5 ten times at its default setting, the only one it accepts. It gave three names: “Rise & Crumb” five times, “The Rising Crust” four times and “Rise & Ferment Sourdough Co.” once. That is fewer names than the open model gave at temperature 1, but still not one answer.
Longer answers still drift, because your request shares a GPU
A short name is the easy case. Each extra word is another draw, and a long answer gives tiny differences more chances to tip a close call. So we asked the same model, at temperature 0 on the same host, to explain to a ten-year-old why the sky is blue in about 80 words. We ran it ten times with no randomness requested.
We got six different explanations. The most common version appeared only three times. One run split from it at the seventh word, writing “a mix of all colors” instead of “made of all the colors of the rainbow”. Others matched for 53 words before parting ways. All ten were correct and readable.
The cause sits in the data centre, not in the model. Cloud providers serve many users’ requests together on the same chips, grouped into batches. In September 2025, Horace He and colleagues at Thinking Machines Lab showed that common GPU code returns slightly different numbers depending on the size of the batch, because it adds numbers up in a different order. Batch size depends on how busy the server is at that instant, which no user controls. A rounding difference far down the decimals is enough to swap two words whose odds are nearly tied.
Their test was bigger than ours. They asked Qwen3-235B “Tell me about Richard Feynman” 1,000 times at temperature 0 and got 80 different completions. All 1,000 matched for the first 102 tokens. At token 103, 992 said Feynman was born in “Queens, New York” and 8 said “New York City”. With code rewritten to give the same result at any batch size, all 1,000 matched. The cost was speed: 42 seconds instead of 26 in their benchmark.
The drift shows up in measured quality too. A study by Berk Atil and colleagues ran five models set up to be deterministic on eight tasks, ten runs each, and found accuracy varying by up to 15% between runs. Anthropic’s glossary states it plainly: even with temperature set to 0, “the results will not be fully deterministic”.
The same open model answers differently at every host
Open models add another layer. Qwen3.8 27B has 20 hosts on OpenRouter alone, and they do not all run the same numbers. Two list 16-bit precision and twelve list 8-bit or 4-bit versions, which shrink the model by rounding its numbers; the rest do not say. Our guide to quantization explains that trade. Each host also runs its own serving software on its own hardware.
We sent the sky prompt at temperature 0 to three more hosts. Chutes, running 8-bit, came closest: it matched DeepInfra’s most common version except for one word, “particles” where DeepInfra said “molecules”. Darkbloom (4-bit) and Cerebras (16-bit) each wrote their own version from the first sentence. The model name was the same every time.
For most people this changes one habit. When you compare two models or test a prompt, the host is part of the setup. A gateway that picks the cheapest host per request can hand you a different version of “the same” model on Tuesday than on Monday. Pinning the host, as we did for these tests, removes that variable.
The biggest cloud models are taking the dial away
The neat fix for all this used to be simple: set temperature to 0. On the newest frontier models, that option is going away.
Anthropic’s thinking documentation lists twelve current Claude models, from Claude Opus 4.7 to Claude Haiku 5.5, on which a non-default temperature, top-p or top-k value will “return a 400 error on every request”. OpenAI’s guide to its latest models says to remove temperature and top-p whenever reasoning effort is set to anything but none. Google still accepts the setting on Gemini 3, but its developer guide strongly recommends leaving it at 1.0 and warns that lower values may cause looping or worse results on hard maths and reasoning.
Google gives the clearest reason: Gemini 3’s reasoning is tuned for the default setting, and pushing it lower can trap the model in loops. OpenAI ties its rule to reasoning effort, and Anthropic’s grew out of an older rule that applied only while Claude was thinking. So the dial is now mostly an open-model feature. DeepSeek, Qwen and Mistral models on most hosts still take it, and so does anything you run locally.
There is a quieter trap, too. OpenRouter’s model listing shows no temperature support for Claude Haiku 5.5, and when we sent temperature 0 anyway, the call came back normally with “The Rising Crust”. The setting appears to have been dropped on the way rather than refused. Code that “sets” temperature 0 can end up running at the default without anyone noticing, so check the response, not just the request.
A seed repeats a picture only where the host honours it
Image models roll dice too, in a different way. A diffusion model starts from a field of random noise and refines it into a picture step by step. The seed decides that starting noise. With the same model, prompt, seed and settings, you should get the same image. Change the seed and you get a new picture.
We tested that on two image models through Runware. FLUX.1 dev, an open model, kept its word: two runs with seed 1234 returned byte-for-byte identical files. Changing “red bicycle” to “blue bicycle” with the same seed kept the lane, the wall and the light, and redrew the bike. A different seed gave a different scene.
Qwen-Image 2.0, through a different route on the same API, did not. It accepted seed 1234, echoed it back in the response, and returned two different pictures. It did the same with our first prompt, so this was not a quirk of one scene. The seed setting existed; the guarantee did not.
A seed that works is what makes careful editing possible. You can keep a composition and change one detail, the same trick behind our guide to keeping a character consistent across images. Before you rely on it with a new model or host, run the same seed twice and compare the results.
Keep the answer you like instead of hoping for it twice
Randomness is a feature until you need a repeat, and then the fix is rarely a setting. Three habits cover most cases.
First, decide which kind of task you have. For names, ideas, drafts and pictures, leave the defaults alone and ask for several options in one go; five names in one answer beat five separate runs. For extraction, labelling and anything a later step depends on, constrain the output instead of trusting the dice. Ask for a fixed format, give examples, and check the result with code.
Second, save the output you like. A generation you keep is the only one guaranteed to be identical tomorrow. For images, store the seed, model, host and every setting next to the file. For text, store the answer itself, not just the prompt that once produced it.
Third, when you truly need the same answer every time, use a setup that can give it. Ollama’s documentation says a fixed seed “will make the model generate the same text for the same prompt”, and on your own computer no stranger’s request shares your batch. In the cloud, pin the model version and the host, use temperature 0 where it is still allowed, and test the promise by sending the same request twice.
Our whole text experiment, 56 calls across two models and five hosts, cost $0.0029. Checking whether your own setup repeats itself costs even less, and it is worth doing before a workflow depends on it. The same prompt gives a different answer because the model rolls weighted dice, and the cloud adds a little noise of its own. Turn the dice down where you can, but the reliable way to get the same answer twice is to keep the first one.
