WHAT MAKES INFERENCE NONDETERMINISTIC?

Summary from the TLDR Dev newsletter

When the same prompt is sent to an AI model multiple times at zero temperature, variations in the results can occur due to the nondeterministic nature of floating-point arithmetic combined with server load dynamics. The model's output can differ based on how requests are batched and how the GPU processes the addition of numbers, leading to discrepancies despite identical settings.