Last month we were brought into a programme where an LLM-powered feature had been in production for five months. The team had shipped fourteen prompt updates in that time. When we asked to see the evaluation results for each update, there was a silence, and then someone said, “We tested it manually.”
Fourteen updates. Five months. Manual testing. On a system that produces different output every time. This is not unusual. It is, in fact, the median.
Why your test suite doesn’t work here
Traditional software testing rests on a simple premise: given the same input, the system produces the same output. You write assertEquals(expected, actual) and you’re done. Models violate this premise by design. The same prompt, the same model, the same temperature can produce different outputs on consecutive calls. This is not a bug. It is the reason models are useful.
So you cannot test a model the way you test a function. You have to evaluate it the way you evaluate a hire: across many scenarios, with a scoring rubric, with statistical confidence that the new version is better than the old one.
What to build instead
Step 1: A scored test set. Not ten examples you tried in the playground. Two thousand scenarios, stratified by input type, difficulty, and business importance. Each one has an expected output (or acceptable output range) and a scoring function. Build this before you build the prompt.
Step 2: Automated evaluation in CI. Every prompt change triggers a full eval run. The results are compared against the current production baseline using statistical tests, not eyeballing. A 0.5% average regression that hides a 40% regression on your highest-value input type will be caught.
Step 3: Multi-dimensional gates. Quality is not the only dimension. Safety (does it refuse when it should?), cost (what does this change cost at production volume?), latency (did the new prompt add 2 seconds?), and bias (did accuracy change differently across demographic segments?) are all gated independently.
Step 4: Continuous monitoring. The eval that passed in CI was run against a test set. Production traffic is not a test set. Monitor the same quality metrics in production, with alerting that fires when the model’s behaviour drifts from its evaluation baseline.
This is slower for the first three weeks and much faster for the following year. It is the single clearest signal of whether a team has shipped AI to production before. We build these systems as part of our AI DevSecOps practice.