Why shipping an AI application without evals is like deploying production code without tests
Imagine your team ships a backend service without a single unit test.
No integration tests.
No regression tests.
No monitoring.
Just a late night, a tired engineer, and this:
git push origin main
Would you trust that system in production? Probably not. You would not trust it. I would not trust it. The person pushing the code would not trust it, not really, not deep in their chest at 2 AM when they are staring at the ceiling wondering if the database migration will eat the customer data alive.
Yet that is exactly how thousands of LLM applications are deployed today.
Developers tweak prompts until the answers look good. They run a few manual tests in a Jupyter notebook. The output seems fine. It sounds smart. It uses the right words. So they ship it to production.
Then the emails start.
Users report hallucinations. The chatbot invents refund policies that do not exist. A RAG pipeline cites documents from 2019 as if they were written yesterday. An agent loops forever, calling the same broken tool, burning through tokens like they are free. After every model update, something subtle breaks. A prompt which worked perfectly on GPT-4 now produces meandering, insecure nonsense on the newer version.
The missing piece is not another prompt. It is not a larger model. It is not more training data.
It’s the evaluation.
Traditional Software Testing
Let us remember what we already know. Traditional software engineering built a safety net so robust that we rarely think about it anymore. It looks like this:
Each layer has a purpose. Unit tests catch the stupid mistakes, the off-by-one errors, the null pointer exceptions. They run fast. They give you immediate feedback. Integration tests make sure your service actually talks to the database, that the API contract holds, that the message queue does not silently drop events. Regression tests ensure that the bug you fixed last Tuesday stays fixed. Monitoring watches the patient after surgery, beeping when the heart rate spikes.
This pyramid exists because we learned the hard way. We shipped bugs that cost money. We broke production on Friday evenings. We lost customer trust. So we built rituals around verification. We made testing a non-negotiable part of the craft.
If you are a software engineer, than this is your home. This is the language we speak. And it is exactly the language we need to use when we talk about AI.
AI Does Not Behave Like Traditional Software
Here is the uncomfortable truth that makes everything harder.
In traditional software:
2 + 2 = 4
Always. Forever. In every environment. On every machine. Whether it is Tuesday or Sunday. Whether the developer slept well or not.
LLMs are probabilistic. The same input, run twice, can generate Answer A, Answer B, or Answer C. All of them might be valid. One might be poetic. One might be terse. One might include a detail the others forgot. There is no single correct output you can assert against with a simple ==.
Testing becomes fundamentally different. we cannot just check that the output equals an expected string. we have to check that the output makes sense. That it is faithful to a source. That it is safe. That it is useful. That it did not drift into nonsense while you were not looking.
This probabilistic nature is not a bug. It is the feature that makes LLMs powerful. But it is also the feature that makes them terrifying to deploy.
Traditional Testing vs AI Evaluation
This is the heart of the article. Let us map the familiar to the unfamiliar.
Think about this table for a moment. Every concept on the left is something you already trust. Every concept on the right is something you need to build. The tools are different, but the instincts are the same. You are still asking: does this thing work? Is it safe to ship? Will it break when I am on vacation?
What Exactly Are AI Evals?
Let us define this properly.
AI evaluations are systematic methods for measuring whether an LLM behaves the way you expect across quality, correctness, safety, reliability, and performance.
Notice what is in that definition. It is not just correctness. It is everything.
A response can be factually correct but dangerously incomplete. It can be accurate but written in a tone that alienates your users. It can be perfect in isolation but take fifteen seconds to generate, which means no one will wait for it. It can be safe but irrelevant.
Evaluation is the discipline of looking at the whole picture and deciding, with evidence, whether your system deserves to face the public.
The Layers of AI Evaluation
I like to think of evaluation as a pyramid. Not because it is ancient and mysterious, but because each layer supports the one above it.
At the bottom, we have dataset quality. If your data is garbage, nothing above it matters. You can have the most sophisticated judge in the world, but it will be measuring garbage against garbage.
Next comes prompt and response evaluation. This is your unit test layer. Fast, focused, repeatable.
Then RAG and tool use evaluation. This is where you check if your retrieval system is actually retrieving, if your tools are being called correctly.
Above that, agent workflow evaluation. This is integration testing for AI. You are not looking at one call anymore. You are looking at a chain of decisions.
At the very top, human review. Slow, expensive, irreplaceable. The final sanity check.
Every layer matters. Skip the bottom, and the top crumbles.
Prompt Evals
Prompt evals are your unit tests. They are the closest analogy to the assert statements you already write.
The pipeline is simple:
we give it a question. The model answers. we measure the answer against what we expected.
But what do we measure?
Relevance. Did the answer actually address the question, or did it go on a tangent about something else entirely?
Correctness. Is the information right? Not just sort of right, but actually right?
Style. Does it sound like your brand? Is it too formal, too casual, too robotic?
Completeness. Did it miss a critical detail that a user would need?
Safety. Does it contain harmful instructions, private data, or biased assumptions?
These are not abstract concerns. I once saw a customer support bot that answered a question about returns perfectly, except it forgot to mention that the user needed a receipt. The answer was relevant, correct, well-written, and complete in every way except the one way that mattered to the business. A good prompt eval catches that.
RAG Evaluation
Retrieval-Augmented Generation changed the game. Instead of hoping the model knows everything, you give it documents. But now you have two systems to break: the retriever and the generator.
The pipeline looks like this:
You need metrics for every stage.
Context Precision. Of the chunks you retrieved, how many were actually useful? If you grab ten chunks and only two matter, you are wasting tokens and confusing the model.
Context Recall. Did you retrieve the chunks you needed? If the answer lives in paragraph four and your retriever never found it, the game is already lost.
Faithfulness. Does the generated answer stick to the retrieved context, or does it start making things up?
Groundedness. Can you trace every claim in the answer back to a specific chunk?
Answer Relevance. Even if the retrieval was perfect, did the final answer actually help the user?
Retriever Recall and Precision. Classic information retrieval metrics. You need them. Do not ignore them just because LLMs are new and shiny.
Chunk Coverage. Are your chunks sized correctly? Too big, and the model drowns in noise. Too small, and meaning fragments.
Citation Accuracy. If the model says “according to document A,” is it actually document A? Or did it hallucinate a citation?
RAG evals are where most production systems fall down. Everyone tests the prompt. Almost no one tests the retriever with the same rigor. That is a mistake.
Agent Evaluation
Modern AI systems are not single prompts. They are workflows. They plan. They remember. They use tools. They reason, or at least they try to.
The flow looks like this:
Evaluating this is hard. You cannot just check the final output. You have to check the journey.
Planning. Did the agent break the task into the right steps? Or did it decide to solve a math problem by searching the web when it could have just calculated?
Tool Selection. When it needed weather data, did it call the weather tool? Or did it call the stock price tool and hope for the best?
Tool Accuracy. Did it pass the right arguments? Did it handle the API response correctly?
Reasoning. Did it think through the problem, or did it jump to a conclusion? Can you even tell?
Recovery. When a tool failed, did the agent adapt? Or did it panic and loop?
Memory. Did it remember what the user said three turns ago? Or did it start every response fresh?
Latency. How long did the whole dance take?
Cost. How many tokens did it burn? An agent that solves a problem for five dollars when a simpler approach costs five cents is not a success. It is a financial liability.
Agent evaluation is integration testing at its most complex. You are not testing a function. You are testing a system that makes decisions.
Hallucination Is Not One Thing
Most articles stop at: hallucination equals bad.
That is not good enough. If you treat every hallucination the same, you will fix the wrong things.
Here is a taxonomy that actually helps:
Unsupported. The model states a fact, but no evidence exists in the context. It might be true in the world, but you cannot verify it.
Fabricated. The model invents something entirely. A study that does not exist. A product feature you never built. A law that was never passed.
Incorrect Citation. The model cites a real source but misrepresents what it says. This is especially dangerous because it looks authoritative.
Wrong Tool Usage. The model calls a tool with the wrong parameters, or calls the wrong tool, and then reports the result as if it were correct.
Missing Context. The model ignores critical information you provided and answers from its training data instead.
Outdated Information. The model uses knowledge from its training cutoff even though you gave it current documents.
Reasoning Error. The model follows a logical chain, but one of the links is broken. The conclusion sounds right, but the path to get there was wrong.
Each of these needs a different fix. Unsupported claims need better grounding. Fabrications need stricter prompting or retrieval. Incorrect citations need citation verification logic. Wrong tool usage needs better tool descriptions. Missing context needs prompt engineering or reranking. Outdated information needs fresher data. Reasoning errors need chain-of-thought verification or different models.
You cannot fight an enemy which you cannot name.
Golden Datasets
If evals are your tests, then datasets are your test cases. And not all datasets are equal.
A golden dataset is:
Curated. Hand-picked for quality, not scraped in bulk.
Human Verified. Real people have looked at the questions and the expected answers and confirmed they are right.
Stable. The expected answers do not change every week because the underlying facts changed.
Used for regression. You run the same dataset against prompt v1, prompt v2, and model v3. You watch the scores move.
A good golden dataset entry looks like this:
Question: "What is your refund policy for international orders?"
Expected Answer: "International orders can be refunded within 30 days of delivery. The customer is responsible for return shipping unless the item was defective."
Expected Sources: ["returns_policy_2024.pdf", section 3.2]
Expected Behaviour: "Answer must mention 30-day window and customer responsibility for shipping.
This is your ground truth. It is expensive to build. It takes time. But it is the only way to know if your system is actually improving or just changing.
Synthetic Datasets
Golden datasets are beautiful, but they do not scale. You cannot hand-craft ten thousand examples. So people turn to synthetic data.
The advantages are real.
Scalable. Generate thousands of examples overnight.
Cheap. No human annotators billing by the hour.
Automatic. A script runs. You wake up to new data.
But the disadvantages are just as real.
Model bias. The model generating your data will encode its own assumptions. It will create questions it knows how to answer. It will avoid edge cases.
Distribution shift. Synthetic data often looks too clean. Real users type typos, use slang, ask ambiguous questions, and get angry. Synthetic data rarely captures that chaos.
Artificial patterns. If you generate ten thousand questions from the same template, your model might learn the template instead of the task.
Synthetic data is a tool. Use it for bootstrapping, for expanding coverage, for stress testing. But do not trust it blindly. Always validate a sample with human eyes. If the synthetic data feels too smooth, it probably is.
Human Evaluation
We talk a lot about automation, and for good reason. But there are places where humans are still irreplaceable.
Creativity. Does the response spark something? Is it surprising in a good way? A metric cannot measure delight.
Tone. Does it sound like a person? Or does it sound like a machine trying to sound like a person? This is subtle. It matters.
Persuasion. If the goal is to convince a user to take action, did the response actually move them? You need human judgment for that.
Brand Voice. Does it sound like your company? Would your best writer read it and nod, or would they wince?
UX. Is the response structured in a way that helps the user? Are the steps clear? Is the formatting readable?
I have seen systems with perfect BLEU scores and terrible user satisfaction. The metrics said success. The humans said confusion. Guess which one mattered to the business?
Automate what you can. But reserve a seat at the table for human judgment.
LLM-as-a-Judge
This is one of the most powerful and most dangerous tools in modern evaluation.
The idea is simple. Instead of a human grading every response, you use another LLM.
The pipeline looks like this:
Response
|
Judge Prompt
|
GPT-5 (or Claude, or Gemini)
|
Score
|
Reason
You feed the judge a rubric. “Score this response from 1 to 5 on helpfulness. Explain your reasoning.” The judge outputs a score and a justification.
The strengths are obvious. It is fast. It is cheap compared to human labelers. It can evaluate thousands of responses while you sleep. It can check for things that are hard to code, like “is this response empathetic?”
But the weaknesses are serious.
Position bias. If you give the judge two responses and ask which is better, it often prefers the one that appears second, regardless of quality.
Verbosity bias. Longer responses often score higher, even if they are worse.
Self-preference. A model used as a judge tends to prefer responses generated by models similar to itself.
Reasoning errors. The judge can be wrong. It can miss subtle factual errors. It can be fooled by confident nonsense.
Cost and latency. Running a large judge model on every evaluation is not free. At scale, it adds up.
LLM-as-a-judge is not a replacement for human evaluation. It is a filter. Use it to surface the worst and best responses. Use it to triage what humans should review. But never let it be the only line of defense.
And please, for the love of all that is holy, do not use the same model to generate and judge. That is like grading your own homework.
Offline vs Online Evaluation
There are two worlds of evaluation, and you need to live in both.
Offline evaluation happens before deployment.
Dataset
|
Run Model
|
Compare
|
Metrics
You have a fixed set of questions and expected behaviors. You run your system against them. You get numbers. You iterate. This is safe. This is controlled. This is where you catch the obvious failures.
Online evaluation happens in production.
Users
|
Traffic
|
Monitoring
|
Alerts
Real people ask real questions. Some of those questions get logged and evaluated automatically. You watch for drift. You watch for hallucination spikes. You watch for latency regressions. You watch for user satisfaction scores dropping.
Offline evaluation tells you if you are ready to ship. Online evaluation tells you if you were wrong.
You need both. Offline without online is naive. Online without offline is reckless. Together, they form a loop.
Regression Testing
This is where the engineering analogy becomes most painful, because almost no one does it for AI.
Imagine this:
Prompt v1
|
94% pass rate
|
Prompt v2
|
91% pass rate
|
Regression detected
You changed the prompt to fix one bug. You did. But you broke three other things you were not watching. Without a regression suite, you will not know until a user tells you.
Or this:
GPT-4
|
Gemini
|
Claude
|
Compare
You switch models because the new one is cheaper. But your evaluation scores drop by twelve percent on edge cases. Do you ship anyway? Maybe. But at least you know.
Regression testing for AI is not optional. It is the difference between a system that improves over time and a system that randomly walks through quality space.
Version your prompts. Version your datasets. Run the old tests against the new code. Every single time.Continuous Evaluation
If you have a CI/CD pipeline, you already know the rhythm.
Git Push
|
GitHub Actions
|
Run Evals
|
Pass?
|
Deploy
This should feel familiar because it is exactly what you already do with unit tests. A developer pushes code. The pipeline runs. If the tests fail, the deploy stops. No human has to remember to check. The system enforces quality.
AI evals belong in the same pipeline.
Push a new prompt. Run the eval suite. Did relevance drop? Block the deploy. Did safety scores fall? Block the deploy. Did latency spike? Block the deploy.
This is not futuristic. This is standard engineering hygiene. The only reason it feels new is because AI tooling is still maturing. But the principle is ancient: do not ship what you have not verified.
Production Monitoring
Once you deploy, the work is not done. It has just begun.
You need to watch the patient. You need dashboards that matter.
Latency. How long are users waiting? If your p99 jumps from two seconds to eight seconds, something is wrong.
Token Usage. Are you burning through your budget? An agent stuck in a loop can cost more in an hour than a human employee makes in a day.
Failures. How often does the model refuse to answer? How often does the API time out?
Hallucination Rate. What percentage of responses contain unsupported claims? Track this over time. Watch it like a heart monitor.
Tool Errors. Are your external APIs failing? Is the agent handling those failures, or is it exposing raw error messages to users?
Retrieval Success. Is the retriever finding relevant chunks, or is it returning noise?
Cost. Per request, per user, per day. AI is not cheap at scale.
Fallback Rate. How often are you falling back to a simpler model because the primary one failed?
User Satisfaction. Direct feedback, thumbs up, thumbs down, NPS. The ultimate metric.
Monitoring is not vanity. It is survival. It is how you know if the plane is still flying or if you are just falling with style.
Popular Frameworks
I do not want to give you a laundry list. I want to tell you when to use each tool for any situations so u don’t need to panics and refer these .
Do not choose a framework because it is popular. Choose it because it solves the problem you actually have. A general-purpose tool is fine for learning. A specialized tool is necessary for production.
Common Mistakes
I have made most of these. So have the teams I have advised. Learn from our bruises.
Only testing manually. You click through a few examples, everything looks fine, you ship. This is not evaluation. This is hope.
Tiny datasets. Five examples tell you nothing. You need hundreds, ideally thousands, covering the long tail of user behavior.
Chasing benchmark scores. A high score on a public benchmark does not mean your system works for your users. Benchmarks are starting points, not destinations.
Ignoring production drift. The world changes. Your users change. Your data changes. A system that scored perfectly in January might be failing by March.
Evaluating only final answers. If you only check the output, you miss the broken retriever, the wrong tool call, the bad reasoning chain. Evaluate the pipeline, not just the product.
Never testing retrieval. RAG systems live and die by retrieval. If you only test generation, you are testing the wrong thing.
No regression suite. You change the prompt, you ship, you move on. Then you find out three weeks later that you broke Spanish language support. Keep old tests. Run them always.
No versioned datasets. If your dataset changes constantly, you cannot compare this week to last week. Version control your data like you version control your code.
Treating eval scores as absolute truth. A score is a signal, not a verdict. A response that scores 4.2 might be worse in practice than one that scores 3.8. Investigate the edge cases.
Optimizing for one metric while hurting others. You squeeze hallucination rate to zero and suddenly the model refuses to answer anything ambiguous. You win on safety and lose on usefulness. Metrics trade off. Watch them all.
The Future
Evaluation is not a solved problem. It is a moving target. Here is where I think we are headed.
AI judging AI. We will see specialized judge models, smaller and faster than general models, trained specifically to evaluate reasoning, safety, and style. They will not be perfect, but they will be better than using a general-purpose model.
Self-improving evaluation loops. Systems that detect their own failures, generate new test cases from those failures, and retrain or reprompt automatically. The loop between production monitoring and offline evaluation will tighten until they are almost the same thing.
Agent-specific benchmarks. Current benchmarks test single-turn question answering. The next generation will test planning, tool use, multi-turn memory, and recovery from failure. We need standards for agents.
Continuous red-teaming. Not a one-time security review, but an automated process that constantly probes your system for jailbreaks, leaks, and failures. Red teaming as a service, running 24/7.
Automated dataset generation. Smarter than today's synthetic data. Systems that mine production logs for edge cases, that use adversarial methods to find your blind spots, that keep your test suite fresh without human curation.
CI/CD-native evaluation pipelines. Evaluation will not be a separate tool you bolt on. It will be a primitive in every deployment platform, as normal as running pytest.
Multi-agent evaluation. As systems involve multiple agents negotiating, competing, or collaborating, we will need new metrics for coordination, consensus, and emergent behavior.
The future is not about better models alone. It is about better confidence. Better trust. Better systems.
Conclusions
A decade ago, shipping software without automated tests was considered reckless. If you did it, other engineers looked at you sideways. You were seen as careless, as someone who valued speed over craft.
Today, shipping an LLM application without evaluations should feel just as risky.
Unit tests gave us confidence that our code behaved as expected. AI evals give us confidence that our models do too.
The tools are different. The syntax is different. The probabilistic nature of the beast makes everything harder. But the goal is the same.
Build systems you can trust.
Build systems that do not break in silence.
Build systems that improve because you can measure the improvement.
Build systems that fail gracefully because you saw the failure coming.
The engineers who master evaluation will be the ones who ship AI that lasts. The ones who skip it will spend their nights answering angry emails and wondering why the magic stopped working.
Do not be the person who pushes to main without tests.
Be the person who ships with confidence.
Because at the end of the day, that is what engineering is. Not perfect code. Not infinite scale. Just the quiet, stubborn belief that what you built will work, and the evidence to back it up.
AI Evals Are the Unit Tests of LLM Applications was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.