Airbnb argues that operating generative AI products reliably requires treating evaluation not as a final pre-launch check, but as an engineering activity designed into the product from the beginning. Teams should derive evaluation criteria from real failures and combine code-based checks, LLM judges, and human judgment. Agent systems in particular require evaluating not only the final answer but also tool calls and the entire execution path. The essential ingredients are not merely a strong model, but clear product criteria, calibrated evaluators, and continuous collaboration across the team.

1. Why Evaluating Generative AI Is Hard

The article begins by describing how Airbnb applies LLM-powered features across products such as review highlights, AI customer support, and smart communication for guests and hosts. Internally, AI also helps teams understand user reactions and product trends so they can identify the next area to improve.

Generative AI, however, breaks the assumptions of conventional software testing. Traditional programs generally produce the same result for the same input, and correctness is comparatively clear. LLM outputs are nondeterministic, while the definition of a "good response" depends on context and human judgment. Teams often use another AI as a judge, but that judging AI can make mistakes as well.

A single LLM interaction can also chain together retrieval, reasoning, tool calls, and text generation. Each stage can fail independently, so checking only whether the final answer is correct is not enough.

Every product team has its own criteria and workflow, while Airbnb also operates shared infrastructure and principles to support them. The infrastructure team provides tools and best practices, and lessons from different product areas are shared across the company. The article cautions that this is not a universal prescription for every team: evaluation must be designed for its specific context.

2. The Foundation: Design Evaluation from the Start

Evaluation of an LLM system must be planned from the beginning of the project. Developing without an evaluation strategy usually creates three problems:

• False confidence: A generic "helpfulness" score may be high even though the evaluation misses a critical failure experienced by real users.
• Undetected regression: A small prompt change can subtly damage an unmeasured dimension of quality.
• Wasteful scaling: A team may build a large evaluation pipeline around metrics unrelated to actual product outcomes.

Evaluation is therefore not unnecessary overhead. It should be treated as a substantial share of the development investment required to make the product work in practice.

One Principle: Read the Data Yourself

The authors' strongest recommendation is remarkably simple:

"When in doubt, look at the data."

Start with a prototype and run about 100 examples, even if the data is synthetic. Then go beyond the aggregate numbers: read the outputs and execution traces yourself to locate exactly where and how the model fails. Classify the errors you find and build evaluations that detect them.

"Read the outputs and traces and find the model's mistakes. Categorize them and build evaluations."

The article argues that this habit has a greater effect on product quality than any framework or tool.

Five Principles of Eval-Driven Development

Eval-Driven Development (EDD) systematizes this habit. It is the generative-AI counterpart to test-driven development. Instead of trying to predict every possible failure in advance, teams continually find failures, encode them as evaluation criteria, and retest. The process also forces stakeholders to agree concretely on what a "good result" means, influencing the product roadmap itself.

EDD rests on five principles:

1. Define goals and release criteria in advance
   Decide what to optimize and which conditions must be met before launch. Early answers may be incomplete, and some criteria may emerge only while exploring the data.

2. Let real errors drive the metrics
   Build criteria from observed failures in collaboration with product, engineering, domain experts, and other stakeholders. Do not invent metrics solely from abstract speculation.

3. Keep judges few and sharp
   Rather than 20–30 noisy evaluators, use three to five well-calibrated LLM judges, each responsible for one dimension of correctness.

4. Name a final decision-maker
   The team should discuss what counts as good or bad, but when opinions diverge, a human decision-maker must make the final call.

5. Collaborate continuously
   Product owners should regularly answer questions such as "Is X better than Y?" and "What is actually wrong with this output?" so the criteria continue to improve.

3. Layering Three Evaluation Methods

The article divides generative AI evaluation into three broad methods, or combinations of them. They form a layered defense, beginning with fast, simple checks and moving toward subtler and more expensive validation.

1. Code-based and heuristic checks: Fast and inexpensive; catch obvious failures.
2. LLM judges (LLM-as-a-Judge): Assess nuanced qualities such as style, faithfulness, and relevance.
3. Human evaluation: Resolve high-risk cases, subtle judgments, and disagreement among automated evaluators.

Use Code-Based Checks to Filter Basic Errors

Deterministic checks that require no LLM call should be the first filter. They can quickly verify JSON format, detect empty or suspiciously long responses, and identify forbidden terms.

Examples include:

• Format validation: Confirm valid JSON or schema conformance; apply this consistently.
• Length and required-value checks: Detect empty, suspiciously short, or unusually long output.
• Keyword and regular-expression checks: Find prohibited language for safety and compliance.
• Precision, recall, and F1: Apply to classification tasks.
• Cosine similarity: Compare semantic similarity when reference answers exist.

To keep downstream data pipelines from breaking, telling the model "output JSON" in the prompt is not enough. The article recommends using structured output such as a JSON schema to enforce the format strictly.

"Do not rely on prompt instructions alone to produce the data format. That will break downstream data pipelines."

Virtual Judges: LLM-as-a-Judge

The second layer uses a more capable LLM to evaluate another model's output against an explicit rubric. This can measure nuanced qualities such as tone, coherence, groundedness, and relevance with far fewer resources than large-scale human review.

An effective virtual judge needs an unambiguous rubric. A question such as "Is the description readable and consistent with our standards?" is hard even for humans to apply consistently, making it even less suitable for an LLM.

The article gives a hypothetical rubric for judging the readability of a property description. A good description should sound like a friendly travel advisor: warm yet professional, natural, and grammatically complete. It should fail if it is excessively formal or jargon-heavy, too casual, strongly promotional, or robotic. It should also avoid internal terminology, quotation marks, bullet points, and sentence fragments, and each sentence should end with a period.

"A good description should be warm but professional like a friendly travel advisor, simple, natural, and grammatically complete."

The judging result should not be free-form text. It should return only an explicit JSON format containing a reason with the error type and explanation, and a binary score of 0 or 1. Narrow criteria and a constrained output format make automated evaluation reliable enough for downstream systems.

Calibrate Virtual Judges Before Trusting Them

An uncalibrated virtual judge can be more dangerous than having no judge at all, because plausible-looking scores create false confidence. Airbnb proposes this calibration process:

1. Build a golden dataset of 50–100 examples that includes both good and bad cases.
2. Run the virtual judge on the golden set.
3. Measure agreement with human labels. A typical target is agreement in the high 80s to 90s percent range.
4. Analyze disagreements, revise the judging prompt and few-shot examples, and repeat.
5. Recalibrate periodically because failure patterns change over time.

Agreement can also be measured with statistics such as Cohen's kappa or Krippendorff's alpha. Humans do not agree perfectly with one another, so 100% agreement need not be the goal.

When Human Evaluation Is Essential

Human judgment remains the basis for ground-truth data, a safety layer in high-risk domains, and the final means of resolving conflict among automated evaluators.

The article recommends beginning with 20–100 examples labeled by domain experts. Large labeling workforces should be considered only when the rubric is very robust and throughput is the true bottleneck.

If experts cannot agree on a label for an example, do not rush to automate it.

"If experts disagree on the label, stop. Resolve human disagreement before automating anything."

All three methods are needed during development, before launch, and in production. Code checks immediately prevent basic failures, virtual judges validate quality at scale, and humans own the important exceptions and judgment calls automation may miss.

4. Agent Systems Require Evaluating the Process, Not Just the Answer

Agentic AI performs multiple reasoning steps, tool calls, branching logic, and intermediate state transitions. Evaluating only the final response is therefore insufficient. An answer can be correct by accident even when the agent used the wrong tool parameters, followed an unnecessarily long path, or reasoned incorrectly.

Airbnb describes agent evaluation at three levels:

• Step-level evaluation: Check whether an individual reasoning step or tool call is correct.
• Trajectory-level evaluation: Examine whether the complete execution path was reasonable and efficient.
• Session-level evaluation: Determine whether the overall interaction ultimately achieved the user's goal.

These evaluations use the agent's traces and spans. The records show which agents and subagents ran, their inputs and outputs, and the tools they called. Once stored in an observability platform or persistent store, the execution path can be reconstructed in memory with a tree traversal such as depth-first search.

This makes it possible to determine whether a particular subagent was invoked at the right moment, whether it used the correct tool, and whether evaluation can be scoped to only the necessary agent or subagent. In other words, evaluation must cover not one model but the entire system, including retrieval, tools, agents, and generation.

5. A Practical Evaluation Process for a Travel Support AI

The article uses a hypothetical AI assistant that answers travel-platform support-policy questions to illustrate the process from initial development through production monitoring.

Read 100 Examples and Find the Failure Modes

First, run 100 inputs through the prototype and read every output. The review reveals:

• 15 responses invented policy details not found in the source material. This is a faithfulness problem.
• Eight were correct but too verbose. This is a conciseness problem.
• Five refused valid questions they could have answered. This is an over-refusal problem.
• Three returned malformed JSON. This is an output-format problem.

The key is to avoid a generic "helpfulness" score and instead convert failures actually found in the product into concrete evaluation targets.

Build Evaluators and Assemble a Golden Set

Next, add code checks for JSON validity and output-length bounds. At the same time, create a virtual judge for faithfulness and a separate one for conciseness. The faithfulness judge should use a different model from the generator, an isolated prompt, and step-by-step reasoning.

A product manager or domain expert labels 60 examples, including failures, to form a golden set. Again, each evaluator focuses on a single quality dimension. The principle is to avoid a so-called "God evaluator" that attempts to judge everything at once.

Calibrate, and Change Only One Variable at a Time

The first faithfulness judge agrees with the PM only 78% of the time, which is not reliable enough. Analysis shows that the judge incorrectly labels accurate paraphrases as unfaithful to the source.

After revising the rubric and adding few-shot examples, human agreement rises to 88%. Improving the retrieval stage then substantially reduces faithfulness errors.

The article stresses that teams should not change several factors at once while improving models and prompts.

"First hold the model constant and change the prompt. Then hold the prompt constant and change the model. Then hold both constant and change the serving configuration."

At each step, virtual-judge results narrow the candidate set, while samples from the leading candidates are used to improve the judges again. Through this loop, the candidate system and evaluator refine each other until both stabilize.

Large-Scale Evaluation and Production Monitoring

Once the evaluators and system are reasonably stable, expand the evaluation to 5,000 examples. In production, sample 5% of de-identified real traffic every day and run code checks and virtual judges. Humans review flagged outputs, and the PM reviews the results weekly.

When a new failure mode emerges, add a new evaluator and improve the system accordingly. Production evaluation is not a one-time pre-launch gate; it is a continuously closing improvement loop. 🔄

Real user data is strongly de-identified before human review, and its use is tightly limited to safety and quality assurance. These controls follow Airbnb's privacy principles.

6. Key Lessons and Conclusion

The article concludes that successful generative AI products do not simply depend on selecting the most capable model. What matters more is the ability to identify failures in actual user experiences, convert them into measurable criteria, and continuously improve by combining automation with human judgment.

The key lessons are:

1. Reading outputs and execution traces yourself is the starting point for every evaluation.
2. Measure product-specific failure modes that happen in practice, not generic metrics.
3. Begin with 50–100 examples so you can fail quickly and iterate cheaply.
4. Make each evaluator responsible for one dimension of quality.
5. Do not trust a virtual judge at scale until it reaches high-80s to 90s percent agreement with humans.
6. Use a layered defense that combines code checks, virtual judges, and human evaluation.
7. Always include bad examples in the golden set.
8. Evaluate retrieval, tool calls, and the whole pipeline—not only the model. For agents, examine the execution path as well as the final answer.
9. Carry pre-launch evaluations into production for continuous monitoring.
10. Evaluation is a team sport. Engineers, PMs, domain experts, and others must jointly define what product success means.

"The teams that succeed in AI are not the ones with the best models. They are the ones with the best communication and the clearest product vision."

Ultimately, Eval-Driven Development is more than a quality-control technique aimed at eliminating AI errors. It is a development culture in which the team explicitly agrees on the experience a product should deliver and continually verifies that promise against real data.