Hamel Husain and Shreya Shankar argue that most AI eval dashboards are theater. Teams reach for ready-made metrics like 'hallucination' or 'toxicity' because they sound rigorous, but those scores rarely correlate with the actual problems users face. The result: pretty dashboards that nobody trusts and that don't drive shipping decisions.
Their alternative is a three-phase process. Phase 1 is error analysis: ground your evals in reality by appointing a single principal domain expert (often the PM) as the 'benevolent dictator' on quality. Arm them with ~100 representative user interactions, sampled randomly at first, and have them do open coding — free-form pass/fail critiques detailed enough that a brand-new employee could understand them. The output is a prioritized list of your product's most common failure modes.
Phase 2 is building the eval suite. Each prioritized failure mode gets the right tool: code-based evaluators for objective, rule-based checks ('does the output contain a user ID?'), and LLM-as-a-judge for subjective qualities like tone or reasoning quality. Don't pick the tool first; pick the failure mode first, then the tool.
Phase 3 is operationalizing — making the suite a flywheel that catches regressions before they ship and feeds back into product improvement. The throughline: evals are a process, not a dashboard. They're the PM's domain, not a thing you outsource to ML engineers.