How do I evaluate whether an AI feature is actually working?

Build an evaluation set of real examples with known-good outputs before you tune anything, and score against it every time you change something. Without that, you're relying on vibes — and vibes are how teams ship regressions confidently. The practical build: 1. **Collect real inputs.** Take 50-200 actual user queries from your logs, spanning the common cases, the edge cases and the ones you know are hard. Real inputs, not invented ones — synthetic test cases are systematically easier than reality and will flatter you. 2. **Define what good looks like, specifically.** For each example, either write the ideal output or write the criteria it must satisfy ('mentions the refund window', 'does not invent a policy', 'under 100 words'). Criteria are usually more practical than exact answers. 3. **Score.** Three approaches, usually combined: - *Deterministic checks* where possible: does it produce valid JSON, does it cite a real source, does it stay under length, does it include the required field. Cheap, exact, run on every change. - *Model-based grading*: a strong model scores outputs against your rubric. Fast and scalable, imperfect, and needs spot-checking against human judgement to confirm the grader itself is sane. - *Human review* on a sample. Slow and expensive, and the ground truth everything else calibrates to. Do it regularly, not once. 4. **Track a small number of metrics over time** and never change the eval set casually. If you change the set, you can't compare to history — which is exactly when regressions slip through. The metrics that usually matter more than 'quality': - **Failure rate on the cases that must not fail.** Often more important than average quality — one confidently wrong answer about a refund policy costs more than fifty mediocre-but-safe ones. - **Refusal calibration**: does it say 'I don't know' when it should, and does it wrongly refuse things it should answer? - **Latency and cost per request.** Real product constraints that quality-only evaluation ignores. - **Downstream behaviour**: did the user rephrase, retry, escalate to support, or abandon? These are honest signals that no offline metric captures. What to avoid: judging by a handful of examples someone tried in a meeting (vivid, unrepresentative, and whoever tried the worst case wins the argument); optimising the average while the tail gets worse; and shipping changes without re-running the eval because 'it's obviously better'. The cultural point that matters most: make the eval set a shared artefact the whole team looks at, and add every reported failure to it as a permanent regression test. Teams that do this improve steadily; teams that argue from anecdotes oscillate.