The Redesigned Shared List Experience.
The challenge
Teams could run an eval. They couldn't act on one.
A run returned a single aggregate: 82% pass. Someone changed a prompt, ran it again, got 79%. That three-point drop was the entire signal — no indication of which behaviors regressed, or whether the drop was real or noise from a flaky grader. So people did the obvious thing: they ignored the number and went back to reading outputs by hand in a spreadsheet.
"I know something got worse. I have no idea what. So I just re-read fifty outputs myself."
— Staff engineer, 6 months on the platform
The Redesigned Shared List Experience.
The Process
Comparison, not the run. Nobody cares about one run — they care about what changed. Two runs side by side became the default view.
Failures grouped by behavior. 400 failing rows have no shape. We clustered them by what went wrong — refusals, hallucinated citations, format breaks. Twelve clusters instead of 400 rows.
Manual grading first. Teams didn't trust LLM-as-judge scores. You hand-grade twenty examples, then see where the auto-grader agreed. Setup got slower on purpose.
Deleted the metrics dashboard. Most-visited screen, least useful. We removed it behind a flag for a month; nobody asked for it back.
Solution
Run comparison as the default view. Open a project and you land on the last two runs, diffed, sorted by largest behavioral change. The aggregate score is still there, in small type, at the top right.
Failure clusters. Grouped by behavior, each with a plain-language label, a count, a delta against the previous run, and three representative examples inline.
Keyboard-first grading. J/K to move, 1–3 to score, one modal, no page loads. Grading twenty examples takes about four minutes.
Trace panel. One click from any failed case to the full input, output, grader reasoning, and token-level cost — so you can check whether the grader was wrong before you go fix the prompt.
Outcome
Week-6 retention went from 31% to 58% over the two months following release. The clearer signal: the number of teams running evals in CI more than doubled, which was the behavior we were actually designing for — evals only pay off if they run on every change, not when someone remembers.
What I'd do differently. The failure clustering was built on a taxonomy the researchers wrote, and it's excellent for chat and RAG products. It's poor for agentic workflows, where the failure is usually a wrong sequence of tool calls, not a bad output. We shipped knowing that and it became the top support request within six weeks. I should have pushed to make clusters user-definable in v1 instead of treating it as a fast-follow.
Collaborative-list engagement
+18%
Return visits to shared lists within 7 days
+24%
We stopped sending twenty links back and forth. Everyone could see what was still in the running.
Meadea John
Chief Innovation Officer