Applied AI & evaluation
Knowing where AI needs human judgment.
Benchmarking AI-assisted review, with attention to the errors that matter in an actual workflow.
Context
An AI score is useful only when its operational consequences are understood.
The challenge
High agreement can conceal errors that cross a decision threshold.
The approach
- Compare models under consistent inputs, rubrics, and prompt conditions.
- Examine category-level errors and keep final decisions with people.
Evaluate the decision, not just the score
The benchmark exposed why aggregate agreement is insufficient. A small scoring error can have a large consequence when it changes which side of a threshold an item falls on.
Keep the boundary explicit
The system writes a score and rationale. It does not automatically write the final decision or send a notification.
What changed
The evaluation logic runs in a spreadsheet-based workflow, producing scores and reasons for review.
What the evidence tells us
A 50% reduction in review effort is a target, not an achieved result. Actual review time has not yet been measured.