The Evaluator Trust Problem
LLM-as-judge is becoming infrastructure. But the evaluator itself is largely treated as a black box. This note examines the reliability characteristics of AI evaluators and proposes a framework for evaluator trust.
The problem
The AI industry has transferred trust from models to evaluators without transferring rigor. We used to ask “can you trust the model’s output?” Now we use LLM-as-judge to answer that question. But almost nobody is asking: can you trust the judge?
This research note is in progress. Check back for the full study.