Build Your Party.

Model evaluation often begins with a number. A benchmark produces a score. A leaderboard assigns a rank. A comparison between two models reduces their respective performances on some collection of tasks to a relationship between two positions on a single scale.

That compression is useful. It makes a large field navigable. It also conceals something worth understanding.

A model that scores 87 on a general benchmark has not produced a single observation about a single capability. It has produced a composite. The benchmark may have measured reasoning, factual accuracy, instruction following, coding ability, retrieval fidelity, and several other dimensions. The score is what happens when those measurements are combined.

Different models can arrive at similar scores through very different capability profiles. One model may reason well but handle retrieval poorly. Another may be less capable in open-ended generation but extremely disciplined about grounding its responses in provided evidence. A third may be fast, inexpensive, and remarkably good at deciding which tool should handle a particular request.

Their scores can be close. Their builds are not the same.

The score is a projection

If an evaluation measures several capabilities, the aggregate score is a function of them. Some aggregation is a simple average. Some is weighted. Some involves more complicated combinations. In every case, the operation takes a multidimensional profile and collapses it into a scalar.

That operation is a projection. Like any projection, it preserves some information while losing the rest.

What it preserves is relative position. One model scored higher than another across the evaluated tasks. That can be genuinely useful. It is also frequently treated as though it contains more information than it actually does.

What gets lost is shape. Two models with similar aggregate scores may have very different strengths. The projection hides those differences because its job is to produce a single axis for comparison.

Aggregate scores are incomplete in a specific way that matters when a model is selected for a specific job.

Shape matters under pressure

I encountered this directly in recent work on evidence utilization. The question was not simply whether models could answer correctly, but how their performance changed as the evidence environment around them became more difficult.

We held the underlying questions constant and changed the context. Models first saw clean, well-matched evidence. Then the same evidence mixed with distracting material. Then a fuller retrieval environment containing more information they had to sort through.

Performance changed. More importantly, it did not change uniformly.

One model lost roughly ten points as distractors were introduced. Another lost closer to five. Both performed strongly under clean evidence conditions. But their ability to preserve that performance under noise was not the same.

That difference is easy to hide inside a general benchmark score. It is much harder to ignore when the model sits inside a retrieval system where noisy context is the normal operating condition.

The relevant question stopped being "which model is better?" and became "better at what, under which conditions?" That question does not appear on a leaderboard. But it is the question that determines which model actually works in the system being built.

The character sheet

Tabletop role-playing games have been solving a version of this problem for decades.

In Dungeons & Dragons, a character has a level. Level tells you something about overall progression. It does not tell you whether the character is strong, perceptive, persuasive, or good at picking a lock. That information lives in the character sheet. Six ability scores describe underlying attributes: Strength, Dexterity, Constitution, Intelligence, Wisdom, and Charisma. Skills and proficiencies represent more specific competence. When a situation arises in the game, the outcome depends on the relevant ability, whether the character has proficiency in the specific task, and the difficulty of the challenge.

A Fighter and a Wizard can be the same level and useful for completely different reasons. One does not replace the other simply by having a higher number somewhere on the sheet.

The analogy is imperfect, but the structural insight is useful. A character's level tells you something important, but it does not describe the character. Two characters at the same level can have very different ability scores, proficiencies, class features, equipment, and roles. The character sheet preserves that shape. A model's aggregate benchmark score works the same way. It indicates something about general capability. The underlying profile, the distribution of strengths across reasoning, retrieval, tool use, calibration, domain knowledge, and many other dimensions, determines what happens when a particular challenge appears.

D&D also distinguishes between raw ability and proficiency. A high Dexterity helps with many tasks, but a character proficient in Stealth is specifically better when the situation calls for it. Models increasingly have the same structure. The base model supplies the underlying capability profile. Fine-tuning, retrieval, tools, prompts, and orchestration determine how that profile is expressed in the system. The sheet matters, but so does the build you bring to the encounter.

Deployment is the encounter

An earlier Field Note argued that a benchmark ends at deployment because deployment changes the object being evaluated. There is a second consequence.

Once the object is a system, the relevant question is not which component has the highest aggregate score. It is which capabilities the system actually needs and whether the selected components provide them.

A retrieval-heavy application cares about evidence fidelity and context discipline. An evaluation layer needs conservative judgment and resistance to surface plausibility.

The application determines which part of the character sheet is being tested. A model that is strong on the dimensions that matter for the role may outperform a higher-scoring model whose strengths are concentrated elsewhere.

The leaderboard narrows the field. The capability profile determines the fit.

And once a system has more than one role, the question extends beyond individual selection. The model that plans does not have to be the model that executes. The model generating an answer does not have to be the model evaluating it. A smaller model may handle classification or routing while a more capable model is reserved for work that requires it.

That does not mean every system should become a collection of specialized models. Sometimes one highly capable model is exactly the right architecture. Four Wizards is a perfectly legal party.

The point is that this is a design choice, not the inevitable result of sorting a leaderboard from highest to lowest.

A leaderboard tells you who ranks highest. Deployment tells you what encounter you are actually walking into. Once you know that, the question changes. You are no longer looking for the biggest number. You are looking at the sheet. And if you are building a system with more than one role, you are looking at the whole party. Build it around the encounter, not the leaderboard.