Explainability has become one of the defining ambitions of enterprise AI. As organizations deploy increasingly capable systems, they want to understand how those systems reach decisions, what information influences their outputs, and why they behave differently under similar conditions. The demand is reasonable. When AI participates in consequential decisions or takes actions on behalf of an organization, simply observing the result is no longer sufficient.
But I think we are at risk of confusing the ability to explain a system with the ability to trust it.
An AI system might provide detailed reasoning, cite supporting documents, expose its tool interactions, and generate a trace of every observable step in its execution. Those capabilities can make its behavior easier to investigate, but none necessarily establishes that the behavior was appropriate, that the evidence actually influenced the outcome, or that the conditions producing an undesirable result can be corrected.
Enterprise AI requires something more: the ability to use that understanding to improve the system.
And the system is considerably larger than the model.
Knowing what happened is not knowing why
Consider an AI assistant tasked with assessing whether a proposed action complies with an organization's internal policy.
The assistant retrieves the relevant documentation, including a provision that explicitly prohibits the action under certain conditions. It nevertheless recommends proceeding. When asked to explain its decision, the assistant cites other relevant provisions and provides an apparently coherent justification.
We can identify at least one problem immediately. The recommendation conflicts with the policy.
But what actually failed?
Perhaps the retrieval system supplied the correct information, but the model did not meaningfully incorporate the exception. Perhaps the relevant provision was buried in a large context window, competing with more prominent but less applicable material. Perhaps instructions in the application harness encouraged the assistant to prioritize completing the user's request. Or perhaps the underlying model simply struggled to reconcile multiple conditions.
The observable failure is the recommendation. Its cause is less obvious.
Even an accurate explanation of the assistant's stated reasoning might not resolve that ambiguity. A citation can establish that information was referenced without establishing that it materially influenced the decision. A generated reasoning trace can expose useful intermediate behavior without necessarily providing a faithful account of the mechanisms that produced the outcome.
More sophisticated techniques, including attribution methods, can help us investigate these questions. But their conclusions also depend on what they measure, the assumptions underlying those measurements, and the degree to which the results have been independently validated.
Understanding what happened is the beginning of diagnosis, not its completion.
The correction may be somewhere else
Much of the conversation around explainable AI continues to treat the model as the primary unit of analysis. Yet the behavior of a deployed AI application emerges from interactions among numerous components, only some of which reside within the model itself.
A modern AI system may include retrieval pipelines, context assembly, system instructions, external tools, and execution permissions. Each contributes to behavior, and each introduces potential failure modes.
More importantly, each represents a possible point of intervention.
Suppose an investigation establishes that an assistant consistently overlooks relevant evidence when presented with lengthy retrieved documents. Fine-tuning might improve the model's ability to use that information. But it might also be more effective to change how documents are selected, reorganize the context, introduce intermediate verification, or select a model better suited to the task.
Those interventions operate at different layers of the system. Their effectiveness cannot be determined simply by observing where the failure appeared.
An agent that attempts an unauthorized tool operation might produce an entirely understandable explanation for its action. It may have interpreted the user's request as sufficient authorization, overlooked a restriction, or prioritized task completion over a competing instruction.
Yet improving that reasoning is not necessarily the most reliable correction.
A permission boundary enforced by the tool or execution environment can prevent an unauthorized operation regardless of whether the model recognizes the restriction. The model's behavior remains relevant, but security no longer depends exclusively on its willingness or ability to comply.
The component responsible for an undesirable behavior is not always the most effective place to correct it.
The scalability problem
There is another challenge hiding beneath the enthusiasm for explainability: the economics of human attention.
A system that produces a handful of consequential decisions each day can reasonably accommodate detailed expert review. A system generating millions of interactions cannot.
Even if every interaction includes a technically meaningful explanation, someone or something must still determine whether the behavior was acceptable, whether the explanation provides sufficient evidence, and whether the system requires correction.
We do not have the luxury of assigning an expert to every output.
Nor should that be the aspiration.
If explainability only makes individual failures easier for specialists to investigate, its operational value remains constrained by the availability of those specialists. As deployment scales, the volume of decisions increases, the number of possible failure conditions expands, and the cost of manual investigation becomes prohibitive.
Rather than treating each incorrect response as an isolated incident, we need mechanisms capable of identifying recurring patterns across large populations of interactions. We need to distinguish systematic problems from incidental variation, determine which failures warrant intervention, and evaluate whether a proposed correction improves behavior beyond the examples that originally exposed the problem.
Consider an assistant that repeatedly fails to incorporate exceptions in retrieved policy documents. Correcting one answer does little if thousands of subsequent interactions reproduce the same behavior.
The more valuable result is determining the conditions under which these failures occur, identifying an intervention that addresses those conditions, and establishing whether that intervention generalizes across the workload.
That last requirement is essential.
A change that corrects a collection of known failures may introduce new ones elsewhere. A more restrictive context strategy might reduce irrelevant information while excluding necessary evidence. Fine-tuning might improve compliance with one class of instructions while degrading performance on another. Additional verification might improve reliability while introducing unacceptable latency or cost.
Correction is therefore not simply the act of changing the system. It is the process of making a change and establishing, through evidence, that the system has meaningfully improved.
At scale, that process must be repeatable, measurable, and increasingly automated.
From explanation to intervention
The value of explainability is not proportional to the amount of information it can produce about an individual decision. It depends on whether that information helps us make better decisions about the system itself.
We should be able to observe behavior across components, evaluate that behavior against defined requirements, and investigate the conditions associated with failure. Those findings should support hypotheses about potential corrections, which can then be tested through controlled interventions.
The mechanisms for those interventions must span the entire AI stack.
A problem that appears in the model's answer may be corrected in retrieval, orchestration, or the permissions governing tool use. And certain boundaries should be enforced independently of the model's behavior rather than left to its interpretation.
There is no universally preferable intervention layer. The appropriate choice depends on the failure, the available evidence, and the operational constraints of the system.
Importantly, greater explanation does not always imply a better correction.
We may be able to establish empirically that an external permission constraint reliably prevents a particular class of unauthorized actions without understanding every aspect of the model's internal decision process. Conversely, we might possess an unusually detailed account of a model's behavior without identifying any intervention that reliably improves it.
This is why explanation, diagnosis, intervention, and validation should be treated as related but distinct capabilities.
The objective is not to maximize the sophistication of any one of them. It is to establish a working relationship among them.
Human oversight must become leverage, not labor
None of this suggests that human expertise becomes unnecessary.
In fact, more capable AI systems may increase the importance of expert judgment, particularly when establishing acceptable behavior, defining authority boundaries, evaluating novel failure modes, and deciding which tradeoffs an organization should tolerate.
But expertise and manual inspection are not the same thing.
The purpose of scalable oversight should be to make a relatively small amount of expert judgment consequential across a much larger population of decisions.
Experts can establish requirements, develop and validate evaluation methods, investigate unfamiliar patterns, and approve consequential interventions. Automated mechanisms can then apply those standards across large workloads, surface meaningful deviations, and test well-characterized corrections under controlled conditions.
This division of responsibility is not merely a matter of efficiency. It changes what meaningful oversight looks like.
An organization that requires experts to review every output has created a process whose reliability depends on the availability of human attention. An organization that can translate expert judgment into repeatable evaluations and enforceable constraints has created a means of extending that judgment across the system.
Neither automation nor explanation should be trusted simply because it exists. Evaluators can be wrong. Attribution signals can be misleading. Automated corrections can overfit the examples that motivated them. Monitoring can miss novel forms of failure.
These mechanisms require independent validation, appropriate escalation, and safeguards against interventions that create new problems.
The ambition should not be fully autonomous correction at any cost. It should be scalable correction with evidence sufficient to justify both the intervention and the confidence placed in it.
Reliability is a system property
There is a tendency to describe trustworthy AI in terms of properties of individual models: accuracy, alignment, robustness, interpretability, or explainability.
Each matters. None is sufficient.
A highly capable model can operate within an unreliable application. A well-aligned model can encounter malicious instructions through retrieved content. A model that generally respects authority boundaries can still make an unauthorized tool call if the execution environment delegates permissions it should have enforced independently.
Likewise, an explainable model does not make the system surrounding it explainable, and a transparent system does not become reliable merely because its failures are visible.
Trustworthy behavior emerges from how components interact, how requirements are enforced, how failures are detected, and how the system responds when expectations are violated.
A model explaining its decisions is not enough to evaluate the deployed system. We also need to know whether the changes we make actually hold up under different conditions.
The challenge is not simply to make AI systems more transparent. It is to make them more amenable to systematic evaluation and improvement.
That is a larger architectural responsibility than explainability alone can satisfy.
What comes after explainability?
I expect explainability techniques to improve considerably. We will develop more sophisticated methods for investigating how models incorporate context, respond to instructions, use tools, and interact with their environments. Better evidence of influence and more faithful representations of system behavior will help expose failures that remain difficult to investigate today.
Those advances are important.
But I would be concerned if our destination were simply an AI system capable of producing an increasingly comprehensive account of its own behavior.
A detailed explanation of a recurring failure is still a recurring failure.
The next challenge is to connect what we can observe with what we can reliably change.
That requires evidence capable of distinguishing among competing explanations, evaluation methods that establish whether behavior is acceptable, and mechanisms for intervention across the context, model, harness, tools, and execution environment. It also requires those mechanisms to operate at the scale of deployment, without assuming an expert will investigate every interaction.
A more meaningful ambition is a system whose failures can be detected, investigated, and addressed systematically, with evidence that the corrections work and safeguards that limit the consequences when they do not.
Explainability helps us understand where an AI system fails. The end state is an AI system we can reliably improve because of what we learn.
New notes, occasionally.
RSS