Accountability Without Interpretability: Who Owns a Model’s Decision?
Executive Summary
When an algorithm influences a clinical or operational decision, one question always comes—eventually:
Who is responsible for this outcome?
Too often, the answer is vague:
- “The model flagged it.”
- “The system recommended it.”
- “The algorithm decided.”
These answers are ethically unacceptable.
This post argues a central point:
Accountability does not come from interpretability.
It comes from ownership, governance, and traceability.
That distinction matters in healthcare AI, where the moral and operational question is not only how a model works, but who approved it, who monitors it, and who is answerable when it changes care (London 2019; Sendak et al. 2020).
A model can be opaque and accountable.
A model can be interpretable and unaccountable.
The difference is the system around it.
Why Interpretability Gets Mistaken for Accountability
Calls for interpretability often stand in for deeper concerns:
- Who approved this?
- Who validated it?
- Who monitors it?
- Who answers when it fails?
Interpretability feels like accountability because it offers:
- a visible mechanism,
- something to point at,
- a sense of control.
But the broader literature on explanation and accountability keeps returning to the same point: explanation can support oversight, yet it does not assign responsibility by itself (Amann et al. 2020; Lipton 2018).
But seeing inside a model does not tell you:
- why it was deployed,
- whether it was appropriate,
- who owned the decision to act on it.
Accountability Is Not a Model Property
A critical distinction:
- Interpretability describes how a model works.
- Accountability describes how responsibility is assigned.
In other words, one is a property of the model or its presentation; the other is a property of the socio-technical system around it (Barocas et al. 2023).
Accountability lives in:
- people,
- processes,
- documentation,
- governance structures.
No coefficient table has ever accepted responsibility for harm.
The Dangerous Comfort of “The Model Did It”
When responsibility is diffused into “the model,” several things happen:
- no one feels empowered to override,
- no one feels obligated to monitor,
- failures are framed as technical glitches,
- ethical review is deferred indefinitely.
This is not a tooling problem.
It is a governance failure.
Who Actually Owns a Model’s Decision?
In a defensible system, ownership is explicit and layered:
Model owner
Responsible for development, validation, and updatesSystem owner
Responsible for integration, thresholds, and deployment contextOperational owner
Responsible for how outputs are used in practiceClinical / decision owner
Responsible for acting (or not acting) on the output
If you cannot name these roles, accountability does not exist.
Why Interpretability Alone Cannot Assign Responsibility
Suppose a clinician asks:
“Why did the model flag this patient?”
An explanation might describe:
- feature contributions,
- patterns learned,
- internal weights.
But it still doesn’t answer:
- Why this threshold?
- Why this moment?
- Why this workflow?
- Why this population?
Those choices are human decisions, not model mechanics.
Accountability Requires Traceability, Not Transparency
A defensible system can answer:
- What data was used?
- What model version?
- What parameters?
- What threshold?
- What output?
- What action followed?
- Who reviewed or overrode it?
- When did this occur?
decision_log <- tibble::tibble(
timestamp = Sys.time(),
subject_id = id,
model_version = "v2.1",
input_data_version = "2026-01-01",
risk_score = score,
threshold = 0.18,
alert_fired = score > 0.18,
action_taken = "Escalated",
decision_owner = "On-call clinician"
)This is accountability in practice.
Human-in-the-Loop Is Not Automatic Accountability
“Human-in-the-loop” is often used as a shield:
“A human reviewed it, so we’re covered.”
But if the human:
- lacks time,
- lacks context,
- is conditioned to accept alerts,
- or is blamed only after failure,
then accountability has not been meaningfully assigned.
Human involvement without authority is moral theater.
Governance Is Where Accountability Lives
Accountable systems have:
- a clear purpose statement,
- defined scope of use,
- validation criteria,
- monitoring plans,
- review cadence,
- update triggers,
- decommission rules.
None of these require interpretability. All of them require ownership.
What Happens When the Model Is Wrong?
This is the accountability stress test.
A responsible system can answer:
- Was the model outside its intended scope?
- Did the data violate assumptions?
- Did performance degrade?
- Was the alert ignored—and why?
- What corrective action follows?
If failure leads only to:
“The model was wrong,”
then no one was accountable.
Ethical Accountability Scales With Impact
The more consequential the decision:
- the greater the documentation burden,
- the clearer the ownership,
- the stronger the monitoring,
- the tighter the governance.
Opacity increases the need for accountability — not the opposite.
How to Explain This to Skeptics
Try this framing:
“We don’t hold medical devices accountable because we can see inside them. We hold people and systems accountable for how they are designed, validated, and used.”
That usually reframes the debate.
A Simple Accountability Checklist
Before deployment, ask:
- Who owns this model?
- Who approves updates?
- Who sets thresholds?
- Who monitors performance?
- Who can stop it?
- Who answers when it fails?
If any answer is “the model,” stop.
When a clinical AI model built on DoDTR data is deployed via MAVEN and a patient outcome is poor, accountability is genuinely ambiguous: the data came from DoD, the model was built by a contractor, the deployment is managed by a program office, and the clinical decision was made by a physician who may not have known the model’s validation history. The DoD AI ethical principles explicitly require clear lines of human accountability, but governance frameworks specifying who owns model failures in military health have not kept pace with deployment speed. Without a designated accountable entity — one with both authority and obligation — model errors surface as system failures with no owner. The result is that accountability exists on paper but dissolves in practice whenever an adverse outcome requires someone to answer for it.
Closing: Accountability Is a Human Obligation
Interpretability can be helpful. Sometimes it is even necessary.
But it is not accountability.
Accountability is the willingness to say:
“This system exists because we built it, deployed it, and chose how to use it.”
Until that sentence can be spoken plainly, no amount of explainability will make a system ethical.
This post is part of the Prediction Modeling Toolkit — a companion reference with model accountability frameworks, ownership and traceability templates, governance checklists, and clinical AI audit scaffolds.
Series Callout
This post is part of a broader Ethics in Trauma Registry Analysis Series:
- Opacity Is Sometimes Ethical: When Black Boxes Save Lives
- Accountability Without Interpretability: Who Owns a Model’s Decision?
- Bias Isn’t Always Where You Think It Is: Ethical Failure Modes in Registry Data
- Prediction vs Responsibility: Why Risk Scores Can Be Ethically Dangerous
- Human-in-the-Loop Is Not a Panacea (and Sometimes a Lie)
- The Ethical Implications of Excluding “Messy” Patients
- Missingness as a Fairness Issue in Machine Learning
- You Can’t Trust What You Don’t Track: AI Performance Monitoring in Clinical Systems
- From Weeks to Minutes: The Ethics of Automating CPG Compliance
- Ontology Is Not Optional: Semantic Infrastructure as Ethical Foundation
- What Responsible AI in Clinical Guidance Actually Requires
- Modernizing the DOD Trauma Registry: An Ethical and Technical Imperative