Before the Answer

During a recent ward discussion, I watched a colleague paste a clinical case into an AI and ask: "what should be done for a lab result this high?" One prompt, and the judgment had changed hands. The machine would now reason through the case; we would accept or reject its verdict. So we tried the opposite. We reworked the question around our own hypothesis, and the same engine returned something different: not a verdict, but the evidence, organized for and against what we were thinking. The decision became ours again.

Same case, same model, same physicians. The only thing that changed was the shape of the interaction, and with it, who thought first.

The erosion lives in the wrapper

Medical AI is being evaluated, purchased and adopted on the strength of one question: is it accurate? That question matters, but it hides a more urgent one: what does the tool do to the judgment of the person using it?

Clinical judgment is a cultivated capacity: "the mental and moral, like the muscular powers, are improved only by being used" (Mill, On Liberty). Diagnostic and management reasoning is the register where a physician's practical wisdom forms: observe, hypothesize, test against reality, commit to a course of action, learn from the outcome. A tool that runs this cycle for the physician, however helpful in the moment, confiscates the exercise that keeps her capacity alive.

This is not speculation. Automation bias in clinical decision support has been documented for over a decade (Goddard et al., 2012). Deskilling has now shown up in the wild: in a multicenter observational study, the adenoma detection rate of experienced endoscopists working without AI fell six percentage points after months of routine exposure to AI assistance (Budzyń et al., 2025). The risk is starker for trainees. Some now speak of "never-skilling": those who, learning medicine from oracles, never form the skill in the first place.

That word, trainees, is where this stops being abstract for me. I have practiced pediatrics for twenty years, and for much of that time I have also been a preceptor. Clinical judgment forming in a resident is something I have watched case by case: hypothesis by hypothesis, mistake by mistake, slowly, and only when the resident does the thinking herself. Whether the tools we are now handing her will let her keep doing that thinking decides what kind of physician she becomes. And right now, nobody is making that decision on purpose. It arrives as a default.

But here is the part I find most consequential: the erosion does not live in the model. It lives in the wrapper, the layer of interaction design between the model and the physician. The same engine that answered my colleague's deferring prompt with a verdict answered our reworked prompt with evidence. On the ward, we could rework the question because we knew to; a wrapper hard-codes one of those two shapes as its default, and the default is what every physician who does not fight the tool receives. The design a model is wrapped in decides whether the physician makes the first move, the exercise of her own judgment, or is invited to abdicate it.

Invisible exactly where it matters

Now try to assess that property before adopting a tool. Nothing will help you. Benchmarks measure the model's accuracy. Reporting guidelines describe the study after it is done. Over-reliance research needs users in a lab. Design guidelines exist, but they advise builders; nothing scores a finished tool. None of these look at the interaction structure of the tool itself: the property that shapes, at each use, whether judgment is exercised or displaced, and the only one that could be inspected before adoption, if anything told you what to look for.

And I am not the one asserting this gap: the field has admitted it. DECIDE-AI, the reporting guideline for early clinical evaluation of AI systems, excluded trust from its consensus because "there is currently no commonly accepted way to measure trust in the context of clinical AI" (Vasey et al., 2022). The field's own guideline names the hole: the human side of clinical AI has no accepted measure. Interaction structure is the tractable end of that hole, the part you can inspect without a user study.

Three inconvenient findings

Two objections come up whenever I discuss this. The first: "explanations solve it; make the AI show its reasoning." The second: "regulate it." The experimental literature is inconvenient to both.

Explanations first. Chain-of-thought prose can be an unfaithful rationalization: a model can reach its conclusion for reasons its stated reasoning never mentions, so reading the chain feels like verification but isn't (Turpin et al., 2023). Explanations reduce over-reliance under a specific condition: when they make verifying cheaper than deferring (Vasconcelos et al., 2023). A wall of confident prose raises that cost; evidence linked to checkable sources, on both sides of the question, lowers it. What protects judgment is not more explanation. It is cheaper contestation.

Regulation second. The designs that best preserve judgment, the ones that ask for your hypothesis first and slow you down at the right moment, are the ones users like least (Buçinca et al., 2021). A market steered by in-use preference alone will not select for judgment preservation. But dispreference in the moment is not considered choice: the physician who resents friction at 3 a.m. can still choose, on reflection, the tool that keeps her skill alive, the way she chooses the harder residency over the easier one. What she cannot do is choose a property she cannot see. A mandate deciding for her would repeat the disease in the name of the cure, protecting her judgment by overriding it. The remedy is disclosure: make the defaults visible, so that the clinician choosing a tool, and the builder designing one, can decide deliberately.

And the same literature hides a third finding, inconvenient only to fatalism: the oracle is not the only wrapper. Decision support can be evaluative: it can take your hypotheses and lay out the evidence for and against each, so that the judgment stays yours (Miller, 2023). That is what our reworked prompt on the ward stumbled into. The failure of the oracle is not in being wrong; it is in answering before you have thought.

Making the defaults visible

So I am building that instrument, a defaults-revealer for medical AI: First, Think. It is a set of low-inference structural items, each answerable by looking at a tool for minutes, not months: Does it elicit your hypothesis before revealing its own? Does it return a single conclusion, or a differential you can contest? Does it ask you to commit to a judgment before proceeding? Does it link claims to sources you can check? The items profile a tool's risk of displacing clinical judgment across five dimensions, never as a single score, because a single score would be one more thing to defer to.

The instrument comes with its proof. A pre-registered catalog of about eight real tools (from general LLMs used as diagnosticians to commercial clinical decision support), scored with a public protocol, showing whether designs actually differ. And a clickable contrast pair: the same clinical case, the same engine, two wrappers, oracle versus augmentation, so that anyone can feel, in two minutes, the difference between being answered and being asked.

It is worth being explicit about what this is not. It is not a benchmark: it does not measure whether the AI is right. It is not a certification, a seal, or a call for regulation: it is a voluntary epistemic instrument in the hands of whoever chooses tools and whoever builds them. And it does not claim to measure behavior or outcomes: it measures structural risk (what the design offers to the physician who does not fight against it) and declares the rest, honestly, as future work.

The instrument is designed to make itself unnecessary. After a few applications, a physician has internalized its questions and no longer needs it to see a tool clearly. A good tool is like a good teacher: it prepares you to need it less.

The defaults are being set right now. The tools being adopted today are shaping what "medical AI" feels like for a generation of physicians. And most of those tools answer first. A physician is a knower, not a dispatcher of cases: her skill exists only as long as it is exercised. Between her and the ready-made answer there remains one point in the chain where the first move is still entirely hers: the choice of the tool. That choice deserves to be an informed one.

The question that matters most about a medical AI tool is not "is it right?" It is: "who thinks first when I use it?"