Healthcare

Clinical systems combine unusually high stakes with unusually strong claims. A diagnostic or triage tool may perform well in evaluation and fail in a particular population, a particular workflow, or in the hands of a clinician given no practical way to disagree with it. The consequences are immediate, individual and sometimes irreversible, which makes healthcare both the most demanding setting we work in and the most instructive.

The failure we document most consistently is not diagnostic error but the erosion of override. A clinician is formally entitled to depart from a system's recommendation, and is simultaneously measured on throughput that departing reduces. Within months the override rate approaches zero, and the institution reports this as evidence of alignment. It is better read as a record of the pressure not to disagree. A safeguard that is never exercised is not functioning; it is being reported as functioning. Alongside this sits the transfer problem: a tool validated on one institution's historical data, deployed in another with a different population and workflow, with the original validation cited as though it carried over.

We work on the conditions for clinical deployment: validation in the population that will actually be treated, workflow design that preserves rather than nominally permits override, honest communication of uncertainty to the clinician at the point of decision, clear liability, and stopping rules defined before launch. We also work in the other direction. Clinical governance is among the most mature ethical infrastructure that exists — staged deployment, adverse event reporting with protection for the reporter, professional obligations that outrank an employer's instruction. Those mechanisms took a century and considerable harm to build, and they transfer further than is generally recognised.