Open Challenge: Measuring Override in Practice
An open call inviting teams anywhere to measure human override of automated decisions using a shared published protocol, so results can be compared rather than merely collected.
Override rates are reported inconsistently, defined differently by every operator, and consequently impossible to compare. This challenge invites teams in any jurisdiction to measure override using one published protocol.
Why an open challenge
The problem is shared but the institutional context is not. A protocol that works in one setting may be unusable in another, and the only way to find out is to have many teams attempt it and report what broke. That is a poor fit for a single research project and a good fit for a parallel open call.
What participants receive
- The measurement protocol and analysis code, published in advance.
- A shared results format so submissions are comparable.
- Publication of every submission that follows the protocol, including null and contradictory results.
Status
Planned. The protocol is being drafted by the Working Group on Automated Decision Review and will be published for comment before the challenge opens.
