This distinction between model risk and institutional responsibility is consequential.
I would extend the inquiry one step upstream: before asking who remains accountable when an AI-enabled decision goes wrong, what did the institution authorize—and did the humans granting that authority recognize what they were delegating?
An AI-generated conclusion can acquire institutional authority long before anyone formally decides to give it authority. It can happen through reliance: one person accepts the output, another assumes it was reviewed, and eventually the machine’s judgment becomes embedded in the organization’s judgment.
A human being “in the loop” matters only if that human remains oriented enough to recognize when judgment is being substituted, not merely assisted.
This is where accountability may begin before the failure—and even before the deployment.
LaMont, your point suggests a useful test: does accepting the AI’s recommendation require less justification than rejecting it? If so, the institution has already begun delegating judgment, regardless of what its formal policy says. Meaningful oversight depends on who bears the burden of proof, and whether people have the time, evidence and organisational support to challenge the default.
Gonçalo, precisely. You’ve identified an important institutional indicator: when accepting machine-generated judgment becomes easier than exercising independent human judgment, authority may have already shifted.
I would add another dimension: does the human recognize that this shift has occurred?
An institution can preserve formal oversight while unintentionally creating conditions that discourage discernment. This is why I see human orientation and Pre-Delegation Literacy as foundational to effective governance.
Your proposed test offers a practical way to make that distinction observable. I suspect there’s considerable value in exploring what this reveals across actual enterprise deployments.
That suggests a second test: can the reviewer explain what would make them disagree with the AI?
If they cannot, the right to override may mean very little.
In actual deployments, I’d also ask people to form an initial view before seeing the recommendation. The gap between those two judgements could reveal assumptions that a routine approval process keeps hidden.
Gonçalo, that’s a consequential distinction. The ability to override an AI recommendation is not the same as the capacity to exercise independent judgment.
I particularly appreciate your suggestion of capturing the human’s initial assessment before introducing the AI recommendation. That comparison could make otherwise invisible shifts in judgment and reliance observable.
I’d extend the inquiry one step further: when the two judgments differ, does the human understand why—and have the confidence, evidence, and institutional support to act on that distinction?
This is where Pre-Delegation Literacy becomes especially relevant. Oversight isn’t simply the permission to disagree. It requires humans who recognize what they’re being asked to judge, what they may be surrendering, and what remains theirs to determine.
We’re getting closer to something institutions could actually examine.
This distinction between model risk and institutional responsibility is consequential.
I would extend the inquiry one step upstream: before asking who remains accountable when an AI-enabled decision goes wrong, what did the institution authorize—and did the humans granting that authority recognize what they were delegating?
An AI-generated conclusion can acquire institutional authority long before anyone formally decides to give it authority. It can happen through reliance: one person accepts the output, another assumes it was reviewed, and eventually the machine’s judgment becomes embedded in the organization’s judgment.
A human being “in the loop” matters only if that human remains oriented enough to recognize when judgment is being substituted, not merely assisted.
This is where accountability may begin before the failure—and even before the deployment.
LaMont, your point suggests a useful test: does accepting the AI’s recommendation require less justification than rejecting it? If so, the institution has already begun delegating judgment, regardless of what its formal policy says. Meaningful oversight depends on who bears the burden of proof, and whether people have the time, evidence and organisational support to challenge the default.
Gonçalo, precisely. You’ve identified an important institutional indicator: when accepting machine-generated judgment becomes easier than exercising independent human judgment, authority may have already shifted.
I would add another dimension: does the human recognize that this shift has occurred?
An institution can preserve formal oversight while unintentionally creating conditions that discourage discernment. This is why I see human orientation and Pre-Delegation Literacy as foundational to effective governance.
Your proposed test offers a practical way to make that distinction observable. I suspect there’s considerable value in exploring what this reveals across actual enterprise deployments.
That suggests a second test: can the reviewer explain what would make them disagree with the AI?
If they cannot, the right to override may mean very little.
In actual deployments, I’d also ask people to form an initial view before seeing the recommendation. The gap between those two judgements could reveal assumptions that a routine approval process keeps hidden.
Gonçalo, that’s a consequential distinction. The ability to override an AI recommendation is not the same as the capacity to exercise independent judgment.
I particularly appreciate your suggestion of capturing the human’s initial assessment before introducing the AI recommendation. That comparison could make otherwise invisible shifts in judgment and reliance observable.
I’d extend the inquiry one step further: when the two judgments differ, does the human understand why—and have the confidence, evidence, and institutional support to act on that distinction?
This is where Pre-Delegation Literacy becomes especially relevant. Oversight isn’t simply the permission to disagree. It requires humans who recognize what they’re being asked to judge, what they may be surrendering, and what remains theirs to determine.
We’re getting closer to something institutions could actually examine.