by Rehan Kauser
Photo courtesy of the author
Consider a large bank that uses an artificial intelligence system to recommend dispositions on anti-money-laundering alerts. The figures are illustrative. Every recommendation is routed to a human analyst for approval. The analysts clear roughly four hundred alerts a day. They spend well under a minute on each. They agree with the system in more than ninety-nine percent of cases. The audit trail shows a named human approval on every single decision.
Then an examiner asks a question the institution has never asked itself: in how many cases did the reviewer change the outcome?
Nobody has measured it. The record proves a human was present. It does not prove anyone was overseeing anything.
This is the gap between a control that exists and a control that works. The Department of Justice’s guidance on evaluating corporate compliance programs asks three fundamental questions, the third being whether it works in practice rather than on paper.[1] The 2024 revisions directed prosecutors to examine how companies assess and mitigate risks from emerging technology, including artificial intelligence, and reinforced the importance of data access, monitoring and testing in determining whether compliance controls work in practice.[2] An institution that relies on human review as its principal safeguard, and cannot demonstrate that reviewers are able to detect and correct errors when they occur, may be unable to show that the control works in practice — while holding a perfect record suggesting that it does. The Department does not state that a low override rate, standing alone, establishes failure; the inference drawn here is a practitioner’s.
Human review has often become the default control for AI decisions because it is legible to existing governance structures and comparatively easy to approve. It fits the vocabulary regulators already use, it allows second-line functions to sign off without resolving harder questions about model behavior, and it maps onto existing structures in which a named person is answerable for a decision. Most of all, it looks like the pre-AI world.
The difficulty is that the control was adopted without testing the assumption underneath it: that a human placed in the loop is able to perform the oversight the policy imagines.
Financial institutions already have a standard for this, and it does not come from AI policy. The revised interagency guidance on model risk management, issued in April 2026, treats effective challenge as critical analysis undertaken by informed parties with appropriate expertise, sufficient independence, organizational standing and the influence necessary to effect change.[3] That guidance expressly places generative and agentic AI systems outside its scope. For tools it does not cover, it directs banking organizations to rely on their own risk-management and governance practices to determine appropriate controls. It is therefore invoked here by analogy rather than as an applicable requirement — but the four attributes translate directly, and the guidance itself points institutions toward their own practices for precisely these systems. Applied to a human reviewing AI-generated decisions, the test has five parts.
Information. The reviewer can see what the system saw and why it produced the recommendation. Shown a conclusion without its basis, the reviewer is not reviewing but ratifying.
Time. The time per decision is proportionate to its consequence. Four hundred alerts a day is not a review rate; it is a throughput target.
Competence. The reviewer can evaluate the specific output, not merely recognize that it resembles the ones before it.
Influence and incentive. The reviewer has authority to disagree and is not measured on a metric that punishes it.
Demonstrated effectiveness. The reviewer can detect an error when one is present, and intervention, when it occurs, changes the outcome. This is the condition that distinguishes a control from a record — and the one that should be tested directly.
In the narrower Article 22 context, European data-protection guidance reaches a similar conclusion: human involvement must be meaningful rather than a token gesture, and must be exercised by someone with the authority and competence to change the decision.[4] Two very different regimes reach the same conclusion: the presence of a human is not the control. The capacity of that human to alter the outcome is.
The European Union’s AI Act names the difficulty directly, requiring that those assigned oversight of high-risk systems remain aware of the tendency to over-rely on automated output.[5] Naming it is easier than solving it. Decades of human-factors research show that people over-rely on automated recommendations, monitor them less attentively over time, and miss errors they would have caught unaided.[6] The effect runs both ways: reviewers accept incorrect advice, and fail to act when the system stays silent about something that warranted attention.[7]
The implication for institutions is uncomfortable. As the model improves, the reviewer’s scrutiny tends to decline — precisely as the institution begins relying on that reviewer to catch the rare, high-consequence failure. The safeguard is weakest where it is needed most. This is largely a property of the task as designed rather than a deficiency in the individuals performing it, and training alone is unlikely to eliminate it.
Ben Green’s survey of forty-one policies requiring human oversight of government algorithms concludes that they rest on an uninterrogated assumption that people can effectively oversee algorithmic decisions, and in practice legitimize flawed systems while allowing vendors and agencies to avoid accountability.[8] Substitute “business line” for “agency,” and the concern will be familiar in many financial-services AI governance programs.
A second problem runs in the opposite direction. Madeleine Clare Elish describes the moral crumple zone: when an automated system fails, responsibility is misattributed to the nearest human operator, who in fact had limited control over the system’s behavior. The human absorbs the impact, protecting the integrity of the technological system.[9]
A compliance program requiring a named approval on every AI-generated decision risks recreating that zone by design. When something goes wrong, the record identifies the nearest approver — one who reviewed four hundred alerts that day and had no realistic opportunity to disagree — rather than the people who designed the system, set its limits and determined the reviewer’s practical ability to intervene. That is not accountability but attribution, operating on the least empowered person in the chain.
The more immediate problem is evidentiary. The approval record intended to demonstrate a functioning control may, on examination, show only that a person was present. A reviewer who has never disagreed is not by itself proof that the system is accurate, nor proof that the review is ceremonial. It is a prompt for further testing rather than a source of reassurance — and the institution that has not run that test cannot say which explanation is true.
An institution claiming human review as a mitigating control should be able to produce the following.
Disagreement rates, disaggregated. Not an aggregate override percentage, but rates by reviewer, by decision type and by consequence tier. A rate near zero may mean the model is accurate, that reviewers see genuinely low-risk cases, or that review has become ceremonial. The number alone does not distinguish between them.
Time on decision, measured against complexity. Where the median review takes forty seconds and the decision requires reconciling several documents, the institution knows what its control is worth.
Controlled testing. Known-problem or synthetic cases presented in a segregated test environment, or introduced into a production-like workflow with appropriate safeguards, with a measured catch rate. This is one of the strongest ways to determine whether reviewers can detect and correct errors, because it tests capability directly rather than inferring it from override statistics. Although the cost will vary by workflow and control environment, this form of testing is often more feasible than institutions assume, and it remains uncommon in many review programs.
Override quality. When reviewers did disagree, were they right? A high override rate producing worse outcomes is a different problem, but a knowable one.
A contemporaneous record of what the reviewer could see — the information actually presented at the moment of decision, not a reconstruction after the fact.
Taken together, these measures allow an institution to distinguish model accuracy from reviewer passivity — a distinction an aggregate approval rate cannot make. The objective is not to manufacture disagreement or to set a preferred override rate. It is to establish, through risk-calibrated evidence, that the reviewer can identify and correct errors when they occur.
The standard follows from the evidence: an institution that cannot demonstrate that reviewers are able to detect and correct errors when warranted should be cautious about recording human review as a mitigating control in its risk assessment.
The conclusion is not that human oversight should be abandoned, but that it should be placed where it can function and replaced elsewhere with controls that do not depend on human attention.
For decisions that are low in consequence, readily reversible and not otherwise subject to a per-decision legal or policy requirement, institutions should consider replacing routine per-decision review with risk-based sampling and monitoring. That approach is often cheaper — and more honest about what is already happening.
For decisions that are consequential or irreversible, the institution should enforce limits before execution — thresholds, blast-radius constraints, mandatory escalation — and retain human review only where all five conditions genuinely hold. A limit enforced by the system does not tire, does not drift with volume, and does not accept a recommendation because the last four hundred were correct.
The governing principle is simple to state and hard to implement: do not place a human where that human cannot realistically say no. Place a limit there instead.
An earlier piece considered what happens when two individually authorized agents produce an outcome no person approved, and observed that authority evaluated at configuration cannot account for a state that did not exist when it was granted.[10] The problem here is the mirror image. There, the record was complete and no one had authorized the outcome. Here, a person has authorized every outcome, and the authorization may mean very little.
Both failures share a cause: each treats the existence of a control as equivalent to its operation. The agent’s permission was checked, so the action was governed. The human clicked approve, so the decision was reviewed. In neither case did anyone ask whether the control was capable of producing a different result.
For boards and second-line functions, the questions are these. Have reviewers been tested under controlled conditions with known-problem cases, and what proportion did they catch? For each AI-assisted decision subject to human review, what proportion of reviews change the outcome, and how is that figure explained? How much time does review take, relative to the complexity of the decision? Are reviewers measured on throughput or on accuracy? And where review cannot realistically disagree, what limit has been put in its place?
An institution that cannot answer the first of those questions has not yet established that it has a control. It has established that it has a record. If the human has never said no, the useful response is not reassurance and not alarm, but a test — and until that test is run, it is worth asking what, exactly, the institution has been relying on.
[1] U.S. Dep’t of Justice, Criminal Division, Evaluation of Corporate Compliance Programs (updated Sept. 2024), at 2 (setting out three fundamental questions: whether the program is well designed; whether it is applied earnestly and in good faith; and whether it works in practice).
[2] Id. at 3–4 (addressing emerging technology and artificial intelligence); id. at 19 (addressing compliance access to data and resources).
[3] Board of Governors of the Fed. Reserve Sys., Fed. Deposit Ins. Corp. & Office of the Comptroller of the Currency, Supervisory Guidance on Model Risk Management, SR 26-2 (Apr. 17, 2026) (attachment), at 6 (effective challenge is performed by individuals with the appropriate expertise to conduct a critical and objective challenge, sufficient independence to maintain objectivity, and the organizational standing and influence to effect any change); id. at 4 n.3 (generative and agentic AI models are not within the scope of the guidance, and a banking organization’s own risk management and governance practices should guide the determination of appropriate governance and controls for tools not covered). SR 26-2 supersedes SR letter 11-7 (Apr. 4, 2011) and SR letter 21-8 (Apr. 9, 2021). The standard is invoked here by analogy rather than as a directly applicable requirement.
[4] Article 29 Data Prot. Working Party, Guidelines on Automated Individual Decision-Making and Profiling for the Purposes of Regulation 2016/679, WP251rev.01 (adopted Feb. 6, 2018), at 21 (in the Article 22 context of decisions based solely on automated processing, human involvement must be meaningful rather than a token gesture, and must be carried out by someone with the authority and competence to change the decision).
[5] Regulation (EU) 2024/1689 (Artificial Intelligence Act), art. 14(4)(b) (applying to high-risk AI systems within the Regulation’s scope, and requiring that persons assigned human oversight remain aware of the possible tendency of automatically relying or over-relying on the system’s output). Not every AI system used in financial services falls within that classification.
[6] Raja Parasuraman & Dietrich H. Manzey, Complacency and Bias in Human Use of Automation: An Attentional Integration, 52 Hum. Factors 381 (2010).
[7] Linda J. Skitka, Kathleen L. Mosier & Mark Burdick, Does Automation Bias Decision-Making?, 51 Int’l J. Hum.-Computer Stud. 991 (1999).
[8] Ben Green, The Flaws of Policies Requiring Human Oversight of Government Algorithms, 45 Computer L. & Sec. Rev. 105681 (2022).
[9] Madeleine Clare Elish, Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction, 5 Engaging Sci., Tech. & Soc’y 40 (2019).
[10] Rehan Kausar, When Two Authorized Agents Do Something No One Approved, NYU Program on Corporate Compliance & Enforcement (Sept. 15, 2026).
Rehan Kausar is Founder and Chief AI Officer at AI Advantages.
The views, opinions and positions expressed within all posts are those of the author alone and do not represent those of the Program on Corporate Compliance and Enforcement (PCCE) or of the New York University School of Law. PCCE makes no representations as to the accuracy, completeness and validity or any statements made on this site and will not be liable any errors, omissions or representations. The copyright of this content belongs to the author and any liability with regards to infringement of intellectual property rights remains with the author.
by Rehan Kauser
Photo courtesy of the author
Consider a large bank that uses an artificial intelligence system to recommend dispositions on anti-money-laundering alerts. The figures are illustrative. Every recommendation is routed to a human analyst for approval. The analysts clear roughly four hundred alerts a day. They spend well under a minute on each. They agree with the system in more than ninety-nine percent of cases. The audit trail shows a named human approval on every single decision.
Then an examiner asks a question the institution has never asked itself: in how many cases did the reviewer change the outcome?
Nobody has measured it. The record proves a human was present. It does not prove anyone was overseeing anything.
This is the gap between a control that exists and a control that works. The Department of Justice’s guidance on evaluating corporate compliance programs asks three fundamental questions, the third being whether it works in practice rather than on paper.[1] The 2024 revisions directed prosecutors to examine how companies assess and mitigate risks from emerging technology, including artificial intelligence, and reinforced the importance of data access, monitoring and testing in determining whether compliance controls work in practice.[2] An institution that relies on human review as its principal safeguard, and cannot demonstrate that reviewers are able to detect and correct errors when they occur, may be unable to show that the control works in practice — while holding a perfect record suggesting that it does. The Department does not state that a low override rate, standing alone, establishes failure; the inference drawn here is a practitioner’s.
Human review has often become the default control for AI decisions because it is legible to existing governance structures and comparatively easy to approve. It fits the vocabulary regulators already use, it allows second-line functions to sign off without resolving harder questions about model behavior, and it maps onto existing structures in which a named person is answerable for a decision. Most of all, it looks like the pre-AI world.
The difficulty is that the control was adopted without testing the assumption underneath it: that a human placed in the loop is able to perform the oversight the policy imagines.
Financial institutions already have a standard for this, and it does not come from AI policy. The revised interagency guidance on model risk management, issued in April 2026, treats effective challenge as critical analysis undertaken by informed parties with appropriate expertise, sufficient independence, organizational standing and the influence necessary to effect change.[3] That guidance expressly places generative and agentic AI systems outside its scope. For tools it does not cover, it directs banking organizations to rely on their own risk-management and governance practices to determine appropriate controls. It is therefore invoked here by analogy rather than as an applicable requirement — but the four attributes translate directly, and the guidance itself points institutions toward their own practices for precisely these systems. Applied to a human reviewing AI-generated decisions, the test has five parts.
Information. The reviewer can see what the system saw and why it produced the recommendation. Shown a conclusion without its basis, the reviewer is not reviewing but ratifying.
Time. The time per decision is proportionate to its consequence. Four hundred alerts a day is not a review rate; it is a throughput target.
Competence. The reviewer can evaluate the specific output, not merely recognize that it resembles the ones before it.
Influence and incentive. The reviewer has authority to disagree and is not measured on a metric that punishes it.
Demonstrated effectiveness. The reviewer can detect an error when one is present, and intervention, when it occurs, changes the outcome. This is the condition that distinguishes a control from a record — and the one that should be tested directly.
In the narrower Article 22 context, European data-protection guidance reaches a similar conclusion: human involvement must be meaningful rather than a token gesture, and must be exercised by someone with the authority and competence to change the decision.[4] Two very different regimes reach the same conclusion: the presence of a human is not the control. The capacity of that human to alter the outcome is.
The European Union’s AI Act names the difficulty directly, requiring that those assigned oversight of high-risk systems remain aware of the tendency to over-rely on automated output.[5] Naming it is easier than solving it. Decades of human-factors research show that people over-rely on automated recommendations, monitor them less attentively over time, and miss errors they would have caught unaided.[6] The effect runs both ways: reviewers accept incorrect advice, and fail to act when the system stays silent about something that warranted attention.[7]
The implication for institutions is uncomfortable. As the model improves, the reviewer’s scrutiny tends to decline — precisely as the institution begins relying on that reviewer to catch the rare, high-consequence failure. The safeguard is weakest where it is needed most. This is largely a property of the task as designed rather than a deficiency in the individuals performing it, and training alone is unlikely to eliminate it.
Ben Green’s survey of forty-one policies requiring human oversight of government algorithms concludes that they rest on an uninterrogated assumption that people can effectively oversee algorithmic decisions, and in practice legitimize flawed systems while allowing vendors and agencies to avoid accountability.[8] Substitute “business line” for “agency,” and the concern will be familiar in many financial-services AI governance programs.
A second problem runs in the opposite direction. Madeleine Clare Elish describes the moral crumple zone: when an automated system fails, responsibility is misattributed to the nearest human operator, who in fact had limited control over the system’s behavior. The human absorbs the impact, protecting the integrity of the technological system.[9]
A compliance program requiring a named approval on every AI-generated decision risks recreating that zone by design. When something goes wrong, the record identifies the nearest approver — one who reviewed four hundred alerts that day and had no realistic opportunity to disagree — rather than the people who designed the system, set its limits and determined the reviewer’s practical ability to intervene. That is not accountability but attribution, operating on the least empowered person in the chain.
The more immediate problem is evidentiary. The approval record intended to demonstrate a functioning control may, on examination, show only that a person was present. A reviewer who has never disagreed is not by itself proof that the system is accurate, nor proof that the review is ceremonial. It is a prompt for further testing rather than a source of reassurance — and the institution that has not run that test cannot say which explanation is true.
An institution claiming human review as a mitigating control should be able to produce the following.
Disagreement rates, disaggregated. Not an aggregate override percentage, but rates by reviewer, by decision type and by consequence tier. A rate near zero may mean the model is accurate, that reviewers see genuinely low-risk cases, or that review has become ceremonial. The number alone does not distinguish between them.
Time on decision, measured against complexity. Where the median review takes forty seconds and the decision requires reconciling several documents, the institution knows what its control is worth.
Controlled testing. Known-problem or synthetic cases presented in a segregated test environment, or introduced into a production-like workflow with appropriate safeguards, with a measured catch rate. This is one of the strongest ways to determine whether reviewers can detect and correct errors, because it tests capability directly rather than inferring it from override statistics. Although the cost will vary by workflow and control environment, this form of testing is often more feasible than institutions assume, and it remains uncommon in many review programs.
Override quality. When reviewers did disagree, were they right? A high override rate producing worse outcomes is a different problem, but a knowable one.
A contemporaneous record of what the reviewer could see — the information actually presented at the moment of decision, not a reconstruction after the fact.
Taken together, these measures allow an institution to distinguish model accuracy from reviewer passivity — a distinction an aggregate approval rate cannot make. The objective is not to manufacture disagreement or to set a preferred override rate. It is to establish, through risk-calibrated evidence, that the reviewer can identify and correct errors when they occur.
The standard follows from the evidence: an institution that cannot demonstrate that reviewers are able to detect and correct errors when warranted should be cautious about recording human review as a mitigating control in its risk assessment.
The conclusion is not that human oversight should be abandoned, but that it should be placed where it can function and replaced elsewhere with controls that do not depend on human attention.
For decisions that are low in consequence, readily reversible and not otherwise subject to a per-decision legal or policy requirement, institutions should consider replacing routine per-decision review with risk-based sampling and monitoring. That approach is often cheaper — and more honest about what is already happening.
For decisions that are consequential or irreversible, the institution should enforce limits before execution — thresholds, blast-radius constraints, mandatory escalation — and retain human review only where all five conditions genuinely hold. A limit enforced by the system does not tire, does not drift with volume, and does not accept a recommendation because the last four hundred were correct.
The governing principle is simple to state and hard to implement: do not place a human where that human cannot realistically say no. Place a limit there instead.
An earlier piece considered what happens when two individually authorized agents produce an outcome no person approved, and observed that authority evaluated at configuration cannot account for a state that did not exist when it was granted.[10] The problem here is the mirror image. There, the record was complete and no one had authorized the outcome. Here, a person has authorized every outcome, and the authorization may mean very little.
Both failures share a cause: each treats the existence of a control as equivalent to its operation. The agent’s permission was checked, so the action was governed. The human clicked approve, so the decision was reviewed. In neither case did anyone ask whether the control was capable of producing a different result.
For boards and second-line functions, the questions are these. Have reviewers been tested under controlled conditions with known-problem cases, and what proportion did they catch? For each AI-assisted decision subject to human review, what proportion of reviews change the outcome, and how is that figure explained? How much time does review take, relative to the complexity of the decision? Are reviewers measured on throughput or on accuracy? And where review cannot realistically disagree, what limit has been put in its place?
An institution that cannot answer the first of those questions has not yet established that it has a control. It has established that it has a record. If the human has never said no, the useful response is not reassurance and not alarm, but a test — and until that test is run, it is worth asking what, exactly, the institution has been relying on.
[1] U.S. Dep’t of Justice, Criminal Division, Evaluation of Corporate Compliance Programs (updated Sept. 2024), at 2 (setting out three fundamental questions: whether the program is well designed; whether it is applied earnestly and in good faith; and whether it works in practice).
[2] Id. at 3–4 (addressing emerging technology and artificial intelligence); id. at 19 (addressing compliance access to data and resources).
[3] Board of Governors of the Fed. Reserve Sys., Fed. Deposit Ins. Corp. & Office of the Comptroller of the Currency, Supervisory Guidance on Model Risk Management, SR 26-2 (Apr. 17, 2026) (attachment), at 6 (effective challenge is performed by individuals with the appropriate expertise to conduct a critical and objective challenge, sufficient independence to maintain objectivity, and the organizational standing and influence to effect any change); id. at 4 n.3 (generative and agentic AI models are not within the scope of the guidance, and a banking organization’s own risk management and governance practices should guide the determination of appropriate governance and controls for tools not covered). SR 26-2 supersedes SR letter 11-7 (Apr. 4, 2011) and SR letter 21-8 (Apr. 9, 2021). The standard is invoked here by analogy rather than as a directly applicable requirement.
[4] Article 29 Data Prot. Working Party, Guidelines on Automated Individual Decision-Making and Profiling for the Purposes of Regulation 2016/679, WP251rev.01 (adopted Feb. 6, 2018), at 21 (in the Article 22 context of decisions based solely on automated processing, human involvement must be meaningful rather than a token gesture, and must be carried out by someone with the authority and competence to change the decision).
[5] Regulation (EU) 2024/1689 (Artificial Intelligence Act), art. 14(4)(b) (applying to high-risk AI systems within the Regulation’s scope, and requiring that persons assigned human oversight remain aware of the possible tendency of automatically relying or over-relying on the system’s output). Not every AI system used in financial services falls within that classification.
[6] Raja Parasuraman & Dietrich H. Manzey, Complacency and Bias in Human Use of Automation: An Attentional Integration, 52 Hum. Factors 381 (2010).
[7] Linda J. Skitka, Kathleen L. Mosier & Mark Burdick, Does Automation Bias Decision-Making?, 51 Int’l J. Hum.-Computer Stud. 991 (1999).
[8] Ben Green, The Flaws of Policies Requiring Human Oversight of Government Algorithms, 45 Computer L. & Sec. Rev. 105681 (2022).
[9] Madeleine Clare Elish, Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction, 5 Engaging Sci., Tech. & Soc’y 40 (2019).
[10] Rehan Kausar, When Two Authorized Agents Do Something No One Approved, NYU Program on Corporate Compliance & Enforcement (Sept. 15, 2026).
Rehan Kausar is Founder and Chief AI Officer at AI Advantages.
The views, opinions and positions expressed within all posts are those of the author alone and do not represent those of the Program on Corporate Compliance and Enforcement (PCCE) or of the New York University School of Law. PCCE makes no representations as to the accuracy, completeness and validity or any statements made on this site and will not be liable any errors, omissions or representations. The copyright of this content belongs to the author and any liability with regards to infringement of intellectual property rights remains with the author.
