The Babysitting Problem: Why Human-in-the-Loop AI Governance Fails at Scale

The Babysitting Problem: Why Human-in-the-Loop AI Governance Fails at Scale

Cyril Treacy

COO and Co-Founder

A practitioner guide to why enterprise AI governance defaults to the wrong oversight model, how each of the three human-agent designs fails in production, and what the monitoring infrastructure for meaningful oversight actually requires.

A practitioner guide to why enterprise AI governance defaults to the wrong oversight model, how each of the three human-agent designs fails in production, and what the monitoring infrastructure for meaningful oversight actually requires.

Key Takeaways

  • Most enterprise AI governance defaults to human-in-the-loop approval, which degrades into rubber-stamping at scale. The cognitive crumple zone is real and predictable.

  • Nominal human presence in a workflow is not the same as meaningful human judgment. The airline automation lesson from the 1990s applies directly to enterprise AI oversight today.

  • The supervisor (Human-on-the-Loop) model is better than the gatekeeper, but only works if the monitoring infrastructure surfaces the right exceptions at the right time.

  • The strategist model (Human-in-the-Lead) preserves the most meaningful human judgment by reserving human attention for policy decisions, not transaction approvals.

  • EU AI Act and FCA standards require substantive human oversight, not nominal. Approval logs from fatigued reviewers will not satisfy a regulatory audit.

  • Moving from gatekeeper to strategist oversight requires runtime observability infrastructure first. Without it, reducing transaction-level approval means losing governance, not improving it.

Key Takeaways

  • Most enterprise AI governance defaults to human-in-the-loop approval, which degrades into rubber-stamping at scale. The cognitive crumple zone is real and predictable.

  • Nominal human presence in a workflow is not the same as meaningful human judgment. The airline automation lesson from the 1990s applies directly to enterprise AI oversight today.

  • The supervisor (Human-on-the-Loop) model is better than the gatekeeper, but only works if the monitoring infrastructure surfaces the right exceptions at the right time.

  • The strategist model (Human-in-the-Lead) preserves the most meaningful human judgment by reserving human attention for policy decisions, not transaction approvals.

  • EU AI Act and FCA standards require substantive human oversight, not nominal. Approval logs from fatigued reviewers will not satisfy a regulatory audit.

  • Moving from gatekeeper to strategist oversight requires runtime observability infrastructure first. Without it, reducing transaction-level approval means losing governance, not improving it.

Every compliance conversation I have starts the same way. The enterprise has a human-in-the-loop design. Someone in legal or risk signed off on it. The team feels covered. Then I ask how many approvals a day the assigned reviewer is processing. The number comes back in the hundreds. At that point, the governance isn't governance. It's babysitting.

This isn't a niche concern. It's the most common failure mode in enterprise AI oversight right now, and it has a name in safety research: the cognitive crumple zone. The human is physically present in the workflow. They've stopped genuinely evaluating anything.

Getting out of this trap requires thinking clearly about three distinct oversight models, why enterprises default to the worst one, and what the better ones actually need to work.

Why the Gatekeeper Model Feels Safe (and Isn't)

The gatekeeper model is the default for a reason. It feels responsible. The agent recommends an action, the human approves each step before anything proceeds. At low volume, in genuinely high-stakes decisions where deliberation is possible, it works. That's the context it was designed for.

At scale, it fails in a specific and entirely predictable way.

A January 2026 Arion Research study on agentic AI deployment put precise language to what practitioners have been observing for years. When the approval volume reaches a point where careful review is no longer operationally realistic, the human shifts from active evaluator to passive monitor. Approval fatigue is real. Skilled professionals who started out reading every decision carefully move, over weeks, into a pattern of scanning and clicking. First reviews: careful. After 200 correct decisions in a row: reflexive. When the system finally produces a failure worth catching, the human is poorly positioned to notice it, diagnose it, or stop it.

The airline industry learned a version of this in the 1980s and 1990s. As cockpit automation increased, pilots spent less time actively flying and more time watching systems that rarely needed intervention. When automation failed unexpectedly, some crews lacked the situational awareness to recover. The solution wasn't less automation. It was better-designed human-automation interaction: keeping pilots meaningfully engaged, not just nominally present.

Enterprise AI oversight has the exact same structural problem. Inserting a human approval step is not the same as maintaining genuine human judgment in the loop. At volume, those two things diverge completely.

There's also a specific regulatory liability worth naming directly. Gatekeeper designs produce the appearance of oversight without the substance. The audit log shows human approvals at every step. An FCA review or EU AI Act compliance check sees a clean record. But if those approvals came from a reviewer clicking through at four seconds per transaction, the documentation is misleading. Regulators are increasingly asking not just whether humans were present in the process, but whether the oversight was substantive. That distinction matters now. It will matter more as enforcement matures.

The Supervisor Model: Better, With Conditions

The supervisor model (Human-on-the-Loop) is a genuine step forward. The agent acts within defined guardrails. The human monitors dashboards and exception alerts, intervening when something falls outside expected parameters. This shifts the human from transaction approver to system monitor.

It handles volume far better than the gatekeeper model. It focuses human attention on genuine exceptions rather than routine approvals. For many mid-risk agentic deployments, it's an appropriate design.

But it has its own failure condition, and enterprises rarely think through it before deploying.

The supervisor model only works if the monitoring infrastructure is good enough to surface the right exceptions at the right time. Coarse alerting produces noise (alerts that need no intervention) or silence (real problems that never trigger an alert). Both degrade the human's ability to respond. A supervisor who receives 200 low-significance alerts a day starts ignoring them. A supervisor who rarely sees alerts builds false confidence about a system that may be drifting quietly.

At Disseqt, we see this play out in production regularly. Organisations deploy supervisor-model oversight and then discover their monitoring only captures the failures they anticipated during design. Distribution shift, prompt injection attempts, and gradual behavioural drift don't always trigger the alerts configured weeks earlier. The human supervisor is present. The monitoring layer is underspecified. The coverage looks complete. It isn't.

The other issue is reaction time. Agents in production can cascade non-deterministic behaviour through connected systems in milliseconds. A human supervisor watching a dashboard cannot respond at that resolution. When a multi-agent pipeline fails, the cascade has usually propagated three or four systems deep before the dashboard reflects it. Supervisor-model oversight needs automated intervention at the infrastructure layer, not a human watching a screen.

Human-in-the-Lead: The Strategist Model

The strategist model is where serious enterprise AI governance lands when it matures. Humans define the objectives, set the policy constraints, and own accountability for outcomes. Agents handle tactical execution. The human role is not approving transactions or watching dashboards. It's setting the rules, reviewing the evidence, and adjusting policy when behaviour drifts from intent.

This preserves the most meaningful form of human judgment, because it reserves human attention for decisions that actually require it. A senior compliance officer who spends her day approving chargeback-automation decisions one by one is wasting her expertise. That same compliance officer defining the policy constraints governing 50,000 daily transactions, reviewing the exception log weekly, adjusting risk thresholds when the evidence warrants it: that's the right use of her judgment.

The Arion Research study describes what they call a co-audit workflow that makes this model defensible in practice. When agents escalate to humans, they present a reasoning trace, not a binary "can I do this?" The human sees what data sources the agent consulted, what the confidence level was, and what specific uncertainty triggered the escalation. The human reviews the reasoning, not just the proposed action. That keeps genuine evaluation in the process even when routine decisions run autonomously.

What makes the strategist model work is real-time transparency into agent behaviour at the level of detail needed for sound policy decisions. Not a weekly summary report. Actual visibility into what the agent is doing, why, and where its confidence is low. That visibility is the foundation of meaningful Human-in-the-Lead oversight.

And this is precisely where most enterprises are stuck. They want the strategist model. They don't yet have the monitoring infrastructure that would make it safe to trust.

Comparing the Three Models

Here's how the three oversight designs compare across the dimensions that matter for enterprise deployment:

Oversight Model

Human Role

Scales to High Volume?

Genuine Judgment Preserved?

Monitoring Requirement

Regulatory Standing

Human-in-the-Loop (Gatekeeper)

Approves every transaction

No

No (degrades to rubber-stamping at volume)

Low (human is the check)

Looks compliant; may not be substantive

Human-on-the-Loop (Supervisor)

Monitors dashboards, responds to alerts

Yes, with caveats

Partial (depends on alert quality and coverage)

High (solid alerting infrastructure needed)

Defensible with good exception records

Human-in-the-Lead (Strategist)

Sets policy, reviews reasoning traces, owns outcomes

Yes

Yes (human attention on highest-value decisions)

Very high (real-time observability required)

Strongest: documented, continuous, evidenced

The regulatory column matters. The EU AI Act requires human oversight of high-risk AI systems. But the Act's expectation is not that a human clicked approve. It's that oversight was substantive, documented, and continuous. A gatekeeper model with fatigued reviewers will not satisfy that bar when an auditor looks closely. The strategist model, backed by monitoring infrastructure that generates the evidence trail, is the design regulators are describing when they write about meaningful human oversight.

The Financial Conduct Authority (FCA) in the UK has been equally direct. The FCA's 2025 AI guidance made clear that firms must demonstrate human oversight was genuine and effective, not nominal. The documentation burden alone is a reason to move toward infrastructure that generates compliance evidence continuously rather than assembling it retrospectively before an audit.

What "Agentic Theater" Looks Like in Practice

There's a version of all three models I'd call agentic theater: the POC worked brilliantly, the governance design looked sound, and then production happened.

The gatekeeper version: a financial institution deploys an agent with a human approval step on every transaction. Six months later, the review queue is 400 items deep, the reviewer is spending under three seconds per item, and a policy violation that should have been caught in week two has been silently approved 1,800 times.

The supervisor version: a regulated enterprise implements dashboard monitoring and configures alerts for the failure modes identified during testing. The system runs cleanly for 90 days. A prompt injection attack then triggers behaviour the alert configuration didn't anticipate. The supervisor sees nothing unusual. The failure propagates for 11 days before a downstream system flags an anomaly.

The strategist version without infrastructure: a CTO decides the team has earned the right to reduce transaction approval and move to policy-level oversight. The policy is set. The agents are given more autonomy. Without runtime observability, the policy isn't being enforced at the inference layer. It's a document. The agents operate against it until something forces a review.

None of those are hypotheticals. They're composites of what I see in enterprise conversations every quarter. The common thread is the gap between the governance model on paper and the operational reality underneath.

The Infrastructure Gap That Keeps Enterprises in Gatekeeper Mode

When I talk to CISOs and compliance leads about moving from gatekeeper to strategist oversight, the honest answer to "why haven't you?" is usually the same: they don't have enough visibility into what their agents are doing to feel confident reducing transaction-level approval. The gatekeeper model is inefficient and hollow. But it feels safer than releasing control without a clear view of what's happening underneath.

That instinct is correct. The strategist model is not just an organisational design choice. It's a technical requirement. You cannot give humans the strategist role if you can't show them in real time what the agents are doing, where they're operating outside expected parameters, and what the reasoning was behind consequential decisions.

This is what the AI Assurance Layer provides in practice. Pre-production testing with the CONSUL structure covers 84+ jailbreak techniques and 65+ input validators in a 3-day cycle (compared to the 4-week manual testing cycles most enterprises were running). That 80% productivity gain and 63% cost reduction are real, but the more important output is the evidence base that justifies moving toward less restrictive oversight post-deployment.

Pre-production testing catches the failure modes you can anticipate before they reach production. But it cannot catch everything. Drift, distribution shift, and policy violations that emerge only under production conditions need continuous monitoring after go-live.

Runtime policy enforcement at the inference layer means the guardrails travel with the agent into production. Not a policy document in a governance folder. Actual enforcement at the point where decisions are made. Millisecond-resolution monitoring across multi-agent pipelines catches and stops cascade failures at the infrastructure layer, rather than waiting for a supervisor to notice something wrong on a dashboard three minutes later.

The progression that works: start with gatekeeper oversight and monitoring active together. Move to supervisor oversight as monitoring proves reliable. Move to strategist oversight as confidence in the policy layer and the observability layer reaches the point where transaction-level approval adds cost without adding genuine governance value. The monitoring infrastructure is not optional at any stage. It's what makes each transition defensible, and what separates genuine governance from agentic theater.

Want to see how this works in practice? Request a technical demo and we'll walk through the monitoring architecture with your specific agent deployment in mind.

Bottom line

The gatekeeper model feels safe because it puts a human in the process. The strategist model is safe because it puts genuine judgment in the process. Those are different things, and regulators are increasingly asking which one you actually have.

Stop defending gatekeeper oversight because it looks responsible on paper. Start building the monitoring infrastructure that makes the strategist model real.

Cyril Treacy is COO of Disseqt AI, the AI Assurance Layer for Enterprises. Disseqt helps organisations test, protect, and monitor AI agents across the full deployment lifecycle. Learn more about the Disseqt platform, explore the CONSUL testing structure, or request a technical demo.

FAQs

01

What is the "cognitive crumple zone" in AI governance?

The cognitive crumple zone describes what happens when humans are assigned to monitor or approve AI agent decisions at a volume that makes genuine evaluation impossible. Over time, the human shifts from active assessment to passive monitoring. Their situational awareness degrades. When a real failure occurs, they're poorly positioned to recognise it or respond effectively. A January 2026 Arion Research study on agentic AI deployment applied this term to enterprise AI oversight, drawing a parallel to automation challenges that led to redesigned human-automation interaction in commercial aviation.

02

What is the difference between Human-in-the-Loop, Human-on-the-Loop, and Human-in-the-Lead?

Human-in-the-Loop means the agent recommends and the human approves each action before the agent proceeds. Human-on-the-Loop means the agent acts within guardrails and the human monitors for exceptions, intervening when something falls outside expected parameters. Human-in-the-Lead means the human sets objectives and policy constraints while the agent handles tactical execution autonomously. Each model requires a higher level of monitoring infrastructure to work as intended. Human-in-the-Lead offers the strongest regulatory standing, but only when paired with real-time observability and runtime policy enforcement.

03

Why does the Human-in-the-Loop model fail at enterprise scale?

At high transaction volumes, the human reviewer cannot maintain genuine evaluation of every decision. Approval fatigue sets in quickly: the first reviews are careful, but after weeks of correct decisions, attention wanes and approvals become reflexive. The governance documentation looks clean, but the oversight is no longer substantive. Under EU AI Act and FCA standards, this distinction matters. Regulators are asking whether human oversight was effective, not just whether it was present.

04

What monitoring infrastructure does the Human-in-the-Lead model require?

The strategist model requires real-time visibility into agent reasoning and decision-making, not just outcome logs. This includes runtime policy enforcement at the inference layer, millisecond-resolution monitoring for multi-agent pipelines, reasoning traces that explain why an agent made a specific decision, and continuous alerting calibrated to genuine policy violations rather than routine activity. Without this infrastructure, reducing transaction-level approval means losing governance, not improving it.

05

How does the EU AI Act define meaningful human oversight for agentic AI?

The EU AI Act requires that human oversight of high-risk AI systems be substantive, documented, and continuous. Nominal approval steps or policies that exist in documentation but aren't operationally enforced do not satisfy it. Organisations must demonstrate that oversight was effective, with evidence trails that support that claim. The Act has been in active enforcement since August 2024, and regulators are examining whether oversight designs would hold up under scrutiny, not just whether a human was technically present in the workflow.

AUTHOR

Cyril Treacy

COO and Co-Founder

Cyril is Co-Founder and COO at Disseqt, leading go-to-market, partnerships, and customer success. He brings 20+ years of enterprise sales, pre-sales leadership, and scaling expertise from Salesforce and the Irish startup ecosystem.

See Disseqt in action
Book a 30-minute walkthrough

Our team will walk you through a live workflow using your own AI environment. No slides. No generic demo. A real walkthrough of how Disseqt fits into your stack.

See Disseqt in action
Book a 30-minute walkthrough

Our team will walk you through a live workflow using your own AI environment. No slides. No generic demo. A real walkthrough of how Disseqt fits into your stack.

See Disseqt in action
Book a 30-minute walkthrough

Our team will walk you through a live workflow using your own AI environment. No slides. No generic demo. A real walkthrough of how Disseqt fits into your stack.