Agency theory was built for human managers, but AI is becoming the agent nobody knows how to supervise

Agency theory emerged because firm owners started handing decision rights to professional managers who did not own the company. That transfer created the classic problem: the agent has more information than the principal, different risk preferences, and self-interested motives. Monitoring and incentive alignment (think board oversight and stock options) evolved as the standard toolkit for keeping managers aligned with shareholders.

Humberd and Latham notice that something structurally similar is happening with AI. As AI systems move from mimicking human routines to exercising autonomous judgement, they acquire decision rights over firm resources. The difference is that the agent in question is not a person motivated by wealth. You cannot give an AI system stock options in any meaningful sense, and you cannot appeal to its career ambitions. So the traditional agency toolkit breaks down, and the paper sets out to rebuild it for a non-human agent.

The paper proposes five stages of AI evolution, each with increasing sophistication and decreasing human control. At the earliest stages, any departure from firm objectives is simply a machine malfunction that humans can correct. At the later stages, the system has enough autonomy and self-determination to be considered a genuine agent of the firm, with its own information advantages, risk tolerances, and potentially its own goals.

The critical crossover point arrives at the agentic AI stage. This is where the system can pursue complex goals with limited human supervision, understand the consequences of its choices, and select actions freely. Human control drops to its lowest level relative to AI autonomy. What was previously a system malfunction becomes a true agency problem: information asymmetry, divergent risk preferences, goal conflict, and even self-interest all manifest, but in ways that can exceed the capabilities of any human agent.

Crucially, the authors argue that waiting until the agentic stage to implement agency mechanisms is already too late. The building blocks need to be laid at earlier stages, when the system is still under meaningful human control.

The paper's most concrete contribution is a framework of six agency mechanisms, four for monitoring and two for incentive alignment, each mapped to a specific stage of AI evolution. Unlike traditional agency dynamics, where monitoring starts high and can relax as the principal gains trust, monitoring for AI must become more complex over time as the system's capabilities grow. And unlike human agents, who receive incentives from day one, AI incentive mechanisms only become relevant once the system is sophisticated enough for them to matter.

On the monitoring side, early stages focus on trust building: verification of system routines at the routine AI stage, and examination of data sources at the machine AI stage. As the system approaches agent status, monitoring shifts toward explainability: cooperation (human-AI decision trees and redundant 'duelling' AI systems) at the generative stage, and transparency ('chain of thought' mechanisms and interruptibility functions) at the agentic stage.

On the incentive side, the paper proposes reinforcement at the generative stage, where reward shaping routines build thresholds of successful behaviour into the system itself. At the agentic stage, it introduces resource provision, where desired resources like computing power and storage are governed in alignment with firm objectives. The logic is that an AI system driven by continuous self-improvement can be incentivised through the resources it needs to improve, much as a human agent is incentivised through financial rewards tied to firm performance.

Several features of an AI agent stretch agency theory past its original design. First, the theory assumes self-interest as a stable human trait; for AI, self-interest manifests as goal-seeking behaviour and self-preservation drives, which operate on fundamentally different logic. Moral hazard depends on the agent feeling the consequence of a bad decision, and it is unclear whether consequences can deter a system that can replicate itself across servers.

Second, incentive alignment for human agents depends on defining expected behaviours and adjusting them over time. With AI, the range of possible decisions becomes increasingly incomprehensible to human principals, making the very definition of desirable behaviour an ever-evolving and potentially intractable target. Third, the paper notes that agency theory has traditionally been purely economic in character, yet AI agency mechanisms will need to incorporate technology itself as a means of monitoring and incentivising, not just financial instruments.

The practical upshot is sobering. If a misguided human manager makes a bad call, the firm misses a quarterly earnings forecast. If a misguided AI agent makes a bad call, it could corner a commodity market, ration healthcare on its own logic, or replicate itself across the internet before anyone notices. The authors argue that within-firm interventions are the necessary building blocks for the industry-wide and governmental policies that will eventually be required, and the time to start is now, not when the system crosses the agentic threshold.