Hope and Agency in the Age of AI

A systems view: three levels of the AI–agency problem

Macro — governance and policy

The clearest case is the Dutch childcare-benefits scandal (toeslagenaffaire). From 2013, the Dutch tax authority used a self-learning risk-scoring algorithm that treated non-Dutch nationality and foreign-sounding names as fraud indicators, with no meaningful human oversight. Tens of thousands of families were wrongly accused; Amnesty International described a black-box system producing a "black hole of accountability," and the scandal brought down the Dutch government in January 2021. In 2024 the Netherlands Institute for Human Rights ruled the practice unlawful discrimination.

The second macro dimension is the tools that decision-makers themselves increasingly lean on, beyond the systems that act on citizens. As officials, regulators and executives use AI to draft, analyze and shape high-stakes choices, the models' embedded assumptions and blind spots enter the exercise of power — quietly, and without the accountability that attaches to a named human judgment. What has shifted is where consequential reasoning happens: increasingly it runs through a handful of opaque systems whose biases are inherited rather than chosen. This dynamic is harder to document than a single algorithm's failure, and more pervasive.

The real threat, then, is opacity and unaccountability — in the systems that judge citizens, and in the tools that now advise those in power. This is the backdrop the analysis responds to; the leverage lies elsewhere. But there are pathways into these larger systems: the micro and meso work — more capable, more agentic people and better-functioning organizations — is what produces the citizens, civil-society actors and institutional practices that demand accountable systems. Agency at the base is a precondition for accountable governance above.

Meso — organizations

AI is improving individual speed without reliably improving organizational outcomes — and that gap, not the technology itself, is the problem. MIT's 2025 GenAI Divide report found that about 95% of enterprise generative-AI pilots produced no measurable profit-and-loss return, despite an estimated $30–40 billion invested; only around 5% captured significant value. Tools like ChatGPT and Copilot lifted individual productivity but did not convert into institutional gains.

The report attributes this to a "learning gap": tools that do not adapt, retain feedback, or fit real workflows — an integration and design failure in how the tools are deployed. This reframes "AI deteriorates human capital" from an indictment of the technology into a solvable design problem.

The deeper distinction is augmentation versus substitution. Brynjolfsson's Turing Trap argues that the field over-invests in AI that imitates and replaces humans — concentrating wealth and power — rather than AI that augments them and distributes new capability. The evidence favors augmentation when it is well-scoped: in a study of 5,179 customer-support agents, AI raised productivity 14% on average and 34% for novices, largely by spreading expert practice to less-experienced workers. But the gains are jagged: in a trial with 758 BCG consultants, AI lifted quality roughly 40% on tasks inside its frontier, yet made consultants markedly more likely to be wrong on tasks outside it, and homogenized their outputs.

The meso lesson: AI builds organizational capability only when designed to augment judgment and spread skill — and erodes it when designed to substitute for both.

Micro — people

As AI becomes a daily "second brain," the central question is whether it builds a person's capacity or quietly replaces it. The honest answer is that it depends on how it is used, and the risk is real but overstated in popular accounts.

The cautionary evidence: Gerlich's 2025 study of 666 adults found a negative correlation between frequent AI use and critical-thinking scores, mediated by cognitive offloading and strongest among younger users. A Microsoft–Carnegie Mellon study presented at CHI 2025 found that the more people trusted an AI's output, the less critical thinking they reported applying.

The restraint the evidence demands: these are correlations rather than proof of causation; they rest partly on self-report; and higher education appears to buffer the effect — engagement is protective. Some researchers reject a blanket "cognitive atrophy" narrative, noting that offloading hampers recall of specific facts rather than thinking in general. Severe-harm cases exist but are rare and anecdotal, and belong in the argument as cautions rather than trends.

The useful frame is fifty years old. Ivan Illich distinguished convivial tools, which extend a person's autonomy and competence, from manipulative tools, which create dependency and degrade the capacity they claim to serve. The design question at this level is exactly this: does a given tool leave the person more able, or more dependent?

Generative tensions this reality brings

1. Speed vs. capability. The same tool can make a person faster today and less able tomorrow. Capability is the thing worth protecting; speed is acceptable only when it does not mortgage it.

2. Augmentation vs. substitution. Designs that extend human judgment distribute capability; designs that replace it concentrate power (the Turing Trap).

3. Individual gain vs. collective value. Faster individuals do not add up to better organizations without deliberate integration (the meso paradox). The relevant unit is often the team or institution's aggregate human capital, not the lone user.

4. Measuring what matters vs. what is easy. The default metric for productivity — output or income per hour — captures speed and volume but is blind to whether judgment, skill or agency actually grew. Optimizing it therefore rewards precisely the speed-over-capability trade this analysis warns against. Measuring agency and hope directly is harder, but it is the only way to know whether a tool builds people or merely rushes them.

5. Autonomy vs. dependency. Every helpful tool risks becoming a crutch. The test is whether use builds the capacity to eventually need the tool less over time, and if it needs it more, whether that more intensive use is translating into value add or decrease.

Implications

The analysis points to a single conclusion: the measure of any AI tool is whether it leaves us more able. Speed as an indicator is a piece, but not the whole puzzle.

That yields a criterion for telling helpful uses from harmful ones:

Does this leave the person or organization with more agency and hope after using it than before — more able to set their own goals and see their own pathways — not merely more efficient?

Three sub-questions make the criterion usable, each drawn from the concepts above:

1. Autonomy and competence (Self-Determination Theory): does use build the person's own capability, or substitute for it?

2. Pathways and agency (Snyder): does the experience leave people seeing more routes to their goals, and more will to pursue them?

3. Convivial, not manipulative (Illich): does the design reduce dependency over time, or deepen it?

By this measure, a tool that improves speed while eroding capability is a net loss. The problem is that trade itself — speed purchased with capability. What to build in response is set out in the theory of change below.

Where this sits among AI risks

Effective altruism's impact lens asks which problems are large in scale, relatively neglected, and tractable. Most AI-risk attention and funding has gone to one failure mode: a powerful, misaligned system that seeks power or deceives its operators. The risk this document describes is quieter and already under way.

It maps closely onto what Jan Kulveit and colleagues call gradual disempowerment: as AI comes to outcompete people in economic, cultural and governmental roles, societal systems depend less on human input and can drift from human interests — with no single catastrophic event, and even while the AI does exactly what it is told. Their three arenas — economy, government, culture — line up closely with this paper's meso, macro and micro levels.

This framing adds the level the systemic account tends to skip: the individual. Gradual disempowerment is usually told at the scale of institutions; agency and hope are where it is first felt, and where it might first be resisted, one person's preserved capacity at a time. On the impact lens, that makes the micro and meso levels attractive — large in scale, since they touch everyone who uses these tools; comparatively neglected, since most effort targets catastrophic alignment; and tractable through design and measurement, not only through governance.

The macro level of this working document is, in the vocabulary of the field, an AI-governance problem — oversight, accountability, and meaningful human control over consequential systems. Agency and hope give governance something it often lacks: a human objective to aim at, measured as whether people can still influence the decisions that affect them, rather than compliance alone. Read that way, the framing connects to several strands of AI risk at once.

• Gradual disempowerment: The systemic form of this thesis; its three arenas — economy, government, culture — map to the meso, macro and micro levels.

• AI governance and accountability: The macro level is a governance problem: oversight, accountability and meaningful human control. Agency supplies a human objective — whether people can influence decisions that affect them (SDG 16.7.2) — and a purpose for oversight beyond compliance.

• Concentration of power: The Turing Trap's warning: automation that substitutes for people concentrates economic and political power and strips their leverage. Preserved agency is the counterweight.

• Epistemic autonomy and information integrity: The culture arena: a second brain that shapes beliefs and choices is a pathway tool that can build or erode independent judgment.

• Catastrophic misalignment: The dominant concern, and a distinct one. This framing addresses harm that arrives with no single failure and even when AI does as it is told, so it complements alignment work rather than competing with it.

• Long-term outcomes: Sustained agency and hope persist even without the tool, and goal attainment increases. Learned helplessness decreases, which complements SDG 4.4 (skills) and SDG 3.4 (well-being)