Photo by Frans van Heerden: https://www.pexels.com/photo/scenic-photo-of-water-dam-during-daytime-2699258/
There’s a question that has been turning in my head for weeks—the kind that arrives at eleven at night when you’re going back over everything that was produced during the day. The question is simple, almost banal in its phrasing: how do we govern what we can no longer read?
I’m not speaking abstractly. I’m speaking about what happens every day in organisations using AI agents. About pull requests containing changes generated automatically, about specification documents co-written with Claude, about configurations suggested by systems that understand the infrastructure better than the people approving them understand it. About that subtle, almost imperceptible feeling of approving things you only partly understand.
The fact that there isn’t a clean answer to this question is, by itself, significant. And the fact that the same question gets asked more and more often—in boardrooms, legal offices, compliance teams—suggests we’re crossing a point of no return. Not a gradual evolution. A point of no return.
The Assumption That Collapses
Traditional governance rests on an assumption so embedded it has become invisible: that there’s enough time to understand what was produced before it goes to production. Code review. Document audit. Contract approval. Analysis validation. Sign-off on deliverables. All of this worked for decades because the bottleneck was production. The time needed to create something was long enough to let the validation process keep up.
But when an AI system generates in minutes what used to take weeks, that assumption collapses. It doesn’t bend, doesn’t adapt: it collapses. And this isn’t only about code. It’s about everything.
Where It Collapses, in Practice
I think about contractual documents. An AI agent today can draft contracts, specific clauses, responses to disputes, preliminary opinions. The quality is often plausible enough to survive a quick read. But “plausible enough” in legal work can mean clauses that look protective and aren’t, obsolete legal references presented with confidence, omissions that surface only in litigation. The risk isn’t the obvious error. It’s the subtlety of the error in a context where subtlety is everything.
I think about financial analysis. Dashboards, projections, scenario analyses, business cases. A CFO who receives an analysis generated or co-generated by AI is approving something they understand? Or are they validating the surface plausibility of numbers that follow a logic no one has actually verified? The problem isn’t whether the numbers add up. The problem is whether the underlying assumptions are correct, whether the models applied are appropriate, whether the data sources are reliable. All things that require domain competence to evaluate, competence the review process presupposes but the production speed makes harder and harder to apply.
I think about external communications. Customer emails, tender responses, product documentation, user manuals, internal policies. Every organisation produces thousands of textual artefacts daily. When a significant share of them gets generated or assisted by AI, who guarantees consistency with the company’s positioning? Who verifies that a customer reply doesn’t create expectations the organisation can’t then meet?
And then there’s compliance. Here the paradox peaks. We’re in the middle of the European regulatory wave: AI Act, Cyber Resilience Act, Product Liability Directive, EAA, GDPR continuing to evolve. Organisations have to produce compliance documentation, risk assessments, registers, policies. The temptation to use AI to accelerate this production is enormous. But we’re talking about documents that attest conformity to rules that, in some cases, are exactly about the use of AI. There’s something circular and unsettling about using an AI system to declare that your AI systems comply with the rules on AI.
An Organisational Epistemology Problem
The problem isn’t only speed. It’s volume combined with opacity. A team using AI to generate code, configurations, documents, analyses, communications, architectural decisions, produces a quantity of artefacts no human review process can realistically cover at 100%. Maybe not even 50%. Maybe not even 20%, if we’re honest with ourselves.
But the worse problem is another: many of those artefacts are plausible enough to survive a superficial check. Which is almost worse than if they were obviously wrong. An obvious error gets caught. A plausible error gets approved, deployed, published, signed, and discovered only when it produces consequences.
I realise we’re facing what could be called an organisational epistemology problem: the organisation knows less than what it produces. And the gap widens every day.
Three Directions, None Decisive
I don’t have definitive solutions. No one does, and anyone who claims to is probably underestimating the problem. But there are a few directions that look sensible, even if none is decisive.
The first is what could be called the move from point-by-point control to structural control. If you can’t check every output, you have to check the system that produces the outputs. That means investing far more in defining upstream constraints, rigorous specifications, automated guardrails, policy-as-code, rather than in downstream review. It’s the point of spec-driven development: if the spec is solid, the range of error in the output narrows. It doesn’t disappear, but it contains.
For legal documents, that means verified templates with clear placeholders, not free-form generation. For financial analyses, validated models with controlled inputs, not blank slates. For configurations, certified baselines with explicit and justified deviations. The underlying principle is simple: shift the intelligence—and therefore the responsibility—into defining the problem, not into generating the solution.
The second direction concerns what we verify. You can’t read all the generated code. You can’t reread every document. You can’t recheck every analysis. But you can define invariants that must be respected and test them automatically. For code: automated tests, type checking, static analysis, security checks. For documents: automated checklists, structure validation, terminological consistency checks. For configurations: compliance policy as code, drift detection, pre-deploy validation.
It’s a paradigm shift: you go from “I read and approved this artefact” to “this artefact respects these verifiable properties”. Which is what we already do with automated tests in software. Only now it becomes the only realistic line of defence for everything.
The limit is obvious: verifiable properties are a subset of desirable properties. You can verify that a contract contains certain clauses, not that it’s strategically appropriate. You can verify that an analysis is internally consistent, not that the assumptions are correct. But it’s still much better than nothing.
The third direction is maybe the most uncomfortable to accept: AI checking AI. Automated reviewers. Specialised validators. Systems that verify the output of other systems. The recursive problem is obvious: who watches the watchers? If I don’t trust the output of the AI that produces, why should I trust the output of the AI that verifies?
But it’s probably inevitable. And the question becomes: how do we keep human intervention points at the moments that actually matter?
My hypothesis, still to be tested in the field, is that the human role will shift toward architectural decisions, ethical choices, exception handling, calibration of the control systems. It’s a different role. Less operational, more strategic. Less doing, more deciding the rules of doing.
The Organisational Risk
What worries me most in all this? Not so much the technical risk. Bugs get found. Errors in documents surface. Bad configurations show up. What worries me is the organisational risk: teams that get used to trusting outputs without understanding them, that gradually lose the ability to assess what they’re approving.
I see it already. Developers accepting generated code without reading it. Managers approving analyses without verifying assumptions. Compliance leads signing documents they only skimmed. It’s understandable—we’re all under pressure, all with more to do than we can manage. But it’s also dangerous.
The most important governance maybe isn’t on systems, but on the competence of the people who should be governing them. If we lose the ability to understand what we’re producing, we lose the ability to govern it. Regardless of how many layers of automated control we put in between.
Who Answers, and Navigating the Uncertainty
There’s also a question that in Italy, and in Europe with the AI Act, becomes particularly pressing: who is responsible? When a contract generated by AI contains an error that causes harm, who answers? The operator who used it? The provider of the AI system? The manager who approved? The organisation as a whole? When a configuration suggested by AI creates a security vulnerability, the chain of responsibility suddenly becomes very important.
The European regulatory framework is trying to give answers. The AI Act defines obligations for providers and deployers. The Product Liability Directive extends liability for defective products. The Cyber Resilience Act introduces security-by-design requirements. But the truth is that regulation runs behind technology, and meanwhile organisations have to make decisions. Decisions that will have legal consequences based on rules that aren’t fully clear yet.
What you can do is build traceability. Know which artefacts were generated or influenced by AI, with which prompts, under which conditions. It doesn’t solve the responsibility problem, but at least it lets you reconstruct what happened.
I write all of this and realise how uncertain the ground is that we’re all moving on. There are operating principles that look sensible: presumption of structural control, explicit invariants, mandatory traceability, strategic human intervention, competence as prerequisite, acceptance of uncertainty. But they’re principles under construction, imperfect, evolving, probably destined to be overtaken quickly by the speed of change.
And there’s something deeply frustrating about this situation. For years anyone working in this field had the job of finding solutions, giving answers, having a clear plan. Now we find ourselves navigating a territory where the questions are clearer than the answers, where every solution opens new problems, where the most honest thing one can say is “I don’t know, but I’m trying to understand”.
Maybe this is the new role of those who govern complex systems: not having all the answers, but asking the right questions and building structures that allow adaptation when the answers change. It’s less reassuring than a framework defined with five best practices and three strategic pillars. But it’s the only honest governance possible when the world produces faster than we can understand.
I often wonder where others in the middle of this transformation feel the tension most. On the speed of production, on the capacity to evaluate, or on something completely different I’m not yet seeing. Because maybe the thing that worries me most is exactly that: not knowing what I’m not seeing.
Key takeaways
Human review of everything is a tacit assumption that simply collapses with AI agents.
Three operational moves: statistically significant sample-based governance, policy-as-code for whole classes of errors, aggressive observability in production.
The most serious risk isn’t the obvious error—it’s the plausible error that survives a superficial review.
PLD and AI Act place responsibility on the producer and the deployer: without assigned roles, you aren’t governing, you’re hoping.
Questions & answers
How do you govern what you can no longer read?
It’s the central question of AI governance in 2026. Pull requests generated automatically, documents co-written with Claude, configurations suggested by systems that understand the infrastructure better than the people approving them. Traditional governance assumes time to understand what was produced before it goes to production—and that assumption has snapped. It doesn’t bend, it collapses.
Why doesn't governance just adapt to the AI cadence?
Because it rests on a human bottleneck: reading. Code review, document audit, contract approval—it all requires a brain reading line by line. When an AI agent generates in minutes what used to take weeks, review doesn’t just get “sped up”: it has to be rethought. Reading everything is no longer an option.
What's the practical answer for CTOs and legal leads?
Three moves: (1) sample-based governance with smart sampling—verify a statistically significant percentage instead of everything; (2) automatic upstream constraints (policy-as-code, type systems, property tests) that catch whole classes of errors without human reading; (3) aggressive observability in production, because what you didn’t verify before, you must be able to correct after. Total ex-ante inspection gives way to verification distributed over time.
Who is responsible if an AI agent gets it wrong?
The legal answer arriving (PLD, AI Act): the producer or the deployer. But operational accountability has to be designed, not discovered after the fact: who wrote the prompt, who approved the deploy, who monitors output in production, who can intervene if an error surfaces. An organisation that hasn’t explicitly assigned these roles isn’t governing—it’s hoping.
The author
