Why Enterprises Can’t Prove What Their Agents Actually Did

Every enterprise has an AI policy now. Almost none can prove what its AI actually did. As autonomous agents move from experimentation into production systems where they can access data, make decisions and initiate actions, that gap is becoming more than an IT governance problem.

It is emerging as a board-level risk because when something goes wrong, organizations need more than a policy showing what an AI system was supposed to do. They need evidence of what actually happened.

I’ve spent most of my career on the infrastructure side of this problem, first building large-scale object storage at Western Digital, then working across the Filecoin ecosystem at Protocol Labs on how decentralized networks prove what actually happened on them. The pattern I keep seeing repeated with AI agents is the same one: Systems get built for speed first, and the ability to prove what happened gets bolted on only after something goes wrong.

Enterprise adoption of AI agents has moved remarkably quickly. According to Writer’s 2026 enterprise AI adoption research, 97% of executives say their organizations deployed AI agents in the past year. Yet its broader AI adoption in the enterprise report found that only 23% of companies are seeing significant ROI from AI agents.

At the same time, organizations are confronting the consequences of AI adoption that has moved faster than governance. Writer reports that 67% of executives believe their company has already suffered a data breach because of unapproved AI tools. The security implications are reinforced by IBM’s 2025 Cost of a Data Breach report, which found that 63% of breached organizations either lacked an AI governance policy or were still developing one.

Together, these numbers expose an adoption-accountability gap. Enterprises have become very good at deploying AI, but far fewer have developed the infrastructure needed to understand and verify what those systems are doing once they are operating across real business data and workflows.

Part of the problem is that organizations have largely approached AI governance as a policy exercise. Policies, model cards, acceptable-use guidelines and risk frameworks can define intended behavior – which models are approved, what data employees may use, which actions an agent can perform and where human oversight is required.

Those controls matter, but they answer a different question from the one an auditor, regulator or security team may eventually ask. A policy tells you what should have happened. It does not necessarily prove which data an agent accessed, what information influenced a particular decision, whether the data changed during a workflow, what action the agent took or whether it remained within its authorized boundaries.

That distinction becomes critical when AI systems are acting autonomously. Policy compliance and provable policy compliance are not the same thing.

The rise of shadow AI illustrates the consequences. IBM found that one in five organizations reported a breach involving shadow AI while organizations with high levels of shadow AI experienced an average of $670,000 in additional breach costs compared with those with little or no shadow AI.

A policy without the ability to verify behavior ultimately still depends on trust. And trust alone is becoming an increasingly weak foundation for enterprise AI governance.

Evidence-based governance

The next stage of AI governance will require organizations to move from trust-based controls toward evidence-based governance. That means creating verifiable records of what AI systems and agents actually do: what data they access, where that data originated, how it moved or changed, what decisions were made and what actions followed. Rather than relying solely on intended behavior, enterprises need the ability to reconstruct actual behavior.

It’s the problem I spend most of my time on now at Akave, building the new infra plumbing to provide and protect data as we manage it today where it should.

This is particularly important because AI agents operate differently from traditional enterprise applications. A conventional application generally follows predefined workflows. An agent may reason across multiple steps, interact with several data sources, call external tools and APIs, generate new information and take actions based on the results.

Traditional system logs can record portions of that activity, but logging alone does not necessarily create a trustworthy record. An evidence-based approach requires decision-level records that capture significant actions and the context surrounding them.

It requires data provenance so an organization can establish where information came from and which data contributed to an output or decision. It also requires chain-of-custody information that can follow data as it moves between systems, applications, models and agents.

Rather than relying solely on intended behavior, enterprises need the ability to reconstruct actual behavior.

Just as important, those records must be resistant to alteration. If evidence can be modified after an event, its value during an audit, investigation or regulatory review is significantly reduced. Tamper-evident or immutable records can provide stronger assurance that the history being examined later reflects what actually occurred.

Finally, the record needs to be reconstructable. When an incident occurs six months after an AI interaction, the relevant question will not simply be, “Do we have logs?” It will be, “Can we reconstruct what happened?”

An enterprise should be able to determine which agent initiated an action, what permissions it had, which data it accessed, where that data originated, what outputs it generated and what subsequent systems or users received those outputs.

That is the difference between observability and accountability.

The need for this capability becomes most apparent after the fact. During normal operations, a dashboard showing that an agent is functioning may be sufficient. During a security incident, compliance audit or regulatory inquiry, it is not. The organization may need to demonstrate not simply that controls existed, but that they were followed in a particular transaction involving particular data. Evidence-based governance makes that possible.

Connecting governance and ROI

Accountability is often discussed primarily as a security and compliance requirement, but it is increasingly an economic issue as well. The gap between widespread deployment and limited ROI suggests putting more agents into production does not automatically translate into business value.

Enterprises need confidence that agents can safely interact with proprietary information and participate in meaningful business processes before they can entrust them with higher-value work. Without that confidence, organizations tend to constrain AI. Agents remain isolated in pilots, access to important datasets is restricted, human approval is inserted into workflows that could otherwise be automated, and security teams become reluctant to expand deployments.

Those restrictions may reduce risk, but they can also reduce the potential value of AI. This is why the ROI and governance problems are increasingly intertwined. Organizations cannot safely give AI systems greater autonomy unless they can establish accountability. And without greater access and autonomy, many AI initiatives may struggle to progress from individual productivity tools to systems capable of transforming business processes.

The security consequences reinforce the economic case. IBM’s research found that organizations with extensive use of AI and automation in their security operations reduced average breach costs by $1.9 million compared with organizations that did not use those capabilities extensively. The lesson is not that AI itself guarantees better security, but that visibility, automation and control can have measurable economic consequences.

Governance, therefore, should not simply be viewed as a brake applied to AI deployment. Done correctly, it can become part of the infrastructure that allows organizations to deploy AI more confidently and at greater scale.

What ‘AI-trusted’ looks like

As enterprises move agents deeper into production, the definition of a trusted AI environment is changing. An ‘AI-trusted’ enterprise is not one that assumes its models will always behave correctly. It is one designed around the possibility that they will not – and that can determine what happens when they don’t.

That requires provenance by default rather than provenance reconstructed manually after an incident. It means auditability is built into AI infrastructure rather than added after deployment. It means knowing not only which models and agents are authorized, but also which identities, permissions and datasets are involved in their operations.

This approach also recognizes that AI governance cannot stop at the model itself. An AI agent exists within a much larger chain that may include enterprise storage, databases, APIs, vector stores, SaaS applications, cloud infrastructure and other agents. Trust therefore has to extend across the data and infrastructure supporting the AI system.

This shift is already becoming more urgent as AI-related security incidents increase. IBM’s research found that more than 20% of organizations experienced a breach targeting AI models or applications. Among the most common causes were compromised APIs, applications or plug-ins and cloud misconfigurations affecting AI workloads.

As AI becomes more autonomous, governance will consequently have to become more granular. Enterprises will need to move beyond proving that they have an AI policy toward proving that individual agents, datasets and actions operated within that policy.

The AI-proof enterprise will not be the organization with the longest governance document or the most restrictive AI policy. It will be the organization that can produce credible evidence of what its AI systems did, which data they touched, why an action occurred and whether established controls were followed.

For boards and executive teams, that may become one of the most important distinctions in enterprise AI. I don’t think that stays hypothetical for long. The organizations that treat provable behavior as core infrastructure, not just a compliance checkbox, will be the ones comfortable giving agents more autonomy. Everyone else will keep the guardrails on, and keep watching faster-moving competitors pull ahead.