By Kay Firth-Butterfield, CEO, Good Tech Advisory
Published in the Key Factor Digital Trust Digest: Download here
As AI develops beyond large language models into generative systems and autonomous agents, we are seeing both extraordinary benefits — and growing risks.
Among the most visible problems are hallucinations, where AI systems produce incorrect or misleading information. Some studies suggest hallucination rates range from 50-80% in certain models, depending on use case and context.
These failures create real-world consequences:
- Employees may rely on incorrect outputs without verification.
- Poor AI-generated work is passed along, requiring hours of correction by others — what many now call “workslop.”
- Worse still, hallucinated outputs can be fed back into corporate systems, contaminating internal data and compounding errors over time.
Imagine a bank relying on AI-generated information that is wrong — stored, reused, and trusted — without employees realizing it.
The risks are obvious, and, in the absence of regulation on developers of Foundational AI Models or other builders, the responsibility lies squarely with the organization deploying the system.
Guardrails Are Not Optional
Public skepticism around AI is growing, and organizations should not dismiss it. These concerns come from the very people companies serve and employ. Responsible AI requires education, transparency, and clear rules about how AI may — and may not — be used.
AI agents are making decisions, initiating transactions, interacting with systems, and sometimes with customers directly.
Trusting such systems without robust constraints invites failure.
For years, we have understood that the safe and effective use of AI requires guardrails, whether imposed by regulation or, more often, by organizations that understand the need to protect their customers, employees, and themselves. Yet as Shadow AI proliferates and intellectual property is exposed through ungoverned use of external tools, organizational liability continues to grow, and courts are increasingly unsympathetic.
With the rise of generative AI and agentic AI, those guardrails are no longer optional.
Responsibility Requires Knowledge And Action
There is broad global agreement on the principles of responsible AI. Yet regulation remains uneven or nascent in many jurisdictions, leaving organizations to define their own safeguards.
This does not grant freedom to act without restraint; it increases responsibility.
Organizations deploying AI should be able to answer fundamental questions[1] :
- How does this system work, and how would we explain its use in court?
- What data protections are in place?
- How do we know which AI agent is interacting with which system — or with another agent?
- What accountability mechanisms exist if something goes wrong?
Responsible AI users cannot operate in ignorance. Efficient deployment requires deliberate choices, documented controls, and ongoing oversight. Doing so should keep your organisation safe.
Verifiable Identity Is Foundational
This brings us to one of the most overlooked — yet critical — guardrails: verifiable digital identity for AI agents.
Today, companies are deploying AI agents internally, in customer-facing roles, and increasingly in interactions with agents from other organizations. Yet many of these agents operate without strong, verifiable identity. When identity is weak or implicit, accountability collapses.
Inside organizations, humans are often kept “in the loop” to review AI actions. While this can reduce risk, it is expensive, slow, and ultimately undermines the efficiency gains AI promises. But removing humans without replacing them with enforceable controls is far worse.
Without verifiable identity:
- It becomes unclear which agent took an action
- Authority and permissions cannot be reliably enforced
- Misbehaving or compromised agents are difficult to stop
In high-stakes environments, these failures can be catastrophic. In military or defense contexts, for example, one agent issuing incorrect instructions to another could lead to serious real-world consequences.
We have already seen organizations held accountable for AI-driven interactions with customers. These incidents underscore a critical point: organizations remain responsible for the actions of the AI systems they deploy. Autonomy does not absolve accountability.
As AI agents increasingly conduct business with one another — booking travel, negotiating services, executing transactions — the need for verification becomes even more urgent.
An organization must be confident that its AI agent is interacting with the right counterpart, under the right constraints.
The Next Frontier of Responsible AI
Verification of AI agents is the next frontier of responsible AI. The risks posed by hallucinations and model error are multiplied when AI agents act autonomously and interact at scale, whether internally, with customers, or across organizational boundaries.
Wise use of AI requires that systems can be trusted, constrained, audited, and stopped when necessary. Verifiable digital identity makes that possible. If safe use of AI is to move beyond aspiration and into practice, organizations must treat identity as a foundational requirement.
Without it, trust erodes. With it, autonomy can be governed — and responsibly scaled.
About the Author
Kay Firth-Butterfield is one of the world’s foremost voices in AI governance and responsible technology. As a TIME100 Impact Award honoree, she has spent over a decade influencing global AI policy, leading initiatives at the World Economic Forum, and advising governments and organizations on wise AI deployment. As the CEO of Good Tech Advisory, she continues to push for AI that is secure, fair, and beneficial to all. Read her latest book Coexisting with AI: Work, Love and Play in a Changing World. Order a copy here: https://coexisting-with-ai.com
