AI Governance Is Solving the Wrong Problem. We Need to Govern What AI Can Do, Not Just What It Says

There's a lot of discussion around AI governance at the moment. Most of it focuses on responsible AI principles, ethical guidelines, risk assessments and policies.

All important, of course.

But I think we're missing something fundamental.

As AI moves from answering questions to performing actions, the problem changes.

It's one thing for ChatGPT to provide an incorrect answer. It's quite another for an AI agent to process a refund, approve a supplier, change a customer record or initiate a payment based on incorrect reasoning.

Yet much of our thinking around AI governance still assumes we're dealing with systems that generate information rather than systems that execute decisions.

I think we need to rethink where governance actually sits.

The Problem With Traditional AI Governance

Most organisations approach AI governance through some combination of policies, committees, risk assessments and approval processes.

We define what AI should and shouldn't do. We establish guidelines for employees. We create acceptable-use policies and introduce additional checks before deploying systems.

There's nothing inherently wrong with this.

The problem is that these controls often exist outside the systems they're supposed to govern.

A policy might state that an AI agent cannot approve a transaction above a certain value without human authorisation.

But what actually prevents it from doing so?

Is that rule written into the application? Is it part of the prompt? Does the system rely on the model to interpret the policy correctly? Or is there an independent control that physically prevents the transaction from being executed?

These are very different approaches, with very different levels of assurance.

And as we give AI systems greater autonomy, I think those distinctions become increasingly important.

We Cannot Make Probabilistic Systems Deterministic

Large language models are probabilistic systems.

They interpret information, reason through problems and produce outputs based on learned patterns and their available context.

We can improve their reliability with better instructions, retrieval, structured outputs, validation and evaluation.

But we cannot guarantee that an LLM will always reason correctly.

Even if a model follows a policy correctly 99% of the time, that remaining 1% becomes significant when the system is executing thousands of actions.

The natural response is to improve the model, add another prompt or introduce another AI-based validation step.

But this creates an interesting problem.

We're effectively asking one probabilistic system to validate another probabilistic system.

That can be useful for identifying errors and interpreting ambiguous information. It isn't the same as enforcing a control.

This led me to a slightly different way of thinking about AI governance:

AI can propose. Evidence must prove. Policy must authorise. Software executes.

The distinction is between allowing AI to make decisions and giving it the authority to execute them.

Introducing Evidence-Bound Action Governance

I've been exploring a framework I call Evidence-Bound Action Governance (EBAG).

The underlying idea is relatively straightforward.

Rather than relying on an AI agent to govern its own behaviour, we introduce an independent governance layer between the agent's proposed action and the system that executes it.

The AI can interpret a request, investigate the context, recommend an action and prepare the necessary parameters.

But before that action is executed, the governance layer independently checks four conditions:

1. Evidence: Is the information supporting the action verified, current and relevant?
2. Invariants: Does the action satisfy the hard rules that must never be violated?
3. Policy: Is the action permitted under the applicable business and regulatory policies?
4. Authority: Is the requesting user or system authorised to perform the action?

For automatic execution, all four conditions must be satisfied.

Conceptually:

Execute(action) = Evidence ∧ Invariants ∧ Policy ∧ Authority

This is a simplified logical model, not a claim that every governance decision can be reduced to four booleans.

Some decisions will require additional assessment, particularly where the evidence is incomplete, the policy is ambiguous or the consequences of an incorrect action are significant.

The important principle is that the model does not get to authorise its own actions.

A Practical Example: An AI Customer Service Agent

Imagine a customer service agent with access to a company's order management and payments systems.

A customer contacts support and requests a refund.

The agent reviews the conversation, retrieves the order details and determines that the customer should receive a refund.

In a typical agent architecture, it might then call a refund API.

We could add instructions telling the agent to check the refund policy. We could also include a separate evaluation step to determine whether the refund is appropriate.

But I would rather have the system independently verify the action before any money moves.

Under the proposed framework, the process would look something like this.

Step 1: The agent proposes an action

Refund AED 450 against order #12345.

Step 2: The governance layer verifies the evidence

Does the order exist? Has it been paid? Is the refund amount valid? Has a refund already been issued?

Step 3: It checks the invariants

The refund cannot exceed the eligible paid amount. The same payment cannot be refunded twice. The refund must be linked to an eligible transaction.

Step 4: It evaluates policy

Does the refund fall within the company's refund policy? Is the return window still valid? Is an exception required?

Step 5: It verifies authority

Can this agent initiate a refund of this value? Does the action require a manager's approval?

Only after these checks does the execution service receive permission to process the refund.

If a condition fails, the system denies the action or escalates it for review.

The AI can still be useful. It can explain the policy to the customer, collect missing information and recommend an exception.

But it cannot bypass the controls simply because its reasoning sounds convincing.

Governance Should Adapt to the Level of Risk

Not every AI action requires the same degree of control.

An agent summarising an internal meeting doesn't need the same governance as one changing employee salary information.

This is where I think a risk-adaptive approach becomes useful.

For example:

Risk level	Example	Governance approach
Low	Summarising an internal document	Basic access controls and suitable data protections
Moderate	Updating a customer support ticket	Identity, data validation and policy checks
High	Issuing a financial refund	Verified evidence, authority limits and execution controls
Critical	Changing payroll or releasing significant payments	Independent verification and authorised human approval

The objective shouldn't be to introduce bureaucracy into every interaction.

It should be to apply controls in proportion to the consequences of the action.

A sensible governance system should make low-risk activities easier while reserving stricter controls for actions with greater potential impact.

From Governance Documents to Governance as a Service

This is where the idea starts to become more interesting.

What if governance wasn't something every AI team had to implement independently?

What if it became a shared service?

A Governance-as-a-Service model could provide a common enforcement layer across multiple AI applications and agents.

Instead of embedding business rules into individual prompts, organisations could manage versioned policies through a central service.

An agent submits a proposed action with its identity, context and supporting evidence. The governance service evaluates the request and returns a decision:

ALLOW, DENY or ESCALATE.

For authorised actions, the system could issue a short-lived, action-specific authorisation that the execution service must validate.

That last part matters.

If an agent can bypass the governance service and call the underlying API directly, the control is largely cosmetic.

The protected system must enforce the authorisation, with no alternative route around it.

There's also an opportunity to use AI to help translate human-readable policies into structured rules.

But I wouldn't allow those generated rules to become automatically enforceable without testing and approval.

Otherwise, we've simply moved the same trust problem from the agent to the policy compiler.

Governance as Part of the Adaptive Enterprise Loop

This also connects to my broader thinking around the Adaptive Enterprise Loop.

The framework describes how organisations move from strategic intent through coordination and delivery into learning and adaptation.

AI can play a role throughout that process.

It can identify patterns, interpret operational information, recommend interventions and help teams respond more quickly.

But there needs to be a distinction between understanding what should happen and deciding what is permitted to happen.

That distinction becomes particularly important at the point where analysis turns into action.

An AI system might identify an operational bottleneck and recommend reallocating resources.

That doesn't necessarily mean it should have the authority to make the change.

A governance layer allows an organisation to benefit from probabilistic intelligence without surrendering control over execution.

It also creates a useful feedback mechanism.

We can measure which actions are approved, which are rejected, where exceptions occur and which policies create unnecessary delays.

Governance then becomes more than a compliance function.

It becomes part of how the organisation learns and improves.

Is This Fundamentally New?

Not entirely, and I think that's important to acknowledge.

We've been using access controls, policy engines, transaction validation, segregation of duties and approval workflows in traditional software for years.

There are also established AI governance frameworks, including the NIST AI Risk Management Framework, that address the broader management of AI-related risks.

The concept of policy-as-code is well established too.

What I'm proposing isn't a replacement for any of those practices.

It's a specific way of bringing them together around the execution boundary of an AI-driven system.

The architectural distinction matters because AI introduces a new source of proposed actions that may be persuasive, contextually convincing and still wrong.

In traditional software, developers generally define the conditions under which an action is triggered.

With AI agents, the model may dynamically determine which action to propose and when.

That creates a different relationship between reasoning, authority and execution.

I think our governance architecture needs to reflect that.

What Organisations Should Do Differently

If I were introducing AI agents into an organisation today, I would start by asking a few practical questions.

What actions can each agent perform? Which systems can it access? What evidence does it use to make decisions? Which actions are reversible? And what actually stops the agent from executing something it shouldn't?

If the answer to that final question is simply that the agent has been instructed not to, I don't think that's sufficient for high-impact actions.

I'd also avoid building a massive governance function before the first agent goes live.

Start with the capabilities that matter. Establish clear boundaries. Make the controls enforceable. Test failure scenarios. Capture evidence of every consequential decision.

Then improve the system based on what happens in production.

This is closer to good software engineering than traditional governance theatre. I've written more about that side of it in Building an AI Agent Is Surprisingly Simple. Getting It Into Production Is Where Things Get Interesting.

If you're working out how to put this into practice across an organisation, we've published a practical guide at ROQ CX on implementing AI governance. It covers risk-based policies, accountability and moving from written policy to technical enforcement.

The Bigger Opportunity

The conversation around AI governance often focuses on how we restrict AI.

I think there's a more useful question.

How do we create systems that allow AI to become more capable without increasing our dependence on it always being correct?

We shouldn't need to trust an AI agent's reasoning completely before allowing it to deliver value.

We should be able to verify the evidence, enforce the boundaries and control the execution independently.

That's the thinking behind Evidence-Bound Action Governance.

Not making AI deterministic.

Making the actions it takes governable.

And I suspect that's going to become increasingly important as AI agents move out of demonstrations and into the systems that actually run our businesses.