What AI Governance Really Means for the People Affected

Governance is about the person on the other end

When people talk about AI governance, they usually talk about the system. What the model can access, what it can do, what rules sit around it. That part matters. But it is the smaller part. The larger part is knowing what a decision meant for the person it affected.

An AI decision in insurance is never just a technical event. Someone made a claim. Someone is waiting to hear back. Someone will live with the answer. Governance, done well, is about that person as much as it is about the model. It is the difference between a system that runs and a system people can actually trust to run their lives through.

I have come to think about governance less as a set of controls on the technology, and more as a way of defining the relationship between the AI and the people around it. Three people, specifically. The policyholder. The claims professional. The insurer. Each of them lives with an AI decision in a different way, and governance has to be clear about all three.

‍

The policyholder needs to know what happened to them

Start with the person the decision is actually about. For a policyholder, governance means understanding how AI shapes the service they receive and the decisions made about them. It also means something more basic. They need to be able to question an outcome, and to escalate it when they do not accept it.

That sounds obvious. In practice it is where a lot of AI deployments fall short. An answer comes back, and the person on the other end has no idea whether a human ever looked at it, no clear way to push back, and no one obvious to push back to. Good governance closes that gap. It makes sure the policyholder can see how a decision was reached at the level they need, and can always reach a person who has the full context of what the AI did and why.

If a policyholder cannot challenge a decision that affects them, it does not matter how accurate the model is. The relationship is broken before it starts.

‍

The claims professional needs to know where they stand

The next person is the one working alongside the AI every day. For a claims professional or an agent, governance means knowing when the AI is there to support their judgment, when they are expected to step in, and what responsibility still sits with them.

This line is easy to leave fuzzy, and fuzzy is where the trouble is. If a claims professional is not sure whether the agent is handling a case or just preparing it for them, work slips through the space in between. People assume the other side has it. That is how things get missed.

AI here is support for the professional's judgment, and support that has to know its own limits. Governance is what draws the limit clearly. It says, in this situation the AI can act on its own, in this one it prepares the work and a person decides, and in this one it stops and asks. That is the human-in-the-loop line in a regulated workflow, and when it is explicit, the professional knows exactly where they stand and can trust the AI instead of second-guessing it.

‍

The insurer needs to know who owns the outcome

The third person is the insurer itself. When AI is involved in a decision that affects a customer, someone inside the company still owns that outcome. Governance means being clear about who.

This is the question that gets skipped in the rush to get something live, and it is the one that hurts most when it is skipped. A decision gets questioned. Something goes wrong, or simply gets challenged. And internally, it is not clear who is accountable. The operator who was in the loop. The team lead. The system. If that has not been settled in advance, it turns into a scramble at the worst possible moment.

Settling ownership before it becomes a problem is not a legal formality. It is what lets the insurer stand behind its own decisions with confidence, and answer for them plainly when asked.

‍

Governance defines a relationship, not a control panel

Put those three people together and you can see why governance cannot only be about controlling the technology. It has to define the relationship between the AI and the people it touches.

That relationship has a few clear parts. Where the AI can act on its own. Where human judgment matters and a person has to decide. How someone can challenge or escalate a decision they do not accept. And who ultimately owns the outcome. Get those four things clear and you have real governance. Miss them and you have a panel that describes the system but says nothing about the people living with what it does.

This is the shift I keep coming back to. The hard questions in AI governance are not really about the model. They are about the people on every side of its decisions.

‍

Why “was this a human or the AI” is becoming the core question

There is one question that pulls all of this together, and it is a simple one. Was this decision made by a human or by the AI. A policyholder has a right to know whether a human reviewed their claim or the AI adjusted it. That single piece of clarity changes how much the person can trust the process, and how well they can act on it if they disagree.

The EU AI Act is making this conversation more urgent. Rules are starting to require exactly this kind of clarity about where AI sits in a decision that affects a person, which is part of aligning AI governance with the rules of a specific industry. But I would not treat it as only a compliance exercise. The reason to make the human or AI line visible is the same reason it will be regulated. It is what the person on the other end deserves in order to understand and question what happened to them.

‍

The real shift is from the system to the people around it

For a long time AI governance was framed as a technical problem. Lock down what the system can do, log what it did, prove it stayed inside the rules. All of that still matters. But the center of gravity is moving. Governance is becoming less about the system itself and more about the people living with its decisions.

That is a healthier place for it to sit. It puts the policyholder's ability to question an outcome, the professional's clarity about their own role, and the insurer's ownership of the result where they belong. The technology serves that relationship. It does not define it.

‍

What this means when you choose who to build with

So what should an insurer actually do differently. The practical change is in who you decide to build with. This is not a feature you buy once and check off. The way regulators handle AI is going to keep changing, and the expectations around how people are treated by these systems will change with it.

That means choosing a partner who will run with you for years, not selling you a capability and moving on. You need something built to hold the regulation that exists now and the regulation that is coming, and to keep the relationship between AI and people clear as both evolve. A single feature cannot do that. A partner who treats governance as the core of the product, and who expects the rules to move, can.

This is where we have put our attention at Notch. We build AI agents for regulated work, and we treat governance as the relationship between the AI and everyone its decisions touch, not as a layer added at the end. In the end, the system is the easy part. The people living with what it decides are the whole point.