When Should Humans Step In — and When Should AI Lead?

Connected Claims USA has a session on the agenda this week called “When Should Humans Step In – and When Should AI Lead?” It’s a roundtable with leaders from The Hartford Steam Boiler, Farm Family Insurance, Sedgwick, CSAA, and Crosstie, moderated by MAIPF. The description asks the panel to work through what works, what doesn’t, and how to balance AI with human judgment when it comes to claims.

My answer to this question is the question itself is the problem.

“Where’s the line between human and AI judgment” assumes the line is something you find in the moment, case by case, relying on an adjuster to catch it fresh every time. That’s a lot to ask of any one person, and in a regulated industry, it can’t be the whole system.

There is a technical distinction that often gets skipped over. An agent isn’t deterministic just because it has guardrails in place. What’s deterministic is the boundary around it, such as which categories of decisions require sign-off and which don’t, and what happens the moment a claim crosses that line. In practice, that boundary lives in infrastructure, not a setting buried in a prompt. The agent runs inside a sandboxed environment inside the carrier’s own cloud, with the reasoning layer and the deterministic guardrail layer built and versioned separately, so the boundary can be audited on its own. Build that boundary into the system, versioned like code, and you don’t need a human standing by hoping to catch unpredictable behavior after the fact. You need the boundary to have been right at the time the human is building it.

If that boundary lives in an adjuster’s head instead, and not in an AI solution you’re implementing, you inherit problems you can’t fix later. That expertise is real. But if it only lives in one person’s head, it isn’t auditable, you can’t defend it to a regulator, and you can’t hand it to the next hire.

We learned this the hard way on our own claims work. Fine-grained document labels looked fine until misclassification forced us to abandon them, and the fix wasn’t more human review. It was a proven testing framework before anything shipped, accuracy checked on a held-out set, with corrections fed back in before the agent ever touched a real claim.

What actually solves this isn’t better judgment in the moment. It’s three things:

• Which categories of decisions require sign-off, specified in advance
• A versioned record of exactly which path each claim took, and why
• The same log, whether a human made the call or an agent did

In practice, that specification looks like this:

A routine claim, low severity, no litigation indicators, no attorney involvement, clear coverage, doesn’t need a human in the loop before it moves. The agent settles it, and the record shows why. A claim with any litigation risk flag, a coverage question that touches an exclusion, or early signs of attorney involvement gets routed to a person before anything is decided, not after. Subrogation potential works differently. The agent can flag and prepare the file, but the decision to pursue it stays with a person. Below the line, the agent doesn’t just act and move on. It surfaces what it did, and why, in a form built for the question the adjuster or underwriter actually asked, not a dump of raw claim data they have to parse themselves. None of this gets discovered mid-claim. It’s decided before the first file ever crosses the line.

“Was this the AI or a person?” shouldn’t be a forensic question you reconstruct after a claim goes sideways. It should already be in the log.

This is the design question we start from at Notch, not one we retrofit later. Every claims agent we run operates inside an authority boundary set before it touches a single file, severity, litigation risk, and sign-off requirements decided at design time, not guessed at mid-claim. The line isn’t a judgment call we ask operators to make fresh every time. It’s specified once, logged every time, and identical for every claim that crosses it.

If you’re at Connected Claims USA, come find us.

‍