The QA Gap in Claims and Underwriting That Sampling Will Never Fix

How Most Carriers Handle QA Today

Most carriers I talk to handle QA in one of two ways. Either they pull a small sample of claims or underwriting files after decisions are made and review those, or they only do a deeper review when a file escalates to a formal compliance check. A manager, a QA team, or a compliance function then goes through the file against the carrier's guidelines, documentation requirements, and process rules.

This gets real work done. You catch errors. You confirm that the files you reviewed followed the right steps. When something goes through a formal compliance review, you get a thorough look at how that decision was made. Most carriers have built a genuine function around this, staffed by people who know the guidelines well and take the work seriously.

But this approach has a structural limit worth being honest about. It was designed for a world where reviewing every file wasn't practically possible. So it samples. And sampling, by design, means most files never get looked at.

What Sampling Misses

When you review a small percentage of files after the fact, you get a picture of a small percentage of your operation. You might catch an individual mistake. You might confirm that a specific adjuster or underwriter handled that file correctly. But you're not seeing what's happening across thousands of decisions at the same time.

That's where the real patterns live. The edge case that keeps creating the same problem because the guideline doesn't address it clearly enough. The documentation step that gets skipped consistently in a particular workflow, not because people don't know the rule, but because the process makes it easy to miss. The type of claim where inconsistencies between the file and the decision show up far more often than they should.

None of that becomes visible from a sample. It only becomes visible when you're looking across everything. And when carriers do eventually see these patterns, it's usually through a regulatory review, a complaint that escalated, or an audit that went deeper than expected. By that point, the pattern has often been running for months.

The result is that QA ends up functioning as a tool for finding individual mistakes after they happen. That's useful. But it's not the same as a system that continuously tells you where your operation deviates from how it's supposed to work, and why.

The Audit and Compliance Layer

There's a second dimension that makes this more pressing than it might seem at first.

Regulators and auditors aren't just asking whether individual decisions were correct. They're increasingly asking whether a carrier's QA process is systematic and defensible. Can you demonstrate, not just assert, that decisions followed the right guidelines and documentation standards? How do you know whether your adjusters are following the same guidelines across different regions? How do you show that documentation standards are being applied consistently, not just in the files you chose to review, but across the operation?

A QA approach built on sampling and escalation-triggered reviews is harder to defend in that context. You can show the files you reviewed. But you can't easily explain what you know about the files you didn't review, or why a pattern that showed up in a regulatory audit wasn't caught internally.

A continuous review process that generates an audit trail across all decisions is a fundamentally different answer to those questions. It doesn't just tell you what happened in the files you sampled. It tells you what happened across your operation, and it creates a record that's much easier to surface in a review. For carriers operating under close regulatory scrutiny, that's not just operationally useful. It's a risk management capability.

What Continuous QA Actually Looks Like

The shift I keep coming back to is moving from QA as a periodic compliance check to QA as a continuous feedback loop for the operation. That means reviewing work as it happens, not just sampling after decisions are done. It means covering a much larger proportion of files. And it means surfacing patterns across thousands of decisions, not just an isolated problem in a single file, but a recurring issue in a specific part of the process.

It also changes what the QA team itself spends time on. Instead of pulling files manually and reviewing them one at a time, the team's job shifts toward interpreting the patterns the agent surfaces, deciding which ones to act on, and making the process changes that prevent the issue from recurring. That's a better use of the expertise they have.

When QA works this way, it stops being a tool for finding problems after they've already happened. It becomes a way to improve how decisions get made before the next wave of files comes through.

What the Agent Checks, and When

The way Notch's QA agent works is that it reviews files continuously across a claims or underwriting workflow, rather than waiting for a periodic sample at the end.

In practice, it checks whether the right information was collected at each step, whether the decision followed the carrier's guidelines and procedures, whether required documentation is present, whether there are inconsistencies between the file and the outcome, and whether there are unusual patterns or exceptions that should be escalated to a person for review.

It runs at two different points in the process. During the workflow, it can flag a problem before a decision is finalized or before a claim moves to its next step, so the issue can be addressed before it creates a downstream problem. After the workflow, it reviews decisions at scale and surfaces recurring patterns across large volumes of files.

That second capability produces pattern-level findings that manual QA can't generate at scale. When a specific type of documentation gap shows up across a meaningful portion of claims in a category, concentrated in one workflow step, the agent surfaces that directly. That's the kind of information that changes how you train adjusters and underwriters, how you update your guidelines, and where you focus your QA team's time going forward.

Getting Live Without a Long Implementation

One of the questions I hear most often from carriers thinking about this is what the integration actually involves. They've seen how long it takes to replace a core system or redesign a major workflow, and they assume a new QA layer means the same kind of project.

It usually doesn't. The integration is lightweight by design. We start with one specific claims or underwriting workflow. We connect Notch to the systems and data the team already uses, including the policy or claims system, documents, emails, guidelines, and procedures, and configure the QA agent around the carrier's own rules and evaluation criteria. The carrier doesn't need to replace existing systems or redesign the process. The agent sits around the workflow, reading the same information the team is already working with, and evaluates files against the agreed QA criteria.

From there, we run the agent in a controlled environment first. We test it against historical or live cases and compare its findings with the carrier's compliance team. We tune the rules and evaluation criteria together before expanding. Because the platform infrastructure, integrations, governance, and audit trail are already built in, this typically takes weeks rather than months.

The goal is to get onto real files quickly, demonstrate that the agent's findings are accurate and useful, and then expand coverage from one workflow to the broader operation.

From One Workflow to an Operational System

Starting with one workflow doesn't mean staying there. The first deployment gives you proof that the agent works, that the carrier's team can trust what it surfaces, and that the integration holds up under real volume. From there, adding a second workflow or expanding to a different product line builds on the same foundation rather than starting over.

Over time, the picture you get across the operation becomes genuinely different from what periodic sampling could produce. You see where QA gaps are systemic rather than individual. You can tell whether a recurring problem is a training issue, a guideline issue, or something in the process design that needs to change. And you can catch those patterns early, before they show up in a regulatory review or a complaint that escalated further than it should have.

That shift from QA as a compliance check to QA as an operational feedback system is what I think the strongest carriers will start treating as a requirement. Not because running an AI agent is the goal, but because genuinely knowing how your operation performs across all your decisions, not just the ones you happened to sample, puts you in a fundamentally better position, operationally, commercially, and in front of a regulator.

The carriers who build that feedback loop now will have a real operational advantage. The ones who wait until an auditor asks them to explain what they know about their own process will have a harder conversation.

‍