5 Things I Mean When I Say AI Guardrails

Last week during a webinar with PropertyCasualty360 someone asked what guardrails we use when building at Notch. A live Q&A only gives you so much time to answer, so I wrote up the longer version I wish I'd had time to talk through.

I’m glad this was asked, as I don't think enough carriers ask it before they sign a vendor. For those not working day in and day out with AI tools, questions about guardrails are usually not asked after something goes wrong. But in the last two years since AI entered insurance, it’s now necessary to consider it at the beginning, as the definition of "guardrails" has changed.

As I mentioned in the webinar, roughly 10% to 15% of our costs with clients go toward guardrails. In this article, I explain why — and what I mean when I say AI guardrails are important. 

If you want the full technical breakdown, our CPO Elool wrote the deeper version of this in the article Guardrails and Escalations for AI Agents in Regulated Industries.

What AI Guardrails Used to Mean

When people in insurance first started implementing AI tools, guardrails were usually just a sub-agent that ran after the main AI produced a response to make sure it was safe to say or a safe action to do. We thought of it like a person standing at a gate reviewing what's about to go out the door. It was a reasonable starting point since at the time most AI in insurance was a single model answering a single kind of question.

As AI in insurance has evolved, this type of checks-and-balances is no longer enough. AI is no longer a single model for one question or action. 

What AI Guardrails Mean Today

AI is now a multi-layered system making decisions, touching data, and taking actions across all areas of the industry, from underwriting and customer service to claims and servicing. A “gatekeeper” checking outbound language doesn't do much if the problem is happening before the response is even given. There are new layers needed in an AI workflow so additional guardrails are needed to keep your business safe. 

1. Sub-agents tuned to context 

The original guardrail of a sub-agent is still part of the system, but it can no longer be one generic doorman.

Multiple sub-agents must be preconfigured per geography and per line of business. Being customizable is necessary, as an acceptable answer for a personal auto claim in one U.S. state isn't automatically acceptable in another.

It’s important that the AI knows the nuances so it can work effectively and relevantly across all business lines and stay compliant. 

2. Deterministic rules on what gets touched 

Before AI ever generates anything there need to be hard rules on what it's allowed to digest and act on in the first place. Otherwise, it could pull answers from anywhere, much of which could be outdated, irrelevant, or noncompliant. 

For example, the AI needs rules so it's clear if only certain document types, only requests from people with certain kinds of policies, or only certain workflows should be touched. Otherwise, you have AI accessing areas it shouldn't be allowed in, with no clear boundary stopping it.

3. Cost guardrails

Runaway cost is often the first obvious symptom of an AI system doing something it shouldn't be doing. It’s important that every use case gets a limit of how much happens per day and per use. This sounds like a detail only relevant for your finance teams, but it's actually a safety mechanism.

4. Deployment guardrails

As I explained in the webinar, at Notch we run three environments, the same discipline any serious software team applies to code that touches production data. Nothing gets tested in the same place it runs live.

5. Evals and regression testing 

Ongoing evaluations, not one-time, are needed to make sure agents stay on course. A model that passed every test at launch can drift and the only way to catch that before a customer or a regulator does is to keep testing against it.

The Rule That Ties it All Together

Here's how I'd summarize what to know about guardrails for any carrier team building their own checklist: whatever measures your business already runs to keep a process safe, your AI needs its own version of every one of them. 

• If your underwriting team has escalation rules for high-value accounts, your AI needs escalation rules. 
• If your claims team has spending authority limits, your AI needs spending limits. 
• If your compliance team has audit requirements, your AI needs an audit trail. 

Guardrails aren’t a separate category you bolt onto AI. They're a mirror of the controls you and your “human” team already use, translated into a system that never clocks out for the night.

My Prediction For the Future

Just like how AI in insurance has changed drastically over the last year, I expect even more changes to come in twelve months from now.

While new workflows, capabilities, and regulations may require new guardrails to be put into place, I still expect to give the same advice to organizations looking to implement AI.

That advice? Figure out what AI actually does well, put the guardrails in place before formally kicking off with a vendor, and start small enough to see what's working before you scale.

Figuring this out now, before you have deployed anything, is the secret to moving faster later. Waiting until something breaks to figure out retroactive guardrails will be a very different kind of blog post.