Start With Audit: The Shallow End of the Pool for AI in Claims

In claims audit, the human decides what the AI reviews and what it’s compared against, so the AI never has control. That’s why it’s the safest place to dip your toes in.

The shallow end of a pool with a 3 FT depth marker and faint digital network lines across the water.

I keep having the same conversation with claims leaders over the last year. They want to adopt AI and they’ve watched the demos but they just don’t know where to start. Some of these CCO’s even have mandates to “do something with AI” but, they stall, because they aren’t sure where to start without creating a problem they’ll have to explain to a regulator, a client or a jury.

This is nothing new and 56% of insurance leaders named regulatory or compliance uncertainty as a top barrier to scaling AI (Grant Thornton’s 2026 AI impact survey). In fact this really isn’t anything new in claims when it comes to adopting any new technology or process.

Here’s the thing, audit is the perfect place to start. Think of it as the shallow end of the pool. You’re in the water and learning how AI behaves on your own files, but you can stand up if something goes wrong. Audit is the one place I’ve found where AI does real work and a person still makes the call.

In my last post, I described a manual audit we finished for a client: 300 claim files, five reviewers reading 7 to 10 files a day for a full week, then more QC passes to keep the scoring consistent. It took about two weeks to get the client their results and that was only because that was their sole responsibility over those 2 weeks. What that review team did is the same thing we have all done when reviewing claim files. We take our experience and knowledge and compare it to what we read in the claim notes. Did the handler do what should have been done, when it should have been done and could they have done it better.

This is exactly what we can have AI do, the comparison of the standard to what was in the actual file. The auditor hands it the standard and what experience has taught us, so it isn’t guessing.
And guess what, this is exactly the best use case for AI in claims.

The caution is justified

If you’re nervous about AI in claims, it’s completely understandable. These tools can and do make mistakes. Even when a model is handed the source material, it can drift. One 2024 study found that about a quarter of the correct answers from the largest model it tested still weren’t fully supported by the sources.

Regulators are paying attention, too. More than half the states have now adopted the NAIC’s model bulletin on insurers’ use of AI. It covers testing and vendor oversight, and it expects insurers to document AI-enabled decisions. The NAIC has also been building a tool examiners can use when they ask carriers about their AI. Twelve states piloted it this year, and a working group took it to the NAIC’s Fall National Meeting in Dallas in November to consider adopting it.

However, those nerves can be significantly reduced in the claims audit setting with the correct approach and proper use of the tools available.

What makes audit different

When AI gets it wrong in claims handling, the risk to the claims department can be extremely problematic to say the least. If a claim gets denied that shouldn’t have been, or a claimant gets bad information then the exposure to the company is real from a financial, regulatory and reputational perspective.

A mistake in an audit looks different. A file gets flagged that shouldn’t have been, and the reviewer catches it. Or a finding gets missed, this is the actual reality of the manual audit which carries the same risk we’ve always lived with in sampling, where most files are never read at all.

The AI is not there in the audit setting to adjudicate the claim, but rather to surface issues or patterns that need to be addressed. If it’s wrong, you lose some of a reviewer’s time. Nobody’s claim or payment changes. This is not to say we are ok with an audit mistake, what we are saying is the impact of that mistake is not financial, regulatory or reputational.

AI in claims audit lets you in the shallow water and learn how AI behaves on your own files, but does not drown you because you can still stand in the shallow end.

The human is already in the loop

AI Audit already has the human in the loop, it’s inherent and built in to setting up an AI audit. There has always been an auditor whose job is to review the work and stand behind the findings. With AI, that person is still there, but they just spend their time on different files. On a review like those 300 files, one auditor can start with the files the AI flagged, check the evidence behind each finding and keep focused on real issues. This can be done in a fraction of the time.

The rest of the industry is landing in the same place. In the Sapiens Insurance AI Maturity Index, released in October, insurers said a human still checks 53% of agentic AI work on average, and people override about 15% of the decisions AI makes. Audit fits that model as it stands. Every finding should point back to the note or document it came from, so the reviewer can confirm it in seconds.

That reference to the source of the finding is exactly how we built The Audit Portal. The AI doesn’t get free rein but rather runs on the client’s own standard, and every check is tied to what a human reviewer would look at in the file. Each finding links straight back to the source note it came from, so the reviewer compares the AI’s read with the file and confirms it before it goes in the report.

A claims auditor checks AI-flagged claim files on a monitor and ticks one off with a pen.

It builds the governance you’ll need anyway

Grant Thornton found that only 24% of insurance leaders were very confident they could pass an independent review of their AI governance in the next 90 days.

Unlike having AI make claims decisions, the AI Audit work is documented by design. That’s why it’s a good place to start building it. There’s a written standard, a set list of files, a finding for each question and a reviewer’s sign-off. Regulators want to see that kind of record. Chris Raimondo, who leads EY’s insurance consulting practice, made the point recently that once a tool goes into daily use, an insurer has to work out how to reconstruct what happened if someone questions the output. In audit, that trail exists before you ever turn the AI on.

The process is governed in the design, where you build a template that works out the policies and what a human would find compliant before the AI even gets a look.

How I’d start

  1. Pick one line and one standard. GL, auto or workers’ comp, with the best-practice standard you already audit against. If the standard is vague, fix that first. The AI will only be as good as the standard you give it. Ensure every question you would expect your auditor to answer is defined and what the expectation would be is settled. This is actually something that manual reviewers don’t always do and that results in scores that differ from reviewer to reviewer.
  2. Run it next to a manual audit. Take files you’ve already reviewed by hand and run them through the AI. Compare the findings. Where it disagrees with your reviewers, find out who was right. Sometimes it’ll be the AI and when that happens you tweak the template, or the context given to AI and run it again. After a few tests, the AI will prove its value.
  3. Require evidence for every finding. No citation back to the note or document means it doesn’t go in the report. Everything needs to point to the notes, the guidance or best practices to justify the basis for the finding.
  4. Keep a person on the sign-off. The auditor owns the findings, not the tool.

A month of that will teach you more about how AI handles your files than a year of vendor demos.

Getting in the water

Starting in the shallow end still gets you in the water. The oversight is already built in, and the expertise stays with the people who know what good claims handling looks like.

The standard is still the expert. Audit is just the safest place to see what the AI does with it.
The Audit Portal was designed around these very principles and effectively uses AI to solve this very issue. Learn more www.theauditportal.ai.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *

*

This site uses Akismet to reduce spam. Learn how your comment data is processed.