Imagine a claims leadership meeting a year after the organization introduced a new AI platform.

The dashboard looks encouraging. Documents are being summarized. Demand packages are being reviewed faster. The vendor estimates that thousands of hours have been returned to the operation.

By nearly every measure on the screen, the implementation is working.

Then someone asks a different question: Did we make better claim decisions?

Most organizations are better equipped to measure what technology does than what it changes. We can count summaries, minutes saved and tasks automated.

But those figures cannot tell us whether severity was recognized sooner, a reserve was more accurate or the right case was referred to counsel. They cannot tell us if the file was negotiated more effectively or the outcome was better. Nor do they reveal when a professional catches what the system missed or accepts an answer that should have been questioned.

The claims industry is entering the next phase of AI adoption. The question is no longer simply whether the technology works but what “working” means.

The Two Sides Are Optimizing for Different Things

There is a widening difference in how the plaintiffs’ bar and the defense industry appear to be using AI. Broadly speaking, plaintiffs’ firms are using it to maximize outcomes: strengthening demands, organizing evidence, shaping narratives and finding arguments likely to increase claim value.

Defense organizations often describe AI in the language of efficiency: faster summaries, fewer touches, shorter cycle times and greater capacity. Those are worthwhile gains, but they reflect a different objective.

If forced to choose between a faster decision and a better one, we would choose better. The ideal is both—speed with quality. A weak analysis delivered in minutes is still weak. A missed severity signal does not become less costly because the file moved quickly.

That changes what the defense should measure. What did AI remove from the process and what did it add to the decision? Did it recognize risk earlier, improve the evaluation, sharpen negotiation strategy or change the outcome? That is how the defense moves from automation to competitive advantage.

The Value We Can See

AI is already proving useful around claim decisions. It can organize medical records, summarize depositions, extract information from demand packages, assemble chronologies, triage claims and flag files requiring earlier attention. None of this is trivial.

Claim professionals spend too much time searching for information and converting unstructured material into something they can evaluate. Removing that friction has real value. Saving 20 minutes on a task performed thousands of times can reduce backlogs and create more time for claimants, counsel and difficult files.

Yet saved time does not automatically become productive capacity. Unless the organization sees what happens afterward, part of the benefit remains theoretical. Did inventory decline? Did the time reappear as better investigation or negotiation preparation? Or did it disappear into an operation already struggling to keep pace?

Time savings begin the ROI discussion. They should not end it.

Claims: Not Just a Production Process

Claims work is a sequence of decisions made under uncertainty. Productivity is not easily defined.

A claim can move quickly and still be handled poorly. A file can close early because the organization recognized an opportunity or failed to recognize its leverage. Legal expenses can fall while indemnity leakage rises. An AI-modeled settlement range can still be wrong for the facts or venue.

Even consistency requires closer examination. Applying a flawed assumption consistently merely distributes the error more evenly. Operational measures still matter, but they should sit alongside decision quality, claim outcomes and human capability. Did technology identify severity earlier? Improve reserve timing? Surface litigation risk? Did it alter the strategy or merely document one already chosen?

Those questions bring us closer to value.

Measure the Decision, Not Merely the Activity

Consider an AI tool reviewing bodily injury demand packages. The easiest measures are speed and usage: review time, volume and adoption. The more meaningful analysis begins afterward.

Did the tool identify missing records, unsupported damages or inconsistencies? Could the professional separate material observations from noise? Did the analysis influence the evaluation, reserve or negotiation plan?

Perhaps most importantly, what did the tool surface that was not included in the injury demand package in the first place? We all know that if we’re working only from the demand package, we’re already working within the plaintiff’s framework and narrative and are losing the negotiation.

Ultimately, we want to know whether comparable claims produce better or worse outcomes. That question requires patience. Severity, venue, counsel, policy limits and timing all affect results. A before-and-after comparison of average settlements proves little by itself.

Claims leaders need baselines, matched comparisons and enough time to separate durable change from ordinary variation. Similar claim cohorts or a credible control group may be necessary.

That work is less exciting than a product demonstration. It is also where the real business case will be found.

Pay Attention When the Human Says No

One of the richest sources of information may be the moment a claim professional rejects the machine’s recommendation. Some organizations view overrides as an adoption problem. Sometimes they are. A professional may reject sound advice because it conflicts with habit. But an override may also reveal a model limitation.

Perhaps the system misunderstood a medical issue, missed a coverage nuance or relied too heavily on a changing venue’s history. Maybe the professional recognized that emotion was driving the negotiation more than economics. The override alone tells us very little. The reasoning and eventual outcome tell us much more.

Organizations should examine what the system recommended, what the professional decided, why they differed and which judgment proved more reliable. Those disagreements can expose model weaknesses, coaching opportunities and especially strong human judgment. Good vendors should be able to provide that override information back to us.

Human disagreement should not be treated as interference with the system. Properly studied, it is part of the system. It is the literal human-in-the-loop that we must require of new technologies.

Gaining Capacity Today But Weakening Bench Strength for Tomorrow?

There is another measurement problem that may not appear on a dashboard for several years. Many tasks being automated are tedious. They are also part of how claim professionals learn.

Reading a complete medical record develops pattern recognition. Building a chronology reveals inconsistencies. Working through a demand teaches a newer professional how a plaintiff attorney’s argument is constructed and where it is vulnerable.

If technology delivers the conclusion before the professional develops the ability to reach it independently, the organization may gain capacity today while weakening its bench for tomorrow.

This does not mean preserving inefficient work. Leaders must separate low-value administration from the experiences through which judgment is formed. That may require raw-file reviews, case conferences and opportunities to reach a conclusion before seeing the system’s answer.

The objective is not to make the human compete with the machine. It is to ensure the human remains capable of recognizing when the machine is wrong.

A Better Scorecard

The next generation of claims AI scorecards should show whether the technology created usable capacity and improved decision quality. They should connect legal expense and indemnity while tracking overrides and what happens afterward.

They should also look for unintended consequences. Are professionals spending more time with claimants and counsel, or simply managing larger inventories? Is AI output the beginning of analysis or a substitute for it? Can newer professionals meaningfully challenge the system? Ultimately, value will be defined not just by AI output but by how the AI output is used by the claim professional.

These are harder questions to answer than just how much activity has there been. But they are questions that today’s change leaders need to start answering.

The organizations that benefit most from AI will not necessarily be those that automate the most tasks. They will connect machine output, human judgment and actual claim results into a continuous learning system.

The defense should not settle for a choice between speed and quality. It should expect AI to deliver both and build scorecards capable of proving it.

AI can make a claims organization faster almost immediately. Competitive advantage comes when it also helps the organization make better decisions.