Over the past decade, property/casualty carriers made serious and necessary investments in front-end digital transformation. Most notably, we built mobile apps and updated portals so policyholders and agents could interact with us far more easily. We learned a valuable lesson during that time: the core systems the front-end technology pulled from needed to be robust and provide a clean “source of truth” foundation.

Now, we’re experiencing a similar transformation as AI evolves in the industry, and the lesson remains the same. AI delivers richer value when it runs on a strong foundational core. That means AI benefits from deep, carrier-specific context through data, workflows and procedures. It also means the output and mechanics of AI work reside in the core platform, creating the audit trail regulators will increasingly demand, which is a theme we’ll return to below.

Across industries, AI tends to arrive as a bolted-on point solution rather than something built into the core. In the P/C industry, that usually looks like an underwriting document summarizer or a claims triage pilot running alongside the systems people actually work in every day. It demos well. It just doesn’t hold up once the core platform underneath it can’t support what the AI is trying to do.

The AI models aren’t the problem. The infrastructure underneath them is. Before signing your next AI vendor contract, put these six questions to your external partners and internal teams.

1. Has this platform actually run at scale, on live core data, through a full regulatory cycle?

Most AI success stories across the industry are still just pilot stories. They run on clean, hand-picked sample data in limited scenarios. Those controlled environments look great in a presentation but offer little insight into how an AI system handles the real-world complexities of a live P/C portfolio, where conditions and requirements are constantly changing.

Implementing a production-ready technology requires a higher standard of excellence. Ask whether the platform has been deployed at carriers of similar complexity, using live system-of-record data, through at least one full regulatory filing or financial reporting cycle. That includes handling complex exceptions, retroactive adjustments and rating plan updates.

A vendor with real production miles can point to specific jurisdictions and transaction volumes. If they can only talk about sandbox trials and innovation labs, your organization may be funding their product development along with the unbudgeted operational risk that comes with it.

2. How many people in core operational roles still use the AI tool regularly 90 days after launch?

AI can only improve operating results if it changes how people work every day, meaning claims adjusters, underwriters and service reps are actually using AI-supported technology inside their systems.

Push vendors for real numbers. What percentage of intended users at similar insurers are still touching the tool 90 days after launch? Get specific about which workflow it’s supporting, whether that’s FNOL triage, subrogation identification or submission clearance. A single strong number here tells you more than a client reference call ever will.

Login frequency and time saved per task are the two metrics that separate a real capability from a pilot people quietly stopped using. When a vendor answers with satisfaction scores instead of usage data, the usage data probably doesn’t look good. Adjusters can genuinely like a tool while loss adjustment expense sits flat, which just means the tool isn’t the thing moving the number you care about.

3. Can you produce an AI action trail that satisfies regulators without custom engineering?

As AI moves from drafting emails to making direct underwriting and claims decisions, regulatory scrutiny will scale exponentially. Regulators and internal compliance teams won’t just ask what an AI model recommended. They’ll want to know how it got there and who signed off.

You should be able to see exactly what happened anytime the system takes an automated action. From day one, it should create a simple trail tied to each policy or claim, showing which model version ran, what information it saw, and whether a human approved or changed the entry.

Ask your vendors to show you the actual audit logs and event histories they use, in a form regulators can review directly. Vendors who say only that they “log outputs” or can “add an audit layer later” are telling you the audit trail doesn’t really exist yet. And when a defensible audit trail requires custom code for every new use case, your compliance team slows each rollout, creating efficiency gaps down the line.

4. Where is the AI actually pulling its data from?

A lot of early insurance AI was really just document retrieval wearing an AI label. Index the policy PDFs and claims notes, let a language model search and summarize them, and call it done. That works fine if someone’s trying to find a paragraph in an endorsement, but it falls apart the moment the decision downstream depends on data that changed an hour ago.

Ask to see the data lineage. If the AI reads and writes into live policy, billing and claims systems through governed APIs, recommendations reflect what’s actually true right now. If it’s working off a synced copy, it’s making decisions on numbers that could already be wrong, and an endorsement issued two hours ago can flip what’s accurate for a given risk.

Carriers need an architecture that uses static documents for context but constantly checks recommendations against live core data before executing actions. Systems relying on static PDFs or files synced overnight will eventually act on outdated information, exposing the business to coverage and compliance risk, along with the leakage that follows.

5. Can underlying language models be swapped without rebuilding core business workflows?

The AI model market moves fast, and today’s model may be superseded in a week or a month. Property/casualty carriers can’t afford to rebuild complex claims and underwriting workflows every time a model provider updates its technology. Ask your vendor directly: If we swap or upgrade the underlying model next year, what breaks? Which workflows, integrations or custom code would need to be rebuilt?

A durable setup keeps the workflow separate from whatever model is powering it. Sorting new claims or running a rating check shouldn’t care which AI engine sits behind it. Your team should be able to swap models through a standard connection, flip them on or off, and move on.

A big coding effort or a workflow redesign every time you change or upgrade an AI provider is a sign the tool was never more than a temporary add-on. Hard-wiring models into application code piles up technical debt and almost always leads to expensive delays later.

6. How much data replication is required to make this architecture function?

Finally, look closely at how much data movement a platform requires. Architectures that pull large amounts of policy and claims data into external sandboxes or third-party databases create latency and compliance gaps. The objective should always be to minimize data movement, using real-time, event-driven integrations so AI works directly on live core data and writes approved actions straight back to the primary system. Complex pipelines, overnight batch syncing and shadow databases that copy core data create unnecessary complexity, recreating the same data fragmentation and technical debt core modernization was designed to eliminate.

The reality across all six questions is straightforward. The biggest risk in insurance AI is architectural fragmentation. Layering automated decisions on fragile core systems just leads to automated mistakes happening faster than human teams can catch them, and that’s a risk companies can’t afford to take.

Success with AI starts with a strong core. When carriers align their AI strategy with core modernization, they can move beyond pilots and turn AI into a real part of how the business operates.