There’s a fine line between a leader’s ambition and an unacceptable level of risk. My experience with a complex claims team put a spotlight on this tension, and it serves as a good example of an instance where risk arises when structure and governance are still evolving. It’s also a helpful analogy for considering design principles for agentic AI in insurance, in an environment in which regulations and best practices continue to evolve.

I was working with a claims function that had experienced a period of rapid growth. The team was made up of expert claims handlers who had been operating as an early-stage firm. Leaders were responsible for multiple roles and had meaningful control over team structure, function and how they were measured. Alignment between teams was informal; there were broad-based targets in place, though reporting and practical visibility across the whole function were relatively low. Similar to a startup, the flexibility of this environment was an ideal space for entrepreneurial leaders to flourish, and many had taken advantage of this opportunity. On the flipside, the risk of over-extending one’s authority was real. In one such case, a leader had independently negotiated an informal arrangement with a vendor in return for access to their services. This agreement was strategically imbalanced. It represented a significant opportunity for the vendor yet realized only localized benefit for the carrier.

The issue we observed was less about the individual and more about the business unit’s structure and governance. While the intention was good, the leader did not have the context and perspective to take into account the bigger picture and the implications of the deal they’d struck. It’s precisely why organizations design structures that coordinate the efficient flow of information to leaders who have delegated decision rights. Mature firms introduce controls and incentives to steer decision-making activity toward alignment with the organization’s strategic objectives.

Who Makes the Decision

A helpful way of thinking about decision-making within organizations is to consider the firm as a machine designed to solve a specific set of problems. These problems are formalized as the firm’s strategic imperatives. The goals and sub-goals within the firm ladder down from those imperatives. In this model, decision authority sits with the board and cascades to the CEO, who delegates it across the firm. Rules, role clarity and incentive targets keep that delegated authority directed toward solving for the problems the firm exists to address.

The structure of the organization serves to route information efficiently to delegated decision-makers, who use it to solve for their piece of the larger puzzle. Often, this information is data that has been transformed either through teams or technologies. It’s this model that is disrupted by agentic AI systems, which are technically capable of transforming data into information, analyzing it and generating recommendations faster than humans alone. Whether this full capability ought to be exercised for every decision is a key question. Where and when to use agentic AI, from data collection through to recommendation and even determination, is a matter of decision-rights delegation.

Agentic AI systems are analogous to simulated teams and typically include one or more AI agents as team members with a coordinating orchestrator functioning as a leader. AI agents are goal-directed programs that use large language models (LLMs) as reasoning engines and may draw on traditional AI/ML predictive models as inputs. Orchestrators, which may not necessarily be AIs but may be a set of deterministic rules, coordinate AI agent activity and constrain system outputs. Agents can collect and sort data, and the orchestrator can evaluate their outputs based on rules. Then, they can either facilitate automatic execution of a decision or expose the information to a user as a recommendation.

Increasingly, regulators and lawmakers have emphasized the need for a human in the loop model for high-risk insurance decisions, such as complex claim determinations that may entail legal or financial risk or underwriting decisions that have potential to disadvantage prospects. There’s been considerable debate about the human’s role—whether they should review and sign off, or whether they play a more active role. We can explore this through an analogy of a small law firm.

Looping Humans in Without Losing the Benefits of AI

Consider a law firm’s associate and paralegal working together on a client matter. The associate holds decision rights, delegated on the basis of role and experience, and engages a paralegal to collect and assemble information within a specific scope. The associate then drafts the memo based on the material the paralegal provides. This structure makes efficient data processing possible (it’s lower cost for a paralegal to collect and sort the data) and optimizes effort in analysis and production (the associate’s time is optimized by receiving only the material required to write the memo). But what if the paralegal, instead of only assembling the material, also drafted a summary that laid out their view of how it added up?

It’s plausible that the paralegal’s summary would appear thorough. After all, they have gathered the material. But a summary reflects the scope it was framed by and may miss context beyond what was assembled, which is particularly important if the matter turns out to be more complex than it appeared at first. Even if the summary was sound, the associate would still need to reconstruct the argument logic to check its rationale, and that’s harder to do when starting from a polished, seemingly complete draft than starting from scratch. It’s easy to anchor one’s read on someone else’s framing rather than construct one’s own. In short, receiving a fully formed summary doesn’t remove the work of reasoning through the matter, but it does risk making the work easier to skip.

The analogy highlights a core design question for agentic AI systems: If we do not intend to delegate full decision rights to agentic AI, how can we best loop humans into the process without sacrificing the benefits of AI? Our analogy suggests that the answer boils down to an effective division of labor between a system that can efficiently collect and sort large volumes of data (as scoped by a user) and a decision-rights delegated user who owns the analysis and decision-making process.

Division of Labor as Design Principle

Agentic AI systems offer unparalleled efficiency and cost benefits to insurers and present an opportunity to create exceptional experiences for prospects and insureds. Yet, there are naturally decisions that bear too much risk to delegate entirely to a technology system the same way we wouldn’t hand full authority to a leader who doesn’t have the context to see the whole picture. Further, owning decision responsibility entails doing the work to reach it. An effective decision-maker ought to be able to walk through their argument process backward and forward. Doing so involves the work of analyzing information and arriving at a considered outcome. And as regulations harden, scrutiny is likely to focus less on whether there is a human in the loop and more on precisely what that human is doing.

In sum, the reasoning for a division of labor matters as much as the division itself. The AI collects and sorts data within bounds the user sets, pulling together disparate sources faster than a person could. But it stops short of analyzing or synthesizing that data into a narrative. That stays with the user, who saves time on assembly and can focus on judgment.