When AI Gets Riskier, Insurers Start Asking Harder Questions

As policymakers and technology companies debate how to make AI safer, insurers are beginning to impose their own test: Can the risk actually be underwritten?

Underwriters are asking more detailed questions about AI governance, human oversight, sensitive data and increasingly autonomous systems as AI moves deeper into business operations. The scrutiny is also shifting insurance coverage away from broad, implicit protection toward policies that address AI risks more explicitly.

In this email Q&A with The AI Innovator, Sherin Ko, vice president of cyber at insurer Relm spoke about what insurers now want to know, what can make an AI company difficult to insure and how the rise of agentic AI is changing the conversation.

The AI Innovator: What are underwriters asking AI companies today during policy renewal time that they weren’t asking a year ago?

Sherin Ko: A year ago, insurance discourse centered tighter on whether a company was using AI at all. It seemed that for many, AI was treated almost like a more advanced Google in chatbot form.

That has changed quickly. Now it’s not a question of if they’re using AI but how, because you’d be hard-pressed to find anyone who isn’t using the tech. For renewal questions, when I’ve been asked this recently, I go back to what a Relm colleague has said: “How are you using AI? Who is responsible for it? How are outputs checked? What governance do you have around the model? Can you evidence any of that?”

Underwriters need to understand what role it plays and how much responsibility has been handed to it. If AI is only assisting a person, that’s one thing. If it’s starting to run a workflow with limited human involvement, then we’re having a very different conversation.

Which governance practices or risk controls most affect the decision on whether or not an AI company can get coverage and how much it pays?

I can’t give a pricing formula because that’s where the secret sauce comes into play. But the controls that matter most are those that show the company understands what the AI is doing and has proper control around it.

First off, we’re looking to see that the company even has governance controls. Then, governance has to match the use case. A low-risk internal tool should not be treated the same way as AI involved in a regulated or legally sensitive process. If the model is using sensitive data, the governance needs to be stronger. If the output could affect a customer or counterparty, the oversight needs to be stronger again.

Human oversight matters, but it isn’t enough to say there is a person in the process because we have to understand what that person is doing. We need to see that the company has a process because it’s helpful when they can reconstruct what happened after a disputed output.

Agentic AI is another one that makes that harder. The purpose of agentic AI is to take more of the process on itself. That means underwriters need to understand how the company has limited the system, who owns the process, and how quickly someone can step in if it goes wrong.

Is there such a thing as an uninsurable AI product or service? Why or why not?

Very little is uninsurable in principle, but some AI products can become uninsurable in practice.

There are cases where the business model itself creates the problem. For instance, we’ve seen AI creator platforms where users could generate and publish sensitive content using people’s likenesses with very limited control over what was created. In that scenario, anyone can see the issue isn’t the AI. If anyone’s pushing products without hefty guardrails then there probably isn’t a case for underwriting.

You said “silent AI” coverage is disappearing from cyber and tech E&O (errors and omissions) policies. What does that mean in practical terms for AI companies?

I would change the premise slightly because this is not only about cyber and tech E&O. If you’re an AI company building a model or selling an AI-enabled service, cyber and technology E&O may be the right place to start. In that market, we are seeing more affirmative AI coverage. That’s positive because the exposure is being addressed directly instead of being left in a grey area.

The harder issue is what happens to everyone else using AI. A law firm may be using AI inside its legal work. An architect or engineer may be using it inside a professional service. Those firms may not be buying cyber and technology E&O as their core cover. They may be relying on traditional professional liability wording that was not built with AI in mind.

The EU has delayed some AI Act obligations until late 2027. How does that change the way AI companies think about product launches, expansion into Europe and insurance coverage, if at all?

The regulatory timetable may have moved but liability definitely hasn’t. If a company is using AI today, the risk exists today. We’ve also been in in this position before because insurance and regulation don’t always magically move at the same speed. In emerging industries, how can they? A delay to parts of the AI Act may give companies more time on formal compliance, but it doesn’t remove the questions they face from customers, boards, investors or insurers.

I would treat the delay as time to get the insurance position into better shape. By the time the regulatory date arrives, companies should already be able to explain how the product works, where liability could sit, and where their insurance responds.

What should AI companies do now to remain insurable as they scale?

For an AI company, the issue is not mapping where AI is being used in the business. AI is the business. The real question is whether the insurer and broker understand how that business is changing as it scales.

These companies can move quickly. A product can begin as an assistive tool and then become part of a much larger workflow. It can start serving a different type of customer. It can begin handling more sensitive data. It can add agentic capability. Each of those changes can move the risk away from what was originally underwritten.

That makes communication important. The company should keep its broker close and keep its insurer informed. It should not wait until renewal to explain that the product has changed in a material way.

The aim is informed capacity which means you can’t renew based on old information or on an old version of the company. You want capacity that understands what the business is doing now, how it is scaling, and where the real exposures sit.