NAIC starts work on an AI evaluation method for insurance regulators
- Proposed questionnaire extends 2023 guidance with practical disclosures
- Regulators have reviewed only Part 1 so far; more sessions set for January and February
- Industry concerns: the plan could create more data than supervisors can use
The National Association of Insurance Commissioners (NAIC) is moving beyond its 2023 AI Model Bulletin with a new AI Systems Evaluation Tool - a standardized questionnaire template for supervisors. Where the Bulletin centers on governance and risk policies, the tool is meant to surface concrete information about how carriers build, deploy, and oversee AI in the business.
At NAIC's fall meeting, materials outlined four parts: (1) quantifying AI use across the enterprise, (2) a governance and risk assessment framework, (3) details for a high-risk model, and (4) details for model data. The Big Data and Artificial Intelligence Working Group discussed a second draft on December 7, led by co-vice chair and Iowa insurance commissioner Doug Ommen, but covered only Part 1 in a four-hour session.
What the tool covers
- Part 1: Inventory and scope - where AI is used, model purposes, business lines, and criticality
- Part 2: Governance - accountability, documentation, testing, monitoring, and risk controls
- Part 3: High-risk model deep dive - design intent, performance, validation, monitoring, and controls
- Part 4: Data - sources, quality, lineage, privacy, and access controls
Revisions to Part 1 after regulator feedback
- How AI will be monitored in market conduct and financial condition exams
- Confidentiality protections for sensitive model and data information
- Coordination methods across states to reduce duplication
- Clearer scope: which AI models and algorithms are in or out, and which insurer functions get scrutiny
Ommen described the tool as a set of templates and exhibits to help regulators start informed conversations with carriers about the AI applications running in production.
Industry response: "Inventory," not evaluation
John Romano, a principal at Baker Tilly focused on insurance audits, views the proposal as more of an inventory than an evaluation device. NAIC wants to understand how insurers use AI and how they assess risk, he said, but he expects pushback: "This could just lead to more disclosure of information than the states know what to do with. They ultimately want to know what they are going to do with all this information."
Why standardization matters - and where it might fall short
Heidi Lawson, partner at Fenwick in insurance, insurtech, and financial services, said the tool helps states align on a common approach given different levels of AI literacy. "If they agree on one thing that could be helpful, because otherwise, it feels like a very high chance it's going to be misunderstood."
She added that third-party diagnostics that test model accuracy and drift could be more efficient than a static questionnaire. A Q&A has value, but its insight will be limited without independent testing and ongoing monitoring.
Adoption to date
According to participants in the December 7 meeting, 24 states have adopted the NAIC AI Model Bulletin, and 10 states have agreed to use the new evaluation tool so far. Broader adoption is expected to hinge on how well the tool balances clarity, workload, and confidentiality.
Federal backdrop
Days after the meeting, on December 11, President Trump issued an order banning AI regulation at the state level. Ommen noted that the order addresses state legislation governing the development of AI, not how industries use AI. "As a state insurance regulator, my concern is not regulating the development," he said. "My concern is making sure that in the use of any tool such as AI, consumers are treated fairly and appropriately in their business with insurance companies."
What insurers should do now
- Build a live inventory of AI systems: business purpose, decision type, deployment status, and criticality.
- Classify model risk: set thresholds for "high-risk" based on impact, scale, and consumer harm potential.
- Tighten governance: define accountable owners, approval gates, and model change controls.
- Level up documentation: model intent, features, training data sources, versioning, and validation evidence.
- Establish monitoring: performance, bias/fairness, drift, complaints data, and triggers for remediation.
- Secure data lineage: sources, quality checks, consent basis, privacy controls, and third-party access.
- Prep for exams: confidentiality protocols and a playbook for multi-state coordination to reduce rework.
- Pilot independent testing: consider external accuracy and drift diagnostics for high-impact models.
Why this matters for your team
Even if your state hasn't adopted the tool yet, regulators are converging on the same questions. Getting your inventory, governance, and monitoring in order now will cut exam friction later and reduce consumer risk today.
Resources
- NAIC: overview and updates on committees and model guidance naic.org
- Practical AI upskilling for insurance functions Complete AI Training - Courses by job
Your membership also unlocks:
