Evaluating AI in Healthcare with Epic’s Open-Source Seismometer Tool
Healthcare organizations often struggle to assess the true performance of artificial intelligence (AI) models once these are integrated into clinical workflows. Epic's open-source Seismometer tool offers a practical way to evaluate AI models locally, helping providers determine their effectiveness based on real-world data and specific workflows.
What is the Seismometer Tool?
The Seismometer ingests an organization’s data to answer key questions about AI models, such as their impact on patient outcomes, speed of treatment, and user acceptance. It also enables side-by-side comparisons of different AI models, a useful feature for health systems that want to validate custom models against vendor solutions.
Since its introduction, Epic and collaborators have refined the tool to better support healthcare organizations that often lack the resources to conduct thorough AI model validation.
How Healthcare Organizations Use the Seismometer
At the University of Wisconsin (UW Health), the tool has become part of standard practice. Brian Patterson, medical informatics director for predictive analytics and AI there, highlights its enhanced features for calculating uncertainty and subgroup analyses—crucial for ensuring fairness in AI applications.
Michigan Medicine has also adopted the Seismometer to evaluate the Epic Sepsis version 2 model. The tool helped identify different alert thresholds for various clinical groups, such as emergency departments and intensive care units. This customization aligns model performance with specific workflows, addressing challenges in reacting to alerts.
Sean Meyer, data science and AI engineering lead at Michigan Medicine, emphasizes the Seismometer’s graphical interface that automates performance plots and filtering, reducing the need for manual coding. The tool pulls historical data on patient outcomes and events—like ICU transfers and antibiotic administration—streamlining validation processes and improving communication with leadership and clinical committees.
Benefits Beyond ROI
Unlike tools focused on return on investment (ROI), the Seismometer prioritizes assessing the quality of AI models—such as improved patient outcomes and workflow integration. Dr. Michael Burns from Michigan Medicine notes that while direct ROI may not always be clear, understanding whether AI improves care quality and efficiency is vital.
“Are we treating patients better, faster, or more efficiently?” he asks. This focus on practical outcomes reflects the complex nature of conditions like sepsis, where patient volumes remain steady despite interventions.
Standardizing AI Validation and Comparisons
The Seismometer simplifies comparing custom-built AI models with vendor solutions, eliminating duplicated evaluation efforts. This capability makes it easier for health systems to decide which models to adopt based on transparent and consistent metrics.
UW Health appreciates how the tool presents key statistics—such as sensitivity and area under the curve (AUC)—broken down by subgroups like age. These metrics provide clear “currency” for conversations about deploying AI models.
Addressing Challenges and Enhancing Transparency
One challenge identified at Michigan Medicine was limited visibility into how the tool handles incomplete patient data. Epic has since improved transparency by making technical mappings clearer, helping users align the Seismometer’s outputs with their own analyses.
Supporting Resource-Limited Organizations
Federally Qualified Health Centers (FQHCs) face additional hurdles in adopting AI due to limited technical expertise and funding. The Community-University Health Care Center in Minneapolis is working with Epic, OCHIN, and the University of Minnesota to implement and validate predictive analytics for patient no-shows.
Through this collaboration, the center aims to reduce staff communication burdens and improve outreach. Although the FQHC cannot independently activate the model, it relies on OCHIN to provide risk scores integrated into its systems.
Once performance data is available, the center plans to evaluate AI tools for managing chronic conditions like asthma and hypertension, monitoring for disparities, and promoting fairness in AI-driven care.
Looking Ahead
The Seismometer is helping healthcare organizations move toward more transparent, efficient, and data-driven AI validation processes. While it currently operates locally, future versions may be server-based, improving scalability and accessibility.
Close collaboration between health systems and Epic’s development teams has encouraged user-driven improvements and new feature recommendations, enhancing the tool’s value.
For healthcare professionals interested in expanding their AI knowledge and skills, exploring courses on AI applications in healthcare can be valuable. Resources like Complete AI Training offer targeted learning paths tailored to healthcare roles.
Your membership also unlocks:
