Structured AI Decision-Making Outperforms Humans in Disaster Management Scenarios

Structured AI Decision-Making in Disaster Management

Abstract

Artificial intelligence (AI) is increasingly used to automate decision-making in safety-critical fields, including aerospace and emergency response. However, ethical concerns arise around ensuring these AI decisions remain reliable and justifiable, especially when human lives depend on them. This article presents a structured decision-making framework designed to bring clarity and accountability to autonomous AI in disaster management.

The framework introduces Enabler agents, distinct decision-making Levels, and specific disaster Scenarios. Its performance was tested against traditional judgement-based systems and human operators experienced in disaster situations—victims, volunteers, and stakeholders. Results showed a 60.94% increase in decision stability over judgement-based systems and a 38.93% higher accuracy than human operators across various scenarios. These outcomes highlight the framework's potential to support more dependable autonomous AI in critical disaster contexts.

Introduction

Disaster management involves three key phases: pre-disaster (warnings and evacuations), disaster (rescue and relief), and post-disaster (damage assessment and rehabilitation). AI, UAVs, and satellite imagery have boosted the ability to analyze data and support decisions in each phase. Autonomous systems have been developed for coordination and response, but safety concerns remain due to the risks errors or biases pose in critical decisions.

This tension between automation benefits and the need for human oversight calls for structured AI decision-making that strengthens reliability and accountability. The framework discussed here addresses this by defining clear roles for AI agents and organizing decision points systematically. The study also includes a human evaluation to benchmark AI performance against experienced operators.

Related Work

Disaster decision-making is complex, dynamic, and often unstructured. Coordination between local agencies, volunteers, and government bodies frequently faces challenges leading to delayed or uneven responses. Past incidents like the Fukushima nuclear disaster and a major UK railway accident revealed how poor coordination and unstructured decisions can hinder effective response.

Machine learning has improved disaster response by processing large data sources like drone footage and satellite images. Multimodal models combining text and images from social media have enhanced situational awareness. Post-disaster, AI models assess damage severity to guide reconstruction. However, these approaches often struggle with inter-agency coordination and integrating diverse, rapidly changing data.

Recent work explores autonomous decision-making with robot teams and big data analytics. Still, many solutions focus on replacing humans rather than structuring how AI assists in decision-making. This framework aims to fill that gap by systematizing decisions and enabling transparent, traceable AI actions.

Framework for Structured Decision-Making

The proposed framework tackles coordination and data overload by organizing decision-making into a structured flow. It uses two types of agents: Enabler agents that analyze incoming data and provide insights, and Decision Maker agents that use those insights to make choices. Decisions unfold within a Scenario—a tree-like structure with five Levels, each representing a critical decision point.

• Level -1: Determines if incoming data is informative to the disaster situation.
• Level -2: Identifies the type of humanitarian effort the data relates to.
• Level -3: Assesses damage severity from data collected by victims or volunteers.
• Level -4: Evaluates damage severity from satellite imagery.
• Level -5: Assesses damage severity from UAV (drone) footage.

This structure supports traceable decisions across emergency response, rehabilitation, and reconstruction activities. Enabler agents analyze data at each Level, feeding judgment insights to the Decision Maker, which can be an AI reinforcement learning (RL) agent or a human operator.

Methods

The framework was implemented and evaluated using datasets representing disaster and post-disaster phases. The CrisisMMD dataset, containing image-text pairs from social media during disasters, supported disaster-phase decisions. For damage assessment in the post-disaster phase, the xBD and RescueNet datasets provided satellite and UAV imagery.

Enabler agents were trained as classification models tailored to each Level’s data type. For example, at Level -1, the agent classifies data as informative or not. At Levels -4 and -5, agents evaluate damage severity from images using multilabel classification.

Performance was measured using precision, recall, F1-score, and accuracy. The Decision Maker agent used reinforcement learning to make decisions based on Enabler insights. A human evaluation involved volunteers using a custom web app, Disaster Maestro, to compare human and AI decision accuracy.

Results

Enabler agents showed strong classification performance across Levels. At Level -1, the agent effectively filtered relevant data, crucial for subsequent decision steps. Similar performance patterns were observed at other Levels, supporting reliable judgment inputs.

Most importantly, the autonomous Decision Maker agent consistently outperformed human operators and judgement-based systems in decision accuracy and stability across multiple scenarios. This confirms the value of structuring AI decision flows and combining AI insights with reinforcement learning for enhanced disaster management.

Conclusion

This study demonstrates that a structured AI decision-making framework improves the reliability and accuracy of autonomous decisions in disaster management. By defining clear roles and organizing decision points, the framework surpasses traditional judgement-based systems and human operators in performance.

Implementing such structured AI solutions can help emergency managers and stakeholders achieve faster, more consistent, and justifiable decisions in safety-critical situations. Continued development and real-world testing will help refine the framework and expand its use across diverse disaster scenarios.

For managers interested in leveraging AI for critical decision-making, exploring structured frameworks can provide greater confidence in autonomous systems. Learn more about building AI skills for practical applications at Complete AI Training.