Deployment of Large Vision and Language Models for Real-Time Robotic Triage in a Mass Casualty Incident

Abstract

In the event of a mass casualty incident, such as a natural disaster or war zone, having a system of triage in place that is efficient and accurate is critical for life-saving intervention, but medical personnel and resources are often strained and struggle to provide immediate care to those in need. This thesis proposes a system of autonomous air and ground vehicles equipped with stand-off sensing equipment designed to detect and localize casualties and assess them for critical injury patterns. The goal is to assist emergency medical technicians in identifying those in need of primary care by using generative AI models to analyze casualty images and communicate with the victims. Large language models are explored for the purpose of developing a chatbot that can ask a casualty where they are experiencing pain and make an informed assessment about injury classifications, and a vision language model is prompt engineered to assess a casualty image to produce a report on designated injury classifiers.