ENABLING SAFE ROBOTIC COMPANIONS THROUGH HUMAN-AWARE SIMULATION, NATURAL COMMUNICATION, AND EMERGENCY RESPONSE

Abstract

Robotic companions have the potential to revolutionize daily life by helping users with ev-eryday tasks, improving safety, and enhancing quality of life in home environments. However, advancing these robotic companions requires addressing three distinct technical challenges. First, representative household simulation environments lack realistic human agents that move and in- teract naturally with scenes, which limits the development and testing of human-robot interaction algorithms. Second, robots must be able to interpret and respond to user commands given in nat- ural language, a task complicated by the inherent ambiguity in human communication. Finally, to meaningfully improve home safety, robots must be able to detect and respond to emergency situations. This dissertation addresses each of these challenges through novel methods spanning simulation, communication, and emergency response. In simulation, we present PAAK and PACE, novel methods for creating realistic virtual human agents in 3D environments. PAAK places motion-captured human animations into 3D scenes, maintaining human-scene interactions by using “keyframes”—the most important frames for modeling those interactions. PACE builds upon this by modifying motion sequences to adapt to dense, cluttered environments, adjusting the high-DOF pose at each frame to account for unique geometric constraints, and received Best Paper Honorable Mention at IEEE VR 2023. For user communication, we present LGX and LBAP, which enable robots to understand natural language commands and handle ambiguity. LGX leverages Large Language Models (LLMs) for language-guided zero-shot object navigation, achieving a 27% improvement in zero-shot success rate over prior methods on RoboTHOR, with successful real-world validation on a TurtleBot 2 platform. LBAP uses Bayesian inference for uncertainty alignment in LLM planners, mitigat- ing hallucinations and better aligning confidence measures with probability of success, reducing human intervention by over 33% at 70% success rate in real-world experiments on a ClearPath TurtleBot. For household emergencies, we present SafetyDetect and HomeEmergency, enabling robots to detect and respond to dangerous situations. SafetyDetect introduces a dataset of 1000 anoma- lous home scenarios and an LLM-based approach using scene graphs to identify unsafe or unsan- itary conditions, achieving a 96% anomaly detection rate in simulation and 89% on a ClearPath TurtleBot in real-world scenarios. HomeEmergency addresses actively occurring emergencies like falls and fires using audio localization through a novel probabilistic dynamic scene graph, where Bayesian inference enables efficient agent localization, with validation across 30 real- world experiments in apartment environments. While PAAK is primarily a simulation tool, our other methods include real-world validation: LGX, LBAP, SafetyDetect, and HomeEmergency all demonstrate effectiveness on physical robotic platforms. Together, these contributions ad- vance robotic companions across research tools, communication capabilities, and safety features, moving toward more helpful and trustworthy robotic companions.