Multi-Modal Deep Learning Framework for Pilot Situation Awareness Assessment Using Eye Gaze and Flight Control Signals

Abstract

Human factors and pilot situation awareness play a critical role in aviation safety, especially during complex flight operations where cognitive overload and loss of attention may lead to accidents. This project presents a multimodal deep learning framework for pilot situation awareness analysis using gaze position and flight control data. The proposed system integrates human eye-tracking information with aircraft flight parameters such as throttle, altitude, speed, pitch, roll, and yaw to evaluate the cognitive awareness level of pilots in real time. The system is implemented using Python and Flask for web-based interaction, while machine learning algorithms including XGBoost, Random Forest, Decision Tree, Support Vector Machine (SVM), K-Nearest Neighbors (KNN), and Logistic Regression are used for classification and comparative analysis. The uploaded aviation dataset undergoes preprocessing techniques such as missing value removal, label encoding, and feature standardization before model training. Multiple classifiers are trained and evaluated, and the model with the highest accuracy is automatically selected as the best prediction model. During prediction, the trained system analyzes pilot gaze coordinates and flight control behavior to classify the pilot’s situation awareness level as either High Awareness or Low Awareness. Based on the prediction results, the system also generates intelligent safety recommendations to assist in maintaining stable flight operations and reducing pilot workload. Experimental results demonstrate that the proposed framework effectively improves awareness detection accuracy and provides real-time decision support for aviation safety monitoring systems. The developed application offers a scalable and intelligent human-centered aviation monitoring solution that can contribute to reducing human-error-related incidents in modern aircraft operations.