Interpretable Multimodal AI for Urgency Prediction and Specialist Referral in Telemedicine

Abstract

Reliable assessment of clinical urgency remains a major challenge in telemedicine, where decisions often depend on incomplete physiological information and brief patient-reported symptoms. This study introduces TeleMed-XAI, an interpretable multimodal learning framework designed to support five-level urgency classification and specialist referral. The framework combines structured demographic and physiological features encoded through a multilayer perceptron with contextual representations of chief complaints extracted using ClinicalBERT. A learnable gating mechanism integrates both modalities, while SHapley Additive exPlanations (SHAP) and token-level attribution provide insights into model predictions. A rule-constrained referral component is further incorporated to support specialty selection and emergency prioritization. To illustrate the proposed evaluation framework, a synthetic dataset comprising 10,000 simulated patient encounters was used to demonstrate classification, calibration, and ablation analyses. In this simulation, TeleMed-XAI achieved 86.5% accuracy, a macro-F1 score of 0.820, and a macro-averaged AUROC of 0.983, with 95.1% sensitivity for high-acuity cases. The illustrative calibration analysis yielded a Brier score of 0.038 and an expected calibration error of 0.040. The simulated ablation comparison also indicated an increase in macro-F1 from 0.787 with feature concatenation to 0.820 with gated fusion. These results demonstrate the proposed evaluation workflow but do not establish predictive or clinical effectiveness. Validation on MIMIC-IV-ED, followed by external testing and independent assessment of specialist referral, is required to determine the framework's practical value.