Analyzing Information Disparities across Modalities in Mortality Prediction

Abstract:

Recent advances in deep learning have enabled the integration of heterogeneous data modalities for clinical prediction, allowing models to exploit complex information embedded within electronic health records (EHRs). Among these modalities, chest radiographs (CXRs) provide a rich source of visual information that can enhance patient outcome prediction for patients in the intensive care unit (ICU). However, the comparative impact of different CXR representations—raw images versus radiology reports—on predictive performance has not been systematically investigated. Such comparisons are essential for identifying the most informative modality and understanding how it complements other data sources. This study compares the predictive utility of raw CXRs versus radiology reports for 30-day post-discharge mortality prediction in ICU patients. We employed a Vision–Language Model (VLM) with patient …