Comparing prognostic performance and reasoning between physicians and large language models

Abstract:

Background Physicians routinely prognosticate to guide shared decision making, particularly when caring for patients with critical illnesses (1). Yet, these physician estimates are prone to inaccuracy and uncertainty (2,3). Artificial intelligence, including large language models (LLMs), shows promise in supporting or improving prognostication (4). However, the performance of contemporary LLMs in prognosticating for the heterogeneous population of critically ill patients remains poorly understood. Objective To compare the performance and reasoning of LLMs and physicians when predicting 6-month mortality for hospitalized, critically ill adults. Methods We randomly selected 100 patient admissions with ICU stays from MIMIC-IV, a publicly available, single-center dataset of hospitalizations. For each case, 4 contemporary LLMs and 7 practicing physicians were asked to use the discharge summary and demographics to …