Multi-task training with in-domain language models for diagnostic reasoning
Published in Proceedings of the 5th Clinical Natural Language Processing Workshop, 78-85, 2023, 2023
Recommended citation: Brihat Sharma, Yanjun Gao, Timothy A Miller, Matthew Churpek, Majid Afshar, and Dmitriy Dligach. 2023. Multi-task training with in-domain language models for diagnostic reasoning. In Proceedings of the 5th Clinical Natural Language Processing Workshop, 78-85, 2023. https://aclanthology.org/2023.clinicalnlp-1.10.pdf
Abstract:
Generative artificial intelligence (AI) is a promising direction for augmenting clinical diagnostic decision support and reducing diagnostic errors, a leading contributor to medical errors. To further the development of clinical AI systems, the Diagnostic Reasoning Benchmark (DR. BENCH) was introduced as a comprehensive generative AI framework, comprised of six tasks representing key components in clinical reasoning. We present a comparative analysis of in-domain versus out-of-domain language models as well as multi-task versus single task training with a focus on the problem summarization task in DR. BENCH. We demonstrate that a multi-task, clinically-trained language model outperforms its general domain counterpart by a large margin, establishing a new state-of-the-art performance, with a ROUGE-L score of 28.55. This research underscores the value of domain-specific training for optimizing clinical diagnostic reasoning tasks.