DEEP LEARNING AND CLINICAL NLP FRAMEWORK FOR AUTOMATED KNOWLEDGE EXTRACTION FROM MULTIMODAL HEALTH

Abstract

Multimodal health records collect heterogeneous information in clinical settings. These records consist of inscriptional data (doctor’s notes), imaging data (X-rays, CT, MRI) , waveform data (ECG, EEG, ECHO), and structured data (lab results). Deep learning has been mostly successful for single-modality data in the medical domain. However, there have been very few systems for knowledge extraction from the entire multimodal health record and consequently for clinical reasoning.

A framework to address these limitations is proposed. The framework performs knowledge extraction through a deep learning and clinical natural language processing architecture from multimodal clinical health records. It processes information from clinical notes; radiology (X-ray, CT); ultrasound; cardiovascular waveform data (ECG, ECHO); and lab test results. The architecture contains five modules, which support ingestion of diverse data types; multimodal fusion; and representation learning for knowledge extraction. It provides appropriate interfaces to ingest data and knowledge at different levels of granularity. It also generates tabular and conceptual network representations that support high-level clinical reasoning.