In a recent study published in the Nature Medicine journal, researchers assessed the efficacy of the machine learning model in the prediction of mental health crises.
The timely recognition of individuals at risk of mental health deterioration has been reported to improve disease outcomes. However, manually reviewing patient records to identify such individuals is impractical due to the high number of mental health problems prevalent in society.
Study: Machine learning model to predict mental health crises from electronic health records. Image Credit: archy13 / Shutterstock
About the study
In the present study, researchers developed a machine learning model that utilized electronic health records of individuals to surveil them for any risk of a mental health crisis for a duration of 28 days.
The team developed a predictive tool to help healthcare workers streamline caseload priorities while enabling professional intervention that could reduce the risk of a crisis. This was achieved by aligning the prediction target with the service-oriented approach in order to define a crisis as a symptom onset that needed the intervention of healthcare resources. The team generated a dataset that included crisis events that were registered whenever a patient urgently required mental health crisis services, including inpatient admission, emergency assessment of the patient, home treatment assessment, or admission to the hospital.
Furthermore, the prediction target was defined as the beginning of a mental health crisis episode, comprising one or more crisis events that were preceded by a minimum of one whole stable week with no occurrence of any crisis event. Accordingly, the team configured the machine learning model to forecast the onset of the first crisis event in an episode within the duration of the following 28 days.
Time series of events are represented with the timestamps and event characteristics in different SQL tables in the hospital’s database. These tables are processed and converted into features for the modeling task. Models are trained, tuned and selected based on the data for the period 2012–2019. The system predicts the risk of crisis onset within the next 28 days (whereby the algorithm is queried every week for every patient). The patients with the highest predicted risk are displayed on the dashboard delivered to clinicians alongside key indicators, patient notes and a questionnaire form about each patient, which the clinician fills out. The icons in this figure were made by Freepik from www.flaticon.com. IP, inpatient.
The team designed the machine learning model such that it allowed queries every week to determine the risk of each patient experiencing a mental health crisis episode in the next 28 days. The model was built by extracting information belonging to three categories: (1) static or semi-static patient data, including information such as sex, age, and International Classification of Diseases 10 (ICD-10)-coded diagnoses; (2) assessments and interactions available with the hospital (3) variables that represented the time duration since the registered events. The model generated a predicted risk score (PRS) between 0 and 1 for each subject.
Machine learning techniques such as decision trees, ensembles, probabilistic, and deep learning-based classifiers were tested. The team subsequently benchmarked the XGBoost (eXtreme gradient boosting) model against two baseline factors, namely the clinical-practice- diagnosis-based baseline model. In addition, the prediction model's performance was evaluated among patients diagnosed with mental health disorders categorized according to the first level of the ICD-10.
Furthermore, the team conducted a prospective study in which machine learning models predicted mental crisis events that were delivered every two weeks to practicing clinicians. This was conducted by querying the model to organize patients in descending order according to the patient’s PRS.
Four representative force plots, depicting how the features contributed to the prediction for four specific data points. From top to bottom: Patient not going to have a crisis during the next four weeks (target=0), the model assigned a prediction value of 0.178. Patient not going to have a crisis during the next four weeks (target=0), the model assigned a prediction value of 0.129. Patient going to have a crisis during the next four weeks (target=1) the model assigned a prediction value of 0.792. Patient going to have a crisis during the next four weeks (target=1) the model assigned a prediction value of 0.725.
Results
The study cohort included 5,816,586 records that were obtained from 17,122 individual patients aged between 16 and 102 years in the period between September 2012 and November 2018. Patients in the group were diagnosed with numerous disorders, including mood, organic, neurotic, psychotic, and personality disorders. The data set also included 60,388 crisis episodes, with an average of 24 crisis events per episode.
The study results showed that the general model performed significantly better in the case of organic disorders with an area under the receiver operating characteristic (AUROC) of 0.89 in comparison to the overall performance of 0.797. When other diagnostic groups were taken into account, the general and overall performance was between 0.770 and 0.814, respectively. The team observed that the lowest performance was for mood disorders, schizophrenia, and schizotypal and delusional disorders. In the comparison of separate models with the general one, the general model performed better than the baseline models while none of the disorder-specific models displayed better performance than the general model.
The team also assessed the performance of the general model in terms of patient age groups. For patients aged less than 18 years, the general model performance reduced to 0.743, while for those aged between 65 and 74 years, the performance rose to 0.840. Notably, similar performance was observed for the patients belonging to other age groups with the AUROC ranging between 0.782 and 0.796.
The study shows that machine learning can provide a foundation for facilitating better resource allocation in mental healthcare. Researchers believe the present study can motivate mental health providers to shift from reactive care to preventative care.
California official comes out of retirement to lead troubled mental health commission