In a recent study published in the journal JAMA Network Open, researchers developed and tested deep ensemble models (DEMs) to automatically evaluate patients' autism spectrum disorder (ASD) status from high-resolution retinal images. They used 1,890 photographs from 958 participants (479 each having typical development [controls]) and ASD [cases]) and found that their models could diagnose ASD with an area under the receiver operating curve (AUROC) of 1.00. Furthermore, their models depicted high specificity and sensitivity of 1.00, respectively. Surprisingly, their models were able to estimate ASD severity with an AUROC of 0.74 while still maintaining a sensitivity of 0.58 and specificity of 0.74, respectively. These findings make a case for the computer-aided diagnosis of ASD using retinal photographs, historically an area presenting limited expert resources for diagnosis (insufficient patient:expert ratios).
Study: Development of Deep Ensembles to Screen for Autism and Symptom Severity Using Retinal Photographs. Image Credit: yogenystocker / Shutterstock
ASD and the need for AI
Autism spectrum disorder (ASD) is a behavioral developmental disorder estimated to affect 1 in every 100 children globally. It is thought to arise due to a combination of genetic, social, and environmental factors. The condition is characterized by two main symptoms – social communication impairments and a tendency to partake in repetitive behaviors without the typical ability for behavioral change. The prevalence of ASD varies widely, mainly due to the condition differing in severity from mild to extremely severe. A large number of individuals live out their lives without ever being formally diagnosed with ASD.
The United States (US) Centers for Disease Control and Prevention (CDC) estimated the prevalence of ASD within the country at 1 in every 36 individuals, with the number rising annually. A major proponent of this growth has been attributed to the growing public and clinical awareness of the condition and expanded patient screening. Novel approaches to screening, including the caregiver-reported Modified Checklist for Autism in Toddlers, have presented high diagnostic accuracy (sensitivity and specificity) while still overcoming the most significant barrier to universal childhood screening for ASD – a dearth of trained professionals.
Previous research has hypothesized that patients' retinal images may form a proxy for ASD due to the retina theoretically representing an indirect representation of brain abnormalities. Studies testing the associations between retinal images and ASD have shown that machine learning (ML) artificial intelligence (AI) models are capable of discerning between patients with ASD (cases) and typical development (TD; controls) based on high-resolution retinal scans alone. Unfortunately, these studies were of the pilot study design and included small sample sizes, thereby making their generalization and widespread adoption challenging.
Furthermore, previous studies utilizing AI for ASD diagnoses have focused on binary ASD positive or negative outcomes, with no reports on whether AI models can discern ASD severity in addition to just its presence. Using ML models for ASD screening would benefit both clinicians and patients due to the reduced workload on the former and broader and earlier screening of the latter. It would be instrumental in remote or rural areas that lack sufficient expert diagnostic support due to the capability of digital retinal images being screened remotely.
About the study
The present study aimed to develop, train, and test deep ensemble models (DEMs) that utilize high-resolution retinal scans for ASD symptoms and severity diagnoses. Ensemble models are a special type of AI model wherein multiple diverse models are created to predict an outcome, either by using many different modeling algorithms or using additional, nonoverlapping training data sets. The final output then aggregates the prediction of each base model and presents a consolidated prediction for the unseen data.
"Deep ensembles offer advantages over single models because they exhibit superior performance and a greater capacity to quantify predictive uncertainty. We used convolutional neural networks with the ResNeXt-50 (32×4d) network as the backbone to construct classification ensemble models to screen for ASD (i.e., ASD vs. TD) and ASD symptom severity (i.e., severe vs. mild to moderate symptoms) using retinal photographs."
This study was carried out in accordance with the Standards for Reporting of Diagnostic Accuracy Studies (STARD) reporting guidelines. It included children and young people under 19 years recruited between April and October 2022 from Yonsei University College of Medicine, Seoul, South Korea. Inclusion criteria comprised all patients whose parents/guardians/caregivers provided written consent who had been clinically diagnosed with ASD using the Autism Diagnostic Observation Schedule–Second Edition (ADOS-2; score cutoff 8) and the Social Responsiveness Scale–Second Edition (SRS-2; score cutoff 76).
Patients presenting severe psychiatric disorders, neurological illnesses, or eye diseases were excluded from the study. Data collection comprised the collation evaluation of ADOS-2 and SRS-2 scores (screening), high-resolution retinal scans of screen patients, and the additional evaluation of participants' full-scale intelligence quotient (FSIQ).
The control (TD) group was chosen retrospectively using deidentified data collected in December 2007 and February 2023. TD participants were age- and sex-matched to ASD participants for a direct 1:1 comparison between cohorts.
Model training involved image preprocessing – cropping retinal images to a uniform 224 × 224 pixels and removing noninformative artifacts. Model diagnostic training was carried out using two parallel approaches: 1. TD versus ASD evaluations solely based on the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5) criteria, and 2. TD versus ASD evaluations that combined both DSM-5 and ADOS-2 scores. Model severity training was conducted using calibrated ADOS-2 and SRS-2 scores.
Researchers developed ten individual AI models trained using 85% of randomly selected case- and control cohort data. A 10-fold cross-validation algorithm was then employed to obtain generalized model performance evaluations. Sequential age-based data and output stratification were employed to account for differences in children's age at screening.
"Moreover, we examined different split ratios (80:20 and 90:10) to assess the robustness and consistency of the predictive performances across diverse splitting proportions. We (also) calculated entropy to estimate the predictive uncertainty for each image in the test and out-of-distribution sets."
Model (individual and consolidated) performance was carried out using area under the receiver operating curve (AUROC) metrics, and comparisons between case and control cohorts were achieved using Chi-squared (χ2) tests and t tests.
Quantitative Validation of the Heat Map With the Progressive Erasing Technique for ADOS-2–Based Symptom Severity Screening. A, Area under the receiver operating characteristic curve (AUROC) with shaded 95% CI obtained from masked images. B, Progressive erasing for severe autism spectrum disorder (ASD) and mild to moderate ASD. ADOS-2 indicates Autism Diagnostic Observation Schedule–Second Edition.
Study findings
The ASD (case) and TD (control) groups comprised 945 individuals each, for a total of 1,890 retinal photographs. The diagnostic test set (15% of the data) resulted in an average model performance (AUROC) of 1.00 across accuracy, sensitivity, and specificity metrics, validating these model's abilities to diagnose ASD based on retinal images alone sufficiently.
"These models had successful calibration performance, as indicated by a mean NLL of 0 and a mean Brier score of 0. Classification and calibration performances were retained even when limited to ASD diagnosis using DSM-5 criteria and ADOS-2 scores."
Severity analyses revealed that models achieved a consolidated accuracy of 0.74 (AUROC). While potentially insufficient in a clinical evaluator setting, this finding highlights the possibility of ASD severity screening, both using remotely acquired retinal images and using AI models (previously untested). Age-stratified screening presented ensemble model accuracy as early as four years of patients' age, earlier than the clinically accepted average of 60.48 months.
"Our findings suggest that the optic disc area is crucial for differentiating between individuals with ASD and TD. Although future studies are required to establish generalizability, our study represents a notable step toward developing objective screening tools for ASD, which may help address urgent issues such as the inaccessibility of specialized child psychiatry assessments due to limited resources."
Anne Bang joins research to understand the genetic underpinnings of psychiatric conditions