Significant flaws found in recently released forensic software

Reviewed by James Ives, M.Psych. (Editor)Jan 16 2019

New research from North Carolina State University and the University of South Florida finds significant flaws in recently released forensic software designed to assess the age of individuals based on their skeletal remains. The researchers report that, on average, the software's age estimates are off by more than 14 years.

"Estimating someone's age at death, based on skeletal remains, helps to build a biological profile of the deceased," says Ann Ross, a professor of biological sciences at NC State and co-author of a paper on the work. "That's important information for identifying unidentified remains, and can also be important in law enforcement contexts."

At issue is a publicly available computer program called DXAGE, which was released in 2018. The program estimates age-at-death based on bone mineral density.

Ross and collaborator Jonathan Bethard, an assistant professor of anthropology at USF, noticed that DXAGE's estimates for adult female remains were based on a fairly small sample size - the remains of only 100 women.

To test DXAGE's accuracy, the researchers used bone mineral density data from 470 women who took part in the National Health and Nutrition Examination Survey conducted by the CDC between 2007 and 2008. Specifically, the researchers plugged the bone mineral density data for each of those women into DXAGE and compared the resulting estimated age with each woman's actual age.

The researchers found that, on average, the DXAGE estimates were off by 14.25 years - though there was significant disparity between age groups. For example, DXAGE could estimate the age-at-death of women who died in their 30s to within an average of 0.65 years. But for women who died in their 70s, DXAGE was - on average - wrong by 24.4 years. In other words, a woman who died in her early 70s could be estimated by DXAGE as having died in her 40s.

"We think this inaccuracy is primarily due to the small sample size used in developing DXAGE," Bethard says. "It may also be due, in part, to DXAGE relying on remains from cemeteries, where they may have been buried for decades. Burial means that some of the minerals in the bones may have leached into the environment."

The researchers stress that basing age on bone mineral density is valuable, because it is a quantitative way of assessing age at death.

"But it's important to have a robust sample size for all age groups, and to use samples that have not been affected by environmental factors after death," Ross says.

Another problem with DXAGE, Ross notes, is that the software makes use of a "black box" neural networking program.

"In other words, it's not clear how the software works," Ross says. "That poses a problem if forensic experts are asked to testify on how they arrived at their age-of-death estimates in a court of law.

"We already have a technique for estimating age-of-death based on bone mineral density, which relies on linear regression; I helped develop it," Ross says. "That technique proved to be more accurate than DXAGE in estimating age for the 470 women we evaluated in this study. And it allows forensic experts to explain their estimates when called on to testify."

The researchers note in their paper that these results highlight the importance of performing validation studies on web?based applications such as DXAGE.

"A lot of new software tools are becoming available to the forensic community," Ross says. "We need to ensure the validity of these tools before putting them into practice."