Capturing vital signs of patients with a cellphone camera

Researchers receive funding from the Bill and Melinda Gates Foundation to capture vital signs via video

As biometric systems -- technologies which measure biological information to identify a person -- continue to advance, their potential impact on health care capabilities surge. Via tools such as fingerprint recognition, face recognition, iris and retina recognition, and vein recognition, health care workers are provided with increasingly sophisticated ways to monitor patients.

Researchers from Penn State and Johns Hopkin University are working to develop additional capabilities by capturing vital signs of patients in resource-constrained environments with a device most Americans use every day -- a cellphone camera.

Funded by an initial $100,000 pilot grant from the Bill and Melinda Gates Foundation, the project, "Non-Contact Estimation of Biomarkers in Resource-Constrained Environments," uses cellphone camera and computer vision techniques to capture patients' vital signs at distances of up to four feet away -- a critical distance needed for highly contagious diseases like Ebola or tuberculosis.

Led by Conrad Tucker, associate professor of engineering design and industrial engineering at Penn State, the mobile application will register natural head and body movements, distinguish between different skin tones and lighting conditions, and capture vital signs such as an individual's pulse rate.

Collaborators at Johns Hopkins include Dr. William Checkley, associate professor of medicine and director of the Johns Hopkins Center for Global Non Communicable Diseases Research and Training, and his team of researchers. Jeffrey Gray, professor in the Department of Chemical and Biomolecular Engineering at Johns Hopkins, played an instrumental role in connecting the Penn State and Johns Hopkins teams.

"Our mobile-based application seeks to expand beyond a 'wellness app' classification by the FDA, to an FDA-approved tool that can be used by patients and healthcare officials for measuring vitals at a distance and in varying environments and populations all across the globe," Tucker said. "This pilot grant is a first step towards this goal, as it will enable the team to evaluate the technology that we've developed in real-world settings involving hardware, environmental and societal constraints."

To test the proposed application in real-world situations, Tucker and doctoral student Sakthi Prakash will travel to India and Sierra Leone in December 2018 and March 2019. Testing locations were selected due to the team's hands-on experience in these countries, along with the opportunity to engage with a diverse group of individuals to test in varying environmental conditions.

Penn State faculty members Soundar Kumara, Allen E. Pearce and Allen M. Pearce Professor of Industrial Engineering, and M. Jeya Chandra, professor emeritus in industrial engineering, have helped the team forge new collaborations in India. University of Illinois faculty member Paul McNamara, associate professor of agricultural and consumer economics, helped the team achieve similar collaborations through his established stakeholder network in Sierra Leone.

Initial findings will be presented to the Bill and Melinda Gates Foundation in early 2019. A return trip to India and Sierra Leone will take place in summer 2019 to collect additional data. Pilot results are expected to be released after the return trip.

The team has tested the application's functionality on the Xiaomi Redmi 6 and the Samsung Galaxy J7 Prime, cellphones that are typically used in these countries, due to their relative affordability and availability in these areas.

"It's one thing to demonstrate the feasibility of an algorithm in a controlled environment in a research lab. It's another thing for technology to function as intended in the field, where researchers have less control of the human and environmental factors," Tucker said. "This will be an excellent opportunity to integrate user-centered design, concepts that form the foundation for many of the engineering design courses that are taught in SEDTAPP here at Penn State."

On Nov. 26, Tucker traveled to the Bill and Melinda Gates Foundation headquarters in Seattle to meet with a program manager from the foundation and members of the team to demo the mobile app that the team has developed and to discuss the team's plan of action for their upcoming international travel.