Multi-modal AI systems for enhanced emotion detection and understanding

Using a fusion of traditional and novel technological methods, researchers are hoping to better quantify emotions to transform the face of the emotion quantification field

Human emotions are complex and are not always easily able to be boiled down to a recognizable pattern. Determining one's emotional state can be difficult human-to-human, and the many nuances of existence as an emotional entity seem impossible to train a non-human entity to understand, identify and learn from. However, a considerable amount of work and research has been put into training artificial intelligence (AI) to observe, quantify and recognize various states of emotion in humans. The fusion of tried and true psychological methods combined with the intelligence and trainability of AI can make emotion recognition technology invaluable in fields such as healthcare and education.

Results were published in CAAI Artificial Intelligence Research on August 21, 2024.

Where conventional techniques are limited, AI can improve. Through the use of a multitude of developments, such as gesture recognition technology, facial emotion recognition (FER) and multi-modal emotional recognition, emotional recognition technology stands a chance to be transformational for many individuals and fields of study as a whole. 

"This technology has the potential to transform fields such as healthcare, education, and customer service, facilitating personalized experiences and enhanced comprehension of human emotions," said Feng Liu, author and researcher of the review.

An artificial intelligence that understands human emotion and can appropriately interact given the emotional input of the human can be revolutionary for human-computer interactions and can be a key in assessing the mental health status of an individual. This isn't done through just one form of input, but instead can also take physiology into account. For example, some techniques can take input from the electrical activity of the brain through an EEG scan and combine that with eye movement technology to monitor people's expressions. Other measurements of emotional arousal such as heart-rate variability and electrical skin response are also tools that are used to convert the intangible "emotion" into patterns and recognizable, readable data for AI to learn from and improve.

Multi-modal emotion recognition similarly combines different perceptual channels, such as sight, hearing and touch to gain a more complete picture of what emotions can entail. The combination of different fields and techniques is necessary to create an accurate and well-rounded representation of the complexities of human emotion.

"It is believed that interdisciplinary collaboration between AI, psychology, psychiatry and other fields will be key in achieving this goal and unlocking the full potential of emotion quantification for the benefit of society," said Liu.

Having AI be able to correctly recognize human emotions can be especially useful in a world where mental health is quickly becoming a top priority. Emotion quantification AI can help in monitoring an individual's mental health and create personalized experiences for that individual, all without having to entangle another person in the process.

Successful use of emotion recognition and quantification AI requires a few major components. One concern that would need to be addressed is safety and transparency, especially as it relates to more sensitive topics such as medical and psychological counseling. Data handling practices and privacy measures taken by the entities using this type of AI will have to be stringent. Additionally, ensuring the AI can adapt to the nuances of cultures is of utmost importance, as this will maintain the integrity and reliability of the AI for future referencing and learning.

Feng Liu of the School of Computer Science and Technology at East China Normal University is the author and researcher of this study.

The Beijing Key Laboratory of Behavior and Mental Health supported this research.