Can AI understand human emotions?

The key reason we became interested in the idea of detecting emotional states through AI is that this challenge formed the foundation of one of our products (more here).

To recap:

In this article, we delve deeper into the specific mechanisms. We’ll explore the AI techniques used to analyze and interpret a user’s psychological state, starting with the numeric data from surveys, devices like EEGs and wearables, and moving towards the written word and more comprehensive data sources like voice and video.

Our goal is to shed light on the technology powering the next generation of mental health tools, examining both its capabilities and its inherent challenges.

When it comes to understanding psychological states through data, two major types of numerical datasets are typically used: physiological measurements and self-reported surveys.

And it works like a charm (not for the subject, though). The real challenge here is that this solution isn’t scalable, ethical, or feasible for most clinics and psychologists. Instead, we want to create an automated system that is accessible to a large number of people. In other words, to automate emotion detection and transform a wearable device into our personal lie detector, though with a slightly different purpose: identifying mental well-being.

To understand which emotions we can detect, let’s first explore what sensors and data are already available from wearable devices today.

Unfortunately, they come with their own set of challenges:

Sometimes, the numerical data is either insufficient or the accuracy leaves much to be desired. The second most natural place to begin AI analysis in a digital mental health setting is through text.

In a nutshell, earlier we could only analyze sentiment (more on that here), classifying content as positive, negative, or neutral. Or, using another technique — topic modeling — we could uncover recurring themes across large volumes of text, shedding light on what truly concerns people.

However, the real breakthrough came with the emergence of transformer-based architectures (like ChatGPT). These models learned to analyze context, detect subtle emotional shifts, capture long-range dependencies in text, and deliver remarkably accurate insights.

So, let’s take a look at which emotions can be detected — and with which models:

When the objective shifts towards identifying more nuanced psychological states, such as chronic stress, anxiety patterns, or the early indicators of conditions like depression, relying solely on textual data reveals significant limitations:

Now, for the most interesting part.

To gain a deeper, more accurate understanding, AI can be employed to analyze richer data streams: audio and video. This allows the technology to tap into the non-verbal cues that are often more revealing than words alone.

Steps to extract mental state from audio using AI:

Finally, let’s look at the example:

However, if the goal is to achieve higher accuracy or to tackle more complex tasks such as detecting stress or depression, foundational models mentioned above would typically require further fine-tuning on specialized datasets, or a custom model might need to be trained from scratch.

Video: Facial Expression Recognition for Videos (Python + TensorFlow + Keras + OpenCV) (YouTube)

Video: eMotion - Facial Expression Recognition (YouTube)

Okay. We’ve already written many words, so let’s speed things up.

From a video as a source, AI can analyze:

How it actually works:

Although models specialized in analyzing individual modalities, such as audio or video, show impressive accuracy in detecting emotions and psychological states, the true power lies in combining these modalities.

Okay, so what should be done if none of the individual approaches yield the desired result: low accuracy, incorrect analysis, and so on?

Fortunately, we can combine approaches using multimodal methods, thereby increasing the accuracy of the analysis.

The rationale behind this is that different modalities capture complementary aspects of an individual’s psychological condition and compensate for the weaknesses of individual modalities, increasing the system’s robustness. For instance, facial expressions might reveal immediate emotions, while the content of speech could provide insights into underlying thoughts, and physiological signals might reflect the body’s response to stress.

You can learn more about these methods and the ideas behind them by following their links: Cascaded RNN-LSTM, DepMamba, MDDformer, DepITCM, Multimodal Early Fusion, MFN-EPD.

While the potential is huge, there are significant hurdles:

Overcoming these challenges, especially ensuring privacy and getting enough diverse, real-world data, is essential before this technology can be widely and responsibly deployed.

In this article, we analyzed various AI approaches to studying human emotions and mental states.

Based on our experience, any approach — whether it’s text, video, audio, or physiological data — is effective depending on the specific task.

The general recommendation is to start with a clear understanding of precisely what you want to analyze and for what purpose you want to use AI. Try to find ready-made models for your task. If there are no exact matches, choose the closest one and fine-tune it using your own data. If that doesn’t work either, consider creating a consensus from several AI models or using a multimodal approach.

It is only important to remember data, safety, and ethics, as we mentioned above.

If you need help with choosing models, adapting them, or even just testing, let us know.

And remember, AI, positioned correctly as a means to augment human expertise, not replace it, offers transformative potential. The key lies in pursuing this future responsibly and ensuring that innovation in mental health technology always serves humanity’s best interests, ethically and effectively.