Researchers use music and empathetic speech in robots to combat loneliness

When soft music is playing in the background, a brief exchange with a robot can evoke unexpected feelings of intimacy.

Researchers were surprised to discover how much of a role the background music played in the interaction.

A team from the Chinese University of Hong Kong, led by Professor Johan Hoorn, investigated how emotional cues affect human-machine interactions over multiple contacts.

The focus of this investigation was loneliness, which is one of today's fastest-growing concerns related to mental health, particularly as more people are becoming socially isolated. If we want artificial robots that have the ability to recognize and respond appropriately to human emotions, we need to have confidence that the machines truly have the ability to understand those emotions.

At the beginning of their investigations, music was an important tool for scientists.

Empathy, in terms of its design.

Empathy, as experienced by humans, is the process by which we develop emotional connections with other humans. While previously it was possible to perceive emotional reactions from computer-generated virtual agents and robots, it was not clear how long these interactions would be perceived by the participants as "empathic."

The most recent study conducted by The Hong Kong Polytechnic University focused on the interaction between participants and an onscreen robot named Zora, who was developed based on a NAO robotic humanoid. Each participant interacted with Zora on three separate occasions during the course of two weeks via online platforms. On each occasion, one of three conditions applied: empathetic response only; background music only; or both empathic response and background music.

The findings showed a statistically significant difference in perceptions of empathy by condition. When people heard the robot speak in a supportive voice while it played music, they consistently rated it as more empathetic compared to when it only used one channel to communicate. Participants did not typically perceive the use of music alone as an expression of the same kind of empathy. Without verbal information, participants typically interpreted the use of music as pleasant, without associating it with any kind of caring attitude.

"Our data indicate that music continued to enhance the robot's similarity to humans through all of the sessions," Hoorn said. "One way to interpret this is that the use of music allowed the conversation to feel more like a personal interaction with a person, similar to what a counsellor would do by using music to support their clients. This allowed users to perceive the robot as more human-like and socially connected."

The results of this study are consistent with the psychological principles that indicate that the emotional meaning of a cue emerges through multiple ways of delivering it. Music enhances the emotional tone, or strength, given to the words that accompanied the music.

The effect of novelty fading.

The increase in perceived empathy created by the use of music was not permanent.

After the first session of using music, users indicated high levels of perceived empathy and connections. However, by the time of later sessions, users had diminished feelings of both. Repeated exposure to the same type of cues reduced their impact.

This sequence of events followed a well-documented pattern in psychology. A new experience generally produces greater emotionality than repeated experiences. It is not until users have had several sessions with the robot that they can shift their focus from the feeling of novelty associated with the robot to other aspects.

Over time, perceived empathy decreased slightly, while users rated the robot as more realistic and relevant in some cases. Users also started to view the robot less as a new technology and more as a social tool. The ability to change from an emotional reaction to a practical assessment of a robot is likely to be an important factor in establishing a long-term relationship with the robot.

The study's most clear result came from examining how empathy led to bonding between participants and the robots. Participants who felt the robot was empathetic toward them tended to perceive the robot to be relevant to their personal needs and therefore ultimately felt more connected to the robot.

This implies that empathy alone does not create a sustained relationship with a robot. Rather, as the participants had more and more interactions with the robot, they felt a greater sense of usefulness toward the robot and developed stronger bonds with it.

Loneliness also contributed significantly to the bond established between participants and the robot at first. Participants who felt more lonely in early sessions tended to display stronger reactions to the robot's empathetic behavior than participants who were not as lonely in those sessions. However, this impact of loneliness decreased over time, indicating that an individual's emotional needs are likely to influence their bonding with a robot during the early sessions, but not during later sessions.

These findings were interpreted using the Interactive Perceiving and Experiencing Fictional Characters (I-PEFiC) model, which explains how people form evaluations of characters or agents based on a variety of criteria, such as perceived capabilities, perceived relevance, and emotional value associated with the agent. It also explains how evaluative responses, such as bond formation, are generated from these evaluations.

Overall, the data indicate that, although empathy initially creates engagement with a robot, a person's cognitive assessment of the ability to utilize the robot for their purposes determines whether or not they will continue to engage with the robot.

Music was used in a variety of ways over the course of the study. In early sessions, music enhanced participants' experiences of empathy for the robot. In later sessions, while the immediate emotional effects of music were decreased, it still enhanced participants' experiences of perceived "realism" of the robot. Some participants, in the past, believed that robots become less human-like when using music. Thus, any future interactions where music was not used will be less emotionally connected to robots.

Habituation can explain this finding, as the sensitivity to an established stimulus decreases over time. For example, while listening to a song for the first time may have a significant emotional impact on an individual, after listening to that same song multiple times, it will become somewhat white noise.

This change shows that robots who rely on static strategies to develop emotional connections with users are likely to have challenges keeping users engaged over an extended period of time. However, using an adaptive approach could help. For instance, adapting to user responses using personalization or changing musical styles could promote greater emotional connections over time.

On-screen robots are important.

On-screen robots represent a fusion of digital avatars and physical robots and are useful because they allow for a realistic connection between users and the robot while avoiding the complexity of using a physical robot.

Remote access to on-screen robots makes them suitable for use in teletherapy, education, mental health care, and other uses for which physically accessing robots may be difficult or impossible. Additionally, the ability to use on-screen robots reduces the amount of money and effort needed to obtain robots.

The timeline of the study was based on the social isolation policies in place during the COVID-19 pandemic, with 144 Cantonese-speaking young adults being recruited for participation. The duration of each interaction was between three and four minutes, and participants answered questions as well as responded to prompts from the robot.

Short interactions did produce measurable emotional reactions, thus demonstrating that even small interactions with a robot can impact how the user perceives the robot's capability to show empathy.

Future robot design should focus on creating adaptable robots.

The results from this study support areas of future research.

Dr. Hoorn is leading another research project with over HK$40 million in funding from Hong Kong’s Research Grants Council to create social robots with embedded large language models that are designed to decrease population stress levels. He is also exploring the study of quantum-inspired approaches to human affect, in which emotions are viewed as probabilistic states rather than fixed categories.

"What inspires me most is the opportunity to create social robots that can acknowledge the complex nature of human emotion and can provide emotional support," said Dr. Hoorn. "We want to provide robots with an emotional intelligence capacity similar to the people for whom they were created to help."

Research findings are available online in the journal ACM Transactions on Human-Robot Interaction.

The original story "Researchers use music and empathetic speech in robots to combat loneliness" is published in The Brighter Side of News.

Like these kind of feel good stories? Get The Brighter Side of News' newsletter.