07 December 2020

It’s easy to get the impression that communication consists mainly of speech. But much of it takes place through gesture, posture, facial expression and sound. At Linköping University, researchers study how our communication uses so much more than just words – in interaction with other humans as well as with robots.

Thor Balkhed

“It’s scary how well synchronised we humans are when we’re together. How skilled we are at interpreting each other and how sensitive we are to deviations. It never ceases to amaze me”, says Mathias Broth, professor of language and culture.

Mathias Broth conducts research together with Leelo Keevallik, also professor of language and culture, and doctoral student Hannah Pelikan, into how we humans communicate with each other. They look at how we carry out acts with each other – or with a robot – and what speechless communication is.Photo with Mathias Broth and Leelo Keevallik, researchers in interaction.Mathias Broth and Leelo Keevallik.

When the researchers study wordless communication, the focus is on the details; therefore the respondents are always recorded. By studying video recordings, it becomes clear how people use their voice, face, and body to interact.

“I studied a French TV team, and how they communicated on air. When the TV producer realised the detail with which we studied him, and the apparently insignificant details we noticed, he was flattered”, says Mathias Broth.

But why do we need to know, in such detail, what we do when we communicate?

“With every little action we do, we maintain roles and norms. With every sigh and every glance. By studying this in detail, we register what is systematic in what people do. On a larger scale, these studies teach us about how people uphold society”, says Leelo Keevallik.

Er, ah, um – Sounds as a way of interacting

Leelo Keevallik’s research focusses on sounds that seemingly mean nothing – sounds that are used to express – for instance – effort, pain, and whether the food tastes good.

“For a long time, these sounds were considered insignificant, because they’re not words. But research has shown that these sounds have a relatively fixed form, and we use them as tools in our social interaction. When someone sighs at a particular moment, we know what it can mean”, she says.

Teaching robots to interact with people 

But interaction doesn’t only take place between people, it can also involve humans and technology. Doctoral student Hannah Pelikan is studying how robots use sound to interact with people. She is assisted by the toy robot Cozmo. Cozmo hardly speaks at all, but can move, recognise faces and make sounds. Photo credit Thor Balkhed

“It’s very simple, but can still interact with people. How does it do that, how do we understand what the robot is doing and wants? And how do we use our understanding to interact with it?”

Those are the questions that Hannah Pelikan hopes to answer. She has studied how people react to Cozmo’s way of showing emotions by way of sounds.

“Wordless interaction might seem vague, but sounds are flexible in a way that words aren’t. With sounds, Cozmo can communicate faster and in more contexts. When Cozmo displays happiness, we humans generally interpret that as a positive response, and thus this works in many different contexts. For instance, it’s appropriate both as response to when Cozmo meets a new person and when it gets a toy.”

Words on the other hand are not as flexible in different contexts. It’s difficult to preprogram the robot so that it can use words to respond to everything a human could possibly say to it, according to Hannah Pelikan.

Among other things, her studies have shown that interaction with Cozmo proceeds well when the robot displays happiness, but that it stops when Cozmo displays sad emotions. In this case, the study participants retry what they have said or done. Showing sad emotions can be an efficient way for the robot to signal that something in the interaction has gone wrong. 

Robots that function together with people 

In order for robots to function in society, they must follow the unwritten rules that form the basis of human interaction. Photo credit Thor Balkhed

“If we want to build machines that are easy for us to interact with and that function in the long term, we must start out from our knowledge about human interaction, and study the interaction between robots and humans in practice.”

Translated by George Farrants.

More research on language and culture

Contact

Latest news from LiU

Person (Jie Zhou) point at a computer screen.

A new world of 2D material is opening up

Materials that are incredibly thin exhibit unique properties that make them appealing for energy storage, catalysis and water purification. LiU-researchers have developed a method that enables the synthesis of hundreds of new 2D materials.

Liqourice in a bowl with liquorice roots next to it.

Small amounts of liquorice raise blood pressure

It is known that large amounts of liquorice cause high blood pressure. A study from LiU now shows that even small amounts of liquorice raise blood pressure. The individuals who react most strongly also show signs of strain on the heart.

He guides his students to top jobs

LiU students are doing well in the Swedish CFA Research Challenge. Year after year, victory goes to Linköping University. Winning the competition is a shortcut to top jobs with major banks and accounting firms. What is the secret to this success?