Communication — what information to convey, when and how to do so — is a key skill for robots to have if they are to effectively assist humans in their tasks.

We are currently living in times of great transformation, an era that has been called the Fourth Industrial Revolution. Similar to how steam power, electricity and mass production, and digitisation drastically altered the lives of those who came before us, emerging technologies such as artificial intelligence (AI) are changing the way we live, work, and play.

Robots and virtual assistants, in particular, have grown increasingly commonplace. “As robots start to be deployed out in the real world, they will help us in many different ways,” says computer scientist. Harold Soh, an assistant professor at the National University of Singapore “One of these ways is via communication — proactively giving us information to help us accomplish our goals.” Such scenarios include guiding drivers through heavy rain or fog, and navigating firefighters and rescue workers towards victims.

The tricky part about these scenarios — and a key feature of human-robot relationships of the future, similar to any solid relationship — is effective communication, says Soh, whose research focuses on developing trustworthy robots capable of collaborating well with humans. But for robots, communication is a skill that doesn’t come easy. The information a machine receives is often in multitude, and it has to tease apart the information that would be relevant to the person it is trying to help. On top of that, it has to figure out the best way to communicate that information.

Take the latter scenario, for instance: a robot assistant analysing a building on fire has to grapple with large swaths of information — floor plans denoting how many storeys there are, alongside where the windows, stairwells, and lifts might be found; heat maps pinpointing the locations of those trapped; archival information about the history of the place, previous and current occupants; details about the firefighting crew and their respective training, experience, and skillset; relative distances of nearby hospitals, and so on. From these, the robot has to decide which information might be most relevant to a firefighter about to enter the burning building. Furthermore, it has to figure out the best way and time to relay that information — would a voice note through earbuds be best, or perhaps text accompanied by visuals popping up on an on-screen dashboard in the fire engine might be more effective?

A hybrid
Given the complexity involved, how can we then develop helpful machine communicators? As Soh sat musing the problem, he recalled a series of famous psychological experiments conducted over a decade ago at the Max Planck Institute for Evolutionary Anthropology in Germany.

In one experiment, toddlers watch as an adult holding a stack of books repeatedly bumps into a cupboard. After some time, the toddlers (who were tested separately) teeter across the room and pull open the cupboard door, allowing the grown-up to place the books on a shelf.

“The toddlers are able to infer what help is needed, something that we seem to develop intuitively as humans,” says Soh.

But figuring out how to assist others is “a difficult challenge,” he adds. “It requires perception and reasoning about the task and the other agent.” People, including young children, are capable of doing so because we have a Theory of Mind. Essentially, humans have mental models of other people that capture their beliefs, intentions, emotions, and knowledge. Having a Theory of Mind allows us to understand how others can have different beliefs and capabilities, which is helpful for communication.

One natural approach to enable robots to communicate well is to imbue them with a Theory of Mind. The problem is that building models of other agents is difficult. Roboticists and computer scientists generally adopt one of two techniques. The first, more traditional `theory-driven’ approach is fashioned upon existing theories within cognitive science and psychology. As Soh explains, “They tend to be ‘handcrafted’ — you have to put in a lot of effort into building a model and being explicit about your assumptions.” Handcrafting a human model tends to work well for narrow well-structured tasks, but the methodology doesn’t scale easily to complex real-environments with high-dimensional observations.

Sitting on the other end of the spectrum is the ‘data-driven’ approach, where scientists create human models by feeding the model data that it is “free to learn from,” says Soh. The data-driven approach is more flexible compared with the theory-driven method, but suffers from one major limiting factor: learning a good model requires lots of data and good data is hard to come by.

The solution, Soh felt, was to combine the two approaches. “The idea is that the model doesn’t have to learn everything from scratch, and that maybe it can leverage prior knowledge it already has access to.”

That became the impetus for the AI Singapore (AISG) grant, says Soh, who was awarded an AI research fund in early 2019.

Mirroring the inner self
The question was then: how best to combine the two approaches? For this, Soh turned to Social Projection, a psychological concept that refers to how people generally expect others to think and behave like themselves. In other words, we humans use ourselves as a reference point when thinking about others.

What if we could incorporate such thinking into the robot, wondered Soh. If so, the robot wouldn’t have to learn a human model entirely from data, but to simply learn how it differs from the human. This allows the robot to use itself as a “baseline model”, which it already knows and doesn’t have to learn about.

“Because it doesn’t involve learning everything from scratch, but just the differences from a suitable reference model, means this approach is much more efficient,” says Soh, whose team has termed their technique Model Implants for Rapid Reflective Other-agent Reasoning/earning, or MIRROR for short.

“To our knowledge, we were the first group to operationalise the theory of social projection in this way and show that learning differences is an effective approach for human modelling,” he adds.

When tested in a simulation involving three tasks — driving, search and rescue, bomb defusal — MIRROR models revealed information more selectively, provided better communication, required fewer data samples, and were able to learn human behaviour quicker as compared to existing techniques.

In a further experiment, 21 human volunteers driving a car using a simulator reported that MIRROR offered useful verbal and visual assistance that helped them successfully navigate a highway under foggy conditions, resulting in fewer collisions.

Reflecting on his team’s work over the past three years since he was awarded the AISG grant, Soh says: “I’m really happy that we took an intuitive concept and formalised it into a model that we could then implement in a robot.” The funding, he adds, has enabled his team to kickstart and develop “a strong body of work so we could go for other grants,” including two worth more than million dollars.

Today, Soh continues to work on his passion for building trustworthy, collaborative robots for the future. “Communication is an essential skill for intelligent agents. It’s something that we’d like to push further — how can we build robots and AI that can proactively give information in order to help people accomplish their goals?”

He adds: “In the future, we would like to deploy these AI systems to help people in complex tasks.”

Check out other AI Research awarded projects here: https://aisingapore.org/research/grant-call-awardees/