“Remember that paper clip that used to appear in Microsoft Word? There you have the problem. For a support system to help you, it has to understand what you’re doing. Otherwise it can be annoying”, says Magnus Bång, associate professor at LiU’s Department of Computer and Information Science.
Magnus Bång and Jonas Lundberg, professor at LiU’s Department of Science and Technology, manage the F-Auto (Flexible Automation) research project, where they, together with LFV (the Air Navigation Services of Sweden), investigate how AI can support air traffic controllers.
The two researchers have built a system that monitors air traffic and that, with the help of AI, can detect conflicts, for instance discover aircraft that are too close to each other, or whose flight paths intersect.
“Conflicts like that should be able to be solved automatically, for instance by the AI system adjusting the aircraft’s route or speed”, explains Magnus Bång.
‘Human-in-the-Loop-AI’
Traditionally, automation has often meant automating as much as the technology can handle. The rest is left to a human operator to sort out. But this approach has proven problematic. If the operator has no work tasks, he or she soon loses the overview of the situation. If the AI system suddenly reports that it cannot handle the situation and is passing responsibility to the operator, the operator has no idea what is going on, and it can be difficult to rapidly make good decisions.
The opposite situation can also arise. When the air traffic controller is busy solving an urgent problem, the AI system must not interrupt.
Instead of automating as much as possible, Magnus Bång and Jonas Lundberg argue that systems must be created where the operator is constantly kept informed. They use the term ‘Human-in-the-Loop-AI’.
“Our research question is ‘How do we make sure that the operator and the AI collaborate?’. We maintain that the AI needs to know what the air traffic controller is doing, and vice versa”, says Magnus Bång.
One step further
To find out what an air traffic controller does, they use eye tracking, where the air traffic controller wears special glasses, making it possible to follow his or her eye movements.
“The information is streamed in real time, so we understand what the controller is looking at. We can know which specific conflict they are working on thanks to their eye movements. That information is sent to the AI. If the controller fixes their eyes on a certain flight object, the AI knows that he or she has seen it.”
AI is a popular research field, however there is not much research like Bång’s and Lundberg’s.
“Of course there’s a lot of cognition research into the problems of automation, but we go one step further, bringing with us the cognitive-science approach when we build these support systems. We don't want to automate processes simply because it’s possible, we build the AI itself, based on a cognitive-science perspective.”
The aim of their system is to support the air traffic controller, leading to increased safety, among other things.
“One thing we’re trying to achieve is being able to detect if the operator has missed anything. When working together, the human and the technology should be able to increase efficiency, monitor a larger air space, and be able to land more aircraft.”
"Our AI system has an industry focus"
The research in the field concerns air traffic control, but the results will be applicable to many other fields.
“The automation of any work environment that requires a human leads to these problems, whether it’s in maritime transport, nuclear power plants or healthcare.”
The research conducted by Magnus Bång and Jonas Lundberg is in progress. Using the system in real-world situations is still in the future.
“Our AI system has an industry focus, but we’re probably ten years away from being able to use it in real-world situations. There are considerable validation processes to deal with. The requirements on safety performance in air transport is extremely high and of course we collaborate with LFV. At the same time both the EU and the US are very interested in automating parts of the air traffic system.”
Meet Magnus Nylin, industry-based PhD student at LFV
Magnus Nylin’s research project is part of the F-Auto project. An industry-based PhD student is a graduate student whose research is part of employment at a company or, in this case, the Air Navigation Services of Sweden, LFV.
What is your background?
I have a master’s degree in cognitive science from Linköping University, which I took because I wanted to continue to a doctor’s degree. I have also trained as an air traffic controller, and I’ve worked at the Swedish air navigation service provider LFV since 1999.Magnus Nylin
Tell us about your research
My research concerns flexible automation, abbreviated as “F-Auto”. The project concerns not only air traffic management, which is what I mainly work with, but also train control, and vessel traffic service from the maritime domain.
We are trying to obtain information about the operator in the system. Currently, this type of automated systems look only outwards at external sources of information, and the system knows nothing about the operator. The crux of the F-Auto project is to bring the system to understand what the operator is doing. The particular focus of my PhD project is the interaction itself. How should the system adapt to the human operator? When and how should a highly automated system request help with a task?
If the system is aware of the air traffic controller’s situation, it can adapt the information it provides to the situation.
Why have flexible automation systems not been constructed earlier?
There are several reasons, one of which is the development of technology. A technique we use to understand the operator’s situation is eye-tracking, and the equipment for this was large and cumbersome. It has not been possible to create a system that would be practical in everyday use. An important part of the F-Auto concept is, of course, that the system should make operator’s work easier and increase the overall capability.
And then developments within AI have also created new opportunities to analyse the data that are collected.
When do you plan to submit your thesis?
I’ve been working on it for three years and expect to need two more, so I’m starting to feel that the end of this project is approaching.
