There’s a cold wind off the sea and the leadership team of the rescue services huddle together in the tents, but overhead two drones are battling against the winds, together with a small boat out on the waves. They are on a mission. They are to search autonomously for people who have fallen overboard.
Jens-Olof Lindh, rescue services leader The rescue services leader, Jens-Olof Lindh, has defined today’s mission: one drone is to maintain surveillance and transmit aerial images to the rescue services centre, while the second drone is to follow the boat, a Piraya model, to help maintain a picture of the situation closer to the water surface. The Piraya is equipped with many types of instrument: cameras, an echo sounding system, communications equipment, gyros, etc. This small boat is a testbed for several sensors. The boat and the drones have exchanged handshakes, and they have let Jens-Olof Lindh know that the mission is within their abilities. They have therefore been given the go-ahead to carry out the mission.
Over the sea, the first drone films the search area from on high. The Piraya is waiting for the second drone to reach the initial point, before starting to search across the water surface.
On land, the command centre is following on computer screens how the boat and the drones travel, and what they see through their cameras. Today is just a test and a demonstration, and both the drones and the boat have human safety operators. But even though the wind is at the limit of what the system can cope with, none of the safety personnel need to take action. The drones carry out the mission, return to base and land without fuss, while the Piraya comes into dock and is moored.
Piotr Rudol, research engineer, artificial intelligence, LiU, with Pontus Hero Ek and Jonas Ekskog, developers from Combitech. Photo credit Thor BalkhedLiU, the Royal Institute of Technology (KTH) and Lund University, with employees from companies such as Axis Communications, various companies in the Saab Group, and Ericsson. The watchers ask questions of Jens-Olof Lindh, from Saab Kockums, who answers while demonstrating the system on the screens.
All those who are present are interested in different parts of the system, which are now being tested together in real-life conditions for the first time. We see, among other things, cameras from Axis Communications, cloud-based computer systems from Ericsson, management systems from Saab Combitech, boats and systems from Saab Kockums, mobile communication solutions from Telia, and the application of university research within artificial intelligence, positioning, computer vision and autonomy.
The doctoral students working here will spend the coming year coupling their university-based work into the system, in order to refine and improve it. Today, however, they are here simply to see how the technology works, and test parts of their research in a real scenario.
Photo credit Thor Balkhedout the mission, and receive the green light from Jens-Olof Lindh. One drone flies rapidly away, and we watch on the screen as it systematically scans a predetermined area. When it finds those in need of help, it contacts the second drone completely autonomously, and sends the coordinates. This second drone carries a mobile phone in a special package, and this is released as soon as the drone is in the right position, to gently float to the ground on a parachute.
“It could just as well be a defibrillator we drop – today, we’re only interested in showing that the system works”, says Jesper Tordenlid, responsible for WARA PS, the WASP arena for public safety.
A slightly larger patrol boat from the Docksta boatyard and Saab Kockums is also part of the system.
“This patrol boat will be equipped with systems for autonomous driving with safety personnel onboard. Kockums is planning to build a research system in which one autonomous patrol boat can work with several autonomous Pirayas, where the smaller and faster boats collaborate with the larger boat”, says Jesper Tordenlid.
Doctoral students Linnea Persson and Inês Lourenco from KTH Photo credit Thor BalkhedThis time the patrol boat takes a few trips around in the choppy seas of the bay at Gränsö Castle, in Västervik. Doctoral students Linnea Persson and Inês Lourenco from KTH are onboard to carry out testing and measurement. The task is to land a drone, a quadcopter, on the boat while it is travelling. Linnea Persson and Inês Lourenco are satisfied with the measurements they have been able to make here.
“The quadcopters have a great potential, but their range is limited. In order to be able to work far from land, they must be able to land on a vessel and recharge their batteries. We wanted to test how the landing platform works, and measure the dynamics of the boat”, Linnea Persson tells us.
The objective of the research she and Inês Lourencos are carrying out is to coordinate the control of the boat and the drone such that landing is possible, even in high winds.
Testing his system in practice is invaluable for Bertil Grelsson.He uses a 360° camera on the Piraya to capture images. The first stage of the subsequent mathematical processing is to compensate for the vertical motion of the boat on the waves, after which the images are combined with satellite images of the area.
“We can then use neural networks to calculate the position of the boat with an accuracy of a few metres”, he claims.
Testing the system in practice is invaluable for Bertil.
“I’ve been able to collect real data here and identified where the challenges are. The sun, for example, may be low in the sky and reflections confuse the sensors; there may be water splash; or another boat may pass between the camera and the shoreline.”
Bertil Grelsson has more than 120,000 images to deal with back at LiU, and the next step will be to teach the system which images to discard – those with splashes, out of focus, or lacking a clearly defined shoreline.
“The Piraya itself should eventually be able to process the images and draw conclusions about its position from them”, he says.
Bertil Grelsson will be writing his doctoral thesis during the autumn, and will then take his newly found knowledge to his colleagues at Saab.
“The main reason I’m here is to increase expertise in neural networks within Saab – what they can be used for, and what doesn’t work”, he concludes.
Over the sea, the first drone films the search area from on high. The Piraya is waiting for the second drone to reach the initial point, before starting to search across the water surface.
On land, the command centre is following on computer screens how the boat and the drones travel, and what they see through their cameras. Today is just a test and a demonstration, and both the drones and the boat have human safety operators. But even though the wind is at the limit of what the system can cope with, none of the safety personnel need to take action. The drones carry out the mission, return to base and land without fuss, while the Piraya comes into dock and is moored.
First demonstration
What we have witnessed is the first demonstration of an autonomous rescue system, developed within the Wallenberg AI Autonomous Systems and Software Program, WASP. Those watching include senior researchers and doctoral students fromAll those who are present are interested in different parts of the system, which are now being tested together in real-life conditions for the first time. We see, among other things, cameras from Axis Communications, cloud-based computer systems from Ericsson, management systems from Saab Combitech, boats and systems from Saab Kockums, mobile communication solutions from Telia, and the application of university research within artificial intelligence, positioning, computer vision and autonomy.
The doctoral students working here will spend the coming year coupling their university-based work into the system, in order to refine and improve it. Today, however, they are here simply to see how the technology works, and test parts of their research in a real scenario.
New mission for the drones
The drones are now despatched on a new mission, this time to search for people who have disappeared on land. The drones exchange handshakes to confirm that they can carry“It could just as well be a defibrillator we drop – today, we’re only interested in showing that the system works”, says Jesper Tordenlid, responsible for WARA PS, the WASP arena for public safety.
A slightly larger patrol boat from the Docksta boatyard and Saab Kockums is also part of the system.
“This patrol boat will be equipped with systems for autonomous driving with safety personnel onboard. Kockums is planning to build a research system in which one autonomous patrol boat can work with several autonomous Pirayas, where the smaller and faster boats collaborate with the larger boat”, says Jesper Tordenlid.
“The quadcopters have a great potential, but their range is limited. In order to be able to work far from land, they must be able to land on a vessel and recharge their batteries. We wanted to test how the landing platform works, and measure the dynamics of the boat”, Linnea Persson tells us.
The objective of the research she and Inês Lourencos are carrying out is to coordinate the control of the boat and the drone such that landing is possible, even in high winds.
Testing in practice
Bertil Grelsson is an industry-based doctoral student at LiU, where he divides his time between the Computer Vision Laboratory at the university and Saab. He is working to enable boats and other vessels to determine their positions without access to the GPS system, using cameras and silhouettes of the shoreline.“We can then use neural networks to calculate the position of the boat with an accuracy of a few metres”, he claims.
Testing the system in practice is invaluable for Bertil.
“I’ve been able to collect real data here and identified where the challenges are. The sun, for example, may be low in the sky and reflections confuse the sensors; there may be water splash; or another boat may pass between the camera and the shoreline.”
Bertil Grelsson has more than 120,000 images to deal with back at LiU, and the next step will be to teach the system which images to discard – those with splashes, out of focus, or lacking a clearly defined shoreline.
“The Piraya itself should eventually be able to process the images and draw conclusions about its position from them”, he says.
Bertil Grelsson will be writing his doctoral thesis during the autumn, and will then take his newly found knowledge to his colleagues at Saab.
“The main reason I’m here is to increase expertise in neural networks within Saab – what they can be used for, and what doesn’t work”, he concludes.
Solving common problems
Jesper Tordenlid is satisfied with today’s exercises. The group of researcher, industrial actors and doctoral students has been here for a few days before the demonstration itself, and tomorrow will be devoted to looking forwards.
Mille Millnert, chair of the WASP AI board, and Lars Nielsen, program director, on site.“It’s great when we meet and spend some days here: things happen when researchers and engineers come together to solve common problems under real-life conditions. The multidisciplinary discussions are the most important result of our time here. Tomorrow we’ll be looking to the future: we hope that in a year’s time most of the 15 doctoral students who are engaged in this part of WASP can demonstrate results coupled to what we have done today.”
“It is truly compelling that the universities and the industries that will build the systems can work so closely together within WASP, somewhat that we have a long tradition of in Sweden”, concludes Gunnar Bark, director of the WASP arenas.
Translation George Farrants
