Technology for augmented intelligence

The screen shows a tiny heart beating in a baby aged five months about to undergo surgery for a heart abnormality. The work or the surgeon is recorded using a technology known as “light field imaging”, and a complete virtual model is in the process of construction. The model then makes it possible for the surgeon to practise the complicated procedure. In the future, it will be extended to become a system that can also assist and provide advice during complicated cases.

The AR headset shows us a photorealistic image of a painting of Vertumnus, god of the seasons, painted in the early 1590s by Giuseppe Arcimboldo. The painting is safely stored Vertumnus, god of the seasons, painted in the early 1590s by Giuseppe Arcimboldo/ Skoklosterin a perfectly adapted environment, while we can examine it down to its finest details in the laboratory. A virtual torch allows us to see each individual fibre that has expanded out from the panel behind, and the researchers can use infrared light to bring out the underlying charcoal drawing. Each nuance of colour has its own characteristic properties and can be separated in virtual reality.

“We’re conducting this project together with National Historical Museums”, Jonas Unger says. He is research leader for computer graphics and image processing, and project manager at the Center for Augmented Intelligence.  

Other researchers from the Division of Media and Information Technology, MIT, at LiU are associated with the centre: Jonas Löwgren, professor of interaction and information design; Patric Ljung, research leader for immersive visualization; and Anders Ynnerman, professor of scientific visualization.

The centre is a platform within Visual Sweden, a Vinnova-financed investment for research and development in visualisation. In addition to Linköping University, the municipalities of Linköping and Norrköping are participating, as are Region Östergötland and 63 companies, of all sizes. 

Jonas Unger and his colleagues give us a guided tour of the newly equipped premises at Campus Norrköping, including a laboratory for interactive design and one for multidimensional computer graphics. It is in the latter that we see the three-dimensional beating heart and Arcimboldo’s masterwork, and where we can wander around and admire the view from the balcony of a yet-to-be-built building in Norrköping.

It’s not just surgeons who need huge quantities of data when training: another application is looking at autonomous vehicles.  Apostolia Tsirikoglou, doctoral student at MIT, describes methods to create realistic synthetic data that can replace otherwise time-consuming data collection. It is also a way to ensure that the control systems in the autonomous vehicles have learnt all they need to enable the vehicle to be driven safely. 

“We want to build unique virtual worlds where we can change different parameters and retain full control. We need large quantities of synthetic data for this, and we have developed a successful way of doing it. Our technology works better than previously used methods”, Apostolia Tsirikoglou assures us.

Boris Ahnberg, product owner for future products at Toyota Material Handling, sees many advantages of the collaboration within Visual Sweden.

Ahmet Börütecene, postdoc at MIT, demonstrates the Laboratory for Interaction Design.“The market for online-enabled trucks is increasing as the degree of automation increases. We see a future in which trucks of different sizes collaborate and distribute work between themselves totally autonomously. We now have equivalent computer power to the machine that beat Kasparov in chess in our telephones. What appears amazing today will be mundane tomorrow”, he concludes. 

Boris Ahnberg and his colleagues have collaborated with researchers in the Laboratory for Interaction Design to explore the possible appearance and abilities of such systems.

The Centre for Augmented Intelligence is currently home to four projects: 

• How we can use large numbers of images for machine learning, such as in the examples above for the preservation of cultural artefacts and heart surgery. The researchers here collaborate with several companies and Skåne University Hospital. 

• How it is possible to sort and assign priorities in a huge amount of information, such as the information that is typically displayed on the various screens in an industrial control room. This project is a collaboration with Toyota Material Handling.

• How an expert and a service engineer can share the same virtual world, allowing the expert to support the engineer on site, facing a concrete and difficult problem. It must be possible to transmit images and mobile descriptions in both directions. This development is proceeding together with Siemens Turbomachinery. 

• How data can be displayed at many different locations while remaining traceable – how can the semantic web be visualised. It must be possible to both visualise and understand the origin of the data and how they have been combined. 

The Center for Augmented Intelligence, CAI, is the first platform, with SEK 2 million in financing from Visual Sweden for three years, and a further SEK 6 million from LiU and project partners. Further platforms will follow, such as a platform for n-dimensional modelling and a platform whose goal is to construct a complete digital copy of a human for medical use.