The research covers a broad range of topics within artificial visual systems with a particular focus on machine learning for robot vision. The group has made major contributions in establishing the concept of computer vision in Sweden.
The development of our methods for artificial vision is motivated by models of the human visual system, since artificial vision is to coexist with people and predict human actions trigged by visual perception. The research results are enabling autonomous vehicles or robot systems that interact with humans.
The Computer Vision Laboratory was founded by Professor Gösta Granlund in the 1970s, and was a pioneer in the use of machine learning for computer vision. Professor Michael Felsberg took over the leadership of the group 2008.
The Computer Vision Laboratory is lifted in the Vinnova report (“Artificial intelligence in Swedish business and society – Analysis of development and potential") as one of the major and most successful research environments in AI in Sweden.
Computer vision Photo credit Göran Billeson
Visual Object Tracking is one of the principal challenges in Computer Vision, where the task is to locate a certain object in all frames of a video, given only its location in the first frame.
Some example frames showing the scale adaptation of our approach.
Due to variations of appearance, the model generated from the first frame needs to be updated on the fly. We address this problem by online machine learning approaches, more concretely, discriminative correlation filters.
This video illustrates online adaptation of feature (color) selection to the model appearance.
Application scenario within WASP where the developed code has been used for people tracking:
International VOT benchmark committee
With the growing availability of depth cameras and lidar, the processing of the resulting point clouds has become its own sub-area of Computer Vision.
Illustration of the probabilistic modelling of point clouds using latent variables.
Data from these sensors is usually sparse, thus requires densification. Also, the position and orientation are often only partly known and a registration of the measurements in a joint reference system is required.
Finally, the point cloud needs to be split into object-related measurements, i.e., 3D segmentation is needed. We approach these problems using novel deep learning methodology based on normalized CNNs and 3D fusion of deep 2D segmentations, as well as probabilistic modelling.
Centauro and Veoneer
Navigation and localization are principal challenges in Robot Vision, where the task is to estimate the geometric relation between a vehicle and its environment, either passively in a virtual model or actively through controlling the robot.Localization at sea using a 360 camera and deep learning for horizon matching.In the active case, the vehicle pose needs to be estimated in real-time, which is achieved by online learning of vision-based regression. To achieve this, problem knowledge and suitable visual representations are combined using machine learning.
Application scenario within WASP where the developed code has been used for people tracking:
Saab Group
Semantic description of humans in images and videos is one of the fundamental problems in computer vision with many applications such as visual surveillance, facial verification, health care, image and video search engines, tagging suggestions and human-computer interaction.
Humans in different shapes.
Humans have an outstanding ability when it comes to recognizing (i) semantic attributes such as age, gender, hair style and clothing style (ii) actions such as riding a horse, climbing, running and walking and (iii) facial expressions such as angry, happy and smiling.
We are currently developing novel deep learning solutions for the challenging problem of semantic description of humans in images and videos. The major emphasis is to investigate the challenging generic sub-problems of efficient image and video description, automatic learning of visual models, joint learning from textual annotations and visual data and learning robust methods with minimal supervision.
Rao Muhammad Anwer
The goal of sensor fusion is to combine the measurements from several sensors into new information that cannot be computed from one sensor alone, or that would require a much more costly solution.
The sensor fusion group is part of the Division of Automatic Control at Linköping University. Sensor fusion as a research area was established in 1995, and it has since then been a research area in many large research centers.
In total, the group has produced more than 25 PhD’s and 40 Licentiates.
The research has major industrial relevance in fields such as ground-based and airborne autonomous vehicles, collaboration between people and robots, and positioning with the aid of mobile telephones, to mention just a few examples.
The research is split into basic research issues and more application oriented projects. The group has done fundamental well-cited contributions in nonlinear filtering and localization theory. Applied research is mostly done in collaboration with Swedish manufacturing industry.
In some cases, the research has resulted in spin-off companies, most notably NIRA Dynamics, Softube and Senion.
The goal is to develop fundamental nonlinear filtering and estimation methods with applications to speed estimation in wheel based vehicles.
Illustration of a spectrogram (time-varying frequency content) computed from a vehicle mounted accelerometer, compared to the estimated harmonics.The manufacturing industry always look for opportunities to replace costly and sensitive sensors with cheaper and more robust ones. The wheel speed sensor is one such sensor that is exposed to harsh environment and is a relatively costly part of vehicles. In some applications, a contact-less accelerometer can be used instead.
By analyzing the time-varying spectrum of the vibrations caused by the wheel, the wheel speed can be computed. Another potential application is to support IoT devices where speed information is needed when located on a vehicle. Here, wired solutions to the existing sensors can be excluded. Speed estimation in cellphones is one further application.
This project has developed algorithms tailored to this problem that extend current theory in nonlinear filtering.
This project aims to develop enabling technologies that allow for human-robot cohabitation in mines. That is, to facilitate safe and efficient interaction between humans, autonomous systems, and manual machinery.
The goal in human-robot cohabitation in mines is to facilitate safe and efficient interaction between humans, autonomous systems, and manually operated machines. Robust and accurate localization of each object is an enabler to achieve this goal, and the first phase of this project aims to develop algorithms onboard autonomous vehicles that localize the vehicle relative to the mine and other stationary and moving objects in the environment.
The use of autonomous vehicles in mines today requires that no manual operation is allowed in the same area. Safety gates separate a production area so the autonomous vehicles can operate without any manual interference. This setup makes the automation very sensitive to external disturbances. For example, an operator entering through a safety gate will potentially shut down the full production area. The first approach is to base the situation awareness on data from a laser scanner, using an adaptive map of the mine.
Illustration of the position dependent information content in a mine, using two laser scanners mounted on a loader. Illustration Epiroc.
Epiroc
The goal of this project is to develop autonomous functions for the joint problem of sensor management and aircraft control to relieve a human system operator. This will be achieved by exploring high level control of the platforms and their resources, e.g. sensors.
Illustration of how a radio signal reaches the receiver in different ways due to multipath effects caused by reflections in the walls.The basic measurements are usually split into timing and signal strength measurements. In this project, we extend these approaches by studying the power delay profile (PDP), that provides a deeper understanding multipath phenomenon that are so hard to mitigate in classical methods.
Both timing and signal strength measurements can be derived from the PDP. However, we investigate the advantages of estimating the position from the full PDP, and relate this to a map of the environment to further understand the multipath effects, which in this way can be turned into an information source rather than a nuisance.
Ericsson
The goal of this project is to develop autonomous functions for the joint problem of sensor management and aircraft control to relieve a human system operator. This will be achieved by exploring high level control of the platforms and their resources, e.g. sensors.
The cognitive focus of pilots has shifted from controlling the aircraft to managing the sensor systems with the introduction of autonomous flight systems. One can compare with modern drones, which are very easy to control, but still the pan, tilt and zoom of the onboard camera is a challenging task.
Further, planning the flight to maximize the information from e.g. a camera poses its own challenge. It requires understanding of how to manage available sensors to achieve the goal at hand; proactively take limitations in the environment into consideration; react to changes; learn from experience; and collaborate as necessary.
Four WASP students demonstrating autonomous drone flight in the Visionen test arena as part of a WASP project course. Foto David Brohede
Saab AB/Aeronautics
The time update is a critical component in navigation and tracking filters. Usually, physical dynamical models based on Newton’s laws are used to predict the motion of the vehicle. In this project, we aim to use data-driven methods and machine learning to learn the dynamics of the vehicle.
In many cases, the motion patterns of objects are very repeatable, but beforehand not known. This might be boats drifting along streams or cars following traffic rules. Today, the standard methods simply assume no more knowledge of the motion than that physical limitations with respect to accelerations etc are fulfilled.
Learning to time updateObviously, better knowledge of the underlying motion patterns would be of huge benefit. Learning these motion patterns is a problem that combines theory from classic estimation theory, as well as modern data driven machine learning methods.
The main challenge lies in exploiting the synergies possible by combining the two approaches in one common framework.The method automatically learns how cars drive in a three-way intersection using position measurements and a constant velocity model combined with a Gaussian Process model of maneuvers, resulting in improved localization and fast maneuver classification.
The division of Vehicular Systems has a clear focus on vehicle control, diagnosis, and autonomy. Our facilities include an engine laboratory, a propulsion laboratory, and a vehicle dynamics and autonomy laboratory. The group is internationally well recognized.
Our strategy is to focus on the system science aspects, and to collaborate with others when it comes to mechanical construction or industrial evaluation. The functions under consideration include advanced engine control, control coordination of vehicle and powertrain systems, and autonomous functions in intelligent vehicles and roadways.
Autonomy also requires automated supervision and prognostics for reliable operation. We investigate purely theoretical problems and application-oriented problems, e.g., autonomous mining.
The Vehicle Lab is in fact a complete building, L-huset (the L-building incorporating three laboratories for vehicle research; The Laboratory for Vehicle Propulsion, The Laboratory for Vehicle Dynamics and Autonomy and The Laboratory for Energy Conversion and Energy Storage.
In the engine lab there are two engine test stands suited for testing passenger car engines. Both test stands are equipped with modern asynchronous machines which can act both as a drive and a load to the engines. These test stands give us the possibility to run both steady state and dynamic tests, with torque rise-times less than 10 ms. We are able to do road load simulations.
Predictive maintenance is essential to achieve reliable autonomous mining, avoiding unplanned stops of mining machinery such as wheel loaders and drills.
Mines are difficult environment for mining machinery.Photo EpirocMines are harsh environments where machine usage has significant influence on component lifetime and when removing the human operator from the system, a useful source of information is eliminated. In this perspective, both in conventional and autonomous mines, optimized maintenance strategies of fleets where individual machines have the ability to autonomously predict their own need of maintenance is a promising research area. Main research topics are development of methodology and theory for machine individual prognostics, and in particular machinery in mining applications.
The difficult environment for mining machinery, e.g., for drilling equipment, not only makes life-time for components short compared to, e.g., personal car components but it also makes it difficult to equip the machines with reliable high accuracy sensors. This means that methods must be robust and rely on available sensor data from sensors not necessarily placed near the critical component. To reach the project’s research objectives, availability of data is therefore key and ambitious data collection campaigns on multiple mines in different countries for extended amount of time has been initiated. This is also complemented with planned experiments on research drill rigs using experimental and new sensor technology.
Detailed physical models of machines are not expected to be available, mainly due to the difficult operating conditions and the low-volume of machines. Therefore, a main research question is the development of methods for fusing coarse maintenance information and operational data, high-resolution and high-frequency on-board measurements, and basic physical models. This process requires development of techniques in signal processing and machine learning together with statistical models and techniques from survival and reliability analysis.
A fundamental research question for the project is thus how all available data and physical models is to be used in order to facilitate individual-based predictive maintenance, and in particular for mining products.
Epiroc Drills AB (http://www.epiroc.com), Robert Pettersson
Projektet handlar om att ta fram optimala manövrar i tids- och säkerhetskritiska situationer för autonoma fordon. Exempel är för hög hastighet i en kurva eller snabba undanmanövrar när hinder plötsligt dyker upp på vägen.
Däckens friktion utnyttjas maximalt. Foto iStock/LeMannaDet här är manövrar som kräver att däckens friktion mot marken utnyttjas maximalt. De viktiga forskningsfrågorna här är hur de olika friktionskrafterna mellan däcken och vägen kan användas på bästa tänkbara sätt, liksom hur fordonens egen dynamik kan regleras och utnyttjas för att få bästa tänkbara beteende. Manöverproblemet kännetecknas av en olinjär dynamik och osäkerhet i modellparametrar för den specifika situationen.
I projektet utnyttjas optimering som ett verktyg för att beräkna fordonsmanövrar i ett antal olika scenarier och under olika dynamiska förhållanden, som ytan på vägen och hur vägen kröker sig. De broms- och styrningsmönster som observeras i de optimala lösningarna i de olika fallen används sedan som grund vid utformningen av styrsystemen för autonoma fordon.
Den här typen av styrsystem måste även kunna hantera den inneboende osäkerheten för att uppnå robusthet vid olika situationer, vilket betyder att vi även inom forskningsprojektet utvecklar metoder för sensorbaserad återkoppling i realtid. Styrsystemen utvärderas också genom att de jämförs med resultat från datorsimuleringar av en i förväg beräknad optimal styrning för samma problem.
En utvärdering av metoderna i en experimentbil planeras inom ramen för WASP:s forskningsarena WARA för automatiserade transportsystem.
University of Lincoln, Storbritannien
The project focuses on the development of new formulations of objective functions and constraints for motion-planning problems solved by trajectory optimization.A time-critical avoidance maneuver. Foto iStock/Toa55This research project addresses planning of time-critical avoidance maneuvers, with applications to autonomous ground vehicles. The project focuses on the development of new formulations of objective functions and constraints for motion-planning problems solved by trajectory optimization. The overall objective is to increase the safety for vehicles by utilizing the new sensor capabilities and situation awareness as well as the extended level of actuation possibilities available or foreseen in ground vehicles. One of the major challenges in application of optimization techniques in vehicle control is to satisfy the computational constraints, in light of the inherent time-critical nature of the maneuver to be performed.
The project is performed along two different complementary paths. The first part of the research considers new formulations of optimization problems for planning a safe motion of a vehicle in different critical scenarios such as where a double lane-change maneuver is required. The second part of the research investigates how the computational complexity of the optimization problem for determining the optimal maneuvers can be reduced. More specifically, methods for decomposition and parallel computation of segments of the full maneuver have been developed. Such methods decrease the total computational time, and are thus of interest towards the goal of being able to online compute a full vehicle maneuver using optimization.
Experimental evaluations are planned within the framework of the WASP Autonomous Research Arena (WARA) for automated transport systems, and specifically with the planned experimental car.
Within the WASP research environment Optimization for Learning and Autonomy within the Division of Automatic Control we work with both development and applications of optimization.
The development of optimization methods is targeting structure-exploitation of classes of optimization problems, e.g. for learning and control, in order to increase the efficiency of optimization solvers.
This development also includes utilization of modern parallel hardware. Furthermore, researched is performed to tightly combine planning methods known from AI with methods known from optimal control. Moreover, ongoing theoretical and algorithmic developments include real-time certification of optimization methods. Applications include motion planning and control for autonomous vehicles as well as implementations of optimization methods.
The work is carried out within the WASP cluster Large-Scale Optimization and Control. We collaborate with universities both in Sweden and abroad, among others UCLA, ETH, TUDelft, DTU, University of Siena, KTH, and Lund University. Industrial research partners include Scania and SAAB.
We are investigating whether it is possible to use algorithms within machine learning to discover new optimization algorithms that function better than the current ones, which are constructed manually.
Planning, decision making and predictive control of autonomous systems including machine learning, are all based on very advanced optimization methods. The choice of optimization algorithm including its tunable parameters is still made manually even if frameworks such as disciplined convex optimization, including cvx and YALMIP, have done major progress in automatizing convex optimization.
Disciplined convex optimization is essentially a convenient way of interfacing to general purpose optimization algorithms for convex optimization. For large-scale convex optimization, distributed or not, and non-convex optimization problems the choice and tuning of the optimization algorithm is still very complicated. Also, there are several equivalent formulations of the optimization problem that impact what efficiency is obtained.
In order to overcome this major difficulty, we investigate if it is possible to directly learn these optimization algorithms and the best formulation from experiments and simulations, that is to apply and extend machine learning algorithms to derive new automated optimization algorithms that perform better than manually designed and tuned ones.
Professor Bo Wahlberg, KTH Royal Institute of Technology
This PhD project investigate network topology such as hierarchical network structure, e.g. chordal graphs, for scalable computations with applications to control in a wide sense, e.g. system identification, machine learning, control design, and estimation.Chordal graph with associated clique tree for distributed computations.
We also study how privacy affects the achievable performance and what amount of communication that is needed between different computational agents in order to achieve the most efficient overall computations.
Different type of computational platforms will be investigated. We also also develop efficient optimization code for generic problem formulations.
Relevant applications are within infrastructure networks for traffic, water, gas, electricity, and building control.
