3D models from cheap video cameras

Hannes Ovrén shows in his doctoral thesis in computer vision how 3D models can be created from video films recorded with simple body-mounted cameras. The research opens new possibilities for both robots and humans, not least for the police and rescue services.

Creation of a 3D modelCreation of a 3D model Photo credit: Hannes OvrénResearch in computer vision has a major significance for the future of artificial intelligence: autonomous systems rely on the ability of robots and other systems to orient themselves and discover objects and people.

Hannes Ovrén’s contribution shows how to create a 3D model of the surroundings, based on video films taken with an inexpensive body-mounted camera. The model reproduces scale accurately, allowing measurements to be made.

Per-Erik Forssén and the small cameraPer-Erik Forssén and the small camera. Photo credit: Thor Balkhed“Currently, seeing robots move rather carefully, in order to keep track of where they are. In some cases, they may even have to stop in order to determine their location. This technology allows robots to move more freely and construct a model of the surroundings while moving”, says Per-Erik Forssén, docent at the Computer Vision Laboratory, and Hannes Ovrén’s principal supervisor.

Other fields of use can be found in, for example, police work or rescue work, where personnel with a body-mounted camera can recreate a crime scene or an accident location in three dimensions, with people and objects at the exact location they had at the instant the photograph was taken.

Rolling shutter problem

The problem with creating 3D images from simple video cameras has until now been that the camera must be stationary, preferably mounted on a tripod. If the camera moves, straight objects may appear to be curved in the image, or appear to be at different heights. Objects wobble, and a distorted image is obtained. This is because cheap cameras have a type of shutter known as a “rolling” shutter, which builds the image up in pixels row-by-row. Smartphones have this type of camera.

“Each image frame contains motion, but it is possible to improve the image significantly by modelling how the camera has moved and compensating for the motion”, says Hannes Ovrén.

To prevent the calculations from becoming too demanding, his method creates a curve, known as a “spline”, that describes how the camera has moved. This curve is constructed from spline knots, where each knot controls the appearance of the curve at a certain point in time. If the knots are placed more densely, the method can deal with more complex motion, but the calculations become more demanding.

Robot equipped with a cheap cameraRobot equipped with a cheap camera Photo credit: Thor BalkhedHannes Ovrén shows in the thesis that it is possible to use significantly fewer knots when the errors that arise due to the straightening and smoothing of the curve are modelled. In order to prevent the errors from becoming too large, the method also uses an inertial measurement unit attached to the camera. This is a small and cheap sensor that tracks acceleration, angular velocity and orientation relative to the ground.

Reliable model

“The measurements from the sensor are included in the calculations and we can in this way increase the distance between knots, reducing the size of the calculations”, says Hannes Ovrén.

The simplification means that the motion of the camera and the spline curve are not exactly the same. It is possible, however, to determine how the difference in pathway affects the magnitude of measurement errors, and in this way increase the reliability of the 3D model and the distances in it.

Hannes Ovrén will defend the thesis on 7 September 2018. He plans to continue work at FOI as a newly graduated doctor.

His thesis work has been part of the project “Learnable Camera Motion Models”, financed by the Swedish Research Council.

Continuous Models for Cameras and Inertial Sensors, Hannes Ovrén, Computer Vision Laboratory, Department of Electrical Engineering, Linköping University, 2018. Principal supervisor Per-Erik Forssén.

Translation George Farrants

Videos showing the construction of a 3D model from a GoPro sports camera: 

Per-Erik Forssén and Hannes Ovrén

Per-Erik Forssén and Hannes Ovrén preparing the robot. Thor Balkhed

Robot carrying a cheap camera

Work in progress. Thor Balkhed

Contact

Research

Latest news from LiU