Mobile Camera Localization Using Depth Maps
First Claim
1. A method of real time camera relocalization comprising:
- receiving a sequence of depth map frames from a moving mobile depth camera each depth map frame comprising a plurality of image elements each having a depth value being related to a distance from the mobile depth camera to a surface in the scene captured by the mobile depth camera;
tracking the position and orientation of the mobile depth camera using the depth map frames and simultaneously forming a 3D model of the environment in which the mobile depth camera is moving using the depth map frames;
detecting a failure in the tracking of the position and orientation of the mobile depth camera; and
relocalizing the mobile depth camera by recomputing its position and orientation using a current depth map captured by the mobile depth camera.
2 Assignments
0 Petitions
Accused Products
Abstract
Mobile camera localization using depth maps is described for robotics, immersive gaming, augmented reality and other applications. In an embodiment a mobile depth camera is tracked in an environment at the same time as a 3D model of the environment is formed using the sensed depth data. In an embodiment, when camera tracking fails, this is detected and the camera is relocalized either by using previously gathered keyframes or in other ways. In an embodiment, loop closures are detected in which the mobile camera revisits a location, by comparing features of a current depth map with the 3D model in real time. In embodiments the detected loop closures are used to improve the consistency and accuracy of the 3D model of the environment.
-
Citations
20 Claims
-
1. A method of real time camera relocalization comprising:
-
receiving a sequence of depth map frames from a moving mobile depth camera each depth map frame comprising a plurality of image elements each having a depth value being related to a distance from the mobile depth camera to a surface in the scene captured by the mobile depth camera; tracking the position and orientation of the mobile depth camera using the depth map frames and simultaneously forming a 3D model of the environment in which the mobile depth camera is moving using the depth map frames; detecting a failure in the tracking of the position and orientation of the mobile depth camera; and relocalizing the mobile depth camera by recomputing its position and orientation using a current depth map captured by the mobile depth camera. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8)
-
-
9. A method of real time camera relocalization comprising:
-
receiving a sequence of depth map frames from a moving mobile depth camera each depth map frame comprising a plurality of image elements each having a depth value being related to a distance from the mobile depth camera to a surface in the scene captured by the mobile depth camera; tracking the position and orientation of the mobile depth camera using the depth map frames and simultaneously forming a 3D model of the environment in which the mobile depth camera is moving using the depth map frames; detecting a failure in the tracking of the position and orientation of the mobile depth camera by any of;
comparing a change in a current and previous tracked position of the mobile depth camera with a threshold;
detecting a failure in convergence of an iterative process used to track the position and orientation of the mobile depth camera; and
comparing a current tracked position and orientation of the mobile depth camera with a position and orientation predicted using a model of motion of the mobile depth camera;relocalizing the mobile depth camera by recomputing its position and orientation using a current depth map captured by the mobile depth camera. - View Dependent Claims (10, 11, 12, 13, 14)
-
-
15. A real time camera relocalization system comprising:
-
an input arranged to receive a sequence of depth map frames from a moving mobile depth camera each depth map frame comprising a plurality of image elements each having a depth value being related to a distance from the mobile depth camera to a surface in the scene captured by the mobile depth camera; a frame alignment engine arranged to track the position and orientation of the mobile depth camera using the depth map frames; a 3D model formation system arranged to form a 3D model of the environment in which the mobile depth camera is moving using the depth map frames; a relocalization engine arranged to detect a failure in the tracking of the position and orientation of the mobile depth camera; and
to relocalize the mobile depth camera by recomputing its position and orientation using a current depth map captured by the mobile depth camera and at least one previous depth map obtained from the 3D model. - View Dependent Claims (16, 17, 18, 19, 20)
-
Specification