Mobile camera localization using depth maps

• US 8,711,206 B2
• Filed: 01/31/2011
• Issued: 04/29/2014
• Est. Priority Date: 01/31/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Newcombe, Richard, Izadi, Shahram, Molyneaux, David, Hilliges, Otmar, Kim, David, Shotton, Jamie Daniel Joseph, Kohli, Pushmeet, Fitzgibbon, Andrew, Hodges, Stephen Edward, Butler, David Alexander
• Primary Examiner(s): Czekaj, David
• Assistant Examiner(s): Owens, Tsion B
• Application Number:
  US13/017,474
• Publication Number:
  US 20120194644A1
• Time in Patent Office: 1,184 Days
• Field of Search: 348/46, 348/142, 348/218
• US Class Current: 348/46
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

Mobile camera localization using depth maps is described for robotics, immersive gaming, augmented reality and other applications. In an embodiment a mobile depth camera is tracked in an environment at the same time as a 3D model of the environment is formed using the sensed depth data. In an embodiment, when camera tracking fails, this is detected and the camera is relocalized either by using previously gathered keyframes or in other ways. In an embodiment, loop closures are detected in which the mobile camera revisits a location, by comparing features of a current depth map with the 3D model in real time. In embodiments the detected loop closures are used to improve the consistency and accuracy of the 3D model of the environment.

Claims

1. A method of real time camera relocalization comprising:

receiving a sequence of depth map frames from a moving mobile depth camera each depth map frame comprising a plurality of image elements each having a depth value being related to a distance from the mobile depth camera to a surface in the scene captured by the mobile depth camera;

tracking the position and orientation of the mobile depth camera using the depth map frames and simultaneously forming a 3D model of the environment in which the mobile depth camera is moving using the depth map frames;

detecting a loop closure error;

modifying the 3D model in response to detecting the loop closure error;

detecting a failure in the tracking of the position and orientation of the mobile depth camera; and

relocalizing the mobile depth camera by recomputing its position and orientation using a current depth map captured by the mobile depth camera,

relocalizing the mobile depth camera comprising;

computing a track distribution of poses of the mobile depth camera being a predicted distribution of poses given known previous motion of the mobile depth camera;

sampling a pose from the track distribution and using that sampled pose to align the current depth map and a previous depth map, or

relocalizing the mobile depth camera comprising;

applying a random decision forest to patches from the current depth map and from a plurality of previous depth maps obtained from the 3D model of the environment to obtain histograms of textons being texture features of a depth map and selecting a previous depth map which is similar to the current depth map in terms of the histograms.

2. A method as claimed in claim 1 wherein detecting a failure in the tracking comprises any of:

comparing a change in a current and previous tracked position of the mobile depth camera with a threshold; and

detecting a failure in convergence of an iterative process used to track the position and orientation of the mobile depth camera.

3. A method as claimed in claim 1 wherein detecting a failure in the tracking comprises comparing a current tracked position and orientation of the mobile depth camera with a position and orientation predicted using a model of motion of the mobile depth camera.

4. A method as claimed in claim 1 wherein relocalizing the mobile depth camera comprises finding a keyframe which is similar to the current depth map from a plurality of keyframes being depth map frames previously gathered by the mobile depth camera each having an associated camera position and orientation.

5. A method as claimed in claim 4 wherein finding a keyframe comprises using a geometry estimation process to selecting a keyframe which has similar arrangement and locations of vertical and horizontal surfaces as the current frame.

6. A method as claimed in claim 4 wherein finding a keyframe comprises searching the plurality of keyframes in an order related to a last known location of the mobile depth camera.

7. A method as claimed in claim 1 wherein relocalizing the mobile depth camera comprises:

computing a track distribution of poses of the mobile depth camera being a predicted distribution of poses given known previous motion of the mobile depth camera;

sampling a pose from the track distribution and using that sampled pose to align the current depth map and a previous depth map.

8. A method as claimed in claim 1 wherein relocalizing the mobile depth camera comprises:

applying a random decision forest to patches from the current depth map and from a plurality of previous depth maps obtained from the 3D model of the environment to obtain histograms of textons being texture features of a depth map and selecting a previous depth map which is similar to the current depth map in terms of the histograms.

9. A method of real time camera relocalization comprising:

receiving a sequence of depth map frames from a moving mobile depth camera each depth map frame comprising a plurality of image elements each having a depth value being related to a distance from the mobile depth camera to a surface in the scene captured by the mobile depth camera;

tracking the position and orientation of the mobile depth camera using the depth map frames and simultaneously forming a 3D model of the environment in which the mobile depth camera is moving using the depth map frames;

detecting a loop closure error;

modifying the 3D model in response to detecting the loop closure error;

detecting a failure in the tracking of the position and orientation of the mobile depth camera by any of;

comparing a change in a current and previous tracked position of the mobile depth camera with a threshold;

detecting a failure in convergence of an iterative process used to track the position and orientation of the mobile depth camera;

and comparing a current tracked position and orientation of the mobile depth camera with a position and orientation predicted using a model of motion of the mobile depth camera;

relocalizing the mobile depth camera by recomputing its position and orientation using a current depth map captured by the mobile depth camera; and

at least one of

computing a track distribution of poses of the mobile depth camera being a predicted distribution of poses given known previous motion of the mobile depth camera;

sampling a pose from the track distribution and using that sampled pose to align the current depth map and a previous depth map, or

applying a random decision forest to patches from the current depth map and from a plurality of previous depth maps obtained from the 3D model of the environment to obtain histograms of textons being texture features of a depth map and selecting a previous depth map which is similar to the current depth map in terms of the histograms.

10. A method as claimed in claim 9 wherein relocalizing the mobile depth camera comprises finding a keyframe which is similar to the current depth map from a plurality of keyframes being depth map frames previously gathered by the mobile depth camera and stored as an associated camera position and orientation.

11. A method as claimed in claim 9 wherein finding a keyframe comprises using a geometry estimation process to select a keyframe which has similar arrangement and locations of vertical and horizontal surfaces as the current frame.

12. A method as claimed in claim 9 which comprises:

searching the plurality of keyframes in an order related to a last known location of the mobile depth camera in order to find a keyframe which is similar to the current depth map.

13. A method as claimed in claim 9 comprising computing a track distribution of poses of the mobile depth camera being a predicted distribution of poses given known previous motion of the mobile depth camera;

sampling a pose from the track distribution and using that sampled pose to align the current depth map and a previous depth map.

14. A method as claimed in claim 9 comprising applying a random decision forest to patches from the current depth map and from a plurality of previous depth maps obtained from the 3D model of the environment to obtain histograms of textons being texture features of a depth map and selecting a previous depth map which is similar to the current depth map in terms of the histograms.

15. A real time camera relocalization system comprising:

an input arranged to receive a sequence of depth map frames from a moving mobile depth camera each depth map frame comprising a plurality of image elements each having a depth value being related to a distance from the mobile depth camera to a surface in the scene captured by the mobile depth camera;

a frame alignment engine arranged to track the position and orientation of the mobile depth camera using the depth map frames;

a 3D model formation system arranged to form a 3D model of the environment in which the mobile depth camera is moving using the depth map frames, the 3D model formation system being further arranged to detect a loop closure error and to modify the 3D model in response to the loop closure error;

a relocalization engine arranged to detect a failure in the tracking of the position and orientation of the mobile depth camera; and

to relocalize the mobile depth camera by recomputing its position and orientation using a current depth map captured by the mobile depth camera and at least one previous depth map obtained from the 3D model, the relocalization engine being arranged to apply a random decision forest to patches from the current depth map and from a plurality of previous depth maps obtained from the 3D model of the environment to obtain histograms of textons being texture features of a depth map and to select a previous depth map which is similar to the current depth map in terms of the histograms.

16. A system as claimed in claim 15 wherein the relocalization engine is arranged to find a keyframe which is similar to the current depth map from a plurality of keyframes being depth map frames previously gathered by the mobile depth camera each having an associated camera position and orientation.

17. A system as claimed in claim 16 wherein the relocalization engine is arranged to use a geometry estimator process to select a keyframe which has similar arrangement and locations of vertical and horizontal surfaces as the current frame.

18. A system as claimed in claim 15 wherein the relocalization engine is arranged to find a keyframe by searching the plurality of keyframes in an order related to a last known location of the mobile depth camera.

19. A game system comprising a mobile infra-red time-of-flight depth camera that uses structured light and a system as recited in claim 15 for tracking the mobile depth camera, the mobile depth camera and the real-time tracker being arranged to operate at at least 30 frames per second, the game system being arranged to influence the course of a game in relation to the tracking of the mobile depth camera.