CAMERA POSE ESTIMATION FOR 3D RECONSTRUCTION

• US 20140206443A1
• Filed: 01/24/2013
• Published: 07/24/2014
• Est. Priority Date: 01/24/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Sharp, Toby, Fitzgibbon, Andrew William, Izadi, Shahram
• Granted Patent: US 9,251,590 B2
• Time in Patent Office: Days
• US Class Current: 463/31
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 2009/2435 using a video camera
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration

Abstract

Camera pose estimation for 3D reconstruction is described, for example, to enable position and orientation of a depth camera moving in an environment to be tracked for robotics, gaming and other applications. In various embodiments, depth observations from the mobile depth camera are aligned with surfaces of a 3D model of the environment in order to find an updated position and orientation of the mobile depth camera which facilitates the alignment. For example, the mobile depth camera is moved through the environment in order to build a 3D reconstruction of surfaces in the environment which may be stored as the 3D model. In examples, an initial estimate of the pose of the mobile depth camera is obtained and then updated by using a parallelized optimization process in real time.

Claims

1. A method of real-time camera tracking comprising:

receiving a sequence of depth map frames from a moving mobile depth camera each depth map frame comprising a depth value at each image element that depth value being related to a distance from the mobile depth camera to a surface in the environment captured by the mobile depth camera;

accessing a 3D model of surfaces in the environment;

tracking the position and orientation of the mobile depth camera by computing, for each of a plurality of the depth map frames, an updated position and orientation of the mobile depth camera according to an alignment of the image elements of the depth map frame and a surface depicted in the 3D model of the environment.

2. A method as claimed in claim 1 comprising computing an initial estimate of the position and orientation of the mobile depth camera from two of the depth map frames by computing a camera pose which aligns the two frames.

3. A method as claimed in claim 1 wherein accessing the 3D model of surfaces comprises accessing a signed distance function representing the surfaces in the environment.

4. A method as claimed in claim 1 comprising mapping depth values of image elements from a depth map frame to model positions with respect to the 3D model.

5. A method as claimed in claim 4 comprising calculating depth values for the model positions from the 3D model.

6. A method as claimed in claim 5 comprising computing residuals by comparing the depth values of the model positions and the depth values of corresponding image elements from a depth map frame.

7. A method as claimed in claim 5 comprising interpolating the depth values of the model positions from a plurality of values obtained from the 3D model.

8. A method as claimed in claim 5 comprising using texturing hardware to compute the depth values of the model positions.

9. A method as claimed in claim 1 comprising computing the alignment of the image elements of the depth map frame and a surface depicted in the 3D model of the environment by using a parallel computing device to accumulate quantities related to the alignment over image elements in parallel.

10. A method as claimed in claim 1 comprising computing the alignment of the image elements of the depth map frame and a surface depicted in the 3D model of the environment by using a parallel processing unit to carry out a sum operation to compute quantities related to the alignment per image element.

11. A real-time camera tracker comprising:

a communications interface arranged to receive a sequence of depth map frames from a moving mobile depth camera each depth map frame comprising a depth value at each image element that depth value being related to a distance from the mobile depth camera to a surface in the environment captured by the mobile depth camera;

the communications interface being arranged to access a 3D model of surfaces in the environment;

a camera pose engine arranged to track the position and orientation of the mobile depth camera by computing, for each of a plurality of the depth map frames, an updated position and orientation of the mobile depth camera according to an alignment of the image elements of the depth map frame and a surface depicted in the 3D model of the environment.

12. A real-time camera tracker as claimed in claim 11 the camera pose engine arranged to compute an initial estimate of the position and orientation of the mobile depth camera from two of the depth map frames by computing a camera pose which aligns the two frames.

13. A real-time camera tracker as claimed in claim 11 the communications interface being arranged to access the 3D model of surfaces by accessing a signed distance function representing the surfaces in the environment

14. A real-time camera tracker as claimed in claim 11 the camera pose engine comprising a parallel computing device arranged to compute the alignment of the image elements of the depth map frame and a surface depicted in the 3D model of the environment by accumulating quantities related to the alignment over image elements in parallel.

15. A real-time camera tracker as claimed in claim 11 the camera pose engine comprising a graphics processing unit arranged to compute the alignment of the image elements of the depth map frame and a surface depicted in the 3D model of the environment by carrying out a map reduce operation to compute quantities related to the alignment per image element.

16. A real time camera tracker as claimed in claim 11 the camera pose engine being at least partially implemented using hardware logic selected from any one or more of:

a field-programmable gate array, a program-specific integrated circuit, a program-specific standard product, a system-on-a-chip, a complex programmable logic device, a graphics processing unit (GPU), texturing hardware.

20. A game system comprising a real time camera tracker as claimed in claim 11, the game system being arranged to influence the course of a game in relation to location and orientation of one or more mobile depth cameras tracked by the real time camera tracker.

17. A method of real-time camera tracking comprising:

receiving a sequence of depth map frames from a moving mobile depth camera each depth map frame comprising a depth value at each image element that depth value being related to a distance from the mobile depth camera to a surface in the environment captured by the mobile depth camera;

accessing a signed distance function representing surfaces in the environment;

tracking the position and orientation of the mobile depth camera by computing, for each of a plurality of the depth map frames, an updated position and orientation of the mobile depth camera on the basis of an alignment of the image elements of the depth map frame and at least part of the signed distance function.

18. A method as claimed in claim 17 comprising computing the alignment of the image elements of the depth map frame and a surface depicted in the 3D model of the environment by using a parallel computing device to accumulate quantities related to the alignment over image elements in parallel.

19. A method as claimed in claim 17 comprising computing the alignment of the image elements of the depth map frame and a surface depicted in the 3D model of the environment by using a graphics processing unit (GPU) to carry out a map reduce operation to compute quantities related to the alignment per image element.