Camera pose estimation for 3D reconstruction
First Claim
1. A method of real-time camera tracking comprising:
- receiving, at a processor, a sequence of depth map frames from a moving mobile depth camera each depth map frame comprising a depth value at each image element that depth value being related to a distance from the mobile depth camera to a surface in the environment captured by the mobile depth camera;
accessing, by the processor, a 3D model of surfaces in the environment;
tracking the position and orientation of the mobile depth camera by computing, at the processor, for each of a plurality of the depth map frames, an updated position and orientation of the mobile depth camera according to an alignment of the image elements of the depth map frame and a surface depicted in the 3D model of the environment, the alignment being based at least in part on at least one of;
an energy of a function of a camera pose registration parameter vector being equal to one half of the sum over image elements of the square of a truncated signed distance given by the 3D model at the position of the current image element transformed by a current pose estimate;
oran energy of a function of a camera pose registration parameter vector being equal to the sum over image elements of the squares of the differences between, a depth component of the 3D points in a view space of the image elements of a previous frame, times a rigid transformation from the view space of the previous frame to that of a current frame, and the 3D points in a view space of an image element of the current frame, given a function which represents the projection of a 3D view space coordinate to a 2D image coordinate, where the projection of the 3D view space coordinate is given by a corresponding depth value of the image element of the previous frame times the rigid transformation from the view space of the previous frame to that of the current frame.
5 Assignments
0 Petitions
Accused Products
Abstract
Camera pose estimation for 3D reconstruction is described, for example, to enable position and orientation of a depth camera moving in an environment to be tracked for robotics, gaming and other applications. In various embodiments, depth observations from the mobile depth camera are aligned with surfaces of a 3D model of the environment in order to find an updated position and orientation of the mobile depth camera which facilitates the alignment. For example, the mobile depth camera is moved through the environment in order to build a 3D reconstruction of surfaces in the environment which may be stored as the 3D model. In examples, an initial estimate of the pose of the mobile depth camera is obtained and then updated by using a parallelized optimization process in real time.
-
Citations
20 Claims
-
1. A method of real-time camera tracking comprising:
-
receiving, at a processor, a sequence of depth map frames from a moving mobile depth camera each depth map frame comprising a depth value at each image element that depth value being related to a distance from the mobile depth camera to a surface in the environment captured by the mobile depth camera; accessing, by the processor, a 3D model of surfaces in the environment; tracking the position and orientation of the mobile depth camera by computing, at the processor, for each of a plurality of the depth map frames, an updated position and orientation of the mobile depth camera according to an alignment of the image elements of the depth map frame and a surface depicted in the 3D model of the environment, the alignment being based at least in part on at least one of; an energy of a function of a camera pose registration parameter vector being equal to one half of the sum over image elements of the square of a truncated signed distance given by the 3D model at the position of the current image element transformed by a current pose estimate;
oran energy of a function of a camera pose registration parameter vector being equal to the sum over image elements of the squares of the differences between, a depth component of the 3D points in a view space of the image elements of a previous frame, times a rigid transformation from the view space of the previous frame to that of a current frame, and the 3D points in a view space of an image element of the current frame, given a function which represents the projection of a 3D view space coordinate to a 2D image coordinate, where the projection of the 3D view space coordinate is given by a corresponding depth value of the image element of the previous frame times the rigid transformation from the view space of the previous frame to that of the current frame. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10)
-
-
11. A real-time camera tracker comprising:
-
a communications interface arranged to receive a sequence of depth map frames from a moving mobile depth camera each depth map frame comprising a depth value at each image element that depth value being related to a distance from the mobile depth camera to a surface in the environment captured by the mobile depth camera; the communications interface being arranged to access a 3D model of surfaces in the environment; a camera pose engine arranged to track the position and orientation of the mobile depth camera by computing, for each of a plurality of the depth map frames, an updated position and orientation of the mobile depth camera according to an alignment of the image elements of the depth map frame and a surface depicted in the 3D model of the environment, the alignment being based at least in part on at least one of; an energy of a function comprising terms related at least to the square of a truncated signed distance, a transformation matrix, and a formatting matrix;
oran energy of a function comprising terms related at least to a depth component of the 3D points in a view space of the image elements of a previous frame, a rigid transformation from the view space of the previous frame to that of a current frame, and a projection of a 3D view space coordinate to a 2D image coordinate. - View Dependent Claims (12, 13, 14, 15, 16, 17)
-
-
18. A computer storage media storing device-executable instructions that, when executed, cause a system to execute steps comprising:
-
receiving, at a processor, a sequence of depth map frames from a moving mobile depth camera each depth map frame comprising a depth value at each image element that depth value being related to a distance from the mobile depth camera to a surface in the environment captured by the mobile depth camera; accessing, by the processor, a signed distance function representing surfaces in the environment; and tracking the position and orientation of the mobile depth camera by computing, at the processor, for each of a plurality of the depth map frames, an updated position and orientation of the mobile depth camera on the basis of an alignment of the image elements of the depth map frame and at least part of the signed distance function, the alignment being based at least in part on at least one of; an energy of a function comprising terms related at least to the square of a truncated signed distance, a transformation matrix, and a formatting matrix;
oran energy of a function comprising terms related at least to a depth component of the 3D points in a view space of the image elements of a previous frame, a rigid transformation from the view space of the previous frame to that of a current frame, and a projection of a 3D view space coordinate to a 2D image coordinate. - View Dependent Claims (19, 20)
-
Specification