Method of pose estimation and model refinement for video representation of a three dimensional scene
First Claim
1. A method for accurately estimating a pose of a camera within a scene using a three dimension model of the scene, comprising the steps of:
- (a) generating an initial estimate of the pose;
(b) selecting a set of relevant features of the three dimensional model based on the initial estimate of the pose;
(c) creating a virtual projection of the set of relevant features responsive to the initial estimate of the pose;
(d) matching a plurality of features of an image received from the camera to the virtual projection of the set of relevant features and measuring a plurality of matching errors; and
(e) updating the estimate of the pose to reduce the plurality of matching errors.
3 Assignments
0 Petitions
Accused Products
Abstract
The present invention is embodied in a video flashlight method. This method creates virtual images of a scene using a dynamically updated three-dimensional model of the scene and at least one video sequence of images. An estimate of the camera pose is generated by comparing a present image to the three-dimensional model. Next, relevant features of the model are selected based on the estimated pose. The relevant features are then virtually projected onto the estimated pose and matched to features of the image. Matching errors are measured between the relevant features of the virtual projection and the features of the image. The estimated pose is then updated to reduce these matching errors. The model is also refined with updated information from the image. Meanwhile, a viewpoint for a virtual image is selected. The virtual image is then created by projecting the dynamically updated three-dimensional model onto the selected virtual viewpoint.
147 Citations
42 Claims
-
1. A method for accurately estimating a pose of a camera within a scene using a three dimension model of the scene, comprising the steps of:
-
(a) generating an initial estimate of the pose;
(b) selecting a set of relevant features of the three dimensional model based on the initial estimate of the pose;
(c) creating a virtual projection of the set of relevant features responsive to the initial estimate of the pose;
(d) matching a plurality of features of an image received from the camera to the virtual projection of the set of relevant features and measuring a plurality of matching errors; and
(e) updating the estimate of the pose to reduce the plurality of matching errors. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)
-
-
12. A method for accurately estimating a pose of an image of a scene using a three dimension model of the scene, comprising the steps of:
-
(a) performing a pyramid decomposition of the image to generate a set of pyramid levels of the image;
(b) selecting one pyramid level from the set of pyramid levels;
(c) generating an estimate of the pose using the selected pyramid level;
(d) selecting a set of relevant features of the three dimensional model based on the estimate of the pose;
(e) creating a virtual projection of the set of relevant features responsive to the estimate of the pose;
(f) matching a plurality of features of the selected pyramid level to the virtual projection of the set of relevant features and measuring a plurality of matching errors;
(g) updating the estimate of the pose to reduce the plurality of matching errors; and
(h) repeating to steps (e), (f), and (g) using the updated estimate of the pose until the plurality of matching errors are less than predetermined matching criteria which is responsive to the selected pyramid level. - View Dependent Claims (13, 14)
-
-
15. A method for refining a three dimensional model of a scene using an image of the scene taken by a camera having an unknown pose, comprising the steps of:
-
(a) comparing the image to the three dimension model of the scene to generate an estimate of the pose; and
(b) updating the three dimensional model of the scene based on data from the image and the estimate of the pose. - View Dependent Claims (16)
-
-
17. A method for accurately estimating a position of remote vehicle using a three dimension model and an image from a camera having a known orientation relative to the remote vehicle, comprising the steps of:
-
(a) comparing the image to the three dimension model of the scene to generate an estimate of the pose;
(b) selecting a set of relevant features of the three dimensional model based on the estimate of the pose;
(c) matching a plurality of features of the image to the set of relevant features and measuring a plurality of matching errors;
(d) updating the estimate of the pose based on the plurality of matching errors; and
(e) determining the position of the remote vehicle based on the estimate of the pose and the orientation of the camera. - View Dependent Claims (18)
-
-
19. A method for refining a three dimension model of a scene containing an object using a plurality of images of the scene, each image including the object, comprising the steps of:
-
(a) comparing a first image of the plurality of images to the three dimension model of the scene to generate an estimate of a first viewpoint corresponding to the first image;
(b) comparing a second image of the plurality of images to the three dimension model of the scene to generate an estimate of a second viewpoint corresponding to the second image;
(c) selecting a first set of relevant features of the three dimensional model based on the first viewpoint;
(d) matching a plurality of first features of the first image to the first set of relevant features and measuring a plurality of first matching errors;
(e) selecting a second set of relevant features of the three dimensional model based on the second viewpoint;
(f) matching a plurality of second features of the second image to the second set of relevant features and measuring a plurality of second matching errors; and
(g) updating a position estimate of the object within the three dimensional model of the scene based on the plurality of first matching errors and the plurality of second matching errors. - View Dependent Claims (20, 21, 22, 23)
-
-
24. A method for refining a three dimension model of a scene containing an object using a plurality of images of the scene, each image including the object, comprising the steps of:
-
(a) selecting a subset of images from the plurality of images of the scene, the subset of frames containing at least two of the images;
(b) determining a plurality of approximate relative viewpoints of the subset of images;
(c) comparing each image in the subset of images to the three dimensional model to generate a subset of estimated viewpoints corresponding to the subset of images, the subset of estimated viewpoints constrained by the plurality of approximate relative viewpoints;
(d) selecting a set of relevant features of the three dimensional model corresponding to each estimated viewpoint;
(e) matching a plurality of features of the each image in the subset of images to the corresponding set of relevant features and measuring a plurality of matching errors; and
(f) updating a position estimate of the object within the three dimensional model of the scene based on the plurality of matching errors. - View Dependent Claims (25, 26)
-
-
27. A method for creating a hybrid three dimension model of a scene using a plurality of images of the scene, comprising the steps of:
-
(a) creating a polyhedral model the scene including at least one polygonal surface;
(b) determining a first set of images containing at least a section of a first polygonal surface; and
(c) comparing a plurality of images selected from the first set of images to generate a local surface shape map corresponding to the first polygonal surface. - View Dependent Claims (28, 29, 30, 31)
-
-
32. A method for creating a textured three dimension model of a scene using a plurality of images of the scene, comprising the steps of:
-
(a) creating a polyhedral model the scene including at least one polygonal surface;
(b) identifying at least one portion of the one polygonal surface;
(c) determining a first subset of images containing the at least one portion of the one polygonal surface;
(d) selecting at least one selected image of the first subset of images;
(e) projecting a corresponding section of each of the at least one selected image onto the at least one portion of the one polygonal surface of the textured three dimension model as a local color map;
(f) repeating steps (c), (d), and (e) for each remaining portion of the one polygonal surface of the polyhedral model - View Dependent Claims (33, 34, 35, 36)
-
-
37. A method for creating a dynamic sequence of virtual images of a scene using a dynamically updated three dimension model of the scene, comprising the steps of:
-
(a) updating the three dimension model using a video sequence of images of the scene including the steps of;
(a1) determining a present viewpoint of a present image of the video sequence of images;
(a2) determining a relevant portion of the three dimensional model corresponding to the present image of the video sequence of images;
(a3) updating the relevant portion of the three dimensional model by projecting the present image onto the relevant portion of the three dimension model;
(b) selecting a first virtual viewpoint of a first virtual image of the dynamic sequence of virtual images;
(c) creating the first virtual image by projecting the dynamic three dimensional model onto the first virtual viewpoint; and
(d) repeating steps (b) and (c) for each remaining virtual image of the dynamic sequence of virtual images. - View Dependent Claims (38, 39, 40)
-
-
41. A computer readable medium adapted to instruct a general purpose computer to update a three dimensional model of a scene using the three dimensional model of the scene, an image received from a camera having an unknown pose, the method comprising the steps of:
-
(a) generating an estimate of the pose;
(b) selecting a set of relevant features of the three dimensional model based on the estimate of the pose;
(c) creating a virtual projection of the set of relevant features responsive to the estimate of the pose;
(d) matching a plurality of features of an image received from the camera to the virtual projection of the set of relevant features and measuring a plurality of matching errors;
(e) updating the estimate of the pose to reduce the plurality of matching errors; and
(f) updating the three dimensional model of the scene based on data from the image and the estimate of the pose.
-
-
42. An automatic three-dimensional model updating apparatus for accurately estimating a point of view of an image of a scene, relative to a three-dimensional model of the scene, and updating the three-dimensional model comprising:
-
(a) estimating means for providing an estimate of the point of view of the image;
(b) relevant feature selecting means for selecting a set of relevant features of the three dimensional model based on the estimate of the point of view;
(c) virtual projection means for creating a virtual projection of the set of relevant features responsive to the estimate of the point of view;
(d) matching means for matching a plurality of features of the image to the virtual projection of the set of relevant features;
(e) measurement means for measuring a plurality of is matching errors;
(f) point of view refinement means for updating the estimate of the point of view to reduce the plurality of matching errors; and
(g) model refinement means, responsive to the estimated point of view and to the image, for updating the three-dimensional model.
-
Specification