Method of pose estimation and model refinement for video representation of a three dimensional scene

• US 6,985,620 B2
• Filed: 03/07/2001
• Issued: 01/10/2006
• Est. Priority Date: 03/07/2000
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: Sarnoff Corporation (SRI International, Inc.)
• Inventors: Samarasekera, Supun, Kumar, Rakesh, Sawhney, Harpreet Singh, Hsu, Steve
• Primary Examiner(s): Mehta, Bhavesh M.
• Assistant Examiner(s): CHAWAN, SHEELA C
• Application Number:
  US09/800,550
• Publication Number:
  US 20010043738A1
• Time in Patent Office: 1,770 Days
• Field of Search: 382/103, 382/154, 382/268, 382/284, 382/294, 382/282, 382/296, 382/291, 382/190, 382/172, 382/168, 348/42, 348/47, 348/56, 348/135, 348/139, 348/148, 348/168, 348/580, 348/583, 348/169, 345/425, 345/421, 345/419, 345/475, 345/423, 345/418, 345/630, 345/653, 345/647, 345/634, 345/757, 345/664, 345/420
• US Class Current: 382/154
• CPC Class Codes: G01S 5/163 Determination of attitude using inertial means G01C9/00; control of attitude G05D1/49
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

The present invention is embodied in a video flashlight method. This method creates virtual images of a scene using a dynamically updated three-dimensional model of the scene and at least one video sequence of images. An estimate of the camera pose is generated by comparing a present image to the three-dimensional model. Next, relevant features of the model are selected based on the estimated pose. The relevant features are then virtually projected onto the estimated pose and matched to features of the image. Matching errors are measured between the relevant features of the virtual projection and the features of the image. The estimated pose is then updated to reduce these matching errors. The model is also refined with updated information from the image. Meanwhile, a viewpoint for a virtual image is selected. The virtual image is then created by projecting the dynamically updated three-dimensional model onto the selected virtual viewpoint.

Claims

1. A method for accurately estimating a pose of a camera within a scene using a three dimension model of the scene, comprising the steps of:

(a) generating an initial estimate of the pose;

(b) selecting a set of relevant features of the three dimensional model based on the initial estimate of the pose, the relevant features representing discontinuities in at least one of a surface normal or material properties in the scene;

(c) creating a virtual projection of the set of relevant features responsive to the initial estimate of the pose;

(d) matching a plurality of features of an image received from the camera to the virtual projection of the set of relevant features and measuring a plurality of matching errors; and

(e) updating the estimate of the pose to reduce the plurality of matching errors.

2. The method for claim 1, further comprising the step of:

(f) repeating to steps (c), (d), and (e) using the updated estimate of the pose until the plurality of matching errors are less than predetermined matching criteria.

3. The method of claim 1, further comprising the steps of:

(f) adding the updates to the estimate of the pose made in step (e) to a total change value of the estimate of the pose;

(g) if the total change value is less than predetermined pose change criteria, resetting the total change value and repeating to steps (b), (c), (d), (e) and (f) using the updated estimate of the pose; and

(h) repeating to steps (c), (d), (e), (f) and (g) using the updated estimate of the pose until the plurality of matching errors are less than predetermined matching criteria.

4. The method of claim 1, wherein step (a) includes the step of comparing an image received from the camera to the three dimensional model of the scene to generate the estimate of the pose.

5. The method of claim 1, wherein step (a) includes the step of using a preceding pose estimate of a preceding image received from the camera to generate the estimate of the pose.

6. The method of claim 1, wherein step (b) includes the steps of:

(b1) Z-buffering the three dimensional model based on the estimated pose to determine a set of visible features of the three dimensional model; and

(b2) selecting the set of relevant features from the set of visible features.

7. A method for accurately estimating a pose of a camera within a scene using a three dimension model of the scene, comprising the steps of:

(a) generating an initial estimate of the pose;

(b) selecting a set of relevant features of the three dimensional model based on the initial estimate of the pose including the steps of

(b1) creating a set of features of the three dimensional model including a plurality of edges of at least one object represented in the three dimensional model, each edge having a dihedral angle of greater than a predetermined angle; and

(b2) selecting the set of relevant features from the set of features of the three dimensional model.

(c) creating a virtual projection of the set of relevant features responsive to the initial estimate of the pose;

(d) matching a plurality of features of an image received from the camera to the virtual projection of the set of relevant features and measuring a plurality of matching errors; and

(e) updating the estimate of the pose to reduce the plurality of matching errors.

8. A method for accurately estimating a pose of a camera within a scene using a three dimension model of the scene, comprising the steps of:

(a) generating an initial estimate of the pose;

(b) selecting a set of relevant features of the three dimensional model based on the initial estimate of the pose;

(c) creating a virtual projection of the set of relevant features responsive to the initial estimate of the pose including the step of removing a model feature from the set of relevant features if the model feature is less than a predetermined distance from a remaining relevant feature in the virtual projection.

(d) matching a plurality of features of an image received from the camera to the virtual projection of the set of relevant features and measuring a plurality of matching errors; and

(e) updating the estimate of the pose to reduce the plurality of matching errors.

9. A method for accurately estimating a pose of a camera within a scene using a three dimension model of the scene, comprising the steps of:

(a) generating an initial estimate of the pose;

(b) selecting a set of relevant features of the three dimensional model based on the initial estimate of the pose;

(c) creating a virtual projection of the set of relevant features responsive to the initial estimate of the pose;

(d) matching a plurality of features of an image received from the camera to the virtual projection of the set of relevant features and measuring a plurality of matching errors;

(e) updating the estimate of the pose to reduce the plurality of matching errors; and

(f) perturbing the estimate of the pose and repeating to steps (c), (d), and (e) using the perturbed estimate of the pose until the plurality of matching errors are less than predetermined matching criteria.

10. A method for accurately estimating a pose of a camera within a scene using a three dimension model of the scene, comprising the steps of:

(a) generating an initial estimate of the pose;

(b) selecting a set of relevant features of the three dimensional model based on the initial estimate of the pose, the relevant features representing discontinuities in at least one of a surface normal or material properties in the scene;

(c) creating a virtual projection of the set of relevant features responsive to the initial estimate of the pose;

(d) matching a plurality of features of an image received from the camera to the virtual projection of the set of relevant features and measuring a plurality of matching errors including the steps of;

(d1) computing an oriented energy image of the image received from the camera; and

(d2) integrating the oriented energy image along the virtual projection of the set of relevant features and measuring the plurality of matching errors; and

(e) updating the estimate of the pose to reduce the plurality of matching errors.

11. A method for accurately estimating a pose of a camera within a scene using a three dimension model of the scene, comprising the steps of:

(a) generating an initial estimate of the pose;

(b) selecting a set of relevant features of the three dimensional model based on the initial estimate of the pose;

(c) creating a virtual projection of the set of relevant features responsive to the initial estimate of the pose;

(d) matching a plurality of features of an image received from the camera to the virtual projection of the set of relevant features and measuring a plurality of matching errors including the steps of;

(d1) computing an oriented energy image of the image received from the camera;

(d2) generating a set of scaled oriented energy images;

(d3) selecting a scaled oriented energy image from the set of scaled oriented energy images; and

(d4) integrating the selected scaled oriented energy image along the virtual projection of the set of relevant features and measuring the plurality of matching errors.

(e) updating the estimate of the pose to reduce the plurality of matching errors.

12. A method for refining a three dimensional model of a scene using an image of the scene taken by a camera having an unknown pose, comprising the steps of:

(a) comparing the image to a set of relevant features of the three dimension model of the scene to generate an estimate of the pose, the relevant features representing discontinuities in at least one of the surface normal or the material properties in the scene; and

(b) updating the three dimensional model of the scene based on data from the image and the estimate of the pose.

13. The method claim 12, wherein step (b) includes the step of:

(b1) projecting a plurality of color values of the image onto the three dimension model of the scene to update a texture of a surface of the three dimensional model.

14. A method for refining a three dimension model of a scene containing an object using a plurality of images of the scene, each image including the object, comprising the steps of:

(a) comparing a first image of the plurality of images to the three dimension model of the scene to generate an estimate of a first viewpoint corresponding to the first image;

(b) comparing a second image of the plurality of images to the three dimension model of the scene to generate an estimate of a second viewpoint corresponding to the second image;

(c) selecting a first set of relevant features of the three dimensional model based on the first viewpoint;

(d) matching a plurality of first features of the first image to the first set of relevant features and measuring a plurality of first matching errors;

(e) selecting a second set of relevant features of the three dimensional model based on the second viewpoint;

(f) matching a plurality of second features of the second image to the second set of relevant features and measuring a plurality of second matching errors; and

(g) updating a position estimate of the object within the three dimensional model of the scene based on the plurality of first matching errors and the plurality of second matching errors.

15. The method of claim 14, further including the steps of:

comparing the first image and the second image to generate relative pose constraints; and

updating the first viewpoint and the second viewpoint based on the relative pose constraints.

16. The method of claim 15, wherein step of comparing the first image to the second image includes the steps of:

matching the plurality of first features or the first image to the plurality of second features of the second feature to create a set of matched features;

calculating optical flow for each matched feature; and

generating the relative pose constraints responsive to the calculated optical flow.

17. The method of claim 15, wherein the step of comparing the first image to the second image includes the steps of;

generating a plurality of parallax measurements between the plurality of first features and the plurality of second features; and

generating the relative pose constraints responsive to the plurality of parallax measurements.

18. The method of claim 15, wherein step of comparing the first image to the second image includes the step of generating the relative pose constraints uses an epipolar geometry of the first image and the second image.

19. A method for refining a three dimension model of a scene containing an object using a plurality of images of the scene, each image including the object, comprising the steps of:

(a) selecting a subset of images from the plurality of images of the scene, the subset of frames containing at least two of the images;

(b) determining a plurality of approximate relative viewpoints of the subset of images;

(c) comparing each image in the subset of images to the three dimensional model to generate a subset of estimated viewpoints corresponding to the subset of images, the subset of estimated viewpoints constrained by the plurality of approximate relative viewpoints;

(d) selecting a set of relevant features of the three dimensional model corresponding to each estimated viewpoint;

(e) matching a plurality of features of the each image in the subset of images to the corresponding set of relevant features and measuring a plurality of matching errors; and

(f) updating a position estimate of the object within the three dimensional model of the scene based on the plurality of matching errors.

20. The method of claim 19, further comprising the step of:

(g) repeating to steps (c), (d), (e), and (f) using the updated position estimate of the object until the plurality of matching errors are less than predetermined matching criteria.

21. The method of claim 19, wherein the subset of images is selected to have viewpoints within a predetermined range.

22. A computer readable medium adapted to instruct a general purpose computer to update a three dimensional model of a scene using the three dimensional model of the scene, an image received from a camera having an unknown pose, the method comprising the steps of:

(a) generating an estimate of the pose;

(b) selecting a set of relevant features of the three dimensional model based on the estimate of the pose, the relevant features representing discontinuities in at least one of a surface normal or material properties in the scene;

(c) creating a virtual projection of the set of relevant features responsive to the estimate of the pose;

(d) matching a plurality of features of an image received from the camera to the virtual projection of the set of relevant features, and measuring a plurality of matching errors;

(e) updating the estimate of the pose to reduce the plurality of matching errors; and

(f) updating the three dimensional model of the scene based on data from the image and the estimate of the pose.

23. An automatic three-dimensional model updating apparatus for accurately estimating a point of view of an image of a scene, relative to a three-dimensional model of the scene, and updating the three-dimensional model comprising:

(a) estimating means for providing an estimate of the point of view of the image;

(b) relevant feature selecting means for selecting a set of relevant features of the three dimensional model based on the estimate of the point of view, the relevant features representing discontinuities in at least one of a surface normal or material properties in the scene;

(c) virtual projection means for creating a virtual projection of the set of relevant features responsive to the estimate of the point of view;

(d) matching means for matching a plurality of features of the image to the virtual projection of the set of relevant features;

(e) measurement means for measuring a plurality of matching errors;

(f) point of view refinement means for updating the estimate of the point of view to reduce the plurality of marching errors; and

(g) model refinement means, responsive to the estimated point of view and to the image, for updating the three-dimensional model.