Method and apparatus for multi-view three dimensional estimation

• US 6,571,024 B1
• Filed: 06/18/1999
• Issued: 05/27/2003
• Est. Priority Date: 06/18/1999
• Status: Expired due to Term

• Current Assignee: SRI International, Inc.
• Original Assignee: Sarnoff Corporation (SRI International, Inc.)
• Inventors: Guo, Yanlin, Kumar, Rakesh, Sawhney, Harpreet Singh, Hanna, Keith James, Asmuth, Jane
• Primary Examiner(s): Mehta, Bhavesh
• Assistant Examiner(s): PATEL, KANJIBHAI B
• Application Number:
  US09/336,319
• Time in Patent Office: 1,439 Days
• Field of Search: 382/151, 382/154, 382/190, 382/195, 382/201, 382/203, 382/162, 382/206, 382/254, 382/274, 382/276, 382/278, 382/284, 382/285, 382/294, 382/295, 382/312, 382/296, 345/418, 345/419, 345/672, 345/679, 345/428, 345/629, 345/630, 345/648, 345/683, 348/42, 348/49, 348/50, 348/263, 348/207.99, 348/580
• US Class Current: 382/294
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/248 involving reference images or patches

Abstract

An apparatus and method for generating automated multi-view three dimensional pose and geometry estimation for the insertion of realistic and authentic views of synthetic objects into a real scene. A multi-view three dimensional estimation routine comprising the steps of feature tracking, pairwise camera pose estimation, computing camera pose for overlapping sequences and performing a global block adjustment to provide camera pose and scene geometric information for each frame of a scene. A match move routine may be used to insert a synthetic object into one frame of a video sequence based on the pose and geometric information of the frame, and calculate all other required object views of the synthetic object for the remaining frames using the pose and geometric information acquired as a result of the multi-view three dimensional estimation routine.

Claims

1. A method for camera pose estimation comprising:

(a) tracking features from frame-to-frame in a video sequence;

(b) estimating pairwise camera poses for frame pairs having overlapping features within said video sequence;

(c) computing camera poses for overlapping subsequences within said video sequence based on the pairwise camera pose estimates;

(d) combining said overlapping subsequences to form longer sequences having a consistent coordinate system using camera pose consistency across the overlapping subsequences; and

(e) performing a global adjustment to the longer sequences to produce final camera pose estimates for said longer sequences.

2. The method of claim 1 further comprising:

(f) inserting a synthetic object using the final camera pose estimates to create an authentic projection of said synthetic object in the video sequence.

3. The method of claim 1 further wherein the tracking step further comprises:

(g) selecting new features in every frame in a sequence having frames T_N;

(h) computing an auto-correlation surface for said selected features;

(i) computing displacement fields for said auto-correlation surfaces of said selected features of said frames T₀ to T_t+1;

(j) checking flow consistency for said auto-correlation surfaces of said selected features of said frames T₀ to T_t+1;

(k) keeping said auto-correlation surfaces of said selected features which are consistent for future tracking;

(l) repeating steps (h), (i) and (j); and

(m) keeping consistent features for future tracking until N=t+1.

4. The method of claim 1 wherein the tracking step further comprises selecting a plurality of key frames for performing tracking.

5. The method of claim 4 wherein the key frames are selected as frames with sufficient parallax motion between frames, as frames that transition between overlapping sets of correspondences, or frames that are regularly sampled if the motion from frame-to-frame is smooth.

6. The method of claim 1 wherein the step of estimating pairwise camera poses comprises:

(b1) computing an initial matrix of camera poses;

(b2) rejecting outliers of said initial matrix;

(b3) computing a final matrix of camera poses using an image based error measurement;

(b4) decomposing said final matrix into a camera pose matrix; and

(b5) converting said camera pose matrix into Euclidean coordinates.

7. The method of claim 1 wherein the step of estimating pairwise camera poses comprises:

(b6) computing an initial matrix of camera poses;

(b7) rejecting outliers of said initial matrix;

(b8) computing a final matrix of camera poses using image based error measurement;

(b9) computing Euclidean pose coordinates.

8. The method of claim 1 wherein the step of computing camera poses for overlapping subsequences comprises:

creating subsequences having consistent camera poses; and

optimizing the subsequences by applying a bundle block adjustment to each subsequence.

9. Apparatus for performing camera pose estimation comprising:

means for frame-to-frame tracking of features within a video sequence;

means for pairwise estimating camera poses from frame pairs within said video sequence having overlapping features;

means for computing camera poses for overlapping subsequences within said video sequence using the pairwise camera pose estimates as initial estimates; and

means for performing a global bundle block adjustment to compute final camera pose estimates.

10. The apparatus of claim 9 further comprising:

means for representing longer sequences in a consistent coordinate system using camera pose consistency across the overlapping subsequences.

11. The apparatus of claim 9 further comprising:

means for inserting a synthetic object into a frame of said video sequence where the orientation and geometry of said synthetic object is based on said computed camera pose for said frame.

12. The apparatus of claim 11 further comprising:

means for placing said synthetic object in a plurality of frames in said video sequence, orienting and sizing said placed synthetic object based upon said inserted frame.

13. A computer-readable medium having stored thereon a plurality of instructions, the plurality of instructions including instructions which, when executed by a processor, cause the processor to perform a method for camera pose estimation comprising:

(a) tracking features from frame-to-frame in a video sequence;

(b) estimating pairwise camera poses for frame pairs having overlapping features within said video sequence;

(c) computing camera poses for overlapping subsequences within said video sequence using the pairwise camera pose estimates as initial estimates;

(d) combining said overlapping subsequences to form longer sequences having a consistent coordinate system using camera pose consistency across the overlapping subsequences; and

(e) performing a global adjustment to the longer sequences to produce final camera pose estimates for said longer sequences.

14. The computer-readable medium of claim 13 further comprising:

(f) inserting a synthetic object using the final camera pose estimates to create an authentic projection of said synthetic object in the video sequence.

15. The computer-readable medium of claim 13 wherein the step of tracking comprises:

(g) selecting new features in every frame in a sequence having frames T_N;

(h) computing an auto-correlation surface for said selected features;

(i) computing displacement fields for said auto-correlation surfaces of said selected features of said frames T₀ to T_t+1;

(j) checking flow consistency for said auto-correlation surfaces of said selected features of said frames T₀ to T_t+1;

(k) keeping said auto-correlation surfaces of said selected features which are consistent for future tracking;

(l) repeating steps (h), (i) and (j); and

(m) keeping consistent features for future tracking until N=t+1.

16. The computer-readable medium of claim 15 further comprising selecting a plurality of key frames for performing tracking, wherein the key frames are selected as frames with sufficient parallax motion between frames, as frames that transition between overlapping sets of correspondences, or frames that are regularly sampled if the motion from frame-to-frame is smooth.

17. The computer-readable medium of claim 13 wherein the step of estimating pairwise camera poses comprises:

(b1) computing an initial matrix of camera poses;

(b2) rejecting outliers of said initial matrix;

(b3) computing a final matrix of camera poses using an image based error measurement;

(b4) decomposing said final matrix into a camera pose matrix; and

(b5) converting said camera pose matrix into Euclidean coordinates.

18. The computer-readable medium of claim 13 wherein the step of computing camera poses for overlapping sequences comprises:

creating subsequences having consistent camera poses; and

optimizing the subsequences by applying a bundle block adjustment to each subsequence.