2-D extended image generation from 3-D data extracted from a video sequence

• US 6,504,569 B1
• Filed: 04/22/1998
• Issued: 01/07/2003
• Est. Priority Date: 04/22/1998
• Status: Expired due to Term

• Current Assignee: Grass Valley USA LLC (Vislink Technologies, Inc.)
• Original Assignee: Grass Valley USA LLC (Vislink Technologies, Inc.)
• Inventors: Jasinschi, Radu S., Naveen, Thumpudi, Babic-Vovk, Paul T., Tabatabai, Ali
• Primary Examiner(s): DIEP, NHON THANH
• Application Number:
  US09/064,889
• Time in Patent Office: 1,721 Days
• Field of Search: H04/N 1.300, 345/426, 345/427, 345/419, 345/502, 345/586, 345/952, 345/949, 345/950, 345/951, 348/43, 348/42, 348/47, 348/36, 348/37, 348/38
• US Class Current: 348/43
• CPC Class Codes: G06T 2200/32 involving image mosaicing
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: Y10S 345/95 Sprite processing

Abstract

A method of generating a 2-D extended image from a video sequence representing a natural 3-D scene first determines motion parameters for a camera that recorded the scene with respect to a bakcground object from the video sequence using a structure-from-motion algorithm. The motion parameters include a rotation matrix, a translation vector and a depth map representing the depth of each point in the background object from the camera. Next from the motion parameters and depth map the 2-D extended image is generated for the background object as a composition of the images from the video sequence using a plane perspective projection technique. The background object may be layered as a function of depth and flatness criteria to form a set of layered 2-D extended images for the background object from the video sequence.

Claims

1. A method of generating a 2-D extended image from a video sequence representing a natural 3-D scene comprising the steps of:

for a background object segmented from the video sequence, determining motion parameters for a camera that recorded the video sequence with respect to a 3-D coordinate system, and from the motion parameters determining a depth map representing the depth of each point in the background object from the camera; and

from the motion parameters and depth map, generating the 2-D extended image for the background object as a composition of contiguous images from the video sequence, wherein the determining step comprises the steps of;

extracting feature points for the background object from an image of the video sequence;

tracking the features points for the background object into a next image of the video sequence to produce feature point correspondence pairs;

performing a structure-from-motion algorithm on the feature point correspondence pairs to produce a rotation matrix and a translation vector as the motion parameters as well as a depth value for each feature point in the image; and

from the depth values generating the depth map.

2. The method as recited in claim 1 further comprising the step of pre-segmenting the background object from the video sequence prior to the determining step.

3. The method as recited in claim 2 further comprising the step of detecting scene cuts in the video sequence to divide the video sequence into a plurality of scene shots, each scene shot being processed separately by the pre-segmenting, determining and generating steps.

4. The method as recited in claim 1 wherein the tracking step further comprises the step of removing outliers from the feature point correspondence pairs prior to the performing step.

5. The method as recited in claim 1 wherein the performing step comprises the steps of:

pre-processing the feature point correspondence pairs to produce a plurality of sets of estimated motion parameters; and

selecting from among the sets of estimated motion parameters a best set as the motion parameters.

6. The method as recited in claim 5 wherein the selecting step comprises the step of computing an overall velocity estimation error from input and output velocity vectors for every feature point, with the combination that minimizes the overall velocity estimation error being selected as the best set.

7. The method as recited in claim 5 wherein the selecting step comprises the step of statistically finding the best set from among the sets of estimated motion parameters.

8. The method as recited in claim 1 further comprising the step of deciding from the motion parameters and depth map whether to proceed with the generating step based upon a flatness criterion for the object.

9. The method as recited in claim 8 wherein the deciding step further comprises the step of providing a segmentation signal for use in further segmenting the background object.

10. A method of generating a 2-D extended image from a video sequence representing a natural 3-D scene comprising the steps of:

for a background object segmented from the video sequence, determining motion parameters for a camera that recorded the video sequence with respect to a 3-D coordinate system, and from the motion parameters determining a depth map representing the depth of each point in the background object from the camera; and

from the motion parameters and depth map, generating the 2-D extended image for the background object as a composition of contiguous images from the video sequence, wherein the generating step comprises the steps of;

predicting a next image of the background object as a coarse predicted image from a current version of the 2-D extended image;

refining the coarse predicted image to generate a predicted image;

pasting the predicted image to the current version of the 2-D extended image to generate the 2-D extended image; and

repeating the predicting, refining and pasting steps until an end condition is achieved.

11. The method as recited in claim 10 wherein the predicting step comprises the step of integrating the current version of the 2-D extended image with the motion parameters to produce the coarse predicted image.

12. The method as recited in claim 10 further comprising deciding from the 2-D extended image relative to a first image whether the end condition is achieved.

13. The method as recited in claim 10 further comprising deciding from the output of a scene-cut detector to which the video sequence is input whether a scene-cut has occurred as the end condition.

14. The method as recited in claim 10 further comprising the step of pre-segmenting the background object from the video sequence prior to the determining step.

15. The method as recited in claim 14 further comprising the step of detecting scene cuts in the video sequence to divide the video sequence into a plurality of scene shots, each scene shot being processed separately by teh pre-segmenting, determining and generating steps.

16. The method as recited in claim 10 further comprising the step of deciding from the motion parameters and depth map whether to proceed with the generating step based upon a flatness criterion for the object.

17. The method as recited in claim 16 wherein the deciding step further comprises the step of providing a segmentation signal for use in further segmenting the background object.