Method and apparatus for synthesizing new video and/or still imagery from a collection of real video and/or still imagery

• US 7,085,409 B2
• Filed: 10/16/2001
• Issued: 08/01/2006
• Est. Priority Date: 10/18/2000
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: Sarnoff Corporation (SRI International, Inc.)
• Inventors: Tao, Hai, Kumar, Rakesh, Hanna, Keith, Sawhney, Harpreet Singh
• Primary Examiner(s): Bali, Vikkram
• Assistant Examiner(s): TUCKER, WESLEY J
• Application Number:
  US09/978,158
• Publication Number:
  US 20020061131A1
• Time in Patent Office: 1,750 Days
• Field of Search: 345/955, 345/419, 345/422, 382/285, 382/154, 348/42, 348/47, 352/86
• US Class Current: 382/154
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 2210/44 Morphing
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85

Abstract

An image-based tele-presence system forward warps video images selected from a plurality fixed imagers using local depth maps and merges the warped images to form high quality images that appear as seen from a virtual position. At least two images, from the images produced by the imagers, are selected for creating a virtual image. Depth maps are generated corresponding to each of the selected images. Selected images are warped to the virtual viewpoint using warp parameters calculated using corresponding depth maps. Finally the warped images are merged to create the high quality virtual image as seen from the selected viewpoint. The system employs a video blanket of imagers, which helps both optimize the number of imagers and attain higher resolution. In an exemplary video blanket, cameras are deployed in a geometric pattern on a surface.

Claims

1. In a system using a plurality of fixed imagers covering a scene, method to create a high quality virtual image, in real-time, as seen from a virtual viewpoint of the scene, comprising the steps of;

a) selecting at least two images corresponding to at least two of the plurality of fixed imagers to be used in creating the high quality virtual image each of the at least two images corresponding to respective viewpoints different from the virtual viewpoint;

b) creating at least two depth maps corresponding to the at least two images;

c) determining at least two sets of warp parameters using the at least two depth maps corresponding to said at least two images, each set of warp parameters corresponding to warping one of the at least two images to the virtual viewpoint;

d) warping the at least two images to generate at least two warped images representing the virtual viewpoint using the at least two sets of warp parameters corresponding to said at least two images; and

e) merging the at least two warped images to create the high quality virtual image representing the virtual viewpoint.

2. The method of claim 1 further comprising the step of selecting the virtual viewpoint based on data supplied by an operator.

4. The method of claim 1 wherein the steps of creating the at least two depth maps comprises the steps of;

b1) calculating a plurality of optical flow values between the at least two images;

b2) calculating a plurality of parallax values corresponding to a plurality of image coordinates in the at least two images from the plurality of optical flow values; and

b3) calculating the at least two depth maps from the plurality of image coordinates and the plurality of parallax values.

5. The method of claim 1 wherein creating the at least two depth maps comprises the steps of;

b1) mounting a plurality of depth sensing sensors viewing the scene coincident with the plurality of fixed imagers;

b2) selecting at least two depth sensing sensors corresponding to the at least two images;

b3) measuring a plurality of depth values corresponding to a plurality of image coordinates in the at least two images with said at least two depth sensing sensors; and

b4) creating the at least two depth maps from the plurality of depth values.

6. The method of claim 1 wherein creating the at least two depth maps comprises the steps of;

b1) separating the at least two images into a plurality of segments, pixels of each segment having substantially homogenous values;

b2) calculating a depth value corresponding to each segment;

b3) optimizing the depth values corresponding to each segment; and

b4) creating the at least two depth maps from the plurality of optimized depth values.

7. The method of claim 1 wherein the step of selecting the at least two image is based on a proximity of the virtual viewpoint to respective viewpoints corresponding to the at least two images.

8. The method of claim 1 wherein the step of selecting the at least two image selects exactly two images.

9. The method of claim 1 wherein the step of selecting the at least two images selects exactly three images.

10. The method of claim 9 wherein the exactly three images correspond to three fixed imagers from among the plurality affixed imagers arranged in a triangle.

11. The method of claim 1 further comprising the step of placing the plurality of fixed imagers in a geometric pattern.

3. In a system using a plurality of fixed imagers covering a scene, a method to create a high quality virtual image, in real-time, as seen from a virtual viewpoint of the scene, comprising the steps of;

a) selecting the virtual viewpoint based on tracking at least one feature as the at least one feature moves within the scene;

b) selecting at least two images corresponding to at least two of the plurality of fixed imagers to be used in creating the high quality virtual image each of the least two images corresponding to respective viewpoints different from the virtual viewpoint;

c) creating at least two depth maps corresponding to the at least two images;

d) determining at least two sets of warp parameters using the at least two depth maps corresponding to said at least two images, each set of warp parameters corresponding to warping one of the at least two images to the virtual viewpoint;

e) warping the at least two images to generate at least two warped images representing the virtual viewpoint using the at least two sets of warp parameters corresponding to said at least two images; and

f) merging the at least two warped images to create the high quality virtual image representing the virtual viewpoint.

12. A virtual camera system to create a high quality virtual image, in real-time, as seen from a virtual viewpoint, comprising;

plurality of fixed imagers;

image selection means for selecting an image from each of at least two of the plurality of fixed cameras for use in creating the high quality virtual image each of the selected images corresponding to a respective viewpoint different from the virtual viewpoint;

depth estimation means for creating at least two depth maps corresponding to the at least two images;

calculation means for calculating, based on the at least two depth maps, at least two sets of warp parameters that define respective warnings of the at least two images to the virtual viewpoint;

an image warper which applies the at least two sets of warp parameters from the calculation means to the at least two images respectively to create at least two warped images; and

an image merger to merge the at least two warped images to generate the high quality virtual image representing the virtual viewpoint.

13. The system of claim 12 wherein the depth estimation means includes view-based volumetric mapping means to create depth maps of the images.

14. The system of claim 12 wherein the depth estimation means includes color segmentation depth calculation means to create depth maps of the images.

15. The system of claim 12 further comprising a plurality of depth sensing sensors aligned to view the scene coincident with the plurality of fixed imagers, whereby the at least two depth maps are generated using data provided by the plurality of depth sensing sensor.