VIRTUAL VIEW SPECIFICATION AND SYNTHESIS IN FREE VIEWPOINT

• US 20070109300A1
• Filed: 08/03/2006
• Published: 05/17/2007
• Est. Priority Date: 11/15/2005
• Status: Active Grant

• Current Assignee: Sharp Kabushiki Kaisha (Hon Hai Precision Industry Co., Ltd.)
• Original Assignee: Sharp Laboratories of America Incorporated (Hon Hai Precision Industry Co., Ltd.)
• Inventors: Li, Baoxin
• Granted Patent: US 7,471,292 B2
• Time in Patent Office: Days
• US Class Current: 345/427
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 2207/20088 Trinocular vision calculations; trifocal tensor
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85

Abstract

A system that receives a first video stream of a scene having a first viewpoint and a second video stream having a second viewpoint wherein camera calibration between the first viewpoint and the second viewpoint is unknown. A viewer selects a viewer viewpoint generally between the first viewpoint and the second viewpoint, and the system synthesizes the viewer viewpoint based upon the first video stream and the second video stream.

Claims

1. A method for synthesizing a viewpoint comprising:

(a) receiving a first video stream of a scene having a first viewpoint;

(b) receiving a second video stream having a second viewpoint wherein camera calibration between said first viewpoint and said second viewpoint is unknown;

(c) a viewer selecting a viewer viewpoint generally between said first viewpoint and said second viewpoint; and

(d) synthesizing said viewer viewpoint based upon said first video stream and said second video stream.

2. The method of claim 1 wherein said viewer selects said first viewpoint and said second viewpoint from a group of three or more video streams each of which has a different viewpoint.

3. The method of claim 1 further comprising receiving a third video stream having a third viewpoint, and a pair-wise calibration is determined for each pair of said first, second, and third viewpoints.

4. The method of claim 3 further comprising selectively excluding said calibration of one of said viewpoints.

5. The method of claim 1 further comprising color based segmentation between said first viewpoint and said second viewpoint.

6. The method of claim 1 further comprising forward warping from said first and second viewpoints to a virtual view based upon a disparity map.

7. The method of claim 6 further comprising using a backward search based upon a third viewpoint to find a dominant and disparity consistent color.

8. The method of claim 1 wherein a relationship between said first and second viewpoints is determined based upon a feature detector and a random sample consensus.

9. The method of claim 6 wherein said disparity is based upon an epipolar constraint.

10. The method of claim 1 wherein said viewer viewpoint is specified by a translation and a rotation.

11. The method of claim 1 wherein said viewer viewpoint is selectable by the user from a plurality of potential viewpoints.