Systems and methods for directly generating a view using a layered approach

• US 20050151759A1
• Filed: 09/08/2004
• Published: 07/14/2005
• Est. Priority Date: 09/08/2003
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company), Regents of the University of California (University of California)
• Original Assignee: Regents of the University of California (University of California)
• Inventors: Tao, Hai, Gonzalez-Banos, Hector H., Kong, Dan
• Granted Patent: US 7,161,606 B2
• Time in Patent Office: Days
• US Class Current: 345/647
• CPC Class Codes: G06T 3/14 Transformations for image registration, e.g. adjusting or mapping for alignment of images

Abstract

Apparatuses and methods are presented that use a direct synthesis approach using layers in order to directly generate a virtual view from a pair of images. In one embodiment, a range space is discretized into a plurality of layers. For each layer, the first input image and the second input image are warped to the layer. A matching score is then determined for each pixel in the new view based on the warped input images. For each pixel in the virtual view, a best layer is determined. After the best layer has been determined, the color of the pixel in that layer is used for that pixel in the generated view.

Claims

1. A method for generating a virtual image of a range space from a first input image of the range space and a second input image of the range space, the method comprising:

discretizing the range space into a plurality of layers;

for each layer of the plurality of layers;

warping the first input image to the layer;

warping the second input image to the layer; and

for each pixel in the virtual image, determining a matching score based on the warped first input image and the warped second input image; and

for each pixel in the virtual image;

determining a layer with a best matching score; and

determining a color of the pixel in the layer with the best matching score.

2. The method of claim 1, further comprising rectifying the first input image and the second input image.

3. The method of claim 1, further comprising locating a viewpoint of the virtual image to be on a baseline.

4. The method of claim 1, further comprising enforcing scene consistency by performing local depth voting.

5. The method of claim 1, further comprising detecting an occlusion in the virtual image by performing local depth voting.

6. The method of claim 1, wherein the plurality of layers comprises a number of layers that is proportional to a volume of the range space.

7. The method of claim 1, wherein the plurality of layers comprises a number of layers that is inversely proportional to a quality of the virtual image.

8. The method of claim 1, wherein warping the first input image to the layer comprises performing inverse warping and bilinear interpolation.

9. The method of claim 1, wherein determining the matching score based on the warped first input image and the warped second input image comprises determining the matching score based on a statistical correlation between the pixel'"'"'s neighborhoods in the warped images.

10. The method of claim 1, wherein determining the matching score based on the warped first input image and the warped second input image comprises:

determining an initial depth; and

restricting a matching search to a range from the initial depth.

11. The method of claim 1, wherein convolution operators are used to determine the matching scores for each pixel in the new view for one depth layer or for a plurality of depth layers.

12. The method of claim 1, wherein hardware acceleration is used to determine the matching scores for each pixel in the new view for one depth layer or for a plurality of depth layers.

13. The method of claim 1, wherein determining the layer with the best matching score comprises determining a layer that yields a greatest matching score.

14. The method of claim 1, wherein determining the color of the pixel in the layer with the best matching score comprises determining a color by a weighted combination of corresponding pixels of the input images according to one of a group containing the pixel'"'"'s depth from a viewpoint of the virtual image and a function of an angle and a distance between the viewpoint of the virtual image and a viewpoint of the input image.

15. A system for generating a virtual image of a range space from a first input image of the range space and a second input image of the range space, the system comprising:

means for discretizing the range space into a plurality of layers;

means for warping the first input image to the layer, for each layer of the plurality of layers;

means for warping the second input image to the layer, for each layer of the plurality of layers;

means for determining a matching score based on the warped first input image and the warped second input image, for each pixel in the virtual image, for each layer of the plurality of layers;

means for determining a layer with a best matching score, for each pixel in the virtual image; and

means for determining a color of the pixel in the layer with the best matching score, for each pixel in the virtual image.

16. The system of claim 15, further comprising means for rectifying the first input image and the second input image.

17. The system of claim 15, further comprising means for locating a viewpoint of the virtual image to be on a baseline.

18. The system of claim 15, further comprising means for enforcing scene consistency by performing local depth voting.

19. The system of claim 15, further comprising means for detecting an occlusion in the virtual image by performing local depth voting.