SYSTEM AND METHOD FOR GENERATING A DEPTH MAP AND FUSING IMAGES FROM A CAMERA ARRAY

• US 20130010073A1
• Filed: 07/09/2012
• Published: 01/10/2013
• Est. Priority Date: 07/08/2011
• Status: Active Grant

• Current Assignee: Personify Incorporated (Wilson Human Capital Group, Inc.)
• Original Assignee: Personify Incorporated (Wilson Human Capital Group, Inc.)
• Inventors: Do, Minh N., Nguyen, Quang H., Patel, Sanjay J., Chidester, Benjamin, Dang, Long
• Granted Patent: US 9,300,946 B2
• Time in Patent Office: Days
• US Class Current: 348/46
• CPC Class Codes: G03B 35/12 involving recording of different viewpoint images in different colours on a colour film
• CPC Class Codes: H04N 13/214 using spectral multiplexing
• CPC Class Codes: H04N 13/218 using spatial multiplexing
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A method, apparatus, system, and computer program product for of digital imaging. Multiple cameras comprising lenses and digital images sensors are used to capture multiple images of the same subject, and process the multiple images using difference information (e.g., an image disparity map, an image depth map, etc.). The processing commences by receiving a plurality of image pixels from at least one first image sensor, wherein the first image sensor captures a first image of a first color, receives a stereo image of the first color, and also receives other images of other colors. Having the stereo imagery, then constructing a disparity map and an associated depth map by searching for pixel correspondences between the first image and the stereo image. Using the constructed disparity map, captured images are converted into converted images, which are then combined with the first image, resulting in a fused multi-channel color image.

Claims

1. A method of fusing a plurality of images from a camera array comprising:

receiving a plurality image pixels from at least one first image sensor, wherein the first image sensor captures a first image of a first color, a stereo image of the same first color, and at least one second image of a second color;

constructing a disparity map using the plurality of image pixels, by searching for pixel correspondences between the first image of a first color and the stereo image of the same first color;

converting the at least one second image of the second color into a converted second image using the disparity map; and

combining the first image of the first color and the converted second image into a fused multi-channel color image.

2. The method of claim 1, further comprising producing a fused image by:

capturing at least one third image of a third color;

converting the at least one third image of the third color into a converted third image using the disparity map; and

combining the first image of the first color and the converted third image into the fused multi-channel color image.

3. The method of claim 1, wherein the plurality of images comprise:

first image pixels received from a first green image sensor;

stereo image pixels received from a stereo green image sensor;

second image pixels received from a second red image sensor; and

,

third image pixels received from a third blue image sensor.

4. The method of claim 1, wherein:

the first image pixels received from a first green image sensor corresponds to a respective first lens;

the stereo image pixels received from a stereo green image sensor corresponds to a respective stereo lens;

the second image pixels received from a second red image sensor corresponds to a respective second lens; and

the third image pixels received from a third blue image sensor corresponds to a respective third lens.

5. The method of claim 3 wherein the first green image sensor, and the stereo green image sensor and the second red image sensor third blue image sensor are organized into a two-by-two camera array.

6. The method of claim 3, wherein the first green image sensor, and the stereo green image sensor and the second red image sensor third blue image sensor are organized into a linear array.

7. The method of claim 3, wherein the first green image sensor, and the stereo green image sensor and the second red image sensor third blue image sensor are organized into a floating array.

8. The method of claim 3, wherein constructing the disparity map comprises estimating a plurality of image depths for the first image pixels compared to the stereo image pixels.

9. The method of claim 1, wherein mapping the first image and the stereo image from among the plurality of images to at least a second image from among the plurality of images is performed according to the equation:

10. The method of claim 1, wherein the pixel correspondences are identified using a plurality of scanline pairs of the first image pixels and the stereo image pixels.

11. The method of claim 1, wherein the plurality of correspondences is identified using a restricted search range.

12. The method of claim 1, further comprising correcting for visual artifacts in the fused multi-channel color image using at least one of, a bilateral filtering technique, or an inpainting technique.

13. The method of claim 1, further comprising generating a depth map using the disparity map.

14. The method of claim 13, further comprising processing the generated depth map to extract one or more features of a user gesture.

15. The method of claim 1, wherein constructing the disparity map comprises identifying a plurality of correspondences between the first image pixels and the stereo image pixels, the correspondences comprising pairings of pixels that represent the same point in space.

16. The method of claim 14, wherein constructing the disparity map further comprises computing a plurality of pixel shifts for the correspondences using at least one of, the image depths, a camera focal length, a horizontal distance between the first green image sensor and the stereo green image sensor, a vertical distance between the first green image sensor and the stereo green image sensor, and a pixel pitch.

17. The method of claim 1, wherein constructing the disparity map performs correspondence searches only near edges present in the first image.

18. An apparatus for fusing images from a camera array, comprising:

a computer processor to execute a set of program code instructions; and

a memory to hold the program code instructions, in which the program code instructions comprises program code to perform,

receiving a plurality image pixels from at least one first image sensor, wherein the first image sensor captures a first image of a first color, a stereo image of the same first color, and at least one second image of a second color;

constructing a disparity map using the plurality of image pixels, by searching for pixel correspondences between the first image of a first color and the stereo image of the same first color; and

converting the at least one second image of the second color into a converted second image using the disparity map;

combining the first image of the first color and the converted second image into a fused multi-channel color image.

19. The apparatus of claim 18, further comprising producing a fused image by:

capturing at least one third image of a third color;

converting the at least one third image of the third color into a converted third image using the disparity map; and

combining the first image of the first color and converted third image into the fused multi-channel color image.

20. The apparatus of claim 18, wherein the plurality of images comprise:

first image pixels received from a first green image sensor;

stereo image pixels received from a stereo green image sensor;

second image pixels received from a second red image sensor; and

third image pixels received from a third blue image sensor.

21. The apparatus of claim 20, wherein:

the first image pixels received from a first green image sensor corresponds to a respective first lens;

the stereo image pixels received from a stereo green image sensor corresponds to a respective stereo lens;

the second image pixels received from a second red image sensor corresponds to a respective second lens; and

the third image pixels received from a third blue image sensor corresponds to a respective third lens.

22. A computer program product embodied in a non-transitory computer readable medium, the computer readable medium having stored thereon a sequence of instructions which, when executed by a processor causes the processor to execute a process to implement fusing images from a camera array, the process comprising:

receiving a plurality image pixels from at least one first image sensor, wherein the first image sensor captures a first image of a first color, a stereo image of the same first color, and at least one second image of a second color;

constructing a disparity map using the plurality of image pixels, by searching for pixel correspondences between the first image of a first color and the stereo image of the same first color; and

converting the at least one second image of the second color into a converted second image using the disparity map;

combining the first image of the first color and the converted second image into a fused multi-channel color image.