Compound camera sensor and related method of processing digital images

• US 8,436,909 B2
• Filed: 10/21/2008
• Issued: 05/07/2013
• Est. Priority Date: 10/21/2008
• Status: Active Grant

• Current Assignee: STMicroelectronics SRL (STMicroelectronics NV)
• Original Assignee: STMicroelectronics SRL (STMicroelectronics NV)
• Inventors: Farina, Marco, Guarnera, Mirko Ignazio, Mancuso, Massimo, Messina, Giuseppe, Capra, Alessandro
• Primary Examiner(s): Velez, Roberto
• Assistant Examiner(s): Kao, Yih-Sien
• Application Number:
  US12/288,626
• Publication Number:
  US 20100097491A1
• Time in Patent Office: 1,659 Days
• Field of Search: 348/223.1, 348/234, 348/222.1, 348/260, 348/E9.052, 348/218.1, 348/239, 345/629, 345/660, 382/162, 382/299, 382/300, 382/312, 382/284, 382/255
• US Class Current: 348/222.1
• CPC Class Codes: H04N 23/13 with multiple sensors
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements

Abstract

In one embodiment, a light sensor includes four cell arrays, one for each color of the Bayer pattern, and four lenses each focusing the light coming from the scene to be captured on a respective cell array. The lenses are oriented such that at least a second green image, commonly provided by the fourth cell array, is both horizontally and vertically shifted (spaced) apart by half a pixel pitch from a first (reference) green image. In a second embodiment, the four lenses are oriented such that the red and blue images are respectively shifted (spaced) apart by half a pixel pitch from the first or reference green image, one horizontally and the other vertically, and the second green image is shifted (spaced) apart by half a pixel pitch from the reference green image both horizontally and vertically.

Claims

1. A method of generating an RGB image of pixels of a scene to be captured, comprising:

generating by an image capture device four monochromatic images of said scene, the monochromatic images including a red image, a blue image, a first green image, and a second green image shifted vertically and horizontally from the first green image;

generating enhanced red, blue, first green, and second green images by enhancing by two the resolution of all said red, blue, first green, and second green images;

obtaining a final green image by combining the enhanced second green image with the enhanced first green image;

calculating respective high frequency blue and red images using said final green image;

generating corresponding final blue and red images by adding the high frequency blue image to the enhanced blue image and adding the high frequency red image to the enhanced red image; and

generating a output RGB image by combining said final green, red and blue images.

2. The method of claim 1, wherein:

said final green image is generated by processing with a super resolution algorithm said enhanced first and second green images,

said high frequency red and blue images are obtained as a difference between the final green image and said enhanced second green image.

3. The method of claim 1, wherein:

said final green image is generated by averaging homologous pixels of said enhanced first and second green images,

said high frequency red and blue images are obtained as a difference between the final green image and said enhanced first green image.

4. The method of claim 1, wherein the four monochromatic images of said scene are generated with a light sensor including:

a multi-cell photo-detector composed of four cell arrays of cells, each cell defining a pixel of the image and being configured to generate an electric signal representative of the intensity of light impinging thereon, the four cell arrays including a first cell array sensitive to red light, a second cell array sensitive to blue light, and third and fourth cell arrays sensitive to green light; and

four lenses focusing light coming from the scene on the cell arrays, respectively, the lens focusing light on the fourth cell array having an axis oriented off parallelism from an axis of the lens facing the third cell array.

5. The method of claim 1, wherein generating the enhanced red, blue, first green, and second green images, obtaining the final green image, calculating the respective high frequency blue and red images, generating the corresponding final blue and red images, and generating the output RGB image are performed as part of processing pixel values of said four monochromatic images by a pipeline coupled to said cell arrays.

6. A method of generating an RGB image of pixels of a scene to be captured, comprising:

generating four monochromatic images of said scene with a light sensor including a photo-detector composed of four cell arrays of cells, each cell defining a pixel of the image and being configured to generate an electric signal representative of the intensity of light impinging thereon, the four cell arrays including a first cell array sensitive to red light, a second cell array sensitive to blue light, and third and fourth cell arrays sensitive to green light, the light sensor also including four lenses focusing light coming from the scene on the cell arrays, respectively, said lenses being placed at different distances from the respective cell array to define different depths of field of the corresponding monochromatic images, said depths of field being staggered without gaps;

identifying which of the four monochromatic images is sharpest;

for the other three monochromatic images, calculating respective high frequency images using said identified sharpest monochromatic image as a reference image, and generating three corresponding corrected monochromatic images by correcting intensity values of each pixel of the other three monochromatic images by adding the intensity values of the corresponding pixel of the respective high frequency image; and

processing said corrected monochromatic images by elaborating said sharpest monochromatic image and said corrected monochromatic images to output a corresponding RGB image.

7. The method of claim 6 wherein calculating the respective high frequency images using said identified sharpest monochromatic image as a reference image includes determining a point spread function using a difference between a first distance between the lens and cell array associated with the sharpest monochromatic image and a second distance between the lens and cell array associated with each of the other respective monochromatic images.

8. A method of generating an RGB image of pixels of a scene to be captured, comprising:

generating four monochromatic images of said scene with a light sensor including a photo-detector composed of four cell arrays of cells, each cell defining a pixel of the image and being configured to generate an electric signal representative of the intensity of light impinging thereon, the four cell arrays including a first cell array sensitive to red light, a second cell array sensitive to blue light, and third and fourth cell arrays sensitive to green light, the light sensor also including four lenses focusing light coming from the scene on the four cell arrays, respectively, said lenses being placed at different distances from the respective cell array to define different depths of field of the corresponding monochromatic images, said depths of field being staggered with gaps;

identifying which of the four monochromatic images is sharpest;

generating an enhanced sharp image by reducing blurring of the sharpest monochromatic image, reducing blurring including enhancing high frequency components of the sharpest monochromatic image by using a sharpening filter;

for the other three monochromatic images of the scene, calculating respective high frequency images using said enhanced sharp image as a reference image, and generating three corresponding corrected monochromatic images by correcting intensity values of each pixel of the other three monochromatic images by adding the intensity values of the corresponding pixel of the respective high frequency image;

processing said corrected monochromatic images by elaborating said enhanced sharp image and said corrected monochromatic images to output a corresponding RGB image.

9. The method of claim 8 wherein calculating respective high frequency images using said enhanced sharp image as a reference image includes determining a point spread function using a difference between a first distance between the lens and cell array associated with the sharpest monochromatic image and a second distance between the lens and cell array associated with each of the other respective monochromatic images.

10. A light sensor for capturing an image of a scene, comprising:

a multi-cell photo-detector composed of a plurality of cell arrays of cells, each cell defining a pixel of the image and being configured to generate an electric signal representative of the intensity of light impinging thereon, the cell arrays including a first cell array sensitive to red light, a second cell array sensitive to blue light, and third and fourth cell arrays sensitive to green light, the cells of each of said first, second, third, and fourth cell arrays being organized in singularly addressable rows and columns as follows;

odd rows of cells are staggered with respect to even rows of cells and odd columns of cells are staggered with respect to even columns of cells, such that the distance between any pixel and neighboring pixels belonging to the same row or to the same column is a multiple by the square root of two of the distance between said pixel and any of the closest pixels adjacent thereto;

first, second, third, and fourth lenses focusing light coming from the scene on the first, second, third, and fourth cell arrays, respectively, the third lens having an axis oriented off parallel from an axis of the fourth lens such that an odd number of rows and of columns of cells of the third cell array are never illuminated by light coming from the scene, to generate a green image shifted vertically and horizontally by half a pixel pitch from a reference green image generated by the fourth cell array;

a processing pipeline coupled to said cell arrays and configured to elaborate image pixels captured by said cell arrays and to output a corresponding image.

11. The light sensor of claim 10, wherein the first and second lenses have axes that are not parallel to the axis of the fourth lens, the axes of the first and second lenses being oriented such that an odd number of columns of cells and an odd number of rows of cells, respectively, are never illuminated by light coming from the scene, to generate shifted red and blue images, respectively, by a half of a pixel pitch from the reference green image generated by the fourth cell array.