Compound camera sensor and related method of processing digital images

• US 20100097491A1
• Filed: 10/21/2008
• Published: 04/22/2010
• Est. Priority Date: 10/21/2008
• Status: Active Grant

• Current Assignee: STMicroelectronics SRL (STMicroelectronics NV)
• Original Assignee: STMicroelectronics SRL (STMicroelectronics NV)
• Inventors: Messina, Giuseppe, Guarnera, Mirko Ignazio, Farina, Marco, Capra, Alessandro, Mancuso, Massimo
• Granted Patent: US 8,436,909 B2
• Time in Patent Office: Days
• US Class Current: 348/223.1
• CPC Class Codes: H04N 23/13 with multiple sensors
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements

Abstract

In one embodiment, a light sensor includes four cell arrays, one for each color of the Bayer pattern, and four lenses each focusing the light coming from the scene to be captured on a respective cell array. The lenses are oriented such that at least a second green image, commonly provided by the fourth cell array, is both horizontally and vertically shifted (spaced) apart by half a pixel pitch from a first (reference) green image. In a second embodiment, the four lenses are oriented such that the red and blue images are respectively shifted (spaced) apart by half a pixel pitch from the first or reference green image, one horizontally and the other vertically, and the second green image is shifted (spaced) apart by half a pixel pitch from the reference green image both horizontally and vertically.

Claims

1. A method of generating an RGB image of 2N×

2M pixels of a scene to be captured, comprising;

generating four N×

M monochromatic images of said scene, the monochromatic images including a red image, a blue image, a first green image, and a second green image shifted vertically and horizontally from the first green image;

generating enhanced red, blue, first green, and second green images by enhancing by two the resolution of all said red, blue, first green, and second green images;

obtaining a final N×

M green image by combining the enhanced second green image with the enhanced first green image;

calculating respective high frequency blue and red images using said final N×

M green image;

generating corresponding final N×

M blue and red N×

M images by adding the high frequency blue image to the enhanced blue image and adding the high frequency red image to the enhanced red image; and

generating a 2N×

2M output RGB image by combining said final N×

M green, red and blue images.

2. The method of claim 1, wherein:

said final N×

M green image is generated by processing with a super resolution algorithm said enhanced first and second green images,

said high frequency red and blue images are obtained as a difference between the final N×

M green image and said enhanced second green image.

3. The method of claim 1, wherein:

said final N×

M green image is generated by averaging homologous pixels of said enhanced first and second green images,

said high frequency red and blue images are obtained as a difference between the final N×

M green image and said enhanced first green image.

4. The method of claim 1, wherein the four N×

M monochromatic images of said scene are generated with a light sensor including;

a multi-cell photo-detector composed of four cell arrays of cells, each cell defining a pixel of the image and being configured to generate an electric signal representative of the intensity of light impinging thereon, the four cell arrays including a first cell array sensitive to red light, a second cell array sensitive to blue light, and third and fourth cell arrays sensitive to green light; and

four lenses focusing light coming from the scene on the cell arrays, respectively, the lens focusing light on the fourth cell array having an axis oriented off parallelism from an axis of the lens facing the third cell array.

5. The method of claim 1, wherein generating the enhanced red, blue, first green, and second green images, obtaining the final N×

M green image, calculating the respective high frequency blue and red images, generating the corresponding final N×

M blue and red N×

M images, and generating the 2N×

2M output RGB image are performed as part of processing pixel values of said four monochromatic images by a pipeline coupled to said cell arrays.

6. A method of generating an RGB image of 2N×

2M pixels of a scene to be captured, comprising;

generating four N×

M monochromatic images of said scene, the monochromatic images including a red image, a blue image, a first green image, and a second green image, the red and blue images being shifted by half a pixel pitch horizontally and vertically, respectively, from said first green image and the second green image being shifted by half a pixel pitch both horizontally and vertically from said first green image;

generating a corresponding 2N×

2M Bayer image having odd rows of pixels that are pixels of the first green image and of the red image, and even rows of pixels that are pixels of the blue image and of the second green image; and

generating an output 2N×

2M RGB image by interpolating said 2N×

2M Bayer image.

7. The method of claim 6, wherein the four N×

M monochromatic images of said scene are generated with a light sensor including;

a multi-cell photo-detector composed of four cell arrays of cells, each cell defining a pixel of the image and being configured to generate an electric signal representative of the intensity of light impinging thereon, the four cell arrays including a first cell array sensitive to red light, a second cell array sensitive to blue light, and third and fourth cell arrays sensitive to green light; and

first, second, third, and fourth lenses focusing light coming from the scene on the four cell arrays, respectively, the first, second, and fourth lenses having axes that are off parallel from an axis of the third lens, the axes of the first and second lenses being oriented such to generate the red and blue images shifted by half a pixel pitch horizontally and vertically, respectively, from said first green image and the axis of the fourth lens being oriented such to generate the second green image shifted by half a pixel pitch both horizontally and vertically from said first green image

8. A method of generating an RGB image of 2N×

2M pixels of a scene to be captured, comprising;

generating four N×

M monochromatic images of said scene with a light sensor including a photo-detector composed of four cell arrays of cells, each cell defining a pixel of the image and being configured to generate an electric signal representative of the intensity of light impinging thereon, the four cell arrays including a first cell array sensitive to red light, a second cell array sensitive to blue light, and third and fourth cell arrays sensitive to green light, the light sensor also including four lenses focusing light coming from the scene on the cell arrays, respectively, said lenses being placed at different distances from the respective cell array such to define different depths of field of the corresponding N×

M monochromatic images, said depths of field being staggered without gaps;

identifying which of the four N×

M monochromatic images is sharpest;

for the other three N×

M monochromatic images, calculating respective high frequency images using said identified sharpest N×

M monochromatic image as a reference image, and generating three corresponding corrected N×

M monochromatic images by correcting intensity values of each pixel of the other three N×

M monochromatic images by adding the intensity values of the corresponding pixel of the respective high frequency image; and

processing said corrected N×

M monochromatic images by elaborating said sharpest N×

M monochromatic image and said corrected N×

M monochromatic images to output a corresponding RGB image.

9. A method of generating an RGB image of 2N×

2M pixels of a scene to be captured, comprising;

generating four N×

M monochromatic images of said scene with a light sensor including a photo-detector composed of four cell arrays of cells, each cell defining a pixel of the image and being configured to generate an electric signal representative of the intensity of light impinging thereon, the four cell arrays including a first cell array sensitive to red light, a second cell array sensitive to blue light, and third and fourth cell arrays sensitive to green light, the light sensor also including four lenses focusing light coming from the scene on the four cell arrays, respectively, said lenses being placed at different distances from the respective cell array such to define different depths of field of the corresponding N×

M monochromatic images, said depths of field being staggered with gaps;

identifying which of the four N×

M monochromatic images is sharpest;

generating an enhanced sharp image by reducing blurring of the sharpest N×

M monochromatic image, reducing blurring including enhancing high frequency components of the sharpest N×

M monochromatic image by using a sharpening filter;

for the other three N×

M monochromatic images of the scene, calculating respective high frequency images using said enhanced sharp image as a reference image, and generating three corresponding corrected N×

M monochromatic images by correcting intensity values of each pixel of the other three N×

M monochromatic images by adding the intensity values of the corresponding pixel of the respective high frequency image;

processing said corrected N×

M monochromatic images by elaborating said enhanced sharp image and said corrected N×

M monochromatic images to output a corresponding RGB image.

10. A light sensor for capturing an image of a scene, comprising:

a multi-cell photo-detector composed of a plurality of cell arrays of cells, each cell defining a pixel of an image and being configured to generate an electric signal representative of the intensity of light impinging thereon;

a plurality of lenses focusing the light coming from the scene on the cell arrays, respectively, wherein

said lenses are placed at different distances from the respective cell arrays such to define different depths of field of the detected images, said depths of field being staggered.

11. The light sensor of claim 10, wherein said depths of field are staggered without gaps.

12. The light sensor of claim 10, wherein said depths of field are staggered with gaps.

13. The light sensor of claim 10, wherein said lenses lay in a same plane and said cell arrays are defined on different planes.

14. The light sensor of claim 10, wherein said cell arrays are defined on a same plane and said lenses lay in different planes.

15. The light sensor of claim 10, wherein the plurality of lenses includes four lenses, and the plurality of cell arrays includes a first cell array sensitive to red light, a second cell array sensitive to blue light, and third and fourth cell arrays sensitive to green light, disposed according to a Bayer pattern.

16. The light sensor of claim 15, wherein the lenses focusing light on said third and fourth cells arrays are the closest and the farthest from the respective cell arrays.

17. The light sensor of claim 15, further comprising a processing pipeline coupled to said cell arrays and configured to elaborate image pixels captured by said cell arrays and to output a corresponding Bayer image.

18. A light sensor for capturing an image of a scene, comprising:

a multi-cell photo-detector composed of a plurality of N×

M cell arrays of cells, each cell defining a pixel of the image and being configured to generate an electric signal representative of the intensity of light impinging thereon, the N×

M cell arrays including a first cell array sensitive to red light, a second cell array sensitive to blue light, and third and fourth cell arrays sensitive to green light, the cells of each of said first, second, third, and fourth cell arrays being organized in singularly addressable rows and columns as follows;

odd rows of cells are staggered with respect to even rows of cells and odd columns of cells are staggered with respect to even columns of cells, such that the distance between any pixel and neighboring pixels belonging to the same row or to the same column is a multiple by the square root of two of the distance between said pixel and any of the closest pixels adjacent thereto;

first, second, third, and fourth lenses focusing light coming from the scene on the first, second, third, and fourth cell arrays, respectively, the third lens having an axis oriented off parallel from an axis of the fourth lens such that an odd number of rows and of columns of cells of the third cell array are never illuminated by light coming from the scene, such to generate a green image shifted vertically and horizontally by half a pixel pitch from a reference green image generated by the fourth cell array;

a processing pipeline coupled to said cell arrays and configured to elaborate image pixels captured by said cell arrays and to output a corresponding image.

19. The light sensor of claim 18, wherein the first and second lenses have axes that are not parallel to the axis of the fourth lens, the axes of the first and second lenses being oriented such that an odd number of columns of cells and an odd number of rows of cells, respectively, are never illuminated by light coming from the scene, such to generate shifted red and blue images, respectively, by a half of a pixel pitch from the reference green image generated by the fourth cell array.