Enhancing spatial resolution in a stereo camera imaging system

• US 10,531,067 B2
• Filed: 03/26/2018
• Issued: 01/07/2020
• Est. Priority Date: 03/26/2017
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Attar, Ziv, Hubel, Paul M., Volfin, Ilana, Marineau-Mes, Sebastien
• Primary Examiner(s): Schnurr, John R
• Application Number:
  US15/936,275
• Publication Number:
  US 20180278913A1
• Time in Patent Office: 652 Days
• CPC Class Codes: H04N 13/15 for colour aspects of image signals
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 2013/0077 Colour aspects
• CPC Class Codes: H04N 23/10 for generating image signals from different wavelengths
• CPC Class Codes: H04N 23/687 by shifting the lens or sensor position
• CPC Class Codes: H04N 9/43 Conversion of monochrome picture signals to colour picture signals for colour picture display
• CPC Class Codes: H04N 9/78 for separating the brightness signal or the chrominance signal from the colour television signal, e.g. using comb filter

Abstract

Techniques are disclosed for capturing stereoscopic images using one or more high color density or “full color” image sensors and one or more low color density or “sparse color” image sensors. Low color density image sensors, may include substantially fewer color pixels than the sensor'"'"'s total number of pixels, as well as fewer color pixels than the total number of color pixels on the full color image sensor. More particularly, the mostly-monochrome image captured by the low color density image sensor may be used to reduce noise and increase the spatial resolution of an imaging system'"'"'s output image. In addition, the color pixels present in the low color density image sensor may be used to identify and fill in color pixel values, e.g., for regions occluded in the image captured using the full color image sensor. Optical Image Stabilization and/or split photodiodes may be employed on one or more sensors.

Claims

1. A stereoscopic imaging system, comprising:

a first image sensor configured to capture a first color image of a scene, wherein the first color image sensor includes color pixels having color values;

a second image sensor configured in a predetermined spatial relationship to the first image sensor, so as to form a stereoscopic imaging system, wherein the second image sensor is configured to capture, concomitantly with the first color image, a second image of the scene, wherein the second image sensor includes luminance pixels having luminance values and color pixels having color values, and wherein the first image sensor has a greater number of color pixels than the second image sensor;

a color reconstruction circuit configured to generate, using information from the second image, a second color image; and

an output circuit configured to generate, using information from the first and second color images, an output image,

wherein split photodiode pixels are used on the second image sensor, but not the first image sensor.

2. The stereoscopic imaging system of claim 1, wherein the number of color pixels on the second image sensor comprises less than 50% of the total number of pixels on the second image sensor.

3. The stereoscopic imaging system of claim 2, wherein the number of color pixels on the second image sensor further comprises greater than 3% of the total number of pixels on the second image sensor.

4. The stereoscopic imaging system of claim 1, wherein the color pixels of the second image sensor are arranged in at least one of:

a regular pattern, a pseudo-random pattern, a row pattern, or a grid pattern.

5. The stereoscopic imaging system of claim 1, wherein an optical image stabilization (OIS) system is used on the second image sensor, but not the first image sensor.

6. The stereoscopic imaging system of claim 1, wherein the second image sensor is further configured to capture the second image of the scene by:

exposing first rows of sensor elements of the second image sensor for a first duration; and

exposing second rows of sensor elements of the second image sensor for a second duration different than the first duration.

7. The stereoscopic imaging system of claim 1, further comprising a luminance reconstruction circuit configured to generate a luminance image using information from the second image.

8. The stereoscopic imaging system of claim 7, further comprising a registration circuit configured to generate a registered color image using information from the first image and the luminance image.

9. The stereoscopic imaging system of claim 8, wherein the output circuit is configured to merge the registered color image and the second color image to generate the output image.

10. The stereoscopic imaging system of claim 1, wherein the stereoscopic imaging system is configured to perform auto focus operations using information captured by the first image sensor but not information captured by the second image sensor.

11. An image processing method, comprising:

capturing, using a first image sensor, a first color image of a scene, the first color image including color pixels having color values;

capturing, using a second image sensor configured in a predetermined spatial relationship to the first image sensor, a second image of the scene, wherein the second image sensor is configured to capture the second image of the scene concomitantly with the first color image, wherein the second image sensor includes luminance pixels having luminance values and color pixels having color values, and wherein the first image sensor has a greater number of color pixels than the second image sensor;

generating, using a color reconstruction circuit, a second color image based, at least in part, on the second image;

generating, using a luminance reconstruction circuit, a luminance image based, at least in part, on the second image;

generating, using a registration circuit, a registered color image based, at least in part, on the first image and the luminance image; and

merging, using the output circuit, the registered color image and the second color image to generate an output image.

12. The method of claim 11, wherein the number of color pixels on the second image sensor comprises less than 50% of the total number of pixels on the second image sensor.

13. The method of claim 12, wherein the number of color pixels on the second image sensor further comprises greater than 3% of the total number of pixels on the second image sensor.

14. The method of claim 13, wherein the color pixels of the second image sensor are arranged in at least one of:

a regular pattern, a pseudo-random pattern, a row pattern, or a grid pattern.

15. The method of claim 11, wherein capturing the second image of the scene further comprises:

exposing first rows of sensor elements of the second image sensor for a first duration; and

exposing second rows of sensor elements of the second image sensor for a second duration,

wherein the first rows comprise rows of sensor elements that include both color elements and luminance elements, and

wherein the second rows comprise only luminance elements.

16. An imaging system, comprising:

a first image sensor, comprising;

a first plurality of luminance pixel elements; and

a second plurality of color pixel elements,

wherein a number of the second plurality of color pixel elements comprises less than 50% of a sum of a number of the first plurality of luminance pixel elements and the number of the second plurality of color pixel elements, and

wherein the second plurality of color pixel elements are arranged in a predetermined pattern on the image sensor; and

a second image sensor, comprising;

a third plurality of color pixel elements,

wherein a number of the third plurality of color pixel elements is greater than the number of the second plurality of color pixel elements, and wherein the imaging system is configured to combine image information captured concomitantly by the first image sensor and the second image sensor to generate an output image, and wherein split photodiode pixels are used on the first image sensor, but not the second image sensor.

17. The imaging system of claim 16, wherein an optical image stabilization system is used on the first image sensor, but not the second image sensor.

18. The imaging system of claim 16, wherein the imaging system is configured to perform auto focus operations using information captured by the first image sensor but not information captured by the second image sensor.

19. The imaging system of claim 16, further comprising:

a luminance reconstruction circuit configured to generate a luminance image using information captured by the first image sensor.

20. The imaging system of claim 19, further comprising:

a registration circuit configured to generate a registered color image using information captured by the second image sensor and the luminance image,

wherein generating the output image further comprises merging the registered color image and the information captured using the first image sensor.