Method and apparatus for parallax correction in fused array imaging systems

• US 8,581,995 B2
• Filed: 05/18/2011
• Issued: 11/12/2013
• Est. Priority Date: 01/25/2011
• Status: Active Grant

• Current Assignee: Aptina Imaging Corporation (ON Semiconductor Corporation)
• Original Assignee: Aptina Imaging Corporation (ON Semiconductor Corporation)
• Inventors: Lin, Peng, Smith, Scott, Black, Robert A.
• Primary Examiner(s): SELBY, GEVELL V
• Application Number:
  US13/110,252
• Publication Number:
  US 20120188389A1
• Time in Patent Office: 909 Days
• Field of Search: 348/218.1, 348/222.1, 348/252
• US Class Current: 348/218.1
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 25/41 Extracting pixel data from a plurality of image sensors simultaneously picking up an image, e.g. for increasing the field of view by combining the outputs of a plurality of sensors
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

Electronic devices may include camera modules. A camera module may include an array camera having an array of lenses and an array of corresponding image sensors. Parallax correction and depth mapping methods may be provided for array cameras. A parallax correction method may include a global and a local parallax correction. A global parallax correction may be determined based on one-dimensional horizontal and vertical projections of edge images. Local parallax corrections may be determined using a block matching procedure. Further improvements to local parallax corrections may be generated using a relative block color saturation test, a smoothing of parallax correction vectors and, if desired, using a cross-check between parallax correction vectors determined for multiple image sensors. Three dimensional depth maps may be generated based on parallax correction vectors.

Claims

1. A method of global parallax correction for an array camera having at least first and second image sensors and processing circuitry, the method comprising:

capturing first and second images using the first and second image sensors, respectively, wherein the first and second images contain at least one object having edges;

with the processing circuitry, computing first and second edge images based on the first and second images, respectively, wherein the first and second edge images contain the edges of the at least one object in the first and second images; and

with the processing circuitry, generating a first global parallax correction based on the first and second edge images, wherein generating the first global parallax correction based on the first and second edge images comprises;

with the processing circuitry, computing a first horizontal projection curve based on a horizontal projection of the first edge image and a second horizontal projection curve based on a horizontal projection of the second edge image.

2. The method defined in claim 1 wherein computing the first and second edge images based on the first and second images, respectively, comprises:

with the processing circuitry, convolving the first and second images with an edge detection operator such that the first and second edge images contain only the edges of the at least one object, and wherein other portions of the at least one object are suppressed in the first and second edge images.

3. The method defined in claim 2 wherein generating the first global parallax correction based on the first and second edge images further comprises:

with the processing circuitry, computing a first vertical projection curve based on a vertical projection of the first edge image and a second vertical projection curve based on a vertical projection of the second edge image.

4. The method defined in claim 3 wherein generating the first global parallax correction based on the first and second edge images further comprises:

with the processing circuitry, generating a global horizontal parallax correction based on a first offset between the first and second horizontal projection curves; and

with the processing circuitry, generating a global vertical parallax correction based on a second offset between the first and second vertical projection curves.

5. The method defined in claim 4 wherein the array camera comprises additional image sensors, the method further comprising:

with the additional image sensors, capturing additional images wherein the additional images contain the at least one object having edges;

with the processing circuitry, generating additional edge images based on the additional images, wherein each of the additional edge images contain the edges of the at least one object in each of the additional images; and

with the processing circuitry, computing additional global parallax corrections based on the additional edge images.

6. The method defined in claim 5 wherein the second image sensor is configured to receive light through a green color filter, wherein the first global parallax correction comprises a shift between the first image and the second image, and wherein the additional global parallax corrections comprise additional shifts between the additional images and the second image.

7. A method of local parallax correction for an array camera having at least first and second image sensors and processing circuitry, the method comprising:

capturing first and second images using the first and second image sensors respectively, wherein the first image comprises a plurality of image blocks;

with the processing circuitry, generating a local parallax correction map, wherein the local parallax correction map comprises a plurality of block parallax correction vectors corresponding to the plurality of image blocks and wherein generating the local parallax correction map comprises;

with the processing circuitry, generating the plurality of block parallax correction vectors corresponding to the plurality of image blocks by matching each one of the plurality of image blocks in the first image to a corresponding matched block in the second image;

with the processing circuitry computing a block color energy based on each one of the image blocks in the first image and the corresponding matched block in the second image; and

with the processing circuitry, comparing the block color energy to a threshold.

8. The method defined in claim 7 wherein matching each one of the plurality of image blocks in the first image to the corresponding matched block in the second image comprises:

with the processing circuitry, computing a block energy difference based on each one of the plurality of image blocks in the first image and a plurality of overlapping test blocks in the second image.

9. The method defined in claim 8, further comprising:

with the processing software, generating first and second edge images based on the first and second images respectively, wherein the first and second images contain at least one object having edges, and wherein the first and second edge images contain the edges of the at least one object in the first and second images.

10. The method defined in claim 9 wherein computing the block energy difference based on each one of the plurality of image blocks in the first image and the plurality of overlapping test blocks in the second image further comprises:

with the processing circuitry, computing the block energy difference based on each one of a plurality of image blocks in the first edge image and a plurality of overlapping test blocks in the second edge image.

11. The method defined in claim 10 wherein the array camera comprises additional image sensors, the method further comprising:

with the additional image sensors, capturing additional images, wherein the additional images contain the at least one object having edges;

with the processing circuitry, generating additional edge images based on the additional images, wherein each of the additional edge images contain the edges of the at least one object in each of the additional images; and

with the processing circuitry, computing additional local parallax correction maps based on the additional images and the additional edge images.

12. The method defined in claim 11, further comprising:

with the processing circuitry, cross-checking the local parallax correction map with each of the additional local parallax correction maps.

13. A method of parallax correction for array cameras having an array of image sensors and processing circuitry, the method comprising:

with each image sensor of the array of image sensors, capturing an image, wherein a first one of the images comprises a reference image;

with the processing circuitry, generating a global parallax correction corresponding to each image that represents a shift between each image and the reference image; and

with the processing circuitry, generating a plurality of local parallax correction vectors corresponding to a plurality of portions of each image, wherein each local parallax correction vector represents a shift between one of the plurality of portions of each image and a corresponding portion of the reference image.

14. The method defined in claim 13, further comprising:

with the processing circuitry, generating a depth map based on the plurality of local parallax correction vectors.

15. The method defined in claim 14 wherein generating the global parallax correction corresponding to each image comprises:

with the processing circuitry, generating an edge image from each image;

with the processing circuitry, generating a plurality of projection curves from each edge image;

with the processing circuitry, computing horizontal and vertical offsets between the projection curves of the images and the projection curves of the reference image.

16. The method defined in claim 15 wherein generating the plurality of local parallax corrections corresponding to the plurality of portions of each image comprises:

with the processing circuitry, storing the plurality of local parallax correction vectors corresponding to the plurality of portions of each image in a local parallax correction map corresponding to each image;

with the processing circuitry, eliminating some of the local parallax correction vectors using a block color saturation checking procedure;

with the processing circuitry, smoothing the local parallax correction map; and

with the processing circuitry, performing a cross-check of at least two of the local parallax correction maps corresponding to at least two of the images.

17. The method defined in claim 16 wherein generating the plurality of local parallax correction vectors corresponding to the plurality of portions of each image comprises:

with the processing circuitry, computing a plurality of block energy differences based on a plurality of expanded test blocks corresponding to the plurality of portions of each image.

18. The method defined in claim 17 wherein computing a plurality of block energy differences based on a plurality of expanded test blocks corresponding to the plurality of portions of each image further comprises:

with the processing circuitry, comparing a lowest one of the plurality of block energy differences to a threshold; and

with the processing circuitry, if the lowest one of the plurality of block energy differences is higher than the threshold, computing an additional plurality of block energy differences based on a plurality of expanded test blocks corresponding to a plurality of sub-portions of each image, wherein the plurality of sub-portions of each image are smaller the plurality of portions of each image.