SYSTEMS AND METHODS FOR DETERMINING DEPTH FROM MULTIPLE VIEWS OF A SCENE THAT INCLUDE ALIASING USING HYPOTHESIZED FUSION

• US 20130070060A1
• Filed: 09/19/2012
• Published: 03/21/2013
• Est. Priority Date: 09/19/2011
• Status: Abandoned Application

• Current Assignee: Fotonation Cayman Limited (Adeia Inc.)
• Original Assignee: Pelican Imaging Corporation
• Inventors: Molina, Gabriel, Lelescu, Dan, Chatterjee, Priyam
• Application Number:
  US13/623,091
• Publication Number:
  US 20130070060A1
• Time in Patent Office: Days
• US Class Current: 348/47
• CPC Class Codes: H04N 13/232 using fly-eye lenses, e.g. arrangements of circular lenses
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 23/16 Optical arrangements associated therewith, e.g. for beam-splitting or for colour correction
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 25/41 Extracting pixel data from a plurality of image sensors simultaneously picking up an image, e.g. for increasing the field of view by combining the outputs of a plurality of sensors

Abstract

Array cameras in accordance with embodiments of the invention perform super resolution processing using images of a scene that contain aliasing. In several embodiments, the depth of pixels is determined by fusing portions of a higher resolution image at a number of hypothesized depths and determining the depth at which the portion of the higher resolution image best matches the scene captured in the lower resolution images used to fuse the higher resolution image.

Claims

1. A method of determining a depth of a point in a scene using light field image data comprising a set of low resolution images that capture the scene, the method comprising:

fusing portions of the set of low resolution images to form a portion of a higher resolution image at each of a plurality of hypothesized depths, where the resolution of the portion of the higher resolution image is higher than the resolutions of the portions of the set of low resolution images used to fuse the portion of the higher resolution image;

comparing the portion of the fused higher resolution image obtained at each hypothesized depth to the scene captured in the set of low resolution images; and

selecting the hypothesized depth at which the portion of the fused higher resolution image is most similar to the scene captured in the set of low resolution images as the depth of at least one point in the scene captured by the set of low resolution images.

2. The method of claim 1, wherein comparing the portion of the fused higher resolution image to the scene captured in the set of low resolution images comprises:

generating a set of forward mapped low resolution image portions by forward mapping the portion of the fused higher resolution image using a mapping based upon the characteristics of the cameras utilized to capture the set of low resolution images; and

comparing the forward mapped low resolution image portions with corresponding portions of corresponding images in the set of low resolution images.

3. The method of claim 1, wherein comparing the portion of the fused higher resolution image to the scene captured in the set of low resolution images comprises determining the similarity of pixels in pixel stacks in the portion of the fused higher resolution image.

4. The method of claim 1, wherein comparing the portion of the fused higher resolution image at a specific hypothesized depth to the scene captured in the set of low resolution images comprises comparing the portion of the fused higher resolution image at the specific hypothesized depth to a portion of at least a second fused higher resolution image formed by fusing a second set of low resolution images at the specific hypothesized depth.

5. The method of claim 4, wherein at least one low resolution image is common to said set of low resolution images and said second set of low resolution images.

6. A machine readable medium containing processor instructions, where execution of the instructions by a processor causes the processor to perform a process comprising:

fusing portions of the set of low resolution images to form a portion of a higher resolution image at each of a plurality of hypothesized depths, where the resolution of the portion of the higher resolution image is higher than the resolutions of the portions of the set of low resolution images used to fuse the portion of the higher resolution image;

comparing the portion of the fused higher resolution image obtained at each hypothesized depth to the scene captured in the set of low resolution images; and

selecting the hypothesized depth at which the portion of the fused higher resolution image is most similar to the scene captured in the set of low resolution images as the depth of at least one point in the scene captured by the set of low resolution images.

7. A method of determining a depth of a point in a scene using light field image data comprising a set of low resolution images that capture the scene, the method comprising:

fusing portions of a first subset of a set of low resolution images to form a portion of a first higher resolution image at each of a plurality of hypothesized depths, where the resolution of the portion of the first higher resolution image is higher than the resolutions of the portions of the first subset of the set of low resolution images used to fuse the portion of the first higher resolution image;

fusing portions of a second subset of the set of low resolution images to form a portion of a second higher resolution image at each of the plurality of hypothesized depths, where the resolution of the portion of the second higher resolution image is higher than the resolutions of the portions of the second subset of the set of low resolution images used to fuse the portion of the second higher resolution image;

comparing at least the portions of the first and second higher resolution images fused at each of the plurality of hypothesized depths; and

selecting the hypothesized depth at which the portions of the compared higher resolution images are most similar as the depth of at least one point in the scene imaged by pixels within portions of the first and second subsets of the set of low resolution images.

8. The method of claim 7, wherein at least one low resolution image in the set of low resolution images is common to both the first and second subsets of the set of low resolution images.

9. The method of claim 8, wherein:

the viewpoint of one of the low resolution images in the set of low resolution images is selected as the reference viewpoint used to fuse the portions of the first and second high resolution images at each of the plurality of hypothesized depths; and

the low resolution image selected as the reference viewpoint is common to both the first and second subsets of the set of low resolution images.

10. The method of claim 9, wherein low resolution images captured from viewpoints above, below, to the left, and to the right of the reference viewpoint are common to both the first and second subsets of the set of low resolution images.

11. The method of claim 7, wherein the viewpoint of one of the low resolution images in the set of low resolution images is selected as the reference viewpoint used to fuse the portions of the first and second high resolution images at each of the plurality of hypothesized depths

12. The method of claim 7, wherein fusing portions of a subset of the set of low resolution images to form a portion of a higher resolution image at a hypothesized depth comprises:

identifying pixels within the subset of the set of low resolution images based upon the hypothesized depth and the viewpoints of the low resolution images;

fusing the identified pixels onto a higher resolution grid generated from a chosen reference viewpoint using known calibration information; and

performing hole filing to fill holes in locations in the higher resolution grid.

13. The method of claim 7, wherein comparing at least the portions of the first and second higher resolution images fused at each of the plurality of hypothesized depths comprises comparing the portions of the first and second higher resolution images for matching error at each of the plurality of hypothesized depths.

14. The method of claim 13, wherein matching error is determined using at least one selected from the group of the L₁-norm and the L₂-norm of the difference of the portions of the first and second higher resolution image.

15. The method of claim 7, further comprising:

fusing portions of a third subset of the set of low resolution images to form a portion of a third higher resolution image at each of a plurality of hypothesized depths, where the resolution of the portion of the third higher resolution image is higher than the resolutions of the portions of the third subset of the set of low resolution images used to fuse the portion of the first higher resolution image;

wherein comparing at least the portions of the first and second higher resolution images fused at each of the plurality of hypothesized depths further comprises comparing the portions of the first, second, and third higher resolution images fused at each of the plurality of hypothesized depths.

16. An array camera, comprising:

an array camera module, comprising;

an imager array, comprising;

a plurality of focal planes, where each focal plane comprises a two dimensional arrangement of pixels having at least two pixels in each dimension and each focal plane is contained within a region of the imager array that does not contain pixels from another focal plane;

control circuitry configured to control the capture of image information by the pixels within the focal planes; and

sampling circuitry configured to convert pixel outputs into digital pixel data;

interface circuitry configured to transmit digital pixel data;

an optic array of lens stacks, where an image including aliasing is formed on each focal plane by a separate lens stack in the optic array of lens stacks; and

a processor configured to receive digital pixel data from the array camera module via the interface circuitry; and

memory containing a image processing pipeline application and a controller application;

wherein the processor is configured via the controller application to read digital pixel data from the imager array;

wherein the image processing pipeline application configures the processor to;

obtain a set of low resolution images of a scene that include aliasing by reading digital pixel data from the imager array; and

synthesize a higher resolution image of the scene from a reference viewpoint using the set of low resolution images.

17. The array camera of claim 16, wherein the image processing pipeline application configures the processor to determine a depth of at least one pixel in the synthesized higher resolution image by:

fusing portions of the set of low resolution images to form a portion of a higher resolution image at each of a plurality of hypothesized depths, where the resolution of the portion of the higher resolution image is higher than the resolutions of the portions of the set of low resolution images used to fuse the portion of the higher resolution image;

comparing the portion of the fused higher resolution image obtained at each hypothesized depth to the scene captured in the set of low resolution images; and

selecting the hypothesized depth at which the portion of the fused higher resolution image is most similar to the scene captured in the set of low resolution images as the depth of at least one point in the scene captured by the set of low resolution images.

18. The array camera of claim 17, wherein the image processing pipeline application configures the processor to compare the portion of the fused higher resolution image to the scene captured in the set of low resolution images by:

generating a set of forward mapped low resolution image portions by forward mapping the portion of the fused higher resolution image using a mapping based upon the characteristics of the array camera module; and

comparing the forward mapped low resolution image portions with corresponding portions of corresponding images in the set of low resolution images.

19. The array camera of claim 17, wherein the image processing pipeline application configures the processor to compare the portion of the fused higher resolution image to the scene captured in the set of low resolution images by determining the similarity of pixels in pixel stacks in the portion of the fused higher resolution image.

20. The array camera of claim 16, wherein the image processing pipeline application configures the processor to determine a depth of at least one pixel in the synthesized higher resolution image by:

comparing portions of fused higher resolution images formed by fusing at least two subsets of the set of low resolution images at a plurality of hypothesized depths; and

selecting the depth of at least one pixel in the synthesized higher resolution image based upon the hypothesized depth at which the compared portions of fused higher resolution images are most similar.

21. The array camera of claim 20, wherein at least one low resolution image in the set of low resolution images is common to the at least two subsets of the set of low resolution images.

22. The array camera of claim 21, wherein the at least two subsets of the set of low resolution images include a common low resolution image having a viewpoint that is the reference viewpoint.

23. The array camera of claim 22, wherein the at least two subsets of the set of low resolution images include common low resolution images captured from viewpoints above, below, to the left, and to the right of the reference viewpoint.

24. The array camera of claim 20, wherein the reference viewpoint is the viewpoint of one of the low resolution images in the set of low resolution images.

25. The array camera of claim 20, wherein the image processing pipeline application configures the processor to fuse at least two subsets of the set of low resolution images at a hypothesized depth by:

identifying pixels within the subset of the set of low resolution images based upon the hypothesized depth and the viewpoints of the low resolution images;

fusing the identified pixels onto a higher resolution grid generated from a chosen reference viewpoint using known calibration information; and

performing hole filing to fill holes in locations in the higher resolution grid.

26. The array camera of claim 20, wherein the image processing pipeline application configures the processor to compare portions of fused higher resolution images by comparing matching error of the portions of the fused higher resolution images.

27. The array camera of claim 26, wherein matching error is determined using at least one selected from the group of the L₁-norm and the L₂-norm of the difference of the portions of the first and second higher resolution image.

28. The array camera of claim 17, wherein:

the pixels in the plurality of focal planes in the imager array comprise a pixel stack including a microlens and an active area, where light incident on the surface of the microlens is focused onto the active area by the microlens and the active area samples the incident light to capture image information; and

the pixel stack defines a pixel area and includes a pixel aperture, where the size of the pixel apertures is smaller than the pixel area.

29. The array camera of claim 28, wherein the pixel aperture is formed by a microlens that is smaller than the pixel area.

30. The array camera of claim 29, wherein gaps exist between adjacent microlenses in the pixel stacks of adjacent pixels in a focal plane.

31. The array camera of claim 30, wherein light is prevented from entering the pixel stacks through the gaps between the microlenses by a light blocking material.

32. The array camera of claim 28, wherein the pixel stack includes a color filter.

33. The array camera of claim 32, wherein the color filters in the pixel stacks of the two dimensional arrangement of pixels within a focal plane are the same.

34. The array camera of claim 32, wherein the color filters in the pixel stacks of the two dimensional arrangement of pixels within at least one focal plane form a Bayer filter pattern.

35. The array camera module of claim 28, wherein at least one of the plurality of lens stacks includes a color filter and the pixel stacks of the two dimensional arrangement of pixels within the focal plane on which said at least one of the plurality of lens stacks forms an image do not include color filters.

36. The array camera module of claim 28, wherein the pixel aperture is formed using at least one light blocking material.

37. The array camera module of claim 16, wherein the optic array of lens stacks is constructed using wafer level optics.

38. The array camera module of claim 16, wherein the plurality of lens stacks include polymer optical components.

39. The array camera module of claim 16, wherein the plurality of lens stacks include glass optical components.