Systems and methods for reducing motion blur in images or video in ultra low light with array cameras

• US 10,547,772 B2
• Filed: 10/01/2018
• Issued: 01/28/2020
• Est. Priority Date: 03/14/2013
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Limited (Adeia Inc.)
• Inventors: Molina, Gabriel
• Primary Examiner(s): Camargo, Marly S
• Application Number:
  US16/148,317
• Publication Number:
  US 20190037116A1
• Time in Patent Office: 484 Days
• Field of Search: 3482221, 3482086, 3482181, 3482211, 348262, 348272, 348294, 348302, 348193, 348241, 348533, 348362, 348382, 382302, 382254-301, 359621, 2502081
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 3/4015 Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/682 Vibration or motion blur correction
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 23/71 Circuitry for evaluating the brightness variation
• CPC Class Codes: H04N 5/265 Mixing
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

Systems and methods for reducing motion blur in images or video in ultra low light with array cameras in accordance with embodiments of the invention are disclosed. In one embodiment, a method for synthesizing an image from multiple images captured using an array camera includes capturing image data using active cameras within an array camera, where the active cameras are configured to capture image data and the image data includes pixel brightness values that form alternate view images captured from different viewpoints, determining sets of corresponding pixels in the alternate view images where each pixel in a set of corresponding pixels is chosen from a different alternate view image, summing the pixel brightness values for corresponding pixels to create pixel brightness sums for pixel locations in an output image, and synthesizing an output image from the viewpoint of the output image using the pixel brightness sums.

Claims

1. A method for synthesizing an image from multiple images captured from different viewpoints using an array camera, the method comprising:

capturing image data using a plurality of active cameras within an array camera, where the plurality of active cameras are configured to capture image data comprising pixel brightness values that form a reference image and a plurality of alternate view images captured from different viewpoints;

applying geometric corrections to find correspondences between pixels in the plurality of alternate view images and pixels in the reference image using a processor configured by software using depth information;

summing the pixel brightness values for pixels in the reference image with pixel brightness values for corresponding pixels in the alternate view images to create pixel brightness sums for the pixel locations in the reference image using the processor configured by software; and

synthesizing an output image from the viewpoint of the reference image using image data comprising the pixel brightness sums for the pixel locations in the reference image using the processor configured by software.

2. The method of claim 1, further comprising:

performing parallax detection using the processor configured by software to identify scene dependent geometric shifts to apply to the alternate images by comparing the reference image and the alternate images;

wherein applying geometric corrections to find correspondences between pixels in the plurality of alternate images and pixels in the reference image further comprises applying scene dependent geometric shifts to the plurality of alternate images to compensate for parallax.

3. The method of claim 2, further comprising:

identifying pixels in the alternate images that are occluded in the reference image using the processor configured by software; and

leaving occluded pixels out when summing the pixel brightness values for pixels in the reference image with pixel brightness values for corresponding pixels in the alternate images using the processor configured by software.

4. The method of claim 1, further comprising:

performing parallax detection using the processor configured by software to identify scene dependent geometric shifts to apply to at least a portion of the pixels in the alternative images by comparing the reference image and the alternative images; and

when parallax detection identifies at least one pixel within a threshold distance of the reference viewpoint, applying scene dependent geometric shifts to the plurality of alternative images to compensate for parallax.

5. The method of claim 1, further comprising:

performing parallax detection using the processor configured by software to identify scene dependent geometric shifts to apply to at least a portion of the pixels in the alternative images by comparing the reference image and the alternative images; and

when parallax detection determines that a pixel from the reference viewpoint has a depth within a specified depth of field, applying scene dependent geometric shifts to corresponding pixels in the alternative images to compensate for parallax.

6. The method of claim 1, wherein applying geometric corrections to find correspondences between pixels in the plurality of alternate images and pixels in the reference image further comprises applying a fixed parallax shift to pixels in each of the plurality of alternative images.

7. The method of claim 1, further comprising:

capturing a second set of image data using the plurality of active cameras and synthesizing a second output image using the processor configured by software;

calculating motion compensation vectors for the second output image using the processor configured by software;

applying motion compensation shifts to shift the second output image to the viewpoint of the output image using the processor configured by software;

summing the pixel brightness values for pixels in the output image with pixel brightness values for corresponding pixels in the second output image to create pixel brightness sums for the pixel locations in the output image using the processor configured by software; and

synthesizing a motion compensated output image from the viewpoint of the reference image using the pixel brightness sums for the pixel locations in the output image using the processor configured by software.

8. The method of claim 1, wherein the plurality of active cameras that capture the reference image and the alternative images form a first subset of cameras and the method further comprises:

capturing image data using a second subset of active cameras within the array camera, where the second subset of active cameras are configured to capture image data comprising pixel brightness values that form a second reference image and a second set of alternative images captured from different viewpoints;

applying geometric shifts to shift the second set of alternative images to the viewpoint of the second reference image using the processor configured by software;

summing the pixel brightness values for pixels in the second reference image with pixel brightness values for corresponding pixels in the second set of alternative images to create pixel brightness sums for the pixel locations in the second reference image using a processor configured by software;

synthesizing an alternate view output image from the viewpoint of the second reference image using the pixel brightness sums for the pixel locations in the second reference image using the processor configured by software; and

synthesizing a high resolution image using the processor configured by software to perform a super resolution process based upon the output image and the alternate view output image.

9. The method of claim 1, wherein the array camera comprises cameras that capture image data within different spectral bands.

10. The method of claim 1, wherein the reference image is a virtual image synthesized in a location where none of the active cameras exist.

11. An imaging device for synthesizing an image from multiple images captured from different viewpoints, comprising:

an array camera;

memory containing an image synthesizer application for causing a processor to perform the steps of;

capturing image data using a plurality of active cameras within the array camera, where the plurality of active cameras are configured to capture image data comprising pixel brightness values that form a reference image and a plurality of alternate view images captured from different viewpoints;

applying geometric corrections to find correspondences between pixels in the plurality of alternate view images and pixels in the reference image using the processor configured by software using depth information;

summing the pixel brightness values for pixels in the reference image with pixel brightness values for corresponding pixels in the alternate view images to create pixel brightness sums for the pixel locations in the reference image using the processor configured by software; and

synthesizing an output image from the viewpoint of the reference image using image data comprising the pixel brightness sums for the pixel locations in the reference image using the processor configured by software.

12. The imaging device of claim 11, wherein the steps further comprise:

performing parallax detection using the processor configured by software to identify scene dependent geometric shifts to apply to the alternate images by comparing the reference image and the alternate images;

wherein applying geometric corrections to find correspondences between pixels in the plurality of alternate images and pixels in the reference image further comprises applying scene dependent geometric shifts to the plurality of alternate images to compensate for parallax.

13. The imaging device of claim 12, wherein the steps further comprise:

identifying pixels in the alternate images that are occluded in the reference image using the processor configured by software; and

leaving occluded pixels out when summing the pixel brightness values for pixels in the reference image with pixel brightness values for corresponding pixels in the alternate images using the processor configured by software.

14. The imaging device of claim 11, wherein the steps further comprise:

performing parallax detection using the processor configured by software to identify scene dependent geometric shifts to apply to at least a portion of the pixels in the alternative images by comparing the reference image and the alternative images; and

when parallax detection identifies at least one pixel within a threshold distance of the reference viewpoint, applying scene dependent geometric shifts to the plurality of alternative images to compensate for parallax.

15. The imaging device of claim 11, wherein the steps further comprise:

performing parallax detection using the processor configured by software to identify scene dependent geometric shifts to apply to at least a portion of the pixels in the alternative images by comparing the reference image and the alternative images; and

when parallax detection determines that a pixel from the reference viewpoint has a depth within a specified depth of field, applying scene dependent geometric shifts to corresponding pixels in the alternative images to compensate for parallax.

16. The imaging device of claim 11, wherein applying geometric corrections to find correspondences between pixels in the plurality of alternate images and pixels in the reference image further comprises applying a fixed parallax shift to pixels in each of the plurality of alternative images.

17. The imaging device of claim 11, wherein the steps further comprise:

capturing a second set of image data using the plurality of active cameras and synthesizing a second output image using the processor configured by software;

calculating motion compensation vectors for the second output image using the processor configured by software;

applying motion compensation shifts to shift the second output image to the viewpoint of the output image using the processor configured by software;

summing the pixel brightness values for pixels in the output image with pixel brightness values for corresponding pixels in the second output image to create pixel brightness sums for the pixel locations in the output image using the processor configured by software; and

synthesizing a motion compensated output image from the viewpoint of the reference image using the pixel brightness sums for the pixel locations in the output image using the processor configured by software.

18. The imaging device of claim 11, wherein the plurality of active cameras that capture the reference image and the alternative images form a first subset of cameras and the method further comprises:

capturing image data using a second subset of active cameras within the array camera, where the second subset of active cameras are configured to capture image data comprising pixel brightness values that form a second reference image and a second set of alternative images captured from different viewpoints;

applying geometric shifts to shift the second set of alternative images to the viewpoint of the second reference image using the processor configured by software;

summing the pixel brightness values for pixels in the second reference image with pixel brightness values for corresponding pixels in the second set of alternative images to create pixel brightness sums for the pixel locations in the second reference image using a processor configured by software;

synthesizing an alternate view output image from the viewpoint of the second reference image using the pixel brightness sums for the pixel locations in the second reference image using the processor configured by software; and

synthesizing a high resolution image using the processor configured by software to perform a super resolution process based upon the output image and the alternate view output image.

19. The imaging device of claim 11, wherein the array camera comprises cameras that capture image data within different spectral bands.

20. The imaging device of claim 11, wherein the reference image is a virtual image synthesized in a location where none of the active cameras exist.