Systems and methods for capturing digital images

• US 10,469,714 B2
• Filed: 06/28/2017
• Issued: 11/05/2019
• Est. Priority Date: 07/01/2016
• Status: Active Grant

• Current Assignee: Duelight LLC
• Original Assignee: Duelight LLC
• Inventors: Rivard, William Guie, Feder, Adam Barry, Kindle, Brian J.
• Primary Examiner(s): Haskins, Twyler L
• Assistant Examiner(s): Tejano, Dwight Alex C
• Application Number:
  US15/636,324
• Publication Number:
  US 20180007240A1
• Time in Patent Office: 860 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/20192 Edge enhancement; Edge preservation
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/72 Combination of two or more compensation controls
• CPC Class Codes: H04N 23/74 by influencing the scene brightness using illuminating means
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/76 by influencing the image signals
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/81 for suppressing or minimising disturbance in the image signal generation
• CPC Class Codes: H04N 25/60 Noise processing, e.g. detecting, correcting, reducing or removing noise
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20
• CPC Class Codes: H04N 5/44504 Circuit details of the additional information generator, e.g. details of the character or graphics signal generator, overlay mixing circuits

Abstract

A system, method, and computer program product are provided for capturing digital images. In use, at least one ambient exposure parameter is determined, and at least one flash exposure parameter based on the at least one ambient exposure parameter is determined. Next, via at least one camera module, an ambient image is captured according to the at least one ambient exposure parameter, and, via the at least one camera module, a flash image is captured according to the at least one flash exposure parameter. The captured ambient image and the captured flash image are stored. Lastly, the captured ambient image and the captured flash image are combined to generate a first merged image. Additional systems, methods, and computer program products are also presented.

Claims

1. A device, comprising:

a non-transitory memory storing instructions; and

one or more processors in communication with the non-transitory memory, wherein the one or more processors execute the instructions to;

determine at least one ambient exposure parameter, wherein the at least one ambient exposure parameter includes capturing a sequence of images wherein each image is captured with a successively refined estimate until an ambient exposure goal is satisfied;

determine at least one flash exposure parameter based on the at least one ambient exposure parameter;

capture, via at least one camera module and within a first analog storage plane of an image sensor, an ambient image according to the at least one ambient exposure parameter;

capture, via the at least one camera module and within a second analog storage plane of the image sensor, a flash image according to the at least one flash exposure parameter;

store the captured ambient image and the captured flash image; and

combine the captured ambient image and the captured flash image to generate a first merged image.

2. The device of claim 1, wherein the device is configured such that the at least one ambient exposure parameter includes at least one of an exposure time, an exposure sensitivity (ISO value), a white balance, or a lens aperture.

3. The device of claim 1, wherein the one or more processors further execute the instructions to implement at least one of de-noising the ambient image prior to the combining, de-noising the flash image prior to the combining, or de-noising the first merged image.

4. The device of claim 3, wherein the device is configured such that at least one of the de-noising of the ambient image prior to the combining, the denoising of the flash image prior to the combining, or the de-noising of the first merged image generates at least one de-noised pixel, wherein the one or more processors further execute the instructions to:

calculate a first intermediate noise estimate based on an ambient image ISO value;

calculate a second intermediate noise estimate based on the ambient image ISO value and an ambient pixel intensity;

calculate a third intermediate noise estimate based on the ambient pixel intensity and a flash pixel intensity; and

combine the first intermediate noise estimate, the second intermediate noise estimate, and the third intermediate noise estimate to create an overall pixel noise estimate.

5. The device of claim 4, wherein the device is configured such that the first intermediate noise estimate is calculated as a smoothstep function configured to receive the ambient image ISO value as an input, a first ISO value as a left edge value, and a second ISO value as a right edge value.

6. The device of claim 4, wherein the device is configured such that the second intermediate noise estimate is calculated as a smoothstep function configured to receive the ambient pixel intensity as an input, a left edge value that is a function of the ambient image ISO value, and a right edge value that is an offset from the left edge value.

7. The device of claim 4, wherein the device is configured such that the third intermediate noise estimate is calculated according to a blend surface configured to receive the ambient pixel intensity and the flash pixel intensity.

8. The device of claim 4, wherein the one or more processors further execute the instructions to:

compute a patch-space sample at a location in patch-space that corresponds to a location of the at least one de-noised pixel in a pixel-space;

upon determining that the overall pixel noise estimate is above a predefined threshold, assign a value of the patch-space sample to the at least one de-noised pixel; and

upon determining that the overall pixel noise estimate is below the predefined threshold, assign a value of an ambient pixel to the at least one de-noised pixel.

9. The device of claim 1, wherein the device is configured such that the at least one flash exposure parameter includes at least one of an exposure time, an exposure sensitivity (ISO value), a white balance, a flash duration, or a lens aperture.

10. The device of claim 1, wherein the device is configured such that the at least one flash exposure parameter includes a flash image exposure time which is less than or equal to an ambient image exposure time associated with the at least one ambient exposure parameter.

11. The device of claim 1, wherein the device is configured such that the flash image is captured as one of a sequence of flash images captured according to successively refined flash exposure parameters, the sequence of flash images being captured until a flash exposure goal is satisfied.

12. The device of claim 11, wherein the device is configured such that at least one of a flash duration, a flash intensity, or a combination of the flash duration and the flash intensity is adjusted until the flash exposure goal is satisfied.

13. The device of claim 1, wherein the device is configured such that the ambient image is captured at a different resolution than the flash image.

14. The device of claim 1, wherein the device is configured such that the first merged image includes interpolated pixels from pixel data of the ambient image, the interpolated pixels providing a higher effective resolution.

15. The device of claim 1, wherein the device is configured such that the ambient image is captured at a lower resolution at a specific ISO value, and the flash image is captured at a higher resolution and a higher ISO value.

16. The device of claim 1, wherein the one or more processors further execute the instructions to perform an edge detection pass on ambient pixels of the ambient image to generate an edge-enhanced image, wherein the device is configured such that the edge-enhanced image is used to assign de-noising weights.

17. A computer program product comprising computer executable instructions stored on a non-transitory computer readable medium that when executed by a processor instruct the processor to:

determine at least one ambient exposure parameter, wherein the at least one ambient exposure parameter includes capturing a sequence of images wherein each image is captured with a successively refined estimate until an ambient exposure goal is satisfied;

determine at least one flash exposure parameter based on the at least one ambient exposure parameter;

capture, via at least one camera module and within a first analog storage plane of an image sensor, an ambient image according to the at least one ambient exposure parameter;

capture, via the at least one camera module and within a second analog storage plane of the image sensor, a flash image according to the at least one flash exposure parameter;

store the captured ambient image and the captured flash image; and

combine the captured ambient image and the captured flash image to generate a first merged image.

18. A computer-implemented method, comprising:

determining, using a processor, at least one ambient exposure parameter, wherein the at least one ambient exposure parameter includes capturing a sequence of images wherein each image is captured with a successively refined estimate until an ambient exposure goal is satisfied;

determining, using the processor, at least one flash exposure parameter based on the at least one ambient exposure parameter;

capturing, via at least one camera module and within a first analog storage plane of an image sensor, an ambient image according to the at least one ambient exposure parameter;

capturing, via the at least one camera module and within a second analog storage plane of the image sensor, a flash image according to the at least one flash exposure parameter;

storing the captured ambient image and the captured flash image; and

combining, using the processor, the captured ambient image and the captured flash image to generate a first merged image.

19. A device, comprising:

a non-transitory memory storing instructions; and

one or more processors in communication with the non-transitory memory, wherein the one or more processors execute the instructions to;

determine at least one ambient exposure parameter;

determine at least one flash exposure parameter based on the at least one ambient exposure parameter;

capture, via at least one camera module and within a first analog storage plane of an image sensor, an ambient image according to the at least one ambient exposure parameter;

capture, via the at least one camera module and within a second analog storage plane of the image sensor, a flash image according to the at least one flash exposure parameter;

store the captured ambient image and the captured flash image; and

combine the captured ambient image and the captured flash image to generate a first merged image;

wherein the device is configured such that the flash image is captured as one of a sequence of flash images captured according to successively refined flash exposure parameters, the sequence of flash images being captured until a flash exposure goal is satisfied.

20. The device of claim 19, wherein the device is configured such that at least one of a flash duration, a flash intensity, or a combination of the flash duration and the flash intensity is adjusted until the flash exposure goal is satisfied.

21. A device, comprising:

a non-transitory memory storing instructions; and

one or more processors in communication with the non-transitory memory, wherein the one or more processors execute the instructions to;

determine at least one ambient exposure parameter;

determine at least one flash exposure parameter based on the at least one ambient exposure parameter;

capture, via at least one camera module and within a first analog storage plane of an image sensor, an ambient image according to the at least one ambient exposure parameter;

capture, via the at least one camera module and within a second analog storage plane of the image sensor, a flash image according to the at least one flash exposure parameter;

store the captured ambient image and the captured flash image; and

combine the captured ambient image and the captured flash image to generate a first merged image;

wherein the device is configured such that the first merged image includes interpolated pixels from pixel data of the ambient image, the interpolated pixels providing a higher effective resolution.

22. A device, comprising:

a non-transitory memory storing instructions; and

one or more processors in communication with the non-transitory memory, wherein the one or more processors execute the instructions to;

determine at least one ambient exposure parameter;

determine at least one flash exposure parameter based on the at least one ambient exposure parameter;

capture, via at least one camera module and within a first analog storage plane of an image sensor, an ambient image according to the at least one ambient exposure parameter;

capture, via the at least one camera module and within a second analog storage plane of the image sensor, a flash image according to the at least one flash exposure parameter;

store the captured ambient image and the captured flash image; and

combine the captured ambient image and the captured flash image to generate a first merged image;

wherein the device is configured such that the ambient image is captured at a lower resolution at a specific ISO value, and the flash image is captured at a higher resolution and a higher ISO value.