Mismatched Foreign Light Detection And Mitigation In The Image Fusion Of A Two-Camera System

• US 20170359534A1
• Filed: 06/08/2017
• Published: 12/14/2017
• Est. Priority Date: 06/10/2016
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Li, Xiaoxing, Hubel, Paul M., Tico, Marius, Szumilas, Lech J.
• Granted Patent: US 10,298,864 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 25/61 the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"
• CPC Class Codes: H04N 5/265 Mixing

Abstract

Foreign lighting effects, such as lens flare, are very common in natural images. In a two-camera-system, the two images may be fused together to generate one image of a better quality. However, there are frequently different foreign light patterns in the two images that form the image pair, e.g., due to the difference in lens design, sensor and position, etc. Directly fusing such pairs of images will result in non-photorealistic images, with composed foreign light patterns from both images from the image pair. This disclosure describes a general foreign light mitigation scheme to detect all kinds of foreign light region mismatches. The detected foreign light mismatch regions may be deemphasized or excluded in the fusion step, in order to create a fused image that keeps a natural-looking foreign light pattern that is close to what was seen by the user of an image capture device during an image capture preview mode.

Claims

1. A non-transitory program storage device, readable by a programmable control device, comprising instructions stored thereon that, when executed, cause the programmable control device to:

obtain a reference image and a non-reference image from an imaging system, wherein the reference image and non-reference image share an overlap region;

determine a mask region within the overlap region, wherein the mask region comprises a region wherein a mismatched foreign light effect has been detected between the reference image and the non-reference image; and

fuse the reference image and the non-reference image according to the determined mask region.

2. The non-transitory program storage device of claim 1, wherein the imaging system comprises a first camera unit and a second camera unit.

3. The non-transitory program storage device of claim 2, wherein the reference image is obtained from the first camera unit and the non-reference image is obtained from the second camera unit.

4. The non-transitory program storage device of claim 1, further comprising instructions stored thereon for causing the programmable control device to:

map values of pixels of the non-reference image to values of corresponding pixels of the reference image using a parameter,

wherein the value of the parameter for a particular pixel indicates a likelihood that the particular pixel represents a mismatched foreign light effect between the reference image and the non-reference image at a location of the particular pixel.

5. The non-transitory program storage device of claim 1, wherein the instructions to determine the mask region are only executed when it is determined that there is at least one region comprising a mismatched foreign light effect present in the reference image.

6. The non-transitory program storage device of claim 1, wherein the instructions to determine the mask region further comprise instructions to perform one or more thresholding or morphological operations on values of one or more pixels in the overlap region.

7. The non-transitory program storage device of claim 1, wherein the instructions to determine the mask region further comprise instructions to modulate values of one or more pixels in the overlap region according to a determined foreign light probability value for the respective pixel.

8. The non-transitory program storage device of claim 1, wherein the instructions to fuse the reference image and the non-reference image further comprise instructions to:

fuse the reference image and the non-reference image according to a fusion confidence map,

wherein the fusion confidence map comprises a confidence value for each pixel in the non-reference image, the confidence value indicating a confidence with which the respective pixel should be fused with the corresponding pixel in the reference image.

9. The non-transitory program storage device of claim 1, wherein the instructions to fuse the reference image and the non-reference image further comprise instructions to omit the mask region of the non-reference image from the fusion.

10. An image capture device, comprising:

an imaging system;

a programmable control device; and

a memory coupled to the programmable control device, wherein instructions are stored in the memory, the instructions, when executed, causing the programmable control device to;

capture a reference image and a non-reference image using the imaging system, wherein the reference image and non-reference image share an overlap region;

determine a mask region within the overlap region, wherein the mask region comprises a region wherein a mismatched foreign light effect has been detected between the reference image and the non-reference image; and

fuse the reference image and the non-reference image according to the determined mask region.

11. The image capture device of claim 10, wherein the imaging system comprises a first camera unit and a second camera unit.

12. The image capture device of claim 11, wherein the reference image is obtained from the first camera unit and the non-reference image is obtained from the second camera unit.

13. The image capture device of claim 10, further comprising instructions stored in the memory for causing the programmable control device to:

map values of pixels of the non-reference image to values of corresponding pixels of the reference image using a parameter,

wherein the value of the parameter for a particular pixel indicates a likelihood that the particular pixel represents a mismatched foreign light effect between the reference image and the non-reference image at a location of the particular pixel.

14. The image capture device of claim 10, wherein the instructions to determine the mask region are only executed when it is determined that there is at least one region comprising a mismatched foreign light effect present in the reference image.

15. The image capture device of claim 10, wherein the instructions to determine the mask region further comprise instructions to perform one or more thresholding or morphological operations on values of one or more pixels in the overlap region.

16. The image capture device of claim 10, wherein the instructions to determine the mask region further comprise instructions to modulate values of one or more pixels in the overlap region according to a determined foreign light probability value for the respective pixel.

17. The image capture device of claim 10, wherein the instructions to fuse the reference image and the non-reference image further comprise instructions to:

fuse the reference image and the non-reference image according to a fusion confidence map,

wherein the fusion confidence map comprises a confidence value for each pixel in the non-reference image, the confidence value indicating a confidence with which the respective pixel should be fused with the corresponding pixel in the reference image.

18. The image capture device of claim 10, wherein the instructions to fuse the reference image and the non-reference image further comprise instructions to omit the mask region of the non-reference image from the fusion.

19. An image fusion method, comprising:

obtaining a reference image and a non-reference image using an imaging system, wherein the reference image and non-reference image share an overlap region;

determining a mask region within the overlap region, wherein the mask region comprises a region wherein a mismatched foreign light effect has been detected between the reference image and the non-reference image; and

fusing the reference image and the non-reference image according to the determined mask region.

20. The method of claim 19, further comprising:

mapping values of pixels of the non-reference image to values of corresponding pixels of the reference image using a parameter,

wherein the value of the parameter for a particular pixel indicates a likelihood that the particular pixel represents a mismatched foreign light effect between the reference image and the non-reference image at a location of the particular pixel.

21. The method of claim 19, wherein the act of determining the mask region is only performed when it is determined that there is at least one region comprising a mismatched foreign light effect present in the reference image.

22. The method of claim 19, wherein the act of determining the mask region further comprises modulating values of one or more pixels in the overlap region according to a determined foreign light probability value for the respective pixel.

23. The method of claim 19, wherein the act of fusing the reference image and the non-reference image further comprises:

fusing the reference image and the non-reference image according to a fusion confidence map,

wherein the fusion confidence map comprises a confidence value for each pixel in the non-reference image, the confidence value indicating a confidence with which the respective pixel should be fused with the corresponding pixel in the reference image.

24. The method of claim 19, wherein the act of fusing the reference image and the non-reference image further comprises omitting the mask region of the non-reference image from the act of fusing.