Flare Detection and Mitigation in Panoramic Images

• US 20130329132A1
• Filed: 06/11/2012
• Published: 12/12/2013
• Est. Priority Date: 06/06/2012
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Tico, Marius, Hubel, Paul M., Doepke, Frank, Sachs, Todd S.
• Granted Patent: US 9,253,373 B2
• Time in Patent Office: Days
• US Class Current: 348/571
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/30242 Counting objects in image
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 25/60 Noise processing, e.g. detecting, correcting, reducing or removing noise
• CPC Class Codes: H04N 25/61 the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"
• CPC Class Codes: H04N 5/14 Picture signal circuitry for video frequency region cameras or camera modules comprising electronic image sensors, or control thereof H04N23/00

Abstract

Lens flare mitigation techniques determine which pixels in images of a sequence of images are likely to be pixels affected by lens flare. Once the lens flare areas of the images are determined, unwanted lens flare effects may be mitigated by various approaches, including reducing border artifacts along a seam between successive images, discarding entire images of the sequence that contain lens flare areas, and using tone-mapping to reduce the visibility of lens flare.

Claims

1. A non-transitory program storage device, readable by a programmable control device, comprising instructions stored thereon for causing the programmable control device to:

capture a sequence of images with a programmable imaging device; and

reduce lens flare effects in the sequence of images, comprising instructions for causing the programmable control device to;

identify an overlap region between successive images of the sequence of images;

detect lens flare regions in the overlap region; and

reduce lens flare effects in the overlap region.

2. The program storage device of claim 1, wherein the instructions stored thereon further comprise instruction for causing the programmable control device to:

generate a panoramic image from the sequence of images.

3. The program storage device of claim 1, wherein the instructions for causing the programmable device to detect lens flare regions in the overlap region comprise instructions for causing the programmable device to:

examine auto-exposure parameters of the sequence of images; and

detect discontinuities in the auto-exposure parameters, wherein the discontinuities indicate a lens flare is present in a corresponding image of the sequence of images.

4. The program storage device of claim 1, wherein the instructions for causing the programmable device to detect lens flare regions in the overlap region comprise instructions for causing the programmable device to:

determine a color locus of lens flare regions in a predetermined color space; and

calculate an average intensity gradient magnitude in a spatial region surrounding a pixel of an image of the sequence of images corresponding to the color locus.

5. The program storage device of claim 1, wherein the instructions for causing the programmable device to detect lens flare regions in the overlap region further comprise instructions for causing the programmable device to:

select pixels with a flare probability that exceeds a predetermined threshold probability value.

6. The program storage device of claim 1, wherein the instructions for causing the programmable device to detect lens flare regions in the overlap region comprise instructions for causing the programmable device to:

identify an image of the sequence of images containing lens flare; and

identify pixels in the image corresponding to lens flare.

7. The program storage device of claim 1, wherein the instructions for causing the programmable device to detect lens flare regions in the overlap region comprise instructions for causing the programmable device to:

omit an image containing a lens flare region in the overlap region.

8. The program storage device of claim 1, wherein the instructions for causing the programmable device to reduce lens flare effects in the overlap region comprise instructions for causing the programmable device to:

identify a seam between successive images; and

adjust a seam trajectory in a vertical direction if the seam is in a lens flare region.

9. A non-transitory program storage device, readable by a programmable control device, comprising instructions stored thereon for causing the programmable control device to:

capture a sequence of images using an image sensor of a programmable device;

identify an overlap region between successive images of the sequence of images and a seam between the successive images;

calculate an average intensity gradient magnitude in a spatial region surrounding pixels in the overlap region that have a color associated with lens flare regions;

calculate a probability that a pixel is a lens flare pixel based on the average intensity gradient magnitude;

identify the pixel as a lens flare pixel if the probability that the pixel is a lens flare pixel exceeds a predetermined threshold probability value;

determine a count of pixels of a row of pixels in the overlap region that are lens flare pixels;

adjust the seam to be vertical across the row of pixels if the count exceeds a predetermined threshold count value; and

generate a panoramic image from the sequence of images.

10. The program storage device of claim 9, further wherein the instructions further comprise instructions for causing the programmable control device to:

omit images of the sequence of images that contain lens flare pixels.

11. The program storage device of claim 9, further wherein the instructions further comprise instructions for causing the programmable control device to:

recolor lens flare pixels.

12. A programmable device, comprising:

an image sensor;

a programmable control device; and

a memory coupled to the programmable control device, wherein instructions are stored in the memory, the instructions causing the programmable control device to;

capture a sequence of images using the image sensor;

mitigate lens flare effects in the sequence of images, comprising instructions causing the programmable control device to;

identify an overlap region between successive images of the sequence of images;

detect lens flare regions in the overlap region; and

mitigate lens flare effects in the overlap region; and

generate a panoramic image from the sequence of images.

13. The programmable device of claim 12, wherein the instructions causing the programmable control device to mitigate lens flare effects in the overlap region comprise instructions causing the programmable control device to:

omit images containing lens flare regions from the sequence of images.

14. The programmable device of claim 12, wherein the instructions causing the programmable control device to mitigate lens flare effects in the overlap region comprise instructions causing the programmable control device to:

recolor lens flare pixels in the overlap region.

15. The programmable device of claim 14, wherein the instructions causing the programmable control device to recolor lens flare pixels in the overlap region comprise instructions causing the programmable control device to:

determine a mapping function that matches a color of lens flare pixels with a color of corresponding pixels of a successor or predecessor image of the sequence of images; and

recolor the lens flare pixels by tone mapping the lens flare pixels based on the mapping function.

16. The programmable device of claim 12, wherein the instructions causing the programmable control device to mitigate lens flare effects in the overlap region comprise instructions causing the programmable control device to:

bias a seam between successive images to vertical across lens flare regions of the overlap region.

17. The programmable device of claim 12, wherein the instructions for causing the programmable control device to detect lens flare regions in the overlap region comprise instructions for causing the programmable control device to:

examine auto-exposure parameters of the sequence of images; and

detect discontinuities in the auto-exposure parameters, wherein the discontinuities indicate a lens flare is present in a corresponding image of the sequence of images.

18. The programmable device of claim 12, wherein the instructions for causing the programmable control device to detect lens flare regions in the overlap region comprise instructions for causing the programmable control device to:

calculate an average intensity gradient magnitude in a spatial region surrounding a pixel of an image of the sequence of images having a color corresponding to lens flare regions.

19. The programmable device of claim 12, wherein the instructions for causing the programmable control device to detect lens flare regions in the overlap region further comprise instructions for causing the programmable control device to:

select pixels with a flare probability that exceeds a predetermined threshold probability value.

20. The programmable device of claim 12, wherein the instructions for causing the programmable control device to detect lens flare regions in the overlap region comprise instructions for causing the programmable control device to:

identify an image on the sequence of images containing lens flare; and

identify pixels in the image corresponding to lens flare.

21. The programmable device of claim 12, wherein the instructions for causing the programmable device to detect lens flare regions in the overlap region comprise instructions for causing the programmable device to:

omit an image containing a lens flare region in the overlap region.

22. The programmable device of claim 12, wherein the instructions for causing the programmable control device to reduce lens flare effects in the overlap region comprise instructions for causing the programmable control device to:

identify a seam between successive images; and

adjust a seam trajectory in a vertical direction if the seam is in a lens flare region.

23. A method of mitigating lens flare effects in a sequence of images, comprising:

identifying an overlap region between successive images of the sequence of images;

detecting a lens flare region in the overlap region; and

reducing lens flare effects in the lens flare region.

24. The method of claim 23, wherein reducing lens flare effect in the lens flare region comprises:

omitting an image containing the lens flare region from the sequence of images.

25. The method of claim 23, wherein reducing lens flare effects in the lens flare region comprises:

determining a mapping function that matches a color of lens flare pixels with a color of corresponding pixels of a successor or predecessor image of the sequence of images; and

tone mapping the lens flare pixels based on the mapping function.

26. The method of claim 23, wherein reducing lens flare effects in the lens flare region comprises:

biasing a seam between successive images to vertical across lens flare regions of the overlap region.

27. The method of claim 23, wherein detecting a lens flare region comprises:

examining auto-exposure parameters of the sequence of images; and

detecting discontinuities in the auto-exposure parameters, wherein the discontinuities indicate a lens flare is present in a corresponding image of the sequence of images.

28. The method of claim 23, wherein reducing lens flare effects in the lens flare region comprises:

calculating an average intensity gradient magnitude in a spatial region surrounding a pixel of an image of the sequence of images having a color corresponding to lens flare regions

calculating a probability that a pixel is a lens flare pixel based on the average intensity gradient magnitude; and

identifying the pixel as a lens flare pixel if the probability that the pixel is a lens flare pixel exceeds a predetermined threshold value;

29. A programmable device, comprising:

an image sensor;

a programmable control device; and

a memory coupled to the programmable control device, wherein instructions are stored in the memory, the instructions causing the programmable control device to;

capture a sequence of images using the image sensor;

mitigate lens flare effects in a lens flare region of a first image of the sequence of images overlapped by a second region of the sequence of images; and

generate a composite image from the sequence of images.

30. The programmable device of claim 29, wherein the instructions for causing the programmable control device to mitigate lens flare effects comprise instructions for causing the programmable control device to:

identify an image of the sequence of images containing lens flare;

identify pixels in the image corresponding to the lens flare as lens flare pixels; and

bias a seam between successive images of the sequence of images crossing lens flare pixels to vertical.