Detecting illumination in images

• US 8,611,660 B2
• Filed: 02/12/2013
• Issued: 12/17/2013
• Est. Priority Date: 12/17/2009
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Finlayson, Graham D., Drew, Mark Samuel, Fredembach, Clement
• Primary Examiner(s): SHAH, UTPAL D
• Application Number:
  US13/764,833
• Publication Number:
  US 20130170743A1
• Time in Patent Office: 308 Days
• Field of Search: 382/173, 382/191
• US Class Current: 382/173
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10152 Varying illumination
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06V 10/60 relating to illumination properties, e.g. using a reflectance or lighting model

Abstract

An image having m light sources, with m preferably equaling 2 or 3, is segmented into different regions, each of which is lit by only one of the m light sources, by obtaining paired imaged with different filtering, for example a filtered and an unfiltered image, applying to the image pairs sets of m pre-computed mappings at the pixel or region level, and selecting the most appropriate. The rendering of the information in the image maybe adjusted accordingly.

Claims

1. A non-transitory program storage device comprising instructions stored thereon to cause one or more processors to:

obtain a first image of a scene that is illuminated with m light sources, wherein the first image is associated with a first spectral component;

obtain a second image of the scene, wherein the second image is associated with a second spectral component that is different from the first spectral component;

obtain mappings between regions in the first image and spatially corresponding regions in the second image;

apply the mappings to regions of the first image to obtain values for the spatially corresponding regions of the second image, wherein each value is indicative of the likelihood that the value'"'"'s corresponding region is illuminated by 1 of the m light sources;

identify that mapping whose associated value is most indicative of one of the m light sources; and

identify the one light source as an illuminant of the scene.

2. The non-transitory program storage device of claim 1, wherein the instructions to cause the one or more processors to obtain first and second images comprise instructions to cause the one or more processors to capture the first and second images with an imaging device.

3. The non-transitory program storage device of claim 2, wherein the instructions to cause the one or more processors to capture the first and second images comprise instructions to cause the one or more processors to capture the first image with a first sensor and the second image with a second different sensor of the imaging device.

4. The non-transitory program storage device of claim 1, wherein the instructions to cause the one or more processors to obtain first and second images comprise instructions to cause the one or more processors to retrieve first and second images from a storage device.

5. The non-transitory program storage device of claim 1, wherein the first image comprises an RGB image and the second image comprises a filtered RGB image.

6. The non-transitory program storage device of claim 1, wherein the instructions to cause the one or more processors to obtain mappings comprise instructions to cause the one or more processors to obtain N mappings, where N>

m.

7. The non-transitory program storage device of claim 6, wherein the instructions to cause the one or more processors to apply the mappings comprise instructions to cause the one or more processors to apply N-choose-m combinations of the N mappings.

8. The non-transitory program storage device of claim 6, wherein the instructions to cause the one or more processors to apply the mappings comprise instructions to cause the one or more processors to segment the first image into different regions, each of which is lit by only one of the m light sources.

9. The non-transitory program storage device of claim 7, wherein the instructions to cause the one or more processors to identify that mapping whose associated value is most indicative of one of the m light sources, comprise instructions to cause the one or more processors to identify that mapping that has the lowest prediction error.

10. The non-transitory program storage device of claim 1, wherein the instructions to cause the one or more processors to identify that mapping whose associated value is most indicative of one of the m light sources comprise instructions to cause the one or more processors to identify that mapping that, based on the first image, most closely predicts the second image.

11. The non-transitory program storage device of claim 1, further comprising instructions to cause the one or more processors to render the first image based, at least in part, on the identification of the m illuminants,

12. The non-transitory program storage device of claim 1, wherein the instructions to cause the one or more processors to obtain mappings comprise instructions to cause the ne or more processors to apply robust statistical procedures to generate the mappings.

13. A method of processing an image, comprising:

obtaining a first image of a scene that is illuminated with m light sources, wherein the first image is associated with a first spectral component;

obtaining a second image of the scene, wherein the second image is associated with a second spectral component that is different from the first spectral component;

obtaining mappings between regions in the first image and spatially corresponding regions in the second image;

applying the mappings to regions of the first image to obtain values for the spatially corresponding regions of the second image, wherein each value is indicative of the likelihood that the value'"'"'s corresponding region is illuminated by 1 of the m light sources;

identifying that mapping whose associated value is most indicative of one of the m light sources; and

identifying the one light source as an illuminant of the scene.

14. The method of claim 13, wherein the act of obtaining first and second images comprises capturing the first and second images with an imaging device.

15. The method of claim 14, wherein the act of capturing the first and second images comprises capturing the first image with a first sensor and the second image with a second different sensor of the imaging device.

16. The method of claim 13, wherein the act of obtaining first and second images cot prises retrieving first and second images from a storage device.

17. The method of claim 13, wherein the first image comprises an RGB image and the second image comprises a filtered RGB image.

18. The method of claim 13, wherein the act of obtaining mappings comprises obtaining N mappings, where N>

m.

19. The method of claim 18, wherein the act of applying the mappings comprises applying N-choose-m combinations of the N mappings.

20. The method of claim 18, wherein the act of applying the mappings comprises segmenting the first image into different regions, each of which is lit by only one of the m light sources.

21. The method of claim 19, wherein the act of identifying that mapping whose associated value is most indicative of one of the m light sources, comprises identifying that mapping that has the lowest prediction error.

22. The method of claim 13, wherein the act of identifying that mapping whose associated value is most indicative of one of the m light sources comprises identifying that mapping that, based on the first image, most closely predicts the second image.

23. The method of claim 13, further comprising rendering the first image based, at least in part, on the identification of the m illuminants.

24. The method of claim 13, wherein the act of obtaining mappings comprises applying robust statistical procedures to generate the mappings.

25. An electronic device, comprising:

memory;

one or more image capture sensors operatively coupled to the memory;

a display operatively coupled to the memory; and

one or more processors operatively coupled to the memory, one or more image capture sensors and the display and configured to execute instructions stored in the memory to cause the one or more processors to—

obtain a first image of a scene that is illuminated with m light sources, wherein the first image is associated with a first spectral component,

obtain a second image of the scene, wherein the second image is associated with a second spectral component that is different from the first spectral component,

obtain mappings between regions in the first image and spatially corresponding regions in the second image,

apply the mappings to regions of the first image to obtain values for the spatially corresponding regions of the second image, wherein each value is indicative of the likelihood that the value'"'"'s corresponding region is illuminated by 1 of the m light sources,

identify that mapping whose associated value is most indicative of one of the m light sources, and

identify the one light source as an illuminant of the scene.