Detecting illumination in images

• US 8,385,648 B2
• Filed: 11/08/2007
• Issued: 02/26/2013
• Est. Priority Date: 11/08/2006
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Finlayson, Graham D., Drew, Mark Samuel, Fredembach, Clement
• Primary Examiner(s): Mehta, Bhavesh
• Assistant Examiner(s): SHAH, UTPAL D
• Application Number:
  US12/514,079
• Publication Number:
  US 20100098330A1
• Time in Patent Office: 1,937 Days
• Field of Search: 382/173, 382/191
• US Class Current: 382/173
• CPC Class Codes: G06T 2207/20012 Locally adaptive
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/143 involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: H04N 23/88 for colour balance, e.g. white-balance circuits or colour temperature control

Abstract

An image having m light sources, with m preferably equaling 2 or 3, is sequenced into different regions, each of which is lit by only one of the m light sources, by obtaining paired imaged with different filtering, for example a filtered and an unfiltered image, applying to the image pairs sets of m pre-computed mappings at the pixel or region level, and selecting the most appropriate. The rendering of the information in the image maybe adjusted accordingly.

Claims

1. A method for identifying light sources in an image, comprising:

obtaining a first image having a first set of spectral components;

obtaining a second image that corresponds to the first image, the second image having a second set of spectral components that differs from the first set of spectral components; and

segmenting the first image into one or more regions that are illuminated by different light sources by;

applying a plurality of sets of two or more pre-computed mappings to the first image; and

determining which of the plurality of sets most closely maps pixels in the first image to corresponding pixels in the second image.

2. The method of claim 1 wherein each of the pre-computed mappings is a linear transform and wherein the act of applying a plurality of sets of two or more pre-computed mappings to the first image comprises applying each of two or more linear transforms in each of the plurality of sets to each pixel in the first image to determine predicted filtered pixel values.

3. The method of claim 1 wherein the act of determining which of the plurality of sets most closely maps pixels in the first image to corresponding pixels in the second image comprises performing a statistical analysis on results of the application of each of the plurality of sets to the first image.

4. The method of claim 1 wherein the first and second images are counterparts of each other with different filtering.

5. The method of claim 1 wherein the first image is an unfiltered image and the second image is a filtered counterpart of the first image.

6. The method of claim 1 wherein the first image is produced by an RGB camera and the second image is a filtered RGB counterpart of the first image.

7. The method of claim 1 wherein the first image is produced by an RGB camera and the second image corresponds to a set of one or more sensor responses corresponding to the first image.

8. The method of claim 7 wherein the set of sensor responses includes CMY values.

9. The method of claim 1 wherein the first image is a multispectral image taken by a camera with at least two sensors and the second image is a multispectral image taken by at least one further sensor of the camera.

10. The method of claim 1 further comprising rendering the first image based, at least in part, on the one or more regions.

11. The method of claim 1, wherein each of the plurality of sets comprises two pre-computed mappings.

12. The method of claim 11, wherein the act of determining which of the plurality of sets most closely maps pixels in the first image to corresponding pixels in the second image comprises:

determining a predicted filtered value for a plurality of pixels in the first image by applying each of the pre-computed mappings in each set to the plurality of pixels;

determining an error value, for each of the pre-computed mappings, between the predicted filtered value of each of the plurality of pixels and an actual value of a corresponding pixel in the second image;

determining, for each set, which of the pre-computed mappings in the set minimizes the error value for each of the plurality of pixels;

assigning, for each set, the error value for the determined pre-computed mapping to each of the plurality of pixels;

summing, for each set, the assigned error values to determine an aggregate error value for the set; and

identifying the set having a lowest aggregate error value as the set that most closely maps pixels in the first image to corresponding pixels in the second image.

13. The method of claim 1, wherein the act of segmenting the first image into one or more regions that are illuminated by different light sources further comprises assigning a label to a plurality of pixels in the first image, wherein the assigned label corresponds to one of the pre-computed mappings in the set that most closely maps pixels in the first image to corresponding pixels in the second image.

14. The method of claim 13, wherein the act of assigning a label to a plurality of pixels in the first image comprises assigning a label corresponding to the pre-computed mapping that minimizes an error between a calculated pixel value and an actual pixel value for a corresponding pixel in the second image.

15. The method of claim 13, wherein the act of segmenting the first image into one or more regions that are illuminated by different light sources further comprises:

dividing the first image into one or more image segments;

identifying, for each segment, the label that is assigned to the highest number of pixels in the segment; and

assigning the identified label to the one or more segments.

16. The method of claim 15, wherein the act of segmenting the first image into one or more regions that are illuminated by different light sources further comprises identifying each of the one or more segments that are assigned different labels as illuminated by different light sources.

17. A non-transitory program storage device having instructions stored thereon to cause one or more processors to:

receive a first image having a first set of spectral components;

receive a second image that corresponds to the first image, the second image having a second set of spectral components that differs from the first set of spectral components;

segment the first image into one or more regions that are illuminated by different light sources by;

applying a plurality of sets of two or more pre-computed mappings to the first image; and

determining which of the plurality of sets most closely maps pixels in the first image to corresponding pixels in the second image.

18. The non-transitory program storage device of claim 17, wherein the instructions to cause the one or more processors to apply a plurality of sets of two or more pre-computed mappings to the first image comprise instructions to cause the one or more processors to generate the plurality of sets from a plurality of pre-computed mappings.

19. The non-transitory program storage device of claim 18, wherein each of the plurality of pre-computed mappings maps an image to a filtered counterpart for a particular light source.

20. The non-transitory program storage device of claim 17, wherein the first image is an unfiltered image and the second image is a filtered counterpart of the first image.

21. The non-transitory program storage device of claim 17, wherein the instructions to cause the one or more processors to segment the first image into one or more regions that are illuminated by different light sources further comprise instructions to cause the one or more processors to assign a label to a plurality of pixels in the first image, wherein the assigned label corresponds to one of the pre-computed mappings in the set that most closely maps pixels in the first image to corresponding pixels in the second image.

22. The non-transitory program storage device of claim 21, wherein the instructions to cause the one or more processors to assign a label to a plurality of pixels in the first image comprise instructions to cause the one or more processors to assign a label corresponding to the pre-computed mapping that minimizes an error between a calculated pixel value and an actual pixel value for a corresponding pixel in the second image.

23. The non-transitory program storage device of claim 21, wherein the instructions to cause the one or more processors to segment the first image into one or more regions that are illuminated by different light sources further comprise instructions to cause the one or more processors to:

divide the first image into one or more image segments;

identify, for each segment, the label that is assigned to the highest number of pixels in the segment; and

assign the identified label to the one or more segments.