Removing illumination variation from images

• US 8,798,393 B2
• Filed: 11/30/2011
• Issued: 08/05/2014
• Est. Priority Date: 12/01/2010
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Kwatra, Vivek, Han, Mei
• Primary Examiner(s): Kassa, Yosef
• Application Number:
  US13/308,411
• Publication Number:
  US 20120141044A1
• Time in Patent Office: 979 Days
• Field of Search: 382/260, 382/274, 382/275, 382/278, 358/3.26, 358/3.27, 358/463
• US Class Current: 382/274
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/30181 Earth observation
• CPC Class Codes: G06T 5/94 based on local image properties, e.g. for local contrast enhancement

Abstract

An image comprising varying illumination is selected. Patches of pixels from among the plurality of pixels with the image are identified. Similarities between pairs of patches of pixels based on pixel intensities associated with the pairs of patches of pixels are calculated. Illumination values for the plurality of pixels within the image based on the calculated similarities between the pairs of patches of pixels is calculated. The illumination variation from the image is removed based on the calculated illumination values for the plurality of pixels within the image.

Claims

1. A computer-implemented method of removing illumination variation from an image, the method comprising:

selecting, by one or more computers, an image containing varying illumination across a plurality of pixels within the selected image;

identifying, by the one or more computers, patches of pixels from among the plurality of pixels within the selected image;

calculating, by the one or more computers, similarities between pairs of patches of pixels from within the selected image based on pixel intensities associated with the pairs of patches of pixels;

calculating, by the one or more computers, illumination values for the plurality of pixels within the selected image based on the calculated similarities between the pairs of patches of pixels from within the selected image; and

removing, by the one or more computers, the illumination variation from the selected image based on the calculated illumination values for the plurality of pixels within the selected image.

2. The computer-implemented method of claim 1, wherein identifying the patches of pixels comprises:

determining a plurality of adjacent pixels that are adjacent to a pixel in the selected image; and

assigning the plurality of adjacent pixels and the pixel to a patch that is associated with the pixel.

3. The computer-implemented method of claim 2, wherein calculating the similarities comprises:

randomly selecting a second pixel from the selected image;

determining a second plurality of adjacent pixels that are adjacent to the second pixel in the selected image;

assigning the second plurality of pixels and the second pixel to a second patch that is associated with the second pixel; and

comparing the patch and the second patch based on a similarity of the pixel and the first patch and a similarity of the second pixel and the second patch.

4. The computer-implemented method of claim 1, wherein calculating the similarities between the pairs of patches comprises, for each patch:

calculating a reflectance value of a pixel included in the patch.

5. The computer-implemented method of claim 4, further comprising:

calculating a similarity between a first patch and a second patch from the selected image based on a difference of reflectance values associated with the first patch and the second patch; and

calculating a weight for the similarity between the first patch and the second patch, the weight describing an influence that the similarity between the first patch and the second patch has on the calculation of the illumination values for the plurality of pixels within the selected image.

6. The computer-implemented method of claim 5, wherein calculating the similarity between the first patch and the second patch comprises:

calculating a difference of a first reflectance value of a first pixel from the first patch and a second reflectance value of a second pixel from the second patch, wherein the first pixel is located at a center of the first patch and the second pixel is located at a center of the second patch.

7. The computer-implemented method of claim 5, wherein calculating the similarity between the first patch and the second patch comprises for each corresponding pair of pixels from the first patch and the second patch:

calculating a difference of a first reflectance value of a first pixel from the first patch and a second reflectance value of a second pixel from the second patch, wherein the first pixel and second pixel are respectively located at a same position in the first patch and the second patch.

8. The computer-implemented method of claim 5, wherein calculating the similarity between the first patch and the second patch comprises:

calculating a difference of a first distribution of reflectance values of pixels within the first patch and a second distribution of reflectance values of pixels within the second patch.

9. The computer-implemented method of claim 1, wherein calculating the illumination values comprises:

calculating an illumination value for a pixel within the selected image based on a difference between the illumination value for the pixel and an illumination value for a second pixel from one of the identified patches and a difference between a pixel intensity of the pixel and a pixel intensity of the second pixel.

10. The computer-implemented method of claim 9, wherein calculating the illumination values comprises:

optimizing an energy function to calculate the illumination value for the pixel within the selected image based on the difference between the illumination value for the pixel and the illumination value for the second pixel from one of the identified patches and the difference between the pixel intensity of the pixel and the pixel intensity of the second pixel and further based on a difference between the illumination value for the pixel and an illumination value for each third pixel that is directly adjacent to the pixel in the selected image; and

wherein an illumination value for a defined region of the selected image is used as an initial illumination value to optimize the energy function.

11. The computer-implemented method of claim 1, wherein the calculated illumination values are inverse illumination values and wherein removing the illumination variation from the selected image comprises:

for each pixel in the selected image, multiplying a pixel intensity of the pixel by an inverse illumination value calculated for the pixel.

12. A non-transitory computer-readable storage medium storing computer-executable code for removing illumination variation from an image, the code executable to perform steps comprising:

selecting an image containing varying illumination across a plurality of pixels within the selected image;

identifying patches of pixels from among the plurality of pixels within the selected image;

calculating similarities between pairs of patches of pixels from within the selected image based on pixel intensities associated with the pairs of patches of pixels;

calculating illumination values for the plurality of pixels within the selected image based on the calculated similarities between the pairs of patches of pixels from within the selected image; and

removing the illumination variation from the selected image based on the calculated illumination values for the plurality of pixels within the selected image.

13. The non-transitory computer-readable storage medium of claim 12, wherein the code is executable to perform further steps comprising:

determining a plurality of adjacent pixels that are adjacent to a pixel in the selected image; and

assigning the plurality of adjacent pixels and the pixel to a patch that is associated with the pixel.

14. The non-transitory computer-readable storage medium of claim 12, wherein the code is executable to perform further steps comprising, for each patch:

calculating a reflectance value of a pixel included in the patch.

15. The non-transitory computer-readable storage medium of claim 14, wherein the code is executable to perform further steps comprising:

calculating a similarity between a first patch and a second patch from the selected image based on a difference of reflectance values associated with the first patch and the second patch; and

calculating a weight for the similarity between the first patch and the second patch, the weight describing an influence that the similarity between the first patch and the second patch has on the calculation of the illumination values for the plurality of pixels within the selected image.

16. The non-transitory computer-readable storage medium of claim 15, wherein calculating the similarity between the first patch and the second patch comprises:

calculating a difference of a first reflectance value of a first pixel from the first patch and a second reflectance value of a second pixel from the second patch, wherein the first pixel is located at a center of the first patch and the second pixel is located at a center of the second patch.

17. The non-transitory computer-readable storage medium of claim 15, wherein calculating the similarity between the first patch and the second patch comprises for each corresponding pair of pixels from the first patch and the second patch:

calculating a difference of a first reflectance value of a first pixel from the first patch and a second reflectance value of a second pixel from the second patch, wherein the first pixel and second pixel are respectively located at a same position in the first patch and the second patch.

18. The non-transitory computer-readable storage medium of claim 15, wherein calculating the similarity between the first patch and the second patch comprises:

calculating a difference of a first distribution of reflectance values of pixels within the first patch and a second distribution of reflectance values of pixels within the second patch.

19. The non-transitory computer-readable storage medium of claim 12, wherein the code is executable to perform further steps comprising:

calculating an illumination value for a pixel within the selected image based on a difference between the illumination value for the pixel and an illumination value for a second pixel from one of the identified patches and a difference between a pixel intensity of the pixel and a pixel intensity of the second pixel.

20. The non-transitory computer-readable storage medium of claim 19, wherein the code is executable to perform further steps comprising:

optimizing an energy function to calculate the illumination value for the pixel within the selected image based on the difference between the illumination value for the pixel and the illumination value for the second pixel from one of the identified patches and the difference between the pixel intensity of the pixel and the pixel intensity of the second pixel and further based on a difference between the illumination value for the pixel and an illumination value for each third pixel that is directly adjacent to the pixel in the selected image; and

wherein an illumination value for a defined region of the selected image is used as an initial illumination value to optimize the energy function.

21. The non-transitory computer-readable storage medium of claim 12, wherein the calculated illumination values are inverse illumination values and wherein the code is executable to perform further steps comprising:

for each pixel in the selected image, multiplying a pixel intensity of the pixel by an inverse illumination value calculated for the pixel.

22. A computer system for removing illumination variation from an image, the system comprising:

one or more computer processors; and

a non-transitory computer-readable storage medium storing executable instructions configured to execute on the one or more computer processors, the instructions when executed by the one or more computer processors are configured to perform steps comprising;

selecting an image containing varying illumination across a plurality of pixels within the selected image;

identifying patches of pixels from among the plurality of pixels within the selected image;

calculating similarities between pairs of patches of pixels from within the selected image based on pixel intensities associated with the pairs of patches of pixels;

calculating illumination values for the plurality of pixels within the selected image based on the calculated similarities between the pairs of patches of pixels from within the selected image; and

removing the illumination variation from the selected image based on the calculated illumination values for the plurality of pixels within the selected image.

23. The computer system of claim 22, wherein the instructions when executed by the computer processor are configured to perform further steps comprising:

determining a plurality of adjacent pixels that are adjacent to a pixel in the selected image; and

assigning the plurality of adjacent pixels and the pixel to a patch that is associated with the pixel.

24. The computer system of claim 22, wherein the instructions when executed by the computer processor are configured to perform further steps comprising for each patch:

calculating a reflectance value of a pixel included in the patch.

25. The computer system of claim 24, wherein the instructions when executed by the computer processor are configured to perform further steps comprising:

calculating a similarity between a first patch and a second patch from the selected image based on a difference of reflectance values associated with the first patch and the second patch; and

calculating a weight for the similarity between the first patch and the second patch, the weight describing an influence that the similarity between the first patch and the second patch has on the calculation of the illumination values for the plurality of pixels within the selected image.

26. The computer system of claim 25, wherein calculating the similarity between the first patch and the second patch comprises:

calculating a difference of a first reflectance value of a first pixel from the first patch and a second reflectance value of a second pixel from the second patch, wherein the first pixel is located at a center of the first patch and the second pixel is located at a center of the second patch.

27. The computer system of claim 25, wherein calculating the similarity between the first patch and the second patch comprises for each corresponding pair of pixels from the first patch and the second patch:

calculating a difference of a first reflectance value of a first pixel from the first patch and a second reflectance value of a second pixel from the second patch, wherein the first pixel and second pixel are respectively located at a same position in the first patch and the second patch.

28. The computer system of claim 25, wherein calculating the similarity between the first patch and the second patch comprises:

calculating a difference of a first distribution of reflectance values of pixels within the first patch and a second distribution of reflectance values of pixels within the second patch.

29. The computer system of claim 22, wherein the instructions when executed by the computer processor are configured to perform further steps comprising:

calculating an illumination value for a pixel within the selected image based on a difference between the illumination value for the pixel and an illumination value for a second pixel from one of the identified patches and a difference between a pixel intensity of the pixel and a pixel intensity of the second pixel.

30. The computer system of claim 29, wherein the instructions when executed by the computer processor are configured to perform further steps comprising:

optimizing an energy function to calculate the illumination value for the pixel within the selected image based on the difference between the illumination value for the pixel and the illumination value for the second pixel from one of the identified patches and the difference between the pixel intensity of the pixel and the pixel intensity of the second pixel and further based on a difference between the illumination value for the pixel and an illumination value for each third pixel that is directly adjacent to the pixel in the selected image; and

wherein an illumination value for a defined region of the selected image is used as an initial illumination value to optimize the energy function.

31. The computer system of claim 22, wherein the instructions when executed by the computer processor are configured to perform further steps comprising:

for each pixel in the selected image, multiplying a pixel intensity of the pixel by an inverse illumination value calculated for the pixel.