Detection and manipulation of shadows in an image or series of images

• US 20070104389A1
• Filed: 11/09/2005
• Published: 05/10/2007
• Est. Priority Date: 11/09/2005
• Status: Active Grant

• Current Assignee: IP3 2017, Series 200 of Allied Security Trust I (Allied Security Trust)
• Original Assignee: AEPX Animation Incorporated
• Inventors: Wells, Barton
• Granted Patent: US 7,305,127 B2
• Time in Patent Office: Days
• US Class Current: 382/274
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20192 Edge enhancement; Edge preservation
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/94 based on local image properties, e.g. for local contrast enhancement
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06V 10/273 removing elements interfering with the pattern to be recognised

Abstract

Systems, methods, and computer program products that identify and allow manipulation of shadows in a still frame (photo mode), or series of frames (movie mode). Manipulation of shadows includes such things as adjustment of shadow boundaries, softening of shadow boundaries, adjustment of shadow properties, and even removal of shadows altogether.

Claims

1. A method for identifying pixels in shadow in an image, comprising:

analyzing an image for determining several properties thereof, the properties including hue (H), saturation (S), brightness (V), red color content (R), green color content (G), and blue color content (B);

creating a first histogram of H/S values calculated for each pixel in the image;

creating a second histogram of S/V values calculated for each pixel in the image;

identifying a line feature in the first histogram;

identifying a line feature in the second histogram; and

marking pixels having an H/S value above the line feature in the first histogram and an S/V value above the line feature in the second histogram as being in shadow.

2. A method as recited in claim 1, further comprising storing leveling data, wherein the finding the objects in the image uses the leveling data.

3. A method as recited in claim 1, further comprising smoothing the image and detecting edges in the image.

4. A method as recited in claim 1, further comprising detecting edges in the image and refining lines based on the detected edges in the image.

5. A method as recited in claim 1, further comprising detecting edges in the image and adjusting a boundary of the detected shadows in the image based on a proximity to the detected edges.

6. A method as recited in claim 1, wherein marking the pixels as being in shadow further comprises:

flagging pixels having an H/S value below the line feature in the first histogram as potentially not being in shadow;

flagging pixels having an H/S value above the line feature in the first histogram as potentially being in shadow;

flagging pixels having an S/V value below the line feature in the second histogram as potentially not being in shadow;

flagging pixels having an S/V value above the line feature in the second histogram as potentially being in shadow; and

marking pixels having two flags as being in shadow.

7. A method as recited in claim 1, wherein the method is performed for a series of images.

8. A computer program product, comprising:

a computer readable medium having computer code thereon for processing an image, the computer code including;

computer code for analyzing an image for determining several properties thereof, the properties including hue (H), saturation (S), brightness (V), red color content (R), green color content (G), and blue color content (B);

computer code for creating a first histogram of H/S values calculated for each pixel in the image;

computer code for creating a second histogram of S/V values calculated for each pixel in the image;

computer code for identifying a line feature in the first histogram;

computer code for identifying a line feature in the second histogram; and

computer code for marking pixels having an H/S value above the line feature in the first histogram and an S/V value above the line feature in the second histogram as being in shadow.

9. A computer program product, comprising:

a computer readable medium having computer code thereon for processing an image, the computer code including;

computer code for generating slide show and movie output;

computer code for performing batch processing of an image;

computer code for allowing image labeling;

computer code for matching color schemes across multiple images;

computer code for performing color replacement for changing a color of selected portions of an image;

computer code for performing shadow and highlight correction for improving contrast of over- or under-exposed areas of an image;

computer code for performing photo touch-up functions including removal of redeye, dust, scratches, blemishes, wrinkles;

computer code for performing dimensional effects on an image including image wrapping, stretching, curling, and bending;

computer code for removing image blurring;

computer code for correcting lens distortion;

computer code for image noise reduction;

computer code for cropping and straightening an image;

computer code for providing painting and drawing tools;

computer code for addition of text and labels; and

computer code for manipulating shadows in an image.

10. A method for manipulating shadows of an image, comprising:

analyzing an image for determining several properties thereof;

determining which pixels of the image are in shadow; and

performing at least one of shadow boundary adjustment and shadow boundary softening of the image for a group of pixels determined to be in shadow, an outer periphery of the pixels determined to be in shadow defining the shadow boundary.

11. A method as recited in claim 10, wherein the properties include hue (H), saturation (S), and brightness (V);

wherein determining which pixels of the image are in shadow further comprises creating a first histogram of H/S values calculated for each pixel in the image;

creating a second histogram of S/V values calculated for each pixel in the image;

identifying a line feature in the first histogram;

identifying a line feature in the second histogram; and

marking pixels having an H/S value above the line feature in the first histogram and an S/V value above the line feature in the second histogram as being in shadow.

12. A method as recited in claim 10, further comprising detecting edges in the image and performing shadow boundary adjustment, wherein the shadow boundary adjustment includes adjusting a boundary of the group of pixels in shadow based on a proximity to the detected edges.

13. A method as recited in claim 12, wherein the proximity to the detected edge is based on a predefined number of pixels.

14. A method as recited in claim 12, wherein the proximity to the detected edge is based on a percentage of total pixels in the image.

15. A method as recited in claim 12, wherein the proximity to the detected edge is based on a percentage of horizontal pixel width of the image.

16. A method as recited in claim 12, wherein the proximity to the detected edge is based on a percentage of vertical pixel height of the image.

17. A method as recited in claim 10, wherein shadow boundary softening is performed, wherein the shadow boundary softening includes adjusting at least one of the properties of pixels along the boundary of the shadow based on a linear interpolation between properties of pixels in shadow and properties of pixels not in shadow.

18. A method as recited in claim 10, wherein shadow boundary softening is performed, wherein the shadow boundary softening includes analyzing pixels positioned towards the shadow boundary, and adjusting at least one of the properties of the pixels being analyzed based on an average of similar properties of pixels surrounding each pixel being analyzed.

19. A method as recited in claim 10, further comprising providing the following features:

slide show and movie output of multiple images,

batch processing of multiple images,

image labeling,

matching color schemes across multiple images,

color replacement for changing a color of selected portions of an image,

shadow and highlight correction for improving the contrast of over- or underexposed areas of an image

photo touch-up including removal of redeye, dust, scratches, blemishes, wrinkles,

dimensional effects including image wrapping, stretching, curling, and bending,

removing image blurring,

correcting lens distortion,

image noise reduction,

cropping and straightening an image,

painting and drawing tools, and

allowing addition of text and labels.

20. A method as recited in claim 10, wherein the method is performed for a series of images.

21. A computer program product, comprising:

a computer readable medium having computer code thereon for processing an image, the computer code including;

computer code for analyzing an image for determining several properties thereof;

computer code for determining which pixels of the image are in shadow; and

computer code for performing at least one of shadow boundary adjustment and shadow boundary softening of the image for a group of pixels determined to be in shadow, an outer periphery of the pixels determined to be in shadow defining the shadow boundary.

22. A method for identifying an object in an image, comprising:

analyzing an image for determining several properties thereof;

finding objects in the image based at least in part on the properties of the image;

selecting a first object determined to be at least primarily in shadow;

selecting a second object adjacent the first object;

analyzing the two objects using the properties of the image for determining whether the two objects are part of a same overall object.

23. A method as recited in claim 22, wherein the objects in the image are found using a flood fill algorithm.

24. A method as recited in claim 22, wherein the objects in the image are found using a histogram.

25. A method as recited in claim 22, wherein the method is performed for a series of images.

26. A method as recited in claim 25, wherein at least some of the objects in the image are found using motion-based detection.

27. A method as recited in claim 22, wherein the second object is not in shadow.

28. A method as recited in claim 22, wherein analyzing the two objects includes:

comparing a ratio of an average red (R) value of pixels in the first object to an average R value of pixels in the second object against a predetermined value;

comparing a ratio of an average green (G) value of pixels in the first object to an average G value of pixels in the second object against a predetermined value;

comparing a ratio of an average blue (B) value of pixels in the first object to an average B value of pixels in the second object against a predetermined value;

comparing a ratio of an average saturation (S) value of the first object to an average S value of the second object against a predetermined value;

comparing a ratio of an average brightness (V) value of the first object to an average V value of the second object against a predetermined value;

wherein if the comparisons of the ratios to the predetermined values match predefined criteria, the first and second objects are determined to be in the same overall object.

29. A computer program product, comprising:

a computer readable medium having computer code thereon for processing an image, the computer code including;

computer code for analyzing an image for determining several properties thereof;

computer code for finding objects in the image based at least in part on the properties of the image;

computer code for determining whether one of the objects of the image is at least primarily in shadow;

computer code for selecting a first object determined to be at least primarily in shadow;

computer code for selecting a second object adjacent the first object;

computer code for analyzing the two objects using the properties of the image for determining whether the two objects are part of a same overall object.

30. A method for removing shadows from an image, comprising:

analyzing an image for determining several properties thereof;

finding objects in the image based at least in part on the properties of the image;

selecting a first object determined to be at least primarily in shadow;

selecting a second object adjacent the first object, the second object not being primarily in shadow;

analyzing the two objects using the properties of the image for determining whether the two objects are part of a same overall object;

calculating new red (R) values for pixels in the first object based on a ratio of an average R value of the pixels in the first object to an average R value of pixels in the second object;

calculating new green (G) values for pixels in the first object based on a ratio of an average G value of the pixels in the first object to an average G value of pixels in the second object; and

calculating new blue (B) values for pixels in the first object based on a ratio of an average B value of the pixels in the first object to an average B value of pixels in the second object.

31. A method as recited in claim 30, wherein the objects in the image are found using a flood fill algorithm.

32. A method as recited in claim 30, wherein the objects in the image are found using a histogram.

33. A method as recited in claim 30, wherein the method is performed for a series of images.

34. A method as recited in claim 33, wherein at least some of the objects in the image are found using motion-based detection.

35. A method as recited in claim 30, further comprising allowing user manipulation of a hue (H) of the first object.

36. A method as recited in claim 30, further comprising allowing user manipulation of a saturation (S) of the first object.

37. A method as recited in claim 30, further comprising allowing user manipulation of a brightness (V) of the first object.

38. A computer program product, comprising:

a computer readable medium having computer code thereon for processing an image, the computer code including;

computer code for finding objects in the image based at least in part on the properties of the image;

computer code for determining whether one of the objects of the image is at least primarily in shadow;

computer code for selecting a first object determined to be at least primarily in shadow;

computer code for selecting a second object adjacent the first object, the second object not being primarily in shadow;

computer code for analyzing the two objects using the properties of the image for determining whether the two objects are part of a same overall object;

computer code for calculating new red (R) values for pixels in the first object based on a ratio of an average R value of the pixels in the first object to an average R value of pixels in the second object;

computer code for calculating new green (G) values for pixels in the first object based on a ratio of an average G value of the pixels in the first object to an average G value of pixels in the second object; and

calculating new blue (B) values for pixels in the first object based on a ratio of an average B value of the pixels in the first object to an average B value of pixels in the second object.