Foreground/background segmentation in digital images

• US 7,957,597 B2
• Filed: 09/13/2010
• Issued: 06/07/2011
• Est. Priority Date: 08/16/2004
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: Tessera Technologies Ireland Limited (Adeia Inc.)
• Inventors: Steinberg, Eran, Prilutsky, Yury, Bigioi, Petronel, Corcoran, Peter
• Primary Examiner(s): Couso; Jose L
• Application Number:
  US12/881,029
• Publication Number:
  US 20110025859A1
• Time in Patent Office: 267 Days
• Field of Search: 382/173, 382/176, 382/180, 382/181, 382/190, 382/199, 382/218, 382/220, 382/221, 382/224, 382254-275, 382/284, 382/312, 348/333.05, 348/333.11, 348/370, 348/371, 396/374, 345/426, 345/428, 345/581, 345/582
• US Class Current: 382/224
• CPC Class Codes: G06T 2207/10152 Varying illumination
• CPC Class Codes: G06T 2207/20224 Image subtraction
• CPC Class Codes: G06T 2207/30216 Redeye defect
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/40 Analysis of texture depth or shape recovery from texture G06T7/529
• CPC Class Codes: G06V 10/28 Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
• CPC Class Codes: G06V 20/35 Categorising the entire scene, e.g. birthday party or wedding scene
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: H04N 1/624 Red-eye correction control of camera based on recognised facial parts H04N23/611

Abstract

An analysis and classification tool compares at least a portion of a captured image and a reference image of nominally the same scene. One of the captured and reference images is taken with flash and the other is taken without flash. The tool provides a measure of the difference in illumination between the captured image and the reference image. The tool compares the measure with a threshold and segments a foreground region from a background region based on the measure.

Claims

1. A digital image acquisition system having no photographic film, comprising:

(a) an apparatus for capturing digital images, including a lens, an image sensor and a processor;

(b) a flash unit for providing illumination during image capture;

(c) an analysis tool for comparing at least a portion of a captured image and a reference image of nominally the same scene, at least one of said captured and reference images being taken with flash, said analysis tool providing a measure of the difference in illumination between said captured image and said reference image; and

(d) a classification tool for comparing said measure with a threshold and for categorizing said scene as outdoor or indoor based on said measure.

2. A system according to claim 1, further comprising an exposure equalizer for substantially equalizing an overall level of exposure of at least said portion of said captured and reference images prior to analysis by said analysis tool.

3. A system according to claim 1, further comprising a segmentation tool for determining one or more regions that are indicative of a foreground region, or of a background region, or of a background region and a foreground region, within at least one portion of a captured image, and wherein said analysis tool is arranged to analyse said foreground region or said background region, or both.

4. A system according to claim 1, wherein said classification tool is responsive to said measure exceeding a high threshold to classify an image as indoor, and responsive to said measure not exceeding a low threshold to classify an image as outdoor.

5. A system according to claim 4, wherein said high and low threshold are coincident.

6. A system according to claim 1, further comprising a segmentation tool for determining one or more regions that are indicative of a foreground region, or of a background region, or of a background region and a foreground region, within at least one portion of the captured image, wherein said determining comprises comparing said captured image and the same reference image or a different reference image of nominally the same scene, or both.

7. A system according to claim 6, wherein the captured and reference images have different pixel resolutions, and wherein the system further comprises a pixel matching tool which is operative prior to application of the segmentation tool for matching the pixel resolutions of the captured and reference images at least in respect of said at least one portion.

8. A system according to claim 5, wherein said pixel matching tool utilizes up-sampling of the image of lower resolution or sub-sampling of the image of higher resolution, or both.

9. A system according to claim 6, further comprising an alignment tool which is operative prior to application of the segmentation tool for aligning said regions of said captured and reference images at least in respect of said at least one portion.

10. A system according to claim 6, further comprising an exposure equalizer for substantially equalizing an overall level of exposure of said regions or all of said captured and reference images at least in respect of said at least one portion.

11. A system according to claim 10, wherein the substantially equalising of the overall level of exposure of said regions or all of said captured and reference image comprises simulating an ambient exposure of the captured image on the reference image.

12. A system according to claim 11, wherein the simulating of the ambient exposure of the captured image on the reference image comprises digitally simulating a one or a combination of aperture, acquisition speed, color transformations and gain of the captured image on the reference image.

13. A system according to claim 11, wherein the simulating of the ambient exposure of the captured image comprises individual, non-uniform manipulating of individual regions or color channels or combinations thereof.

14. A system according to claim 10, wherein the substantially equalising of the overall level of exposure of said captured and reference image comprises setting an ambient exposure of the reference image to match a calculated exposure of the captured image.

15. A system according to claim 10, wherein at least in respect of said at least one portion, the segmentation tool determines corresponding pixels in the captured and reference images whose values differ by less than a predetermined threshold, and designates segments of the captured image bounded by said determined pixels as foreground or background by comparing pixel values in a segment with pixel values in a corresponding segment of the reference image.

16. A system according to claim 10, wherein at least in respect of said at least one portion, the segmentation tool determines upper and lower thresholds based on a comparison of the overall level of exposure of the captured and reference images and designates pixels of the captured image as foreground or background according to whether their values are greater than the upper threshold or less than the lower threshold.

17. A system according to claim 10, wherein at least in respect of said at least one portion, the segmentation tool designates one or more segments of the captured image as foreground or background by comparing pixel values in the captured and reference images.

18. A system according to claim 10, wherein the reference image comprises a preview image having a lower pixel resolution than the captured image.

19. A system according to claim 6, further comprising a face detection module.

20. A system according to claim 6, further comprising a red-eye detection filter or a red eye correction filter or both, for selective application to the foreground region.

21. A system according to claim 20, further comprising a probability module for changing a probability of a redeye candidate region being an actual redeye region according to whether the candidate appears in the foreground or background of the captured image.

22. A system according to claim 1, further comprising a depth of focus module for reducing a perceived depth of focus according to whether a candidate region appears in the foreground or background of the captured image.

23. A system according to claim 1, wherein the reference image comprises a preview image.

24. A system according to claim 1, wherein the reference image comprises an image captured chronologically after said captured image.

25. A system according to claim 1, wherein the reference image comprises a combination of multiple reference-images.

26. A system according to claim 1, wherein said digital image acquisition system comprises a digital camera.

27. A system according to claim 1, wherein said digital image acquisition system comprises a combination of a digital camera and an external processing device.

28. A system as claimed in claim 27, wherein said segmentation tool is located within said external processing device.

29. A system according to claim 1, wherein said digital image acquisition system comprises a batch processing system including a digital printing device.

30. A system according to claim 1, wherein said digital image acquisition system comprises a batch processing system including a server computer.

31. A computer readable medium having code embodied therein for programming a processor to perform a method of analyzing a captured image, the computer readable medium comprising an analysis and classification tool for comparing at least a portion of a captured image and a reference image of nominally the same scene, at least one of said captured and reference images being taken with flash, said tool providing a measure of the difference in illumination between said captured image and said reference image, and said tool for comparing said measure with a threshold and for categorising said scene as outdoor or indoor based on said measure.

32. The computer readable medium of claim 31, further comprising a digital segmentation tool for determining one or more regions that are indicative of a foreground region, or of a background region, or of a background region and a foreground region, within at least one portion of a captured image, wherein said determining comprises comparing said captured image and a reference image of nominally the same scene, and wherein at least one of said captured and reference images being taken with flash.

33. The computer readable medium of claim 32, wherein the captured and reference images have different pixel resolutions, and wherein the segmentation tool operates in conjunction with a pixel matching tool which is operative prior to application of the segmentation tool for matching the pixel resolutions of the captured and reference images at least in respect of said at least one portion.

34. The computer readable medium of claim 33, wherein said pixel matching tool utilizes up-sampling of the image of lower resolution or sub-sampling of the image of higher resolution, or both.

35. The computer readable medium of claim 32, wherein the segmentation tool operates in conjunction with an alignment tool which is operative prior to application of the segmentation tool for aligning said captured and reference images at least in respect of said at least one portion.

36. The computer readable medium of claim 32, wherein said segmentation tool operates in conjunction with an exposure equalizer for substantially equalizing an overall level of exposure of said captured and reference images at least in respect of said at least one portion.

37. The computer readable medium of claim 32, wherein said segmentation tool operates in conjunction with a face detection module or a red-eye filter, or both, for selective application to the foreground region.

38. The computer readable medium of claim 37, wherein said segmentation tool further operates in conjunction with a probability module for changing a probability of a redeye candidate region being an actual redeye region according to whether the candidate appears in the foreground or background of the captured image.

39. The computer readable medium of claim 37, wherein said segmentation tool further operates in conjunction with a depth of focus module for reducing a perceived depth of focus according to whether a candidate region appears in the foreground or background of the captured image.

40. The computer readable medium of claim 37, wherein said segmentation tool further operates in conjunction with a blurring module for blurring said regions indicative of background of the captured image.

41. The computer readable medium of claim 31, wherein the reference image comprises a preview image.

42. The computer readable medium of claim 31, wherein the reference image comprises an image captured chronologically after said captured image.

43. The computer readable medium of claim 31, wherein the reference image comprises a combination of multiple reference-images.

44. A method of analysis and classification of a digital image, comprising:

using a processor;

comparing at least a portion of a captured image and a reference image of nominally the same scene, at least one of said captured and reference images being taken with flash;

providing a measure of the difference in illumination between said captured image and said reference image;

comparing said measure with a threshold; and

categorising said scene as outdoor or indoor based on said measure.

45. The method of claim 44, further comprising a digital segmentation tool for determining one or more regions that are indicative of a foreground region, or of a background region, or of a background region and a foreground region, within at least one portion of a captured image, wherein said determining comprises comparing said captured image and a reference image of nominally the same scene, and wherein at least one of said captured and reference images being taken with flash.

46. The method of claim 45, wherein the captured and reference images have different pixel resolutions, and wherein the segmentation tool operates in conjunction with a pixel matching tool which is operative prior to application of the segmentation tool for matching the pixel resolutions of the captured and reference images at least in respect of said at least one portion.

47. The method of claim 46, wherein said pixel matching tool utilizes up-sampling of the image of lower resolution or sub-sampling of the image of higher resolution, or both.

48. The method of claim 45, wherein the segmentation tool operates in conjunction with an alignment tool which is operative prior to application of the segmentation tool for aligning said captured and reference images at least in respect of said at least one portion.

49. The method of claim 45, wherein said segmentation tool operates in conjunction with an exposure equalizer for substantially equalizing an overall level of exposure of said captured and reference images at least in respect of said at least one portion.

50. The method of claim 45, wherein said segmentation tool operates in conjunction with a face detection module or a red-eye filter, or both, for selective application to the foreground region.

51. The method of claim 50, wherein said segmentation tool further operates in conjunction with a probability module for changing a probability of a redeye candidate region being an actual redeye region according to whether the candidate appears in the foreground or background of the captured image.

52. The method of claim 50, wherein said segmentation tool further operates in conjunction with a depth of focus module for reducing a perceived depth of focus according to whether a candidate region appears in the foreground or background of the captured image.

53. The method of claim 50, wherein said segmentation tool further operates in conjunction with a blurring module for blurring said regions indicative of background of the captured image.

54. The method of claim 44, wherein the reference image comprises a preview image.

55. The method of claim 44, wherein the reference image comprises an image captured chronologically after said captured image.

56. The method of claim 44, wherein the reference image comprises a combination of multiple reference-images.