Foreground/background segmentation in digital images with differential exposure calculations

• US 7,796,816 B2
• Filed: 09/18/2009
• Issued: 09/14/2010
• Est. Priority Date: 08/16/2004
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Vision Limited (Adeia Inc.)
• Inventors: Steinberg, Eran, Prilutsky, Yury, Bigioi, Petronel, Corcoran, Peter
• Primary Examiner(s): Couso; Jose L
• Application Number:
  US12/562,833
• Publication Number:
  US 20100008577A1
• Time in Patent Office: 361 Days
• Field of Search: 345/426, 345/428, 345/581, 345/582, 348/333.05, 348/333.11, 348/370, 348/371, 382/173, 382/176, 382/180, 382/181, 382/190, 382/199, 382/218, 382/220, 382/221, 382/224, 382254-275, 382/284, 382/312, 396/374
• US Class Current: 382/173
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10152 Varying illumination
• CPC Class Codes: G06T 2207/20224 Image subtraction
• CPC Class Codes: G06T 2207/30216 Redeye defect
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/40 Analysis of texture depth or shape recovery from texture G06T7/529
• CPC Class Codes: G06V 10/28 Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: H04N 1/624 Red-eye correction control of camera based on recognised facial parts H04N23/611
• CPC Class Codes: H04N 23/74 by influencing the scene brightness using illuminating means
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof

Abstract

A digital segmentation method and apparatus determines foreground and/or background within at least one portion of a captured image. The determining includes comparing a captured image to a pre-captured or post captured reference image of nominally the same scene. One of the images is taken with flash and the other without. The system can be implemented as part of a digital camera acquisition chain having effective computation complexity.

Claims

1. A handheld digital image acquisition system having no photographic film, comprising:

an apparatus for capturing digital images,

a flash unit for providing illumination during image capture,

a processor, and

a memory having computer-readable code embedded therein for programming the processor to perform a digital segmentation method that comprises;

determining one or more regions that are indicative of a foreground region, or of a background region, or of a background region and a foreground region, within at least one portion of a captured image,

wherein said determining comprises comparing said captured image and a reference image of nominally the same scene captured in a temporal range between just before and just after exposure of the captured image,

wherein the comparing comprises spatially aligning one or more regions of the captured and reference images at least in respect of pixels corresponding to the at least one portion of said captured image,

wherein the captured and reference images have different pixel resolutions, and wherein the method further comprises, prior to said determining, matching the pixel resolutions of the captured and reference images at least in respect of said at least one portion; and

wherein at least one of said captured and reference images being taken with flash.

2. A system according to claim 1, wherein said matching comprises up-sampling of the image of lower resolution or sub-sampling of the image of higher resolution, or both.

3. A system according to claim 1, wherein the spatially aligning of the one or more regions is performed prior to the determining of the one or more regions for aligning said regions of said captured and reference images at least in respect of said at least one portion.

4. A system according to claim 1, wherein the method further comprises substantially equalizing an overall level of exposure of at least said one or more regions or of substantially all of said captured and reference images at least in respect of said at least one portion.

5. A system according to claim 4, wherein at least in respect of said at least one portion, the determining comprises identifying corresponding pixels in the captured and reference images whose values differ by less than a predetermined threshold, and designating one or more segments of the captured image bounded by said determined pixels as foreground or background by comparing pixel values in a segment with pixel values in a corresponding segment of the reference image.

6. A system according to claim 4, wherein at least in respect of said at least one portion, the determining comprises designating upper and lower thresholds based on a comparison of the overall level of exposure of the captured and reference images and further designating pixels of the captured image as foreground or background according to whether their values are greater than the upper threshold or less than the lower threshold.

7. A system according to claim 4, wherein at least in respect of said at least one portion, the determining comprises designating one or more segments of the captured image as foreground or background by comparing pixel values in the captured and reference images.

8. A system according to claim 1, wherein the reference image comprises a preview image having a lower pixel resolution than the captured image, and the captured image comprises an image taken with flash.

9. A system according to claim 8, wherein the reference image comprises an image taken without flash.

10. A system according to claim 9, wherein the method further comprises utilizing another reference image that is taken with flash.

11. A method of digital segmentation involving a handheld digital image acquisition device, the method comprising:

determining one or more regions that are indicative of a foreground region, or of a background region, or of a background region and a foreground region, within at least one portion of a captured image,

wherein said determining comprises comparing said captured image and a reference image of nominally the same scene captured just before, during, just after or temporally-overlapping exposure of the captured image,

wherein the comparing comprises spatially aligning one or more regions of the captured and reference images at least in respect of pixels corresponding to the at least one portion of said captured image,

wherein the captured and reference images have different pixel resolutions, and wherein the method further comprises, prior to said determining, matching the pixel resolutions of the captured and reference images at least in respect of said at least one portion; and

wherein at least one of said captured and reference images being taken with flash.

12. A method according to claim 11, wherein said matching comprises up-sampling of the image of lower resolution or sub-sampling of the image of higher resolution, or both.

13. A method according to claim 11, wherein the spatially aligning of the one or more regions is performed prior to the determining of the one or more regions for aligning said regions of said captured and reference images at least in respect of said at least one portion.

14. A method according to claim 11, further comprising substantially equalizing an overall level of exposure of at least said one or more regions or of substantially all of said captured and reference images at least in respect of said at least one portion.

15. A method according to claim 14, wherein at least in respect of said at least one portion, the determining comprises identifying corresponding pixels in the captured and reference images whose values differ by less than a predetermined threshold, and designating one or more segments of the captured image bounded by said determined pixels as foreground or background by comparing pixel values in a segment with pixel values in a corresponding segment of the reference image.

16. A method according to claim 14, wherein at least in respect of said at least one portion, the determining comprises designating upper and lower thresholds based on a comparison of the overall level of exposure of the captured and reference images and further designating pixels of the captured image as foreground or background according to whether their values are greater than the upper threshold or less than the lower threshold.

17. A method according to claim 14, wherein at least in respect of said at least one portion, the determining comprises designating one or more segments of the captured image as foreground or background by comparing pixel values in the captured and reference images.

18. A method according to claim 11, wherein the reference image comprises a preview image having a lower pixel resolution than the captured image, and the captured image comprises an image taken with flash.

19. A method according to claim 18, wherein the reference image comprises an image taken without flash.

20. A method according to claim 19, further comprising utilizing another reference image that is taken with flash.

21. One or more computer-readable storage devices having code embedded therein for programming a processor of a handheld digital image acquisition device to perform a digital segmentation method that comprises:

determining one or more regions that are indicative of a foreground region, or of a background region, or of a background region and a foreground region, within at least one portion of a captured image,

wherein said determining comprises comparing said captured image and a reference image of nominally the same scene captured in a temporal range between just before and just after exposure of the captured image,

wherein the comparing comprises spatially aligning one or more regions of the captured and reference images at least in respect of pixels corresponding to the at least one portion of said captured image,

wherein the captured and reference images have different pixel resolutions, and wherein the method further comprises, prior to said determining, matching the pixel resolutions of the captured and reference images at least in respect of said at least one portion; and

wherein at least one of said captured and reference images being taken with flash.

22. The one or more processor-readable storage devices of claim 21, wherein said matching comprises up-sampling of the image of lower resolution or sub-sampling of the image of higher resolution, or both.

23. The one or more processor-readable storage devices of claim 21, wherein the spatially aligning of the one or more regions is performed prior to the determining of the one or more regions for aligning said regions of said captured and reference images at least in respect of said at least one portion.

24. The one or more processor-readable storage devices of claim 21, wherein the method further comprises substantially equalizing an overall level of exposure of at least said one or more regions or of substantially all of said captured and reference images at least in respect of said at least one portion.

25. The one or more processor-readable storage devices of claim 24, wherein at least in respect of said at least one portion, the determining comprises identifying corresponding pixels in the captured and reference images whose values differ by less than a predetermined threshold, and designating one or more segments of the captured image bounded by said determined pixels as foreground or background by comparing pixel values in a segment with pixel values in a corresponding segment of the reference image.

26. The one or more processor-readable storage devices of claim 24, wherein at least in respect of said at least one portion, the determining comprises designating upper and lower thresholds based on a comparison of the overall level of exposure of the captured and reference images and further designating pixels of the captured image as foreground or background according to whether their values are greater than the upper threshold or less than the lower threshold.

27. The one or more processor-readable storage devices of claim 24, wherein at least in respect of said at least one portion, the determining comprises designating one or more segments of the captured image as foreground or background by comparing pixel values in the captured and reference images.

28. The one or more processor-readable storage devices of claim 21, wherein the reference image comprises a preview image having a lower pixel resolution than the captured image, and the captured image comprises an image taken with flash.

29. The one or more processor-readable storage devices of claim 28, wherein the reference image comprises an image taken without flash.

30. The one or more processor-readable storage devices of claim 29, wherein the method further comprises utilizing another reference image that is taken with flash.