Methods and apparatuses for foreground, top-of-the-head separation from background

• US 8,633,999 B2
• Filed: 05/28/2010
• Issued: 01/21/2014
• Est. Priority Date: 05/29/2009
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: DigitalOptics Corporation Europe Limited (Adeia Inc.)
• Inventors: Capata, Adrian, Ciuc, Mihai, Zamfir, Adrian, Drimbarean, Alexandru
• Primary Examiner(s): Tran, Sinh
• Assistant Examiner(s): Peterson, Christopher K
• Application Number:
  US12/790,594
• Publication Number:
  US 20110050938A1
• Time in Patent Office: 1,334 Days
• Field of Search: 348370-371, 348/222.1, 382181-231
• US Class Current: 348/239
• CPC Class Codes: G06T 2207/10152 Varying illumination
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: H04N 23/74 by influencing the scene brightness using illuminating means
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

Foreground and background regions of a digital image of a scene are distinguished from each other automatically. Foreground objects are identified in a binary image map that distinguishes between foreground pixels and background pixels. From the foreground objects, a primary foreground object is identified. Within the identified primary foreground object, a head region of the primary foreground object is located. Within the head region, patterns of foreground pixels and background pixels that are indicative of a head crown region are identified. Within the head crown region, pixels identified as background pixels that actually show portions of the primary foreground object are converted to foreground pixels, thus improving the accuracy of the binary image map.

Claims

1. A method comprising steps of:

reading, from a storage device, a first image of a scene that was captured with a first amount of illumination;

reading, from the storage device, a second image of the scene that was captured with a second amount of illumination that differs from the first amount of illumination;

distinguishing a foreground portion of the scene from a background portion of the scene based on a comparison of the first image to the second image;

identifying, in the scene, a crown region that contains a crown of a person'"'"'s head;

identifying portions of the crown region that have been misidentified as belonging to the background portion rather than the foreground portion; and

recharacterizing the misidentified portions of the crown region as belonging to the foreground portion rather than the background portion;

wherein the steps are performed by an automated device.

2. The method of claim 1, wherein identifying the crown region comprises:

identifying a primary foreground object by identifying a largest contiguous region of the foreground portion; and

recharacterizing all parts of the foreground portion that are external to the primary foreground object as belonging to the background portion rather than the foreground portion.

3. The method of claim 1, wherein identifying the crown region comprises:

identifying a largest contiguous region of the foreground portion; and

determining whether the largest contiguous region contacts a bottom of a particular image that is either the first image or the second image;

in response to determining that the largest contiguous region does not contact the bottom of the particular image, identifying, within the foreground portion, as a primary foreground object, a next-largest contiguous region that (a) is not as large as the largest contiguous region and (b) contacts the bottom of the particular image; and

recharacterizing all parts of the foreground portion that are external to the primary foreground object as belonging to the background portion rather than the foreground portion.

4. The method of claim 1, wherein identifying the crown region comprises:

identifying a sub-region of the background portion that is entirely surrounded by the foreground portion;

determining whether the sub-region contacts any portion of a rectangle that bounds an area that contains at least a part of the foreground portion; and

in response to determining that the sub-region does not contact any portion of the rectangle, recharacterizing all parts of the sub-region as belonging to the foreground portion rather than the background portion.

5. The method of claim 1, wherein identifying the crown region comprises:

identifying a sub-region of the background portion that is entirely surrounded by the foreground portion;

determining whether the sub-region contacts any portion of a rectangle that bounds an area that contains at least a part of the foreground portion;

determining whether the sub-region occupies less than a specified percentage of a specified area; and

in response to determining that (a) the sub-region does not contact any portion of the rectangle and (b) the sub-region occupies less than a specified percentage of the specified area, recharacterizing all parts of the sub-region as belonging to the foreground portion rather than the background portion.

6. The method of claim 1, wherein identifying the crown region comprises:

identifying a primary foreground object within the foreground portion based at least in part on the primary foreground object being a largest contiguous region of the foreground portion;

identifying a bounding rectangle whose sides contact uppermost, lowermost, leftmost, and rightmost extents of the primary foreground object; and

identifying, within the bounding rectangle, a head box that is defined have (a) a width that is a specified fraction of a width of the bounding rectangle and (b) a height that is a specified fraction of a height of the bounding rectangle.

7. The method of claim 1, wherein identifying the crown region comprises:

identifying a primary foreground object within the foreground portion based at least in part on the primary foreground object being a largest contiguous region of the foreground portion;

identifying, based at least in part on a face-detection algorithm, a facial region of the primary foreground object; and

identifying a head box whose edges contact uppermost, lowermost, leftmost, and rightmost extents of the facial region.

8. The method of claim 1, wherein identifying the crown region comprises:

identifying a head box that bounds the person'"'"'s head;

locating, within the head box, a trio line that is an uppermost pixel row that also contains pixels from the foreground portion followed by pixels from the background portion followed by pixels from the foreground portion;

identifying a crown box (a) whose leftmost, rightmost, and lowermost sides are the leftmost, rightmost, and lowermost sides of the head box, and (b) whose uppermost side is the trio line;

identifying, within the crown box, one or more sub-regions that both (a) belong to the background portion and (b) do not touch any side of the crown box; and

recharacterizing each of the one or more sub-regions as belonging to the foreground portion rather than the background portion.

9. The method of claim 1, wherein identifying the crown region comprises:

identifying a head box that bounds the person'"'"'s head;

locating, within the head box, a trio line that is an uppermost pixel row that also contains pixels from the foreground portion followed by pixels from the background portion followed by pixels from the foreground portion;

identifying a crown box (a) whose leftmost, rightmost, and lowermost sides are the leftmost, rightmost, and lowermost sides of the head box, and (b) whose uppermost side is the trio line;

identifying, within the crown box, one or more pixels that (a) belong to the background portion and (b) change in luminescence more than a specified amount between the first image and the second image; and

recharacterizing each of the one or more pixels as belonging to the foreground portion rather than the background portion.

10. The method of claim 1, further comprising:

after recharacterizing the misidentified portions of the crown region as belonging to the foreground portion rather than the background portion, storing, on the storage device, a third image in which the background portion has been omitted from the scene.

11. A volatile or non-volatile computer-readable storage medium storing instructions which, when executed by one or more processors, cause the processors to perform steps comprising:

reading, from a storage device, a first image of a scene that was captured with a first amount of illumination;

reading, from the storage device, a second image of the scene that was captured with a second amount of illumination that differs from the first amount of illumination;

distinguishing a foreground portion of the scene from a background portion of the scene based on a comparison of the first image to the second image;

identifying, in the scene, a crown region that contains a crown of a person'"'"'s head;

identifying portions of the crown region that have been misidentified as belonging to the background portion rather than the foreground portion; and

recharacterizing the misidentified portions of the crown region as belonging to the foreground portion rather than the background portion.

12. The computer-readable storage medium of claim 11, wherein identifying the crown region comprises:

identifying a primary foreground object by identifying a largest contiguous region of the foreground portion; and

recharacterizing all parts of the foreground portion that are external to the primary foreground object as belonging to the background portion rather than the foreground portion.

13. The computer-readable storage medium of claim 11, wherein identifying the crown region comprises:

identifying a largest contiguous region of the foreground portion; and

determining whether the largest contiguous region contacts a bottom of a particular image that is either the first image or the second image;

in response to determining that the largest contiguous region does not contact the bottom of the particular image, identifying, within the foreground portion, as a primary foreground object, a next-largest contiguous region that (a) is not as large as the largest contiguous region and (b) contacts the bottom of the particular image; and

recharacterizing all parts of the foreground portion that are external to the primary foreground object as belonging to the background portion rather than the foreground portion.

14. The computer-readable storage medium of claim 11, wherein identifying the crown region comprises:

identifying a sub-region of the background portion that is entirely surrounded by the foreground portion;

determining whether the sub-region contacts any portion of a rectangle that bounds an area that contains at least a part of the foreground portion; and

in response to determining that the sub-region does not contact any portion of the rectangle, recharacterizing all parts of the sub-region as belonging to the foreground portion rather than the background portion.

15. The computer-readable storage medium of claim 11, wherein identifying the crown region comprises:

identifying a sub-region of the background portion that is entirely surrounded by the foreground portion;

determining whether the sub-region contacts any portion of a rectangle that bounds an area that contains at least a part of the foreground portion;

determining whether the sub-region occupies less than a specified percentage of a specified area; and

in response to determining that (a) the sub-region does not contact any portion of the rectangle and (b) the sub-region occupies less than a specified percentage of the specified area, recharacterizing all parts of the sub-region as belonging to the foreground portion rather than the background portion.

16. The computer-readable storage medium of claim 11, wherein identifying the crown region comprises:

identifying a primary foreground object within the foreground portion based at least in part on the primary foreground object being a largest contiguous region of the foreground portion;

identifying a bounding rectangle whose sides contact uppermost, lowermost, leftmost, and rightmost extents of the primary foreground object; and

identifying, within the bounding rectangle, a head box that is defined have (a) a width that is a specified fraction of a width of the bounding rectangle and (b) a height that is a specified fraction of a height of the bounding rectangle.

17. The computer-readable storage medium of claim 11, wherein identifying the crown region comprises:

identifying a primary foreground object within the foreground portion based at least in part on the primary foreground object being a largest contiguous region of the foreground portion;

identifying, based at least in part on a face-detection algorithm, a facial region of the primary foreground object; and

identifying a head box whose edges contact uppermost, lowermost, leftmost, and rightmost extents of the facial region.

18. The computer-readable storage medium of claim 11, wherein identifying the crown region comprises:

identifying a head box that bounds the person'"'"'s head;

locating, within the head box, a trio line that is an uppermost pixel row that also contains pixels from the foreground portion followed by pixels from the background portion followed by pixels from the foreground portion;

identifying a crown box (a) whose leftmost, rightmost, and lowermost sides are the leftmost, rightmost, and lowermost sides of the head box, and (b) whose uppermost side is the trio line;

identifying, within the crown box, one or more sub-regions that both (a) belong to the background portion and (b) do not touch any side of the crown box; and

recharacterizing each of the one or more sub-regions as belonging to the foreground portion rather than the background portion.

19. The computer-readable storage medium of claim 11, wherein identifying the crown region comprises:

identifying a head box that bounds the person'"'"'s head;

locating, within the head box, a trio line that is an uppermost pixel row that also contains pixels from the foreground portion followed by pixels from the background portion followed by pixels from the foreground portion;

identifying a crown box (a) whose leftmost, rightmost, and lowermost sides are the leftmost, rightmost, and lowermost sides of the head box, and (b) whose uppermost side is the trio line;

identifying, within the crown box, one or more pixels that (a) belong to the background portion and (b) change in luminescence more than a specified amount between the first image and the second image; and

recharacterizing each of the one or more pixels as belonging to the foreground portion rather than the background portion.

20. The computer-readable storage medium of claim 11, wherein the steps further comprise:

after recharacterizing the misidentified portions of the crown region as belonging to the foreground portion rather than the background portion, storing, on the storage device, a third image in which the background portion has been omitted from the scene.

21. A digital camera comprising:

an image capture module that (a) captures a first image of a scene with a first amount of illumination and (b) captures a second image of the scene with a second amount of illumination that differs from the first amount of illumination;

a foreground identification module that distinguishes a foreground portion of the scene from a background portion of the scene based on a comparison of the first image to the second image; and

a crown identification module that (a) identifies, in the scene, a crown region that contains a crown of a person'"'"'s head, (b) identifies portions of the crown region that have been misidentified as belonging to the background portion rather than the foreground portion, and (c) recharacterizes the misidentified portions of the crown region as belonging to the foreground portion rather than the background portion.

22. The digital camera of claim 21, wherein identifying the crown region comprises:

identifying a largest contiguous region of the foreground portion; and

determining whether the largest contiguous region contacts a bottom of a particular image that is either the first image or the second image;

in response to determining that the largest contiguous region does not contact the bottom of the particular image, identifying, within the foreground portion, as a primary foreground object, a next-largest contiguous region that (a) is not as large as the largest contiguous region and (b) contacts the bottom of the particular image; and

recharacterizing all parts of the foreground portion that are external to the primary foreground object as belonging to the background portion rather than the foreground portion.

23. The digital camera of claim 21, wherein identifying the crown region comprises:

identifying a sub-region of the background portion that is entirely surrounded by the foreground portion;

determining whether the sub-region contacts any portion of a rectangle that bounds an area that contains at least a part of the foreground portion; and

in response to determining that the sub-region does not contact any portion of the rectangle, recharacterizing all parts of the sub-region as belonging to the foreground portion rather than the background portion.

24. The digital camera of claim 21, wherein identifying the crown region comprises:

identifying a sub-region of the background portion that is entirely surrounded by the foreground portion;

determining whether the sub-region contacts any portion of a rectangle that bounds an area that contains at least a part of the foreground portion;

determining whether the sub-region occupies less than a specified percentage of a specified area; and

in response to determining that (a) the sub-region does not contact any portion of the rectangle and (b) the sub-region occupies less than a specified percentage of the specified area, recharacterizing all parts of the sub-region as belonging to the foreground portion rather than the background portion.

25. The digital camera of claim 21, wherein identifying the crown region comprises:

identifying a primary foreground object within the foreground portion based at least in part on the primary foreground object being a largest contiguous region of the foreground portion;

identifying a bounding rectangle whose sides contact uppermost, lowermost, leftmost, and rightmost extents of the primary foreground object; and

identifying, within the bounding rectangle, a head box that is defined have (a) a width that is a specified fraction of a width of the bounding rectangle and (b) a height that is a specified fraction of a height of the bounding rectangle.

26. The digital camera of claim 21, wherein identifying the crown region comprises:

identifying a primary foreground object within the foreground portion based at least in part on the primary foreground object being a largest contiguous region of the foreground portion;

identifying, based at least in part on a face-detection algorithm, a facial region of the primary foreground object; and

identifying a head box whose edges contact uppermost, lowermost, leftmost, and rightmost extents of the facial region.

27. The digital camera of claim 21, wherein identifying the crown region comprises:

identifying a head box that bounds the person'"'"'s head;

locating, within the head box, a trio line that is an uppermost pixel row that also contains pixels from the foreground portion followed by pixels from the background portion followed by pixels from the foreground portion;

identifying a crown box (a) whose leftmost, rightmost, and lowermost sides are the leftmost, rightmost, and lowermost sides of the head box, and (b) whose uppermost side is the trio line;

identifying, within the crown box, one or more sub-regions that both (a) belong to the background portion and (b) do not touch any side of the crown box; and

recharacterizing each of the one or more sub-regions as belonging to the foreground portion rather than the background portion.

28. The digital camera of claim 21, wherein identifying the crown region comprises:

identifying a head box that bounds the person'"'"'s head;

locating, within the head box, a trio line that is an uppermost pixel row that also contains pixels from the foreground portion followed by pixels from the background portion followed by pixels from the foreground portion;

identifying a crown box (a) whose leftmost, rightmost, and lowermost sides are the leftmost, rightmost, and lowermost sides of the head box, and (b) whose uppermost side is the trio line;

identifying, within the crown box, one or more pixels that (a) belong to the background portion and (b) change in luminescence more than a specified amount between the first image and the second image; and

recharacterizing each of the one or more pixels as belonging to the foreground portion rather than the background portion.

29. The digital camera of claim 21, wherein the crown identification module, after recharacterizing the misidentified portions of the crown region as belonging to the foreground portion rather than the background portion, stores, on the storage device, a third image in which the background portion has been omitted from the scene.