Foreground / background separation in digital images

• US 8,363,908 B2
• Filed: 05/03/2007
• Issued: 01/29/2013
• Est. Priority Date: 05/03/2006
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: DigitalOptics Corporation Europe Limited (Adeia Inc.)
• Inventors: Steinberg, Eran, Corcoran, Peter, Prilutsky, Yury, Bigioi, Petronel, Pososin, Alexei, Ciuc, Mihai, Zamfir, Adrian, Capata, Adrian
• Primary Examiner(s): STREGE, JOHN B
• Application Number:
  US11/744,020
• Publication Number:
  US 20070269108A1
• Time in Patent Office: 2,098 Days
• Field of Search: 382/173
• US Class Current: 382/118
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06V 10/28 Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching

Abstract

A method for providing improved foreground/background separation in a digital image of a scene is disclosed. The method comprises providing a first map comprising one or more regions provisionally defined as one of foreground or background within the digital image; and providing a subject profile corresponding to a region of interest of the digital image. The provisionally defined regions are compared with the subject profile to determine if any of the regions intersect with the profile region. The definition of one or more of the regions in the map is changed based on the comparison.

Claims

1. A method for providing separation between foreground and background regions in a digital image of a scene, comprising:

using a processor-based digital image acquisition device including a processor programmed by processor-readable code embedded within one or more digital storage media;

providing a first map comprising two or more regions within a main digital image, each having one or more pixels with a common characteristic;

comparing values of a second characteristic of the one or more pixels that varies in accordance with a known relationship to the common characteristic;

providing a subject profile corresponding to a region of interest of said main digital image, wherein said providing a subject profile comprises;

determining at least one region of said main digital image including a subject'"'"'s face;

determining an orientation of said at least one face within said main digital image; and

defining said subject profile as comprising said at least one face region including the subject'"'"'s face and a respective region below said face region including at least a portion of the subject'"'"'s neck or shirt, or both, in said main digital image in accordance with said orientation;

and wherein the method further comprises;

comparing one or more of said one or more regions with said subject profile to determine if any of said one or more of said one or more regions intersect with said subject profile; and

designating one or more of said one or more of said one or more regions as a foreground region based on said comparison.

2. A method as claimed in claim 1 wherein providing said first map comprises provisionally defining each of said one or more regions of said main digital image as foreground or background and wherein said one or more regions comprise at least one region provisionally defined as foreground.

3. A method as claimed in claim 2 wherein said designating comprises:

comparing a foreground region with said subject profile; and

responsive to said foreground region not substantially intersecting said subject profile, changing a designation of said foreground region to a background region.

4. A method as claimed in claim 1 wherein the providing of said first map is based on a comparison of two or more images nominally of said scene.

5. A method as claimed in claim 4 wherein one or more of said two or more images that are compared comprises a lower resolution version of said main digital image.

6. A method as claimed in claim 4 wherein one of said two or more images that are compared comprises said main digital image.

7. A method as claimed in claim 4 further comprising aligning said two or more images that are compared.

8. A method as claimed in claim 4 further comprising matching resolutions of said two or more images that are compared.

9. A method as claimed in claim 4 wherein at least one of said two or more images that are compared is captured just before or after said main digital image is captured.

10. A method as claimed in claim 1 wherein the providing of said first map comprises:

providing two of more images each of different focus and nominally of said scene;

calculating from said images a depth of focus map indicating pixels of said main digital image as either foreground or background;

blurring said focus map;

thresholding said blurred map to an intermediate focus map indicating regions as either foreground or background; and

filling regions within said intermediate focus map to provide said first map.

11. A method as claimed in claim 1, wherein the providing of said first map comprises:

providing two or more images each of different focus and nominally of said scene; and

comparing high frequency coefficients of corresponding regions in said images to determine whether said one or more of said one or more regions comprise foreground or background to provide said first map.

12. A method as claimed in claim 1 wherein the providing of said first map comprises:

providing two or more images at different exposure levels nominally of said scene; and

comparing luminance levels of corresponding regions in said images to determine whether said one or more of said one or more regions are foreground or background to provide said first map.

13. A method as claimed in claim 1 operable in a digital image acquisition device, said device being arranged to select said subject profile according to content of said main digital image.

14. A method as claimed in claim 1 operable in a digital image acquisition device, said device being arranged to operate in a portrait mode and wherein said subject profile comprises an outline of a person.

15. A method as claimed in claim 14 wherein said outline comprises one of a number of user selectable outlines.

16. A method as claimed in claim 14 wherein said outline is automatically selected from a plurality of outlines based on content of said main digital image.

17. A method as claimed in claim 1 operable in a general purpose computer arranged to receive said first map in association with said main digital image.

18. A method as claimed in claim 1 operable in a general purpose computer arranged to receive one or more additional images nominally of said scene in association with said main digital image and being arranged to calculate said first map from any combination of said one or more additional images and said main digital image.

19. A method as claimed in claim 1 wherein said providing a first map comprises:

analyzing at least one region of the main digital image in a color space to determine a color distribution within said at least one region, said color distribution having a plurality of distinct color peaks; and

segmenting said at least one region based on proximity of a pixel'"'"'s color to said color peaks.

20. A method as claimed in claim 1 wherein said main digital image is one of a stream of images and wherein determining at least one region of said main image including a face comprises:

detecting a face in at least one image of said stream acquired prior to said main digital image; and

tracking said face through said stream of images to determine said face region in said main digital image.

22. The method of claim 1, wherein said comparing comprises comparing said at least one of said one or more provisionally defined regions with said subject profile to determine if any of said at least one of said one or more provisionally defined regions comprise a shape or luminance distribution, or both, that matches that of said subject profile.

23. The method of claim 1, wherein said one or more regions each comprise one or more approximately relatively still regions, and said comparing comprises comparing one or more approximately relatively still regions with said subject profile to determine if any of said one or more approximately relatively still regions intersect with said subject profile, and wherein said designating comprises designating one or more of said approximately relatively still regions as a foreground region based on said comparison.

24. The method of claim 1, wherein said providing a subject profile comprises providing data including a stored shape or luminance distribution, or both, said subject profile not being acquired from a previous image of substantially a same scene as said main digital image, for comparison with said at least one of said one or more regions.

25. The method of claim 1, wherein said providing a subject profile comprises providing data including a stored face region having a relative size next to a second feature, said subject profile not being acquired from a previous image of substantially a same scene as said main digital image, for comparison with one or more face regions of the one or more regions within said main digital image, and the comparing comprises comparing said relative size of said stored face region next to said second feature with that of said subject profile to determine if at least one of said one or more face regions intersect with said subject profile.

21. An apparatus for providing improved separation between foreground and background regions in a digital image of a scene, comprising a processor and one or more processor-readable media for programming the processor to control the apparatus to perform a method comprising:

acquiring a main digital image;

providing a first map comprising one or more regions provisionally defined as one of foreground or background within said main digital image, each having one or more pixels with a common characteristic;

comparing values of a second characteristic of the one or more pixels that varies between foreground and background distances in accordance with a known relationship to the common characteristic;

providing a subject profile corresponding to a region of interest of said main digital image, wherein said providing a subject profile comprises;

determining at least one region of said image including a subject'"'"'s face

determining a orientation of said at least one face within said main image; and

defining said subject profile as comprising said at least one face region including the subject'"'"'s face and a respective region below said face region including at least a portion of the subject'"'"'s neck or shirt, or both, in said main digital image in accordance with said orientation;

and wherein the method further comprises;

comparing at least one of said one or more provisionally defined regions with said subject profile to determine if any of said at least one of said one or more provisionally defined regions intersect with said profile region; and

changing in said map a designation of one or more of said at least one of said one or more provisionally defined regions as a foreground region based on said comparison.

26. An apparatus for providing improved separation between foreground and background regions in a digital image, comprising a processor and one or more processor-readable media for programming the processor to control the apparatus to perform a method comprising:

acquiring a main digital image;

providing a first map comprising one or more regions within said main digital image, each having one or more pixels with a common characteristic;

comparing values of a second characteristic of the one or more pixels that varies in accordance with a known relationship to the common characteristic;

providing a subject profile that comprises a stored shape or luminance distribution, or both, said subject profile not being acquired from a previous image of substantially a same scene as said main digital image, for comparison with said one or more regions;

comparing one or more of said one or more regions with said subject profile to determine if any of said one or more of said one or more regions intersect with said subject profile, wherein said comparing comprises;

determining at least one region of said image including a subject'"'"'s face;

determining an orientation of said at least one face within said main digital image; and

comparing said subject profile with said at least one face region including the subject'"'"'s face and a respective region below said face region including at least a portion of the subject'"'"'s neck or shirt, or both, in said main digital image in accordance with said orientation;

and wherein the method further comprises

designating one or more of said one or more of said one or more regions as a foreground region based on said comparison.

27. An apparatus for providing improved separation between foreground and background regions in a digital image of a scene, comprising a processor and one or more processor-readable media for programming the processor to control the apparatus to perform a method comprising:

acquiring a main digital image;

providing a first map comprising one or more face regions within a main digital image;

comparing values of a second characteristic of the one or more pixels that varies in accordance with a known relationship to the common characteristic;

providing a subject profile that comprises a stored face region having a relative size next to a second feature, said subject profile not being acquired from a previous image of substantially a same scene as said main digital image, for comparison with said one or more face regions within said main digital image;

comparing said relative size of said stored face region next to said second feature with that of said subject profile to determine if any of said one or more face regions intersect with said subject profile, wherein said comparing comprises;

determining at least one region of said main digital image including a subject'"'"'s face;

determining an orientation of said at least one face within said main digital image; and

comparing said subject profile with said at least one face region including the subject'"'"'s face and a respective region below said face region including at least a portion of the subject'"'"'s neck or shirt, or both, in said main digital image in accordance with said orientation;

and wherein the method further comprises;

designating at least one of said one or more regions as a foreground region based on said comparison.

28. An apparatus for providing improved separation between foreground and background regions in a digital image of a scene, comprising a processor and one or more processor-readable media for programming the processor to control the apparatus to perform a method comprising:

acquiring a main digital image;

providing a first map comprising one or more regions within a main digital image, each having one or more pixels with a common characteristic;

comparing values of a second characteristic of the one or more pixels that varies in accordance with a known relationship to the common characteristic;

providing a subject profile corresponding to a region of interest of said main digital image, wherein said providing a subject profile comprises;

determining at least one region of said main digital image including a subject'"'"'s face;

determining an orientation of said at least one face within said main digital image; and

defining said subject profile as comprising said at least one face region including the subject'"'"'s face and a respective region below said face region including at least a portion of the subject'"'"'s neck or shirt, or both, in said main digital image in accordance with said orientation;

and wherein the method further comprises;

comparing one or more of said one or more regions with said subject profile to determine if any of said one or more of said one or more regions comprise a shape or luminance distribution, or both, that matches that of said subject profile; and

designating one or more of said one or more of said one or more regions as a foreground region based on said comparison.

29. An apparatus for providing improved separation between foreground and background regions in a digital image of a scene, comprising a processor and one or more processor-readable media for programming the processor to control the apparatus to perform a method comprising:

acquiring a main digital image;

providing a first map comprising one or more approximately relatively still regions within said a main digital image, each having one or more pixels with a common characteristic;

comparing values of a second characteristic of the one or more pixels that varies in accordance with a known relationship to the common characteristic;

providing a subject profile corresponding to a region of interest of said main digital image, wherein said providing a subject profile comprises;

determining at least one region of said main digital image including a subject'"'"'s face;

determining an orientation of said at least one face within said main digital image; and

defining said subject profile as comprising said at least one face region including the subject'"'"'s face and a respective region below said face region including at least a portion of the subject'"'"'s neck or shirt, or both, in said main digital image in accordance with said orientation;

and wherein the method further comprises;

comparing said one or more approximately relatively still regions with said subject profile to determine if any of said one or more approximately relatively still regions intersect with said subject profile; and

designating at least one of said one or more of said approximately relatively still regions as a foreground region based on said comparison.