Method for automatic determination of main subjects in photographic images

• US 6,282,317 B1
• Filed: 12/31/1998
• Issued: 08/28/2001
• Est. Priority Date: 12/31/1998
• Status: Expired due to Term

• Current Assignee: Monument Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Eastman Kodak Company
• Inventors: Singhal, Amit, Luo, Jiebo, Etz, Stephen
• Primary Examiner(s): Johns, Andrew W.
• Assistant Examiner(s): Azarian, Seyed
• Application Number:
  US09/223,860
• Time in Patent Office: 971 Days
• Field of Search: 382/203, 382/204, 382/205, 382/206, 382/207, 382/168, 382/169, 382/173, 382/218, 382/195, 382/155, 382/156, 382/159, 382/160, 382/190, 382/202, 382/227
• US Class Current: 382/203
• CPC Class Codes: G06F 18/29 Graphical models, e.g. Bayesian networks
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/143 involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/84 using probabilistic graphical models from image or video features, e.g. Markov models or Bayesian networks
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching

Abstract

A method for detecting a main subject in an image, the method comprises: receiving a digital image; extracting regions of arbitrary shape and size defined by actual objects from the digital image; grouping the regions into larger segments corresponding to physically coherent objects; extracting for each of the regions at least one structural saliency feature and at least one semantic saliency feature; and integrating saliency features using a probabilistic reasoning engine into an estimate of a belief that each region is the main subject.

Claims

1. A method for detecting a main subject in an image, the method comprising the steps of:

a) receiving a digital image;

b) extracting regions of arbitrary shape and size defined by actual objects from the digital image;

c) extracting for each of the regions at least one structural saliency feature and at least one semantic saliency feature; and

,

d) integrating the structural saliency feature and the semantic feature using a probabilistic reasoning engine into an estimate of a belief that each region is the main subject.

2. The method as in claim 1, wherein step (b) includes using a color distance metric defined in a color space, a spatial homogeneity constraint, and a mechanism for permitting spatial adaptivity.

3. The method as in claim 1, wherein the structural saliency feature of step (c) includes at least one of a low-level vision feature and a geometric feature.

4. The method as in claim 1, wherein step (c) includes using either individually or in combination a color, brightness and/or texture as a low-level vision feature;

a location, size, shape, convexity, aspect ratio, symmetry, borderness, surroundedness and/or occlusion as a geometric feature; and

a flesh, face, sky, grass and/or other green vegetation as the semantic saliency feature.

5. The method as in claim 1, wherein step (d) includes using a collection of human opinions to train the reasoning engine to recognize the relative importance of the saliency features.

6. The method as in claim 1, wherein step (c) includes using either individually or in combination a self-saliency feature and a relative saliency feature as the structural saliency feature.

7. The method as in claim 6, wherein step (c) includes using an extended neighborhood window to compute a plurality of the relative saliency features, wherein the extended neighborhood window is determined by the steps of:

(c1) finding a minimum bounding rectangle of a region;

(c2) stretching the minimum bounding rectangle in all four directions proportionally; and

(c3) defining all regions intersecting the stretched minimum bounding rectangle as neighbors of the region.

8. The method as in claim 4, wherein step (c) includes using a centrality as the location feature, wherein the centrality feature is computed by the steps of:

(c1) determining a probability density function of main subject locations using a collection of training data;

(c2) computing an integral of the probability density function over an area of a region; and

,

(c3) obtaining a value of the centrality feature by normalizing the integral by the area of the region.

9. The method as in claim 4, wherein step (c) includes using a hyperconvexity as the convexity feature, wherein the hyperconvexity feature is computed as a ratio of a perimeter-based convexity measure and an area-based convexity measure.

10. The method as in claim 4, wherein step (c) includes computing a maximum fraction of a region perimeter shared with a neighboring region as the surroundedness feature.

11. The method as in claim 4, wherein step (c) includes using an orientation-unaware borderness feature as the borderness feature, wherein the orientation-unaware borderness feature is categorized by the number and configuration of image borders a region is in contact with, and all image borders are treated equally.

12. The method as in claim 4, wherein step (c) includes using an orientation-aware borderness feature as the borderness feature, wherein the orientation-aware borderness feature is categorized by the number and configuration of image borders a region is in contact with, and each image border is treated differently.

13. The method as in claim 4, wherein step (c) includes using the borderness feature that is determined by what fraction of an image border is in contact with a region.

14. The method as in claim 4, wherein step (c) includes using the borderness feature that is determined by what fraction of a region border is in contact with an image border.

15. The method as in claim 1, wherein step (d) includes using a Bayes net as the reasoning engine.

16. The method as in claim 1, wherein step (d) includes using a conditional probability matrix that is determined by using fractional frequency counting according to a collection of training data.

17. The method as in claim 1, wherein step (d) includes using a belief sensor function to convert a measurement of a feature into evidence, which is an input to a Bayes net.

18. The method as in claim 1, wherein step (d) includes outputting a belief map, which indicates a location of and a belief in the main subject.

19. A method for detecting a main subject in an image, the method comprising the steps of:

a) receiving a digital image;

b) extracting regions of arbitrary shape and size defined by actual objects from the digital image;

c) grouping the regions into larger segments corresponding to physically coherent objects;

d) extracting for each of the regions at least one structural saliency feature and at least one semantic saliency feature; and

,

e) integrating the structural saliency feature and the semantic feature using a probabilistic reasoning engine into an estimate of a belief that each region is the main subject.

20. The method as in claim 19, wherein step (b) includes using a color distance metric defined in a color space, a spatial homogeneity constraint, and a mechanism for permitting spatial adaptivity.

21. The method as in claim 19, wherein step (c) includes using either individually or in combination non-purposive grouping and purposive grouping.

22. The method as in claim 19, wherein step (d) includes using either individually or in combination at least one low-level vision feature and at least one geometric feature as the structural saliency feature.

23. The method as in claim 19, wherein step (d) includes using either individually or in combination a color, brightness and/or texture as a low-level vision feature;

a location, size, shape, convexity, aspect ratio, symmetry, borderness, surroundedness and/or occlusion as a geometric feature; and

a flesh, face, sky, grass and/or other green vegetation as the semantic saliency feature.

24. The method as in claim 19, wherein step (e) includes using a collection of human opinions to train the reasoning engine to recognize the relative importance of the saliency features.

25. The method as in claim 19, wherein step (d) includes using either individually or in combination a self-saliency feature and a relative saliency feature as the structural saliency feature.

26. The method as in claim 25, wherein step (d) includes using an extended neighborhood window to compute a plurality of the relative saliency features, wherein the extended neighborhood window is determined by the steps of:

(c1) finding a minimum bounding rectangle of a region;

(c2) stretching the minimum bounding rectangle in all four directions proportionally; and

,

(c3) defining all regions intersecting the stretched minimum bounding rectangle as neighbors of the region.

27. The method as in claim 23, wherein step (d) includes using a centrality as the location feature, wherein the centrality feature is computed by the steps of:

(c1) determining a probability density function of main subject locations using a collection of training data;

(c2) computing an integral of the probability density function over an area of a region; and

,

(c3) obtaining a value of the centrality feature by normalizing the integral by the area of the region.

28. The method as in claim 23, wherein step (d) includes using a hyperconvexity as the convexity feature, wherein the hyperconvexity feature is computed as a ratio of a perimeter-based convexity measure and an area-based convexity measure.

29. The method as in claim 23, wherein step (d) includes computing a maximum fraction of a region perimeter shared with a neighboring region as the surroundedness feature.

30. The method as in claim 23, wherein step (d) includes using an orientation-unaware borderness feature as the borderness feature, wherein the orientation-unaware borderness feature is categorized by the number and configuration of image borders a region is in contact with, and all image borders are treated equally.

31. The method as in claim 23, wherein step (d) includes using an orientation-aware borderness feature as the borderness feature, wherein the orientation-aware borderness feature is categorized by the number and configuration of image borders a region is in contact with, and each image border is treated differently.

32. The method as in claim 23, wherein step (d) includes using the borderness feature that is determined by what fraction of an image border is in contact with a region.

33. The method as in claim 23, wherein step (d) includes using the borderness feature that is determined by what fraction of a region border is in contact with an image border.

34. The method as in claim 19, wherein step (e) includes using a Bayes net as the reasoning engine.

35. The method as in claim 19, wherein step (e) includes using a conditional probability matrix that is determined by using fractional frequency counting according to a collection of training data.

36. The method as in claim 19, wherein step (e) includes using a belief sensor function to convert a measurement of a feature into evidence, which is an input to a Bayes net.

37. The method as in claim 19, wherein step (e) includes outputting a belief map, which indicates a location of and a belief in the main subject.