Method and apparatus for adaptive context-aided human classification

• US 20070237355A1
• Filed: 03/31/2006
• Published: 10/11/2007
• Est. Priority Date: 03/31/2006
• Status: Active Grant

• Current Assignee: Fujifilm Corporation (Fujifilm Holdings Corporation)
• Original Assignee: Fuji Photo Film Co Limited (Fujifilm Holdings Corporation)
• Inventors: Song, Yang, Leung, Thomas
• Granted Patent: US 7,864,989 B2
• Time in Patent Office: Days
• US Class Current: 382/100
• CPC Class Codes: G06F 18/2323 based on graph theory, e.g. minimum spanning trees [MST] or graph cuts
• CPC Class Codes: G06F 18/24155 Bayesian classification
• CPC Class Codes: G06V 10/7635 based on graphs, e.g. graph cuts or spectral clustering
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

A method and an apparatus process digital images. The method according to one embodiment accesses digital data representing a plurality of digital images including a plurality of persons; performs face recognition to determine first scores relating to similarity between faces of the plurality of persons; performs clothes recognition to determine second scores relating to similarity between clothes of the plurality of persons; provides a plurality of formulas for estimating a probability of a face from the faces and a clothes from the clothes to belong to a person from the plurality of persons, wherein at least one formula of the plurality of formulas utilizes a first score and a second score, and at least one formula of the plurality of formulas utilizes only one score of a first score and a second score; and selects a formula from the plurality of formulas based on availability of a first score from the first scores for two persons from the plurality of persons, and availability of a second score from the second scores for the two persons, the selected formula estimating a probability relating to similarity of identities of the two persons.

Claims

1. A digital image processing method, said method comprising:

accessing digital data representing a plurality of digital images including a plurality of persons;

performing face recognition to determine first scores relating to similarity between faces of said plurality of persons;

performing clothes recognition to determine second scores relating to similarity between clothes of said plurality of persons;

providing a plurality of formulas for estimating a probability of a face from said faces and a clothes from said clothes to belong to a person from said plurality of persons, wherein

at least one formula of said plurality of formulas utilizes a first score and a second score, and

at least one formula of said plurality of formulas utilizes only one score of a first score and a second score; and

selecting a formula from said plurality of formulas based on availability of a first score from said first scores for two persons from said plurality of persons, and availability of a second score from said second scores for said two persons, said selected formula estimating a probability relating to similarity of identities of said two persons.

2. The digital image processing method as recited in claim 1, wherein said clothes of said plurality of persons include at least one of clothing, shoes, watch, and eyeglasses.

3. The digital image processing method as recited in claim 1, wherein said selecting step includes selecting a formula from said plurality of formulas based on a time when digital images associated with said two persons were taken.

4. The digital image processing method as recited in claim 1, wherein said selecting step includes selecting a formula from said plurality of formulas based on a place where digital images associated with said two persons were taken.

5. The digital image processing method as recited in claim 1, wherein said selecting step includes selecting a formula from said plurality of formulas based on whether said two persons are associated with the same image from said plurality of digital images and said two persons are wearing similar clothes.

6. The digital image processing method as recited in claim 1, wherein said selecting step includes selecting a formula from said plurality of formulas when a first score for said two persons is not available.

7. The digital image processing method as recited in claim 1, wherein said selecting step includes selecting a formula from said plurality of formulas when a first score and a second score for said two persons are available.

8. The digital image processing method as recited in claim 1, wherein said plurality of formulas are derived using logistic regression.

9. The digital image processing method as recited in claim 8, wherein said step of providing a plurality of formulas includes learning parameters for said plurality of formulas using logistic regression.

10. The digital image processing method as recited in claim 1, wherein said step of performing clothes recognition includes

detecting clothes regions below faces of said plurality of persons,

segmenting clothes regions by maximizing a difference between said clothes regions to determine clothes areas, and

removing clutters not belonging to said clothes areas.

11. The digital image processing method as recited in claim 10, wherein said step of performing clothes recognition further includes performing clothes feature extraction for said clothes areas by

extracting representative image patches from said clothes areas,

quantizing said representative image patches using vector quantization to obtain patch vectors,

clustering said patch vectors to obtain patch clusters and code-words as centers of said patch clusters, and

representing said clothes areas by code-word feature vectors of frequencies of appearance of said code-words in said clothes areas.

12. The digital image processing method as recited in claim 11, wherein said step of performing clothes recognition further includes

weighting said code-word feature vectors such that higher priorities are given to code-words occurring less frequently, and

calculating said second scores as a scalar product of said weighted code-word feature vectors for clothes areas pairs from said clothes areas.

13. The digital image processing method as recited in claim 1, wherein said selecting step is performed for a plurality of pairs of persons from said plurality of persons, to estimate a plurality of probabilities relating to similarities of identities of persons in said plurality of pairs.

14. The digital image processing method as recited in claim 13, further comprising:

clustering said plurality of persons using said plurality of probabilities to obtain clusters relating to identities of persons from said plurality of persons.

15. The digital image processing method as recited in claim 13, further comprising:

performing a spectral analysis to obtain eigenvector results from an arrangement of said plurality of probabilities, and

performing discretization of said eigenvector results by clustering said eigenvector results to obtain clusters relating to identities of persons from said plurality of persons.

16. The digital image processing method as recited in claim 13, further comprising:

incorporating in an arrangement of said plurality of probabilities at least one hard constraint relating to persons from said plurality of persons, to obtain constrained inter-relational data results,

performing a spectral analysis to obtain eigenvector results from said constrained inter-relational data results, and

performing discretization of said eigenvector results by clustering said eigenvector results to obtain clusters relating to identities of persons from said plurality of persons.

17. The digital image processing method as recited in claim 13, further comprising:

incorporating in an arrangement of said plurality of probabilities at least one hard constraint relating to persons from said plurality of persons, to obtain constrained inter-relational data results,

performing a spectral analysis to obtain eigenvector results from said constrained inter-relational data results, and

performing discretization of said eigenvector results using constrained clustering with a criterion to enforce said at least one hard constraint to obtain clusters relating to identities of persons from said plurality of persons.

18. The digital image processing method as recited in claim 17, wherein said sub-step of performing discretization uses constrained K-means clustering.

19. The digital image processing method as recited in claim 18, wherein said at least one hard constraint includes a hard negative constraint that two persons found in the same image from said plurality of digital images have different identities.

20. The digital image processing method as recited in claim 18, wherein said at least one hard constraint includes a positive constraint based on predetermined knowledge that two persons found in different images from said plurality of digital images are the same person.

21. A digital image processing apparatus, said apparatus comprising:

an image data unit for providing digital data representing a plurality of digital images including a plurality of persons;

a face recognition unit for determining first scores relating to similarity between faces of said plurality of persons;

a clothes recognition unit for determining second scores relating to similarity between clothes of said plurality of persons; and

a formula selection unit for

providing a plurality of formulas to estimate a probability of a face from said faces and a clothes from said clothes to belong to a person from said plurality of persons, wherein at least one formula of said plurality of formulas utilizes a first score and a second score, and at least one formula of said plurality of formulas utilizes only one score of a first score and a second score, and

selecting a formula from said plurality of formulas based on availability of a first score from said first scores for two persons from said plurality of persons, and availability of a second score from said second scores for said two persons, said selected formula estimating a probability relating to similarity of identities of said two persons.

22. The apparatus according to claim 21, wherein said clothes of said plurality of persons include at least one of clothing, shoes, watch, and eyeglasses.

23. The apparatus according to claim 21, wherein said formula selection unit selects a formula from said plurality of formulas based on a time when digital images associated with said two persons were taken.

24. The apparatus according to claim 21, wherein said formula selection unit selects a formula from said plurality of formulas based on a place where digital images associated with said two persons were taken.

25. The apparatus according to claim 21, wherein said formula selection unit selects a formula from said plurality of formulas based on whether said two persons are associated with the same image from said plurality of digital images and said two persons are wearing similar clothes.

26. The apparatus according to claim 21, wherein said formula selection unit selects a formula from said plurality of formulas when a first score for said two persons is not available.

27. The apparatus according to claim 21, wherein said formula selection unit selects a formula from said plurality of formulas when a first score and a second score for said two persons are available.

28. The apparatus according to claim 21, wherein said plurality of formulas are derived using logistic regression.

29. The apparatus according to claim 28, further comprising:

a similarity measurement unit for learning parameters for said plurality of formulas using logistic regression.

30. The apparatus according to claim 21, wherein said clothes recognition unit performs clothes recognition by

detecting clothes regions below faces of said plurality of persons,

segmenting clothes regions by maximizing a difference between said clothes regions to determine clothes areas, and

removing clutters not belonging to said clothes areas.

31. The apparatus according to claim 30, wherein said clothes recognition unit performs clothes feature extraction for said clothes areas by

extracting representative image patches from said clothes areas,

quantizing said representative image patches using vector quantization to obtain patch vectors,

clustering said patch vectors to obtain patch clusters and code-words as centers of said patch clusters, and

representing said clothes areas by code-word feature vectors of frequencies of appearance of said code-words in said clothes areas.

32. The apparatus according to claim 31, wherein said clothes recognition unit determines said second scores by

weighting said code-word feature vectors such that higher priorities are given to code-words occurring less frequently, and

calculating said second scores as a scalar product of said weighted code-word feature vectors for clothes areas pairs from said clothes areas.

33. The apparatus according to claim 21, further including a similarity measurement unit, wherein

said formula selection unit selects formulas from said plurality of formulas for a plurality of pairs of persons from said plurality of persons, and

said similarity measurement unit estimates a plurality of probabilities relating to similarities of identities of persons in said plurality of pairs.

34. The apparatus according to claim 33, further comprising:

a classification unit for clustering said plurality of persons using said plurality of probabilities to obtain clusters relating to identities of persons from said plurality of persons.

35. The apparatus according to claim 33, further comprising a classification unit for

performing a spectral analysis to obtain eigenvector results from an arrangement of said plurality of probabilities, and

performing discretization of said eigenvector results by clustering said eigenvector results to obtain clusters relating to identities of persons from said plurality of persons.

36. The apparatus according to claim 33, further comprising a classification unit for

incorporating in an arrangement of said plurality of probabilities at least one hard constraint relating to persons from said plurality of persons, to obtain constrained inter-relational data results,

performing a spectral analysis to obtain eigenvector results from said constrained inter-relational data results, and

performing discretization of said eigenvector results by clustering said eigenvector results to obtain clusters relating to identities of persons from said plurality of persons.

37. The apparatus according to claim 33, further comprising a classification unit for

incorporating in an arrangement of said plurality of probabilities at least one hard constraint relating to persons from said plurality of persons, to obtain constrained inter-relational data results,

performing a spectral analysis to obtain eigenvector results from said constrained inter-relational data results, and

performing discretization of said eigenvector results using constrained clustering with a criterion to enforce said at least one hard constraint to obtain clusters relating to identities of persons from said plurality of persons.

38. The apparatus according to claim 37, wherein said classification unit performs discretization using constrained K-means clustering.

39. The apparatus according to claim 38, wherein said at least one hard constraint includes a hard negative constraint that two persons found in the same image from said plurality of digital images have different identities.

40. The apparatus according to claim 38, wherein said at least one hard constraint includes a positive constraint based on predetermined knowledge that two persons found in different images from said plurality of digital images are the same person.