Method and apparatus for context-aided human identification

• US 20070237364A1
• Filed: 03/31/2006
• Published: 10/11/2007
• Est. Priority Date: 03/31/2006
• Status: Abandoned Application

• Current Assignee: Fujifilm Corporation (Fujifilm Holdings Corporation)
• Original Assignee: Fuji Photo Film Co Limited (Fujifilm Holdings Corporation)
• Inventors: Song, Yang, Leung, Thomas
• Application Number:
  US11/394,242
• Publication Number:
  US 20070237364A1
• Time in Patent Office: Days
• US Class Current: 382/115
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

A method and an apparatus process digital images. The method according to one embodiment accesses digital data representing a plurality of digital images including a plurality of persons; performs face recognition to generate face recognition scores relating to similarity between faces of the plurality of persons; performs clothes recognition to generate clothes recognition scores relating to similarity between clothes of the plurality of persons; obtains inter-relational person scores relating to similarity between persons of the plurality of persons using the face recognition scores and the clothes recognition scores; and clusters the plurality of persons from the plurality of digital images using the inter-relational person scores to obtain clusters relating to identities of the persons from the plurality of persons.

Claims

1. A digital image processing method, said method comprising:

accessing digital data representing a plurality of digital images including a plurality of persons;

performing face recognition to generate face recognition scores relating to similarity between faces of said plurality of persons;

performing clothes recognition to generate clothes recognition scores relating to similarity between clothes of said plurality of persons;

obtaining inter-relational person scores relating to similarity between persons of said plurality of persons using said face recognition scores and said clothes recognition scores; and

clustering said plurality of persons from said plurality of digital images using said inter-relational person scores to obtain clusters relating to identities of said persons from said plurality of persons.

2. The digital image processing method as recited in claim 1, wherein said step of performing clothes recognition includes

segmenting clothes to obtain clothes areas in said plurality of digital images, and

removing clutters not belonging to said clothes areas.

3. The digital image processing method as recited in claim 2, wherein said step of performing clothes recognition includes

detecting clothes regions in said plurality of digital images by determining that sections in said plurality of digital images below faces of said plurality of persons are clothes regions associated with said faces,

wherein said sub-step of segmenting clothes determines clothes areas by maximizing a difference between said clothes regions.

4. The digital image processing method as recited in claim 2, wherein said sub-step of removing clutters includes removing data indicative of human skin.

5. The digital image processing method as recited in claim 2, wherein said step of performing clothes recognition includes

performing clothes feature extraction for said clothes areas obtained from said sub-step of removing clutters.

6. The digital image processing method as recited in claim 5, wherein said sub-step of performing clothes feature extraction includes

normalizing said clothes areas based on size of heads of said plurality of persons,

taking small image patches from said normalized clothes areas,

gathering said small image patches from said normalized clothes areas,

quantizing said small image patches using vector quantization to obtain patch vectors,

clustering said patch vectors to obtain patch clusters and code-words as centers of said patch clusters, and

representing said clothes areas by code-word feature vectors of frequencies of appearance of said code-words in said clothes areas.

7. The digital image processing method as recited in claim 6, wherein said step of performing clothes recognition includes computing clothes similarity by

weighting said code-word feature vectors such that higher priorities are given to code-words occurring less frequently, and

calculating said clothes recognition scores as a scalar product of said weighted code-word feature vectors for clothes areas pairs from said clothes areas belonging to different persons from said plurality of persons.

8. The digital image processing method as recited in claim 7, wherein said clothes of said plurality of persons include at least one of clothing, shoes, watch, and eyeglasses.

9. The digital image processing method as recited in claim 1, wherein said step of obtaining inter-relational person scores includes

applying a plurality of formulas for estimating probabilities for pairs of persons from said plurality of persons to represent the same person, based on availability of clothes recognition scores and face recognition scores for said pairs of persons.

10. The digital image processing method as recited in claim 9, wherein said step of obtaining inter-relational person scores

selects at least one formula from said plurality of formulas based on a time when images from said plurality of digital images were taken, to obtain an inter-relational person score between two persons from said plurality of persons,

selects at least one formula from said plurality of formulas based on a place where images from said plurality of digital images were taken, to obtain an inter-relational person score between two persons from said plurality of persons,

selects at least one formula from said plurality of formulas based on whether two persons A and B from said plurality of persons are wearing the same clothes in an image from said plurality of digital images, to obtain an inter-relational person score between said two persons A and B,

selects at least one formula from said plurality of formulas when only a face recognition score but no clothes recognition score is available for two persons C and D from said plurality of persons, to obtain an inter-relational person score between said two persons C and D,

selects at least one formula from said plurality of formulas when only a clothes recognition score but no face recognition score is available for two persons E and F from said plurality of persons, to obtain an inter-relational person score between said two persons E and F, and

selects at least one formula from said plurality of formulas when both a face recognition score and a clothes recognition score is available for two persons H and J from said plurality of persons, to obtain an inter-relational person score between said two persons H and J.

11. The digital image processing method as recited in claim 9, wherein said step obtaining inter-relational person scores further includes

obtaining said plurality of formulas using logistic regression, and

learning parameters for said plurality of formulas using logistic regression, before said sub-step of applying said plurality of formulas.

12. The digital image processing method as recited in claim 1, wherein said step of clustering includes

performing a spectral analysis to obtain eigenvector results from an arrangement of said inter-relational person scores, and

performing discretization of said eigenvector results by clustering said eigenvector results to obtain clusters relating to identities of said persons from said plurality of persons.

13. The digital image processing method as recited in claim 1, wherein said step of clustering includes

incorporating in an arrangement of said inter-relational person scores at least one hard constraint relating to persons from said plurality of persons, to obtain constrained inter-relational data results,

performing a spectral analysis to obtain eigenvector results from said constrained inter-relational data results, and

performing discretization of said eigenvector results by clustering said eigenvector results to obtain clusters relating to identities of persons from said plurality of persons.

14. The digital image processing method as recited in claim 1, wherein said step of clustering includes

incorporating in an arrangement of said inter-relational person scores at least one hard constraint relating to persons from said plurality of persons, to obtain constrained inter-relational data results,

performing a spectral analysis to obtain eigenvector results from said constrained inter-relational data results, and

performing discretization of said eigenvector results using constrained clustering with a criterion to enforce said at least one hard constraint to obtain clusters relating to identities of persons from said plurality of persons.

15. The digital image processing method as recited in claim 14, wherein said sub-step of performing discretization uses constrained K-means clustering.

16. The digital image processing method as recited in claim 15, wherein said at least one hard constraint includes a hard negative constraint that two persons found in the same image from said plurality of digital images have different identities.

17. The digital image processing method as recited in claim 15, wherein said at least one hard constraint includes a positive constraint based on predetermined knowledge that two persons found in different images from said plurality of digital images are the same person.

18. The digital image processing method as recited in claim 1, wherein

said step of obtaining inter-relational person scores includes obtaining an affinity matrix A using said face recognition scores and said clothes recognition scores;

and

said step of clustering includes

incorporating in said affinity matrix A at least one hard negative constraint,

generating a repulsion matrix R using said at least one hard negative constraint,

using said affinity matrix A and said repulsion matrix R to obtain constrained inter-relational data results in the form of an inter-relational data matrix L;

selecting a predetermined number of largest eigenvectors of said inter-relational data matrix L,

stacking selected eigenvectors into columns to obtain a matrix X,

normalizing the rows of said matrix X to unit length to obtain said eigenvector results in the form of a matrix Y,

clustering rows of said matrix Y using K-means clustering to obtain said clusters, and

assigning said persons to clusters to which rows of said matrix Y associated with said persons are assigned.

19. The digital image processing method as recited in claim 1, wherein

said step of obtaining inter-relational person scores includes obtaining an affinity matrix A using said face recognition scores and said clothes recognition scores;

and

said step of clustering includes

incorporating in said affinity matrix A at least one hard constraint,

using said affinity matrix A to obtain constrained inter-relational data results in the form of an inter-relational data matrix L;

selecting a predetermined number of largest eigenvectors of said inter-relational data matrix L,

stacking selected eigenvectors into columns to obtain a matrix X,

normalizing the rows of said matrix X to unit length to obtain said eigenvector results in the form of a matrix Y,

clustering rows of said matrix Y using constrained clustering using a criterion to enforce said at least one hard constraint to obtain said clusters, and

assigning said persons to clusters to which rows of said matrix Y associated with said persons are assigned.

20. The digital image processing method as recited in claim 1, wherein said clustering step assigns digital images of said plurality of digital images to clusters to which persons from said plurality of persons in said digital images are clustered.

21. A digital image processing apparatus, said apparatus comprising:

an image data unit for providing digital data representing a plurality of digital images including a plurality of persons;

a face recognition unit for generating face recognition scores relating to similarity between faces of said plurality of persons;

a clothes recognition unit for generating clothes recognition scores relating to similarity between clothes of said plurality of persons;

a combination unit for obtaining inter-relational person scores relating to similarity between persons of said plurality of persons using said face recognition scores and said clothes recognition scores; and

a classification unit for clustering said plurality of persons from said plurality of digital images using said inter-relational person scores to obtain clusters relating to identities of said persons from said plurality of persons.

22. The apparatus according to claim 21, wherein said clothes recognition unit segments clothes to obtain clothes areas in said plurality of digital images and removes clutters not belonging to said clothes areas.

23. The apparatus according to claim 22, wherein said clothes recognition unit

detects clothes regions in said plurality of digital images by determining that sections in said plurality of digital images below faces of said plurality of persons are clothes regions associated with said faces,

segments clothes to obtain clothes areas by maximizing a difference between said clothes regions.

24. The apparatus according to claim 22, wherein said clothes recognition unit removes clutters by removing data indicative of human skin.

25. The apparatus according to claim 22, wherein said clothes recognition unit

performs clothes feature extraction for said clothes areas obtained after clutters are removed.

26. The apparatus according to claim 25, wherein said clothes recognition unit performs clothes feature extraction by

normalizing said clothes areas based on size of heads of said plurality of persons,

taking small image patches from said normalized clothes areas,

gathering said small image patches from said normalized clothes areas,

quantizing said small image patches using vector quantization to obtain patch vectors,

clustering said patch vectors to obtain patch clusters and code-words as centers of said patch clusters, and

representing said clothes areas by code-word feature vectors of frequencies of appearance of said code-words in said clothes areas.

27. The apparatus according to claim 26, wherein said clothes recognition unit generates clothes recognition scores by

weighting said code-word feature vectors such that higher priorities are given to code-words occurring less frequently, and

calculating said clothes recognition scores as a scalar product of said weighted code-word feature vectors for clothes areas pairs from said clothes areas belonging to different persons from said plurality of persons.

28. The apparatus according to claim 27, wherein said clothes of said plurality of persons include at least one of clothing, shoes, watch, and eyeglasses.

29. The apparatus according to claim 21, wherein said combination unit obtains inter-relational person scores by

applying a plurality of formulas for estimating probabilities for pairs of persons from said plurality of persons to represent the same person, based on availability of clothes recognition scores and face recognition scores for said pairs of persons.

30. The apparatus according to claim 29, wherein said combination unit obtains inter-relational person scores by

selecting at least one formula from said plurality of formulas based on a time when images from said plurality of digital images were taken, to obtain an inter-relational person score between two persons from said plurality of persons,

selecting at least one formula from said plurality of formulas based on a place where images from said plurality of digital images were taken, to obtain an inter-relational person score between two persons from said plurality of persons,

selecting at least one formula from said plurality of formulas based on whether two persons A and B from said plurality of persons are wearing the same clothes in an image from said plurality of digital images, to obtain an inter-relational person score between said two persons A and B,

selecting at least one formula from said plurality of formulas when only a face recognition score but no clothes recognition score is available for two persons C and D from said plurality of persons, to obtain an inter-relational person score between said two persons C and D,

selecting at least one formula from said plurality of formulas when only a clothes recognition score but no face recognition score is available for two persons E and F from said plurality of persons, to obtain an inter-relational person score between said two persons E and F, and

selecting at least one formula from said plurality of formulas when both a face recognition score and a clothes recognition score is available for two persons H and J from said plurality of persons, to obtain an inter-relational person score between said two persons H and J.

31. The apparatus according to claim 29, wherein said combination unit obtains said plurality of formulas using logistic regression, and learns parameters for said plurality of formulas using logistic regression.

32. The apparatus according to claim 21, wherein said classification unit clusters said plurality of persons by

performing a spectral analysis to obtain eigenvector results from an arrangement of said inter-relational person scores, and

performing discretization of said eigenvector results by clustering said eigenvector results to obtain clusters relating to identities of said persons from said plurality of persons.

33. The apparatus according to claim 21, wherein said classification unit clusters said plurality of persons by

incorporating in an arrangement of said inter-relational person scores at least one hard constraint relating to persons from said plurality of persons, to obtain constrained inter-relational data results,

performing a spectral analysis to obtain eigenvector results from said constrained inter-relational data results, and

performing discretization of said eigenvector results by clustering said eigenvector results to obtain clusters relating to identities of persons from said plurality of persons.

34. The apparatus according to claim 21, wherein said classification unit clusters said plurality of persons by

incorporating in an arrangement of said inter-relational person scores at least one hard constraint relating to persons from said plurality of persons, to obtain constrained inter-relational data results,

performing a spectral analysis to obtain eigenvector results from said constrained inter-relational data results, and

performing discretization of said eigenvector results using constrained clustering with a criterion to enforce said at least one hard constraint to obtain clusters relating to identities of persons from said plurality of persons.

35. The apparatus according to claim 34, wherein said classification unit performs discretization uses constrained K-means clustering.

36. The apparatus according to claim 35, wherein said at least one hard constraint includes a hard negative constraint that two persons found in the same image from said plurality of digital images have different identities.

37. The apparatus according to claim 35, wherein said at least one hard constraint includes a positive constraint based on predetermined knowledge that two persons found in different images from said plurality of digital images are the same person.

38. The apparatus according to claim 21, wherein

said combination unit obtains inter-relational person scores by obtaining an affinity matrix A using said face recognition scores and said clothes recognition scores;

and

said classification unit clusters said plurality of persons by

incorporating in said affinity matrix A at least one hard negative constraint,

generating a repulsion matrix R using said at least one hard negative constraint,

using said affinity matrix A and said repulsion matrix R to obtain constrained inter-relational data results in the form of an inter-relational data matrix L;

selecting a predetermined number of largest eigenvectors of said inter-relational data matrix L,

stacking selected eigenvectors into columns to obtain a matrix X,

normalizing the rows of said matrix X to unit length to obtain said eigenvector results in the form of a matrix Y,

clustering rows of said matrix Y using K-means clustering to obtain said clusters, and

assigning said persons to clusters to which rows of said matrix Y associated with said persons are assigned.

39. The apparatus according to claim 21, wherein

said combination unit obtains inter-relational person scores by obtaining an affinity matrix A using said face recognition scores and said clothes recognition scores;

and

said classification unit clusters said plurality of persons by

incorporating in said affinity matrix A at least one hard constraint,

using said affinity matrix A to obtain constrained inter-relational data results in the form of an inter-relational data matrix L;

selecting a predetermined number of largest eigenvectors of said inter-relational data matrix L,

stacking selected eigenvectors into columns to obtain a matrix X,

normalizing the rows of said matrix X to unit length to obtain said eigenvector results in the form of a matrix Y,

clustering rows of said matrix Y using constrained clustering using a criterion to enforce said at least one hard constraint to obtain said clusters, and

assigning said persons to clusters to which rows of said matrix Y associated with said persons are assigned.

40. The apparatus according to claim 21, wherein said classification unit assigns digital images of said plurality of digital images to clusters to which persons from said plurality of persons in said digital images are clustered.