Generic visual categorization method and system

• US 7,756,341 B2
• Filed: 06/30/2005
• Issued: 07/13/2010
• Est. Priority Date: 06/30/2005
• Status: Active Grant

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Perronnin, Florent
• Primary Examiner(s): Mariam; Daniel G
• Assistant Examiner(s): Woldemariam; Aklilu K
• Application Number:
  US11/170,496
• Publication Number:
  US 20070005356A1
• Time in Patent Office: 1,839 Days
• Field of Search: 382/224, 704/245
• US Class Current: 382/224
• CPC Class Codes: G06V 10/464 using a plurality of salient features, e.g. bag-of-words [BoW] representations

Abstract

Generic visual categorization methods complement a general vocabulary with adapted vocabularies that are class specific. Images to be categorized are characterized within different categories through a histogram indicating whether the image is better described by the general vocabulary or the class-specific adapted vocabulary.

Claims

1. A method for assigning one of a plurality of classes to an input image, comprising one or more processors implementing the following actions:

identifying by a key patch detector a plurality of key-patches in the input image;

computing by a feature description module an unordered feature vector for each of the plurality of key-patches;

computing by a multi-histogram computation module a first histogram of fixed length feature vectors for each of the plurality of classes using the plurality of unordered feature vectors computed;

computing by the multi-histogram computation module a second histogram for each of the plurality of classes using the plurality of unordered feature vectors computed and the first histogram of fixed length feature vectors computed; and

,

assigning by a classifier training module at least one of the plurality of classes to the input image using the plurality of histograms computed as input to a classifier;

wherein the first and second histograms are iteratively estimated using an Expectation-Maximization based calculation for mixture weight, mean vector, and covariance matrix parameters of the feature vector, wherein the parameters correspond to relative frequencies, averages, and variations of the visual words.

2. The method according to claim 1, wherein each histogram is computed by estimating occupancy probabilities of the feature vectors of the input image for each class vocabulary.

3. The method according to claim 2, further comprising defining each class vocabulary by merging a general vocabulary and an adapted vocabulary for each class.

4. The method according to claim 3, wherein the histogram computed for each of the plurality of classes indicates whether the input image is better described by the general vocabulary or the adapted vocabulary of its corresponding class.

5. The method according to claim 4, further comprising:

computing key-patches in images of class training sets;

computing feature vectors for key-patches of the images of the class training sets;

computing the general vocabulary by clustering feature vectors of the images of class training sets;

computing each class'"'"' adapted vocabulary by estimating occupancy probabilities, of the feature vectors of the images of its class training set, for each class vocabulary.

6. The method according to claim 5, further comprising training the classifier with histograms computed for each class by estimating occupancy probabilities, of the feature vectors of images of the class trainings set, for each class vocabulary.

7. The method according to claim 2, wherein said assigning further comprises:

computing a score for the input image for each of the plurality of classes using its computed histogram;

assigning the input image a label corresponding to the class with the highest computed score.

8. The method according to claim 7, wherein said computing computes the score using a Support Vector Machine classifier for each class.

9. An apparatus for processing an assignment of one of a plurality of classes to an input image, comprising:

a memory which stores;

a key-patch detector identifying a plurality of key-patches in the input image;

a feature description module for computing an unordered feature vector for each of the plurality of key-patches;

a multi-histogram computation module computing a first histogram of fixed length feature vectors for each of the plurality of classes using the plurality of unordered feature vectors computed;

the multi-histogram computation module computing a second histogram of each of the plurality of classes using the plurality of unordered feature vectors computed and the first histogram of fixed length feature vectors computed;

wherein the first and second histograms are iteratively estimated using an Expectation-Maximization based calculation for mixture weight, mean vector, and covariance matrix parameters of the feature vector, wherein the parameters correspond to relative frequencies, averages, and variations of the visual words

a classifier for assigning at least one of the plurality of classes to the input image using the plurality of histograms computed as input to the classifier; and

a processor in communication with the memory configured for processing the image with the key-patch detector, the feature description module, the multi-histogram computation module, and the classifier.

10. The apparatus according to claim 9, wherein the multi-histogram computation module computes each histogram by estimating occupancy probabilities of the feature vectors of the input image for each class vocabulary.

11. The apparatus according to claim 10, wherein the multi-histogram computation module defines each class vocabulary by merging a general vocabulary and an adapted vocabulary for each class.

12. The apparatus according to claim 11, wherein the histogram computed by the multi-histogram computation module for each of the plurality of classes indicates whether the input image is better described by the general vocabulary or the adapted vocabulary of its corresponding class.

13. The apparatus according to claim 10, wherein the key-patch detector computes key-patches in images of class training sets;

the feature description module computes feature vectors for key-patches of the images of the class training sets;

the multi-histogram computes;

the general vocabulary by clustering feature vectors of the images of class training sets, and each class'"'"' adapted vocabulary by estimating occupancy probabilities, of the feature vectors of the images of its class training set, for each class vocabulary.

14. The apparatus according to claim 13, further comprising a classifier training module for training the classifier with histograms computed for each class by estimating occupancy probabilities, of the feature vectors of images of the class trainings set, for each class vocabulary.

15. The apparatus according to claim 10, wherein the classifier:

computes a score for the input image for each of the plurality of classes using its computed histogram;

assigns the input image a label corresponding to the class with the highest computed score.

16. The apparatus according to claim 15, wherein the classifier comprises a Support Vector Machine classifier for computing the score for each class.

17. A method for a computer-based system for training a classifier, comprising one or more processors implementing the following actions:

identifying by a key patch detector key-patches in images of a plurality of class training sets;

computing by a feature description module feature vectors for the identified key-patches;

computing by a multi-histogram computation module a general vocabulary by clustering the computed feature vectors;

for each of a plurality of classes, computing by a multi-histogram computation module an adapted vocabulary using the general vocabulary;

comparing the adapted vocabulary to the general vocabulary;

storing by a memory the adapted vocabulary if it is changed as compared to the general vocabulary;

computing by a class-specific histogram calculator a histogram for each of the plurality of classes;

training by a classifier training module the classifier using the histograms for each of the plurality of classes;

wherein the first and second histograms are iteratively estimated using an Expectation-Maximization based calculation for mixture weight, mean vector, and covariance matrix parameters of the feature vector, wherein the parameters correspond to relative frequencies, averages, and variations of the visual words.

18. The method according to claim 17, further comprising categorizing an input image with the image classifier.

19. The method according to claim 18, wherein said categorizing further comprises:

identifying a plurality of key-patches in the input image;

computing a feature vector for each of the plurality of key-patches;

computing a histogram for each of the plurality of classes using the plurality of feature vectors computed;

assigning at least one of the plurality of classes to the input image using the plurality of histograms computed as input to the classifier.

20. The method according to claim 19, wherein each histogram is computed by estimating occupancy probabilities of the feature vectors of the input image for each class vocabulary.