Bags of visual context-dependent words for generic visual categorization

• US 20080069456A1
• Filed: 09/19/2006
• Published: 03/20/2008
• Est. Priority Date: 09/19/2006
• Status: Active Grant

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Perronnin, Florent
• Granted Patent: US 7,885,466 B2
• Time in Patent Office: Days
• US Class Current: 382/228
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/464 using a plurality of salient features, e.g. bag-of-words [BoW] representations
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects

Abstract

Category context models (64) and a universal context model (62) are generated including sums of soft co-occurrences of pairs of visual words in geometric proximity to each other in training images (50) assigned to each category and assigned to all categories, respectively. Context information (76) about an image to be classified (70) are generated including sums of soft co-occurrences of pairs of visual words in geometric proximity to each other in the image to be classified. For each category (82), a comparison is made of (i) closeness of the context information about the image to be classified with the corresponding category context model and (ii) closeness of the context information about the image to be classified with the universal context model. An image category (92) is assigned to the image to be classified being based on the comparisons.

Claims

1. an image classification method comprising:

generating a category context model for each of a plurality of image categories including sums of soft co-occurrences of pairs of visual words in geometric proximity to each other in training images assigned to the category;

generating context information about an image to be classified including sums of soft co-occurrences of pairs of visual words in geometric proximity to each other in the image to be classified; and

assigning an image category to the image to be classified based at least on closeness of the context information about the image to the category context models.

2. The image classification method as set forth in claim 1, further comprising:

generating a universal context model including sums of soft co-occurrences of pairs of visual words in geometric proximity to each other in training images assigned to any of the plurality of image categories; and

for each category, generating a comparison of (i) closeness of the context information about the image to be classified with the corresponding category context model and (ii) closeness of the context information about the image to be classified with the universal context model, the assigning of the image category to the image to be classified being based on the comparisons.

3. The image classification method as set forth in claim 1, wherein the soft co-occurrences are computed as products of at least occupation probability values for the visual words of the pair of visual words in neighboring patches of the training image or the image to be classified.

4. An image classifier comprising:

a vocabulary of visual words;

a patch context analyzer configured to generate a context representation for each of a plurality of patches of an image, each context representation being indicative of occurrence probabilities of context words in neighboring patches; and

an image labeler configured to assign an image category to an image based at least on the context representations of a plurality of patches of the image.

5. The image classifier as set forth in claim 4, wherein the context words consist of a sub-set of visual words of the vocabulary that is substantially smaller than the total number of words in the vocabulary.

6. The image classifier as set forth in claim 4, wherein the context words include all visual words of the vocabulary.

7. The image classifier as set forth in claim 4, wherein the context words include at least some derived words that are not included in the vocabulary of visual words but that are derived from visual words of the vocabulary.

8. The image classifier as set forth in claim 4, wherein the patch context analyzer generates the context representation for each patch as a plurality of values, each value being indicative of a statistical occupancy probability of a corresponding context word in neighboring patches weighted by closeness of the neighboring patches to the patch for which the context histogram is generated.

9. The image classifier as set forth in claim 4, wherein the image labeler applies a category context model for each image category, the category context model indicating probabilities of context words being in a neighborhood of an occurrence of a vocabulary word for images of that image category.

10. The image classifier as set forth in claim 9, further comprising:

a category context model generator configured to generate each category context model as sums of soft co-occurrences of pairs of words in geometric proximity to each other in training images assigned to the category.

11. The image classifier as set forth in claim 9, wherein the image labeler further applies a universal context model, the universal context model indicating probabilities of context words being in a neighborhood of an occurrence of a vocabulary word for images regardless of image category.

12. The image classifier as set forth in claim 11, further comprising:

a category context model generator configured to generate each category context model as sums of soft co-occurrences of pairs of words in geometric proximity to each other in training images assigned to the category; and

a universal context model generator configured to generate the universal context model as sums of soft co-occurrences of pairs of words in geometric proximity to each other in training images assigned to all categories.

13. The image classifier as set forth in claim 4, wherein the image labeler includes a plurality of comparators each comparing (i) closeness of the context representations of the patches of the image with a category context model and (ii) closeness of the context representations of the patches of the image with a universal context model, the image category being assigned based on the outputs of the comparators.

14. An image classifier comprising:

a patch context analyzer configured to generate a context representation for each of a plurality of patches of an image; and

an image labeler including a plurality of comparators each comparing (i) closeness of context representations of a plurality of patches of an image with a category context model and (ii) closeness of the context representations of the plurality of patches of the image with a universal context model, the image labeler being configured to assign an image category to the image based on the outputs of the comparators.

15. The image classifier as set forth in claim 14, wherein the patch context analyzer comprises:

a context histogram generator configured to generate the context representation as a plurality of values each indicative of probabilistic number of occurrences of a corresponding context visual word in a neighborhood of a patch whose context representation is being generated.

16. The image classifier as set forth in claim 15, wherein the context histogram generator determines the probabilistic number of occurrences of each context visual word based on occupancy probabilities of the context visual word in a plurality of neighboring patches that neighbor the patch whose context representation is being generated.

17. The image classifier as set forth in claim 16, wherein the plurality of neighboring patches are defined as one of:

patches satisfying selected distance and size constraints respective to the patch whose context representation is being generated, and

pre-selected patches at pre-selected positions respective to the patch whose context representation is being generated.

18. An image classification method comprising:

generating a context representation for each of a plurality of patches of an image based at least on occupancy probabilities of context words in neighboring patches;

for each of a plurality of categories, generating a comparison of (i) closeness of the context representations of the image with a category context model representative of the category and (ii) closeness of the context representations of the image with a universal context model representative of all categories; and

assigning an image category to the image based on the generated comparisons.

19. The image classification method as set forth in claim 18, wherein the generating of the context representations comprises:

generating a context histogram for each patch, each element of the context histogram being indicative of a likelihood of occurrence of a corresponding context word in a neighborhood of the patch.

20. The image classification method as set forth in claim 18, wherein the generating of the comparisons comprises:

generating each category context model as sums of soft co-occurrences of pairs of words in geometric proximity to each other in training images assigned to the category; and

generating the universal context model as sums of soft co-occurrences of pairs of words in geometric proximity to each other in training images assigned to all categories.