SYSTEMS AND METHODS FOR GENERATING A HIGH-LEVEL VISUAL VOCABULARY

• US 20140272822A1
• Filed: 03/14/2013
• Published: 09/18/2014
• Est. Priority Date: 03/14/2013
• Status: Active Grant

• Current Assignee: Canon Ayutthaya Limited (Canon Inc.)
• Original Assignee: Canon Ayutthaya Limited (Canon Inc.)
• Inventors: Yang, Yang, Denney, Bradley Scott, Lu, Juwei, Dusberger, Dariusz, Huang, Hung Khei
• Granted Patent: US 9,342,991 B2
• Time in Patent Office: Days
• US Class Current: 434/167
• CPC Class Codes: G09B 19/00 Teaching not covered by other main groups of this subclass teaching or practice apparatus for gun-aiming or gun-laying F41G3/26
• CPC Class Codes: G09B 5/02 with visual presentation of the material to be studied, e.g. using film strip

Abstract

Systems and methods for learning a high-level visual vocabulary generate inter-visual-word relationships between a plurality of visual words based on visual word-label relationships, map the visual words to a vector space based on the inter-visual word relationships, and generate high-level visual words in the vector space.

Claims

1. A method comprising:

generating inter-visual-word relationships between a plurality of visual words based on visual word-label relationships;

mapping the visual words to a vector space based on the inter-visual-word relationships; and

generating high-level visual words in the vector space.

2. The method of claim 1, wherein respective distances between the visual words in the vector space indicate the inter-visual-word relationships.

3. The method of claim 1, wherein the inter-visual-word relationships are further based on visual word-image relationships and label-image relationships.

4. The method of claim 3, wherein the label-image relationships are one-to-one relationships.

5. The method of claim 1, wherein the visual word-label relationships are based on co-occurrences of respective visual words and labels in one or more images.

6. The method of claim 5, wherein the inter-visual word relationships are based on scores between the visual word-label relationships of respective visual words.

7. The method of claim 6, wherein the scores between the visual word-label relationships of respective visual words are based on respective averages of the KL divergence between each pair of mid-level features according to

S(F_i,F_j)=S(F_j,F_i)=½

[KL(P_i∥

P_j)+KL(P_j∥

P_i)],

where

8. The method of claim 1, further comprising generating clusters of visual words in the vector space based on the respective positions of the visual words in the vector space, wherein the high-level visual words are generated based on the clusters of visual words.

9. The method of claim 1, wherein the vector space is generated via a diffusion map.

10. A device comprising:

one or more computer-readable media configured to store instructions; and

one or more processors configured to cause the device to

generate mid-level visual words based on a plurality of low-level features that were extracted from images;

generate inter-mid-level visual-word relationships based on mid-level visual-word-label relationships; and

map the mid-level visual words to a vector space based on the inter-mid-level visual-word relationships.

11. The device of claim 10, wherein the one or more processors are further configured to cause the device to generate high-level visual words in the vector space based on the inter-mid-level visual words in the vector space.

12. The device of claim 10, wherein the one or more processors are configured to map the mid-level visual words to the vector space by performing operations that include generating a Markov Transition Matrix based on the inter-mid-level visual-word relationships.

13. The device of claim 12, wherein the one or more processors are further configured to cause the device to

calculate dominant eigenvectors based on the Markov Transition Matrix, and

define dimensions of the vector space according to the dominant eigenvectors.

14. One or more computer-readable media storing instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations comprising:

generating inter-visual-word relationships between a plurality of visual words based on visual word-label relationships;

mapping the visual words to a vector space based on the inter-visual-word relationships; and

generating high-level visual words in the vector space based on respective positions of the visual words in the vector space.

15. The one or more computer-readable media of claim 14, wherein the inter-visual-word relationships are represented as distances between the respective visual words in the vector space.

16. The one or more computer-readable media of claim 14, wherein mapping the visual words to the vector space includes generating a weight matrix.

17. The one or more computer-readable media of claim 14, wherein the high-level visual words encode features via soft cluster assignments.

18. A method comprising:

generating inter-visual-word relationships between a plurality of visual words;

generating sets of importance weights for the visual words, wherein a set of importance weights includes a respective weight for each of the visual words;

mapping the visual words to a plurality of vector spaces based on the inter-visual word relationships and on the sets of importance weights, wherein each vector space corresponds to a respective one of the sets of importance weights; and

generating high-level visual words in the plurality of vector spaces.

19. The method of claim 18, further comprising assigning a respective importance score to each of the high-level visual words, and selecting high-level visual words based on their respective importance scores.