SYSTEMS AND METHODS FOR FEATURE FUSION

• US 20140267301A1
• Filed: 03/14/2013
• Published: 09/18/2014
• Est. Priority Date: 03/14/2013
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Denney, Bradley Scott, Lu, Juwei, Dusberger, Dariusz, Huang, Hung Khei, Yang, Yang
• Granted Patent: US 9,190,026 B2
• Time in Patent Office: Days
• US Class Current: 345/467
• CPC Class Codes: G06F 18/231 Hierarchical techniques, i.e. dividing or merging pattern sets so as to obtain a dendrogram
• CPC Class Codes: G06T 11/206 Drawing of charts or graphs
• CPC Class Codes: G06V 10/7625 Hierarchical techniques, i.e. dividing or merging patterns to obtain a tree-like representation; Dendograms
• CPC Class Codes: G09G 5/24 Generation of individual character patterns

Abstract

Systems and methods for generating visual words define initial inter-visual word relationships between a plurality of visual words; define visual word-image relationships between the plurality of visual words and a plurality of images; define inter-image relationships between the plurality of images; generate revised inter-visual word relationships in a vector space based on the initial inter-visual word relationships, the inter-image relationships, and the visual word-image relationships; and generate higher-level visual words in the vector space based on the revised inter-visual word relationships.

Claims

1. A method comprising:

defining initial inter-visual word relationships between a plurality of visual words;

defining visual word-image relationships between the plurality of visual words and a plurality of images;

defining inter-image relationships between the plurality of images;

generating revised inter-visual word relationships in a vector space based on the initial inter-visual word relationships, the inter-image relationships, and the visual word-image relationships; and

generating higher-level visual words in the vector space based on the revised inter-visual word relationships.

2. The method of claim 1, wherein generating the revised inter-visual word relationships includes generating a weight matrix based on the initial inter-visual word relationships, the visual word-image relationships, and the inter-image relationships.

3. The method of claim 1, wherein the initial inter-visual words relationships are based on a visual similarity of the respective visual words and the inter-image relationships are based on a visual similarity of the respective images.

4. The method of claim 1, wherein the visual word-image relationships are based on co-occurrences of respective visual words and images.

5. The method of claim 1, wherein generating revised inter-visual word relationships in the vector space based on the initial inter-visual word relationships, the inter-image relationships, and the visual word-image relationships includes mapping the initial inter-visual word relationships, the inter-image relationships, and the visual word-image relationships to the vector space.

6. The method of claim 5, wherein the revised inter-visual word relationships in the vector space are based on respective Euclidian distances in the vector space.

7. The method of claim 5, wherein the vector space is obtained via a diffusion map of the relationships.

8. The method of claim 6, further comprising generating clusters of visual words in the vector space based on the revised inter-visual word relationships, wherein the higher-level visual words are generated based on the clusters of visual words.

9. The method of claim 1, further comprising:

defining semantic label-visual word relationships between a plurality of semantic labels and the plurality of visual words; and

defining inter-semantic label relationships between the semantic labels,

wherein the revised inter-visual word relationships in the vector space are generated further based on the semantic label-visual word relationships and the inter-semantic label relationships.

10. The method of claim 9, wherein the inter-semantic label relationships are defined according to a hierarchy.

11. A device for generating visual words, the device comprising:

one or more computer-readable media configured to store features; and

one or more processors that are couple to the memory and configured to cause the device to

generate initial inter-feature relationships for a plurality of features,

generate feature-image relationships between the plurality of features and a plurality of images, and

map the features to a vector space based on the initial inter-feature relationships and the feature-image relationships.

12. The device of claim 11, wherein the feature-image relationships are co-occurrence relationships.

13. The device of claim 12, wherein the one or more processors are further configured to cause the device to define the feature-image relationships in a co-occurrence matrix.

14. The device of claim 11, wherein the one or more processors are further configured to cause the device to

generate feature-label relationships, and

map the features to vector space further based on the feature-label relationships.

15. The device of claim 11, wherein the one or more processors are further configured to cause the device to map the feature to the vector space by performing a Markov Random walk.

16. One or more computer-readable media configured to store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations comprising:

defining inter-feature relationships between a plurality of features of a first type and a plurality of features of a second type, wherein the inter-features relationships are based on feature-object relationships between the plurality of features of the first type and intermediate objects and on feature-object relationships between the plurality of features of the second type and the intermediate objects;

defining intra-feature relationships between the features of the first plurality of features;

defining intra-feature relationships between the features of the second plurality of features; and

mapping the features of the first plurality of features and the features of the second plurality of features to a vector space based on the inter-feature relationships and the intra-feature relationships.

17. The one or more computer-readable media of claim 16, wherein respective relative distances between the features in the vector space is based on the inter-feature relationships and the intra-feature relationships.

18. The one or more computer-readable media of claim 16, wherein the intermediate objects include one or more of images, image patches, and labels.

19. The one or more computer-readable media of claim 16, wherein the operations further comprise defining inter-object relationships between the intermediate objects according to a hierarchy of the intermediate objects.

20. A method comprising:

defining initial inter-visual word relationships between a plurality of visual words;

defining semantic label-visual word relationships between a plurality of semantic labels and the plurality of visual words;

defining inter-semantic label relationships between the plurality of semantic labels;

generating revised inter-visual word relationships in a vector space based on the initial inter-visual word relationships, the semantic label-visual word relationships, and the inter-semantic label relationships; and

generating higher-level visual words in the vector space based on the revised inter-visual word relationships.

21. The method of claim 20, further comprising:

defining inter-image relationships between a plurality of images; and

defining visual word-image relationships between the plurality of visual words and the plurality of images,

wherein the revised inter-visual word relationships in the vector space are generated further based on the inter-image relationships and the visual word-image relationships.