Identifying visual contextual synonyms

• US 9,082,040 B2
• Filed: 05/13/2011
• Issued: 07/14/2015
• Est. Priority Date: 05/13/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Cai, Rui, Li, Zhiwei, Zhang, Lei, Tang, Wenbin
• Primary Examiner(s): Perveen, Rehana
• Assistant Examiner(s): Hoffler, Raheem
• Application Number:
  US13/107,717
• Publication Number:
  US 20120290577A1
• Time in Patent Office: 1,523 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/464 using a plurality of salient features, e.g. bag-of-words [BoW] representations

Abstract

Tools and techniques for identifying visual contextual synonyms are described herein. The described operations use visual words having similar contextual distributions as contextual synonyms to identify and describe visual objects that share semantic meaning. The contextual distribution of a visual word is described using the statistics of co-occurrence and spatial information averaged over image patches that share the visual word. In various implementations, the techniques are employed to construct a visual contextual synonym dictionary for a large visual vocabulary. In various implementations, the visual contextual synonym dictionary narrows the semantic gap for large-scale visual search.

Claims

1. A method comprising:

receiving an image;

defining a region of visual context in the image;

partitioning the region of visual context into sectors;

producing a sub-histogram for each sector, wherein the sub-histograms represent which of the sectors contain which of one or more local interest points in the region of visual context;

concatenating the sub-histograms produced for each sector to generate a histogram describing a contextual distribution of the region of visual context; and

storing the histogram describing the contextual distribution of the region of visual context.

2. A method as recited in claim 1, further comprising:

extracting a visual word from the image;

expanding the visual word to a plurality of visual contextual synonyms; and

producing a panel of patches representing a weighted combination of the plurality of visual contextual synonyms.

3. A method as recited in claim 2, further comprising providing the panel of patches to a search application.

4. A method as recited in claim 1, further comprising calculating a circle distance between the sectors in a defined region of interest of the image.

5. A method as recited in claim 1, further comprising aggregating hyper-histograms across multiple instances of a defined region of interest of the image to represent the contextual distribution of a visual word extracted with the image.

6. A method as recited in claim 1, further comprising averaging hyper-histograms across multiple instances of a defined region of interest of the image to represent the contextual distribution of a visual word extracted from the image.

7. A method as recited in claim 1, further comprising:

extracting a visual word from the image based on a defined region of interest of the image; and

expanding the visual word extracted to visual contextual synonyms, wherein the visual contextual synonyms include visual words identified from a visual vocabulary based on having contextual distributions similar to a contextual distribution of the visual word.

8. A method as recited in claim 7, further comprising assigning a weight to the visual word.

9. A method as recited in claim 8, further comprising describing the image with a weighted combination of the visual contextual synonyms.

10. A method as recited in claim 9, the method further comprising:

identifying the contextual distribution of the visual word, the contextual distribution containing an additional visual word; and

creating an inverted list entry from which to build an inverted list, the creating including recording in the inverted list entry information for the additional visual word including an identity of the additional visual word and a weight of the visual word in a contextual distribution of the additional visual word.

11. A method as recited in claim 10, further comprising identifying a visual contextual synonym of the visual word from the inverted list.

12. A computer-readable storage medium having computer-executable instructions recorded thereon, the computer-executable instructions to configure a computer to perform operations comprising:

obtaining an image;

defining a region of visual context in the image;

identifying one or more local interest points in the region of visual context;

partitioning the region of visual context into sectors;

producing a sub-histogram for each sector, wherein the sub-histograms represent which of the sectors contain which of the one or more local interest points;

concatenating the sub-histograms produced for each sector to generate a histogram describing a contextual distribution of the region of visual context; and

storing the histogram describing the contextual distribution of the region of visual context.

13. A computer-readable storage medium as recited in claim 12, the operations further comprising performing a visual contextual search using the region of visual context.

14. A computer-readable storage medium as recited in claim 12, the operations further comprising calculating a circle distance between the sectors in the region of visual context.

15. A computer-readable storage medium as recited in claim 12, the operations further comprising aggregating the histogram describing the contextual distribution of the region of visual context with other histograms representing the region of visual context.

16. A computer-readable storage medium as recited in claim 12, the operations further comprising rotating the region of visual context to a dominant orientation associated with another region of visual context of another image.

17. A computer-readable storage medium as recited in claim 12, the operations further comprising producing a panel of images including a weighted combination of visual contextual synonyms for the region of visual context.

18. A system comprising:

a processor;

a memory coupled to the processor, the memory storing components for identifying visual contextual synonyms, the components configured to perform operations comprising;

obtaining an image;

defining a region of visual context in the image;

identifying one or more local interest points in the region of visual context;

partitioning the region of visual context into sectors;

producing a sub-histogram for each sector, wherein the sub-histograms represent which of the sectors contain which of the one or more local interest points; and

concatenating the sub-histograms produced for each sector to generate a histogram describing a contextual distribution of the region of visual context.

19. A system as recited in claim 18, wherein the operations further comprise storing the histogram describing the contextual distribution of the region of visual context.

20. A system as recited in claim 18, wherein the operations further comprise producing a panel of images including a weighted combination of visual contextual synonyms for the region of visual context.