Image-based semantic distance

• US 8,645,123 B2
• Filed: 12/19/2008
• Issued: 02/04/2014
• Est. Priority Date: 10/27/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Hua, Xian-Sheng, Wu, Lei, Li, Shipeng
• Primary Examiner(s): PULLIAS, JESSE SCOTT
• Application Number:
  US12/340,632
• Publication Number:
  US 20100106486A1
• Time in Patent Office: 1,873 Days
• Field of Search: None
• US Class Current: 704/9
• CPC Class Codes: G06F 18/2321 using statistics or function optimisation, e.g. modelling of probability density functions
• CPC Class Codes: G06V 10/763 Non-hierarchical techniques, e.g. based on statistics of modelling distributions
• CPC Class Codes: G06V 10/768 using context analysis, e.g. recognition aided by known co-occurring patterns
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations
• CPC Class Codes: G06V 30/268 Lexical context

Abstract

Image-based semantic distance technique embodiments are presented that involve establishing a measure of an image-based semantic distance between semantic concepts. Generally, this entails respectively computing a semantic concept representation for each concept based on a collection of images associated with the concept. A degree of difference is then computed between two semantic concept representations to produce the aforementioned semantic distance measure for the pair of corresponding concepts.

Claims

1. A computer-implemented process for establishing a measure of an image-based semantic distance between semantic concepts, comprising:

using a computer to perform the following process actions;

respectively computing a semantic concept representation for each concept based on images associated with the concept; and

computing a degree of difference between two semantic concept representations to produce said image-based semantic distance measure for the pair of corresponding semantic concepts.

2. The process of claim 1, wherein the semantic concept representations each take the form of a statistical distribution model of the images associated with the corresponding semantic concept.

3. The process of claim 2, wherein the statistical distribution models are visual language models, each of which captures the visual characteristics of the images associated with the corresponding semantic concept.

4. The process of claim 2, wherein the degree of difference between the semantic concept representations takes the form of a statistical distribution divergence.

5. The process of claim 4, wherein the statistical distribution divergence is the square root of a Jensen-Shannon divergence computation between the statistical distribution models.

6. The process of claim 1, wherein at least one of the semantic concepts comprises an object depicted in the images associated with the corresponding concept.

7. The process of claim 1, wherein at least one of the semantic concepts comprises a scene depicted in the images associated with the corresponding concept.

8. The process of claim 1, further comprising the process action of computing an image-based semantic distance measure between each of multiple pairs of semantic concepts, and using the measures to construct a visual concept network comprising a graph having a separate node representing each semantic concept and edges connecting the nodes representing the image-based semantic distance measure computed between each connected node.

9. The process of claim 1, further comprising the process action of computing an image-based semantic distance measure between each of multiple pairs of semantic concepts, and using the measures in a semantic concept clustering application.

10. The process of claim 1, further comprising the process action of computing an image-based semantic distance measure between each of multiple pairs of semantic concepts, and using the measures in an image annotation application.

11. The process of claim 1, wherein the images associated with a concept are obtained via a web-based image pool comprising images that are tagged with terms descriptive of the associated image, wherein said terms relate to said semantic concepts.

12. A system for establishing a measure of an image-based semantic distance between a pair of semantic concepts, comprising:

a general purpose computing device; and

a computer program having program modules executable by said computing device, wherein the computing device is directed by the program modules of the computer program to,

respectively compute a semantic concept representation for each semantic concept based on images associated with the concept, wherein each of said representations accounts for variations in the appearance of the associated semantic concept as depicted in the images used to compute the representation, and

compute a degree of difference between the pair of semantic concept representations to produce said image-based semantic distance measure, wherein computing the degree of difference comprises computing a combination of the degree of difference between each appearance variation associated with a first of the semantic concept representations and each appearance variation associated with the other of the semantic concept representations.

13. The system of claim 12, wherein said variations in appearance are attributable to at least one of differences in scale, orientation, object shape, illumination, color and texture.

14. The system of claim 12, wherein the program module for computing a semantic concept representation comprises a sub-module for computing each semantic concept representation using a latent topic visual language model which captures the visual features of said variations in appearance in the form of a conditional distribution for each variation considered.

15. The system of claim 14, wherein prior to computing the latent topic visual language model for a semantic concept representation, each image associated with the semantic concept of the representation is characterized as a collection of visual words.

16. The system of claim 15, wherein the latent topic visual language model associated with each semantic concept representation is based on parameters generated using at least one n-gram approach, wherein said n-gram approach models the image based on each visual word and n-1 of its neighboring visual words.

17. The system of claim 16, wherein generating the visual words of an image comprises:

dividing the image into uniformly sampled, equal sized patches and using a separated 8-dimensional texture histogram to described each patch, wherein each dimension of the texture histogram comprises a texture gradient along a different one of eight quantized directions; and

employing a hash coding approach to convert the texture histogram of each patch to produce a visual word representing the patch.

18. The system of claim 14, wherein the sub-module for computing each semantic concept representation using a latent topic visual language model comprises incorporating a probabilistic latent semantic analysis (pLSA) into the visual language model.

19. The system of claim 14, wherein the program module for computing the degree of difference between a pair of semantic concept representations, comprises a sub-module for computing the difference degree as the square root of a sum of weighted Jensen-Shannon divergences computed between each conditional distribution associated with the first of the semantic concept representations and each conditional distribution associated with the other of the semantic concept representations, wherein for each Jensen-Shannon divergence computed, the weighting is based on the importance of the variation in the images associated with the conditional distribution of the first semantic concept representation and the importance of the variation in the images associated with the conditional distribution of the other semantic concept representation.

20. A computer-implemented process for establishing a measure of an image-based semantic distance between a pair of semantic concepts, comprising using a computer to perform the following process actions:

for each semantic concept, computing a latent topic visual language model based on images associated with the semantic concept, wherein the latent topic visual language model accounts for variations in the appearance of the associated concept in the images used to compute the model, and wherein each variation in appearance it is desired to consider is represented as a separate conditional distribution of the visual features in the images associated with the variation; and

computing the square root of a sum of weighted Jensen-Shannon divergences computed between each conditional distribution associated with the latent topic visual language model computed for a first of the pair of semantic concepts and each conditional distribution associated with the latent topic visual language model computed for the other semantic concept, wherein for each Jensen-Shannon divergence computed, the weighting is based on the probability or frequency of the variation in the images associated with the conditional distribution of the first semantic concept and the probability or frequency of the variation in the images associated with the conditional distribution of the other semantic concept.