Correlative multi-label image annotation

• US 7,996,762 B2
• Filed: 02/13/2008
• Issued: 08/09/2011
• Est. Priority Date: 09/21/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Qi, Guo-Jun, Rui, Yong, Hua, Xian-Sheng, Li, Shipeng, Zhang, Hong-Jiang
• Primary Examiner(s): RIES, LAURIE ANNE
• Application Number:
  US12/030,616
• Publication Number:
  US 20090083010A1
• Time in Patent Office: 1,273 Days
• Field of Search: 715/230, 715/231, 715/233, 715/201, 715/255
• US Class Current: 715/233
• CPC Class Codes: G06F 16/7847 using low-level visual features of the video content
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44

Abstract

Correlative multi-label image annotation may entail annotating an image by indicating respective labels for respective concepts. In an example embodiment, a classifier is to annotate an image by implementing a labeling function that maps an input feature space and a label space to a combination feature vector. The combination feature vector models both features of individual ones of the concepts and correlations among the concepts.

Claims

1. A method for correlative multi-label image annotation, the method comprising:

creating a concept feature modeling portion responsive to low-level features of an image to model connections between the low-level features of the image and individual concepts that are to be annotated;

creating a concept correlation modeling portion to model correlations among at least a subset of the concepts that are to be annotated;

forming a combination feature vector responsive to the concept feature modeling portion and the concept correlation modeling portion;

solving a labeling function responsive to the combination feature vector to produce a concept label vector for the image, the concept label vector including label indicators respectively associated with the concepts that are to be annotated; and

learning a classifier for the labeling function using a kernelized version of the combination feature vector that includes a dot product over at least a vector for low-level features of images to be classified.

2. The method as recited in claim 1, wherein creating the concept correlation modeling portion comprises:

converting the correlations among at least a subset of the concepts that are to be annotated into features that may be incorporated into the combination feature vector.

3. The method as recited in claim 1, wherein the act of creating a concept correlation modeling portion comprises:

using the subset of the concepts that are to be annotated to create the concept correlation modeling portion with the subset to include those concept sets having a correlation value that exceeds a predetermined threshold, wherein each respective correlation value is measured responsive to normalized mutual information among concepts of the respective concept set.

4. The method as recited in claim 1, further comprising:

determining respective ranking scores for respective ones of the concepts that are to be annotated responsive to a conditional expectation of each label value to estimate a confidence level of each label indicator.

5. A system for correlative multi-label image annotation, the system comprising:

a processor; and

computer readable media that includes one or more software components that are executable by the processor, the one or more software components including;

a multi-label classifier that is executable by the processor to produce a concept label vector for an image responsive to low-level features of the image using a labeling function, the concept label vector to include multiple label indicators for multiple concepts that are to be annotated, the labeling function to include a combination feature vector having a concept characteristics portion and a concept correlations portion;

the concept characteristics portion to model connections between the low-level features of the image and detection data for individual concepts, and the concept correlations portion to model correlations among at least a subset of the multiple concepts that are to be annotated, the concept characteristics portion comprising a third-order tensor having three indices representing labels, features, and concepts, and the concept correlations portion comprising a fourth-order tensor having four indices representing positive label indicator values, negative label indicator values, a first concept of each concept pair, and a second concept of each concept pair.

6. The system as recited in claim 5, wherein the subset of the multiple concepts that are to be annotated is to include those concept sets having a correlation value that exceeds a predetermined threshold, wherein each respective correlation value is measured responsive to normalized mutual information among concepts of the respective concept set;

and wherein the multi-label classifier is to exclude from the concept correlations portion those concept sets having a correlation value that fails to exceed the predetermined threshold.

7. The system as recited in claim 5, further comprising:

one or more concept scorers that are executable by the processor to produce respective ranking scores for respective ones of the multiple concepts that are to be annotated, the one or more concept scorers including at least one conditional expectation determiner to determine a conditional expectation of each label value to estimate a confidence level of each label indicator.

8. A system for correlative multi-label image annotation, the system comprising:

a processor; and

computer readable media that includes one or more software components that are executable by the processor, the one or more software components including;

a classifier that is executable by the processor to annotate an image by indicating respective labels for respective concepts to implement a labeling function that maps an input feature space and a label space to a combination feature vector, the combination feature vector to model (i) features of individual ones of the concepts and (ii) correlations among the concepts, the classifier to use a kernelized version of the labeling function such that the combination feature vector is not explicitly calculated, and to evaluate the labeling function to produce a concept label vector having label indicators.

9. The system as recited in claim 8, wherein the combination feature vector is to model positive correlations and negative correlations among the concepts.

10. The system as recited in claim 8, wherein the combination feature vector is to model the correlations among the concepts using pairs of concepts.

11. The system as recited in claim 8, wherein the combination feature vector is to model high-order correlations among the concepts, with the high-order correlations corresponding to correlations among three or more concepts.

12. The system as recited in claim 8, wherein the combination feature vector is to model the features of individual ones of the concepts by modeling connections between low-level features of the image and the concepts.

13. The system as recited in claim 8, wherein the classifier is to form the combination feature vector using at least one indicator function that takes on a value of one (1) if an indicated prediction is true and a value of zero (0) if the indicated prediction is not true.

14. The system as recited in claim 8, wherein the classifier evaluates the labeling function to produce the concept label vector having the label indicators using a Gibbs Random Field (GRF) representation in which edges of the GRF representation correspond to the correlations among the concepts.

15. The system as recited in claim 8, wherein the classifier comprises a thresholding unit to ensure that the combination feature vector models those correlations among the concepts whose correlation values exceed a predetermined threshold.

16. The system as recited in claim 15, wherein each of the correlation values is measured responsive to normalized mutual information among at least two concepts.

17. The system as recited in claim 8, further comprising:

a concept scorer that is executable by the processor to provide a ranking score for a respective labeled concept based on a respective conditional expectation of a labeled value of the respective labeled concept.