Image tagging based upon cross domain context

• US 8,645,287 B2
• Filed: 02/04/2010
• Issued: 02/04/2014
• Est. Priority Date: 02/04/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Baker, Simon John, Kapoor, Ashish, Hua, Gang, Lin, Dahua
• Primary Examiner(s): Wong, Lut
• Application Number:
  US12/699,889
• Publication Number:
  US 20110191271A1
• Time in Patent Office: 1,461 Days
• Field of Search: 706/11, 706/12, 706/52
• US Class Current: 706/11
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06N 5/048 Fuzzy inferencing
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: G06V 10/768 using context analysis, e.g. recognition aided by known co-occurring patterns
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations

Abstract

A method described herein includes receiving a digital image, wherein the digital image includes a first element that corresponds to a first domain and a second element that corresponds to a second domain. The method also includes automatically assigning a label to the first element in the digital image based at least in part upon a computed probability that the label corresponds to the first element, wherein the probability is computed through utilization of a first model that is configured to infer labels for elements in the first domain and a second model that is configured to infer labels for elements in the second domain. The first model receives data that identifies learned relationships between elements in the first domain and elements in the second domain, and the probability is computed by the first model based at least in part upon the learned relationships.

Claims

1. A method comprising the following computer-executable acts:

receiving a digital image, wherein the digital image comprises a first element that corresponds to a first domain and a second element that corresponds to a second domain, wherein the first domain is an event domain representative of events captured in a collection of images, and wherein the second domain is one of a people domain representative of people captured in the collection of images or a location domain representative of locations where the collection of images were captured;

assigning a label that identifies a name of an event corresponding to the digital image to the first element in the digital image based at least in part upon a computed probability that the label corresponds to the first element, wherein the probability is computed through utilization of a first model that is configured to probabilistically infer labels for elements in the first domain and a second model that is configured to probabilistically infer labels for elements in the second domain, wherein the first model receives data that identifies learned relationships between elements in the first domain and elements in the second domain, and wherein the probability is computed by the first model based at least in part upon the learned relationships.

2. The method of claim 1, further comprising extracting a feature from the digital image, wherein the probability is computed based at least in part upon the feature.

3. The method of claim 2, wherein the feature is characteristic of an element in the first domain, wherein the feature is utilized by the first model to compute the probability.

4. The method of claim 2, wherein the feature is characteristic of an element in the second domain, wherein the feature is utilized by the second model to compute a second probability that the second element is to be assigned a second label, and wherein the learned relationships between elements in the first domain and elements in the second domain are computed based at least in part upon the second probability.

5. The method of claim 2, wherein the feature is one of an article of clothing worn by an individual in the digital image, a color signature of the digital image, a timestamp assigned to the digital image, or location data assigned to the digital image.

6. The method of claim 1, wherein the learned relationships are in the form of a computer-readable relational table.

7. The method of claim 1, further comprising providing a graphical user interface that is configured to receive labels pertaining to elements in the first domain and the second domain from a human.

8. The method of claim 1, wherein the learned relationships are between one or more of people and events, locations and events, and people and locations.

9. The method of claim 1, further comprising:

receiving a plurality of digital images; and

automatically assigning tags to each image in the plurality of digital images.

10. The method of claim 1, further comprising:

assigning a second label to an element in the digital image with a particular probability;

comparing the particular probability with a threshold probability;

determining that the particular probability is below the threshold probability; and

providing a graphical user interface that facilitates receipt of human input, wherein the graphical user interface is configured to accept input from a human pertaining to the second label.

11. A system comprising:

a processor;

a memory that comprises a plurality of components that are executed by the processor, the plurality of components comprising;

an extractor component that receives a digital image and extracts at least one feature from the digital image; and

a label assignor component that automatically assigns a first label to first element in the digital image and a second label to a second element in the digital image, wherein the first element is in a first domain, wherein the second element is in a second domain, and wherein the label assignor component assigns the first label to the first element in the digital image based at least in part upon learned contextual relationships between elements in the first domain and elements in a second domain, wherein the elements in the first domain are modeled as a respective first plurality of random variables and the elements in the second domain are modeled as a respective second plurality of random variables.

12. The system of claim 11, wherein the first domain is an event domain, and wherein the second domain is a people domain.

13. The system of claim 11, wherein the label assignor component automatically assigns the first label to the first element in the digital image based at least in part upon learned relationships between elements in the first domain.

14. The system of claim 11, wherein the learned relationships comprise probabilities of elements in different domains co-occurring in the digital image.

15. The system of claim 11, wherein the feature extracted is a facial feature of an individual in the digital image.

16. The system of claim 11, the plurality of components further comprising an interface component that provides a graphical user interface that is configured to receive input from a human pertaining to labels assigned to elements in a plurality of digital images.

17. The system of claim 16, wherein the interface component is configured to receive user input to confirm that the label assigned to the element is correct.

18. The system of claim 11, wherein the feature extracted is one of an article of clothing worn by an individual in the digital image or a color signature of the digital image.

19. A computer-readable data storage device comprising instructions that, when executed by a processor, cause the processor to perform acts comprising:

receiving a first set of labels assigned to a collection of digital images by an individual;

extracting one or more features that are characteristic of a respective element from each of the digital images in the collection of digital images;

inferring a second set of labels for elements characterized by the one or more extracted Features;

based upon co-occurrences of labels in the first set of labels and the second set of labels, learning relationships between elements in a people domain, an event domain, and a location domain;

updating the second set of labels based upon the learning of the relationships between the elements in the people domain, the event domain, and the location domain; and

outputting a recommendation to the individual of a label in the second set of labels assigned to an element in a digital image in the collection of digital images based at least in part upon the inferring of the second set of labels and the updating of the second set of labels for the elements in the collection of digital images.

20. The computer-readable data storage device of claim 19, wherein inferring a label in the second set of labels comprises computing a probability that the label is correct, the acts further comprising repeating the acts of learning relationships between the elements in the first domain, the second domain, and the third domain and updating the second set of labels until either the probability for the label is above a predetermined threshold or until probabilities collectively computed for the second set of labels converge.