VISUAL RECOGNITION USING SOCIAL LINKS

• US 20150262037A1
• Filed: 03/17/2014
• Published: 09/17/2015
• Est. Priority Date: 03/17/2014
• Status: Active Grant

• Current Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Li, Jia, Kong, Xiangnan
• Granted Patent: US 9,710,447 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 18/29 Graphical models, e.g. Bayesian networks
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06V 10/84 using probabilistic graphical models from image or video features, e.g. Markov models or Bayesian networks
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video

Abstract

System, method and architecture for providing improved visual recognition by modeling visual content, semantic content and an implicit social network representing individuals depicted in a collection of content, such as visual images, photographs, etc. which network may be determined based on co-occurrences of individuals represented by the content, and/or other data linking the individuals. In accordance with one or more embodiments, using images as an example, a relationship structure may comprise an implicit structure, or network, determined from co-occurrences of individuals in the images. A kernel jointly modeling content, semantic and social network information may be built and used in automatic image annotation and/or determination of relationships between individuals, for example.

Claims

1. A method comprising:

training, by at least one computing device, a kernel jointly modeling content, semantic and social network information of a training set comprising a plurality of content items, each content item of the plurality having associated content, semantic and social network feature information used in training the kernel; and

identifying, by the at least one computing device, one or more annotations for at least one test content item other than the plurality of content items used to train the kernel, the identifying using the trained kernel and the content item'"'"'s content feature information to identify the one or more annotations.

2. The method of claim 1, the trained kernel modeling relationships among content, semantic and social network features.

3. The method of claim 1, the content item'"'"'s social network information identifying each individual represented in the content item.

4. The method of claim 1, the content item'"'"'s semantic information identifying one or more annotations associated with the content item.

5. The method of claim 1, further comprising:

training, by at least one computing device, a content kernel using at least the content feature information associated with each content item of the plurality of content items in the training set;

training, by at least one computing device, a semantic kernel using at least the semantic feature information associated with each content item of the plurality of content items in the training set;

training, by at least one computing device, a social kernel using at least the social network feature information associated with each content item of the plurality of content items in the training set; and

the training the kernel jointly modeling content, semantic and social network information further comprising training the kernel using the content, semantic and social networking kernels.

6. The method of claim 5, the kernel jointly modeling content, semantic and social network information comprising the content, semantic and social network kernels and a weighting for each of the content, semantic and social network kernels.

7. The method of claim 5, the social network kernel comprising a network of nodes and edges, each node corresponding to an individual represented in at least one content item of the plurality and each edge indicating a relationship between two individuals represented together in at least one content item of the plurality of content items.

8. The method of claim 7, the social network kernel further comprising a weight associated with each edge, the weight identifying a strength of the relationship between the two individuals.

9. The method of claim 8, the weight representing a number of content items in which the two individuals are represented together.

10. The method of claim 1, further comprising:

identifying, by at least one computing device, at least one relationship between at least two individuals using the kernel jointly modeling content, semantic and social network information,

11. A system comprising:

at least one computing device comprising one or more processors to execute and memory to store instructions to;

train a kernel jointly modeling content, semantic and social network information of a training set comprising a plurality of content items, each content item of the plurality having associated content, semantic and social network feature information used in training the kernel; and

identify one or more annotations for at least one test content item other than the plurality of content items used to train the kernel, the identifying using the trained kernel and the content item'"'"'s content feature information to identify the one or more annotations.

12. The system of claim 11, the trained kernel modeling relationships among content, semantic and social network features,

13. The system of claim 11, the content item'"'"'s social network information identifying each individual represented in the content item.

14. The system of claim 11, the content item'"'"'s semantic information identifying one or more annotations associated with the content item.

15. The system of claim 11, the instructions further comprising instructions to:

train a content kernel using at least the content feature information associated with each content item of the plurality of content items in the training set;

train a semantic kernel using at least the semantic feature information associated with each content item of the plurality of content items in the training set;

train a social kernel using at least the social network feature information associated with each content item of the plurality of content items in the training set; and

the instructions to train the kernel jointly modeling content, semantic and social network information further comprising instructions to train the kernel using the content, semantic and social networking kernels.

16. The system of claim 15, the kernel jointly modeling content, semantic and social network information comprising the content, semantic and social network kernels and a weighting for each of the content, semantic and social network kernels.

17. The system of claim 15, the social network, kernel comprising a network of nodes and edges, each node corresponding to an individual represented in at least one content item of the plurality and each edge indicating a relationship between two individuals represented together in at least one content item of the plurality of content items.

18. The system of claim 17, the social network kernel further comprising a weight associated with each edge, the weight identifying a strength of the relationship between the two individuals.

19. The system of claim 18, the weight representing a number of content items in which the two individuals are represented together.

20. The system of claim 11, the instructions further comprising instructions to:

identify at least one relationship between at least two individuals using the kernel jointly modeling content, semantic and social network information.

21. A computer readable non--transitory storage medium for tangibly storing thereon computer readable instructions that when executed cause at least one processor to:

train, a kernel jointly modeling content, semantic and social network information of a training set comprising a plurality of content items, each content item of the plurality having associated content, semantic and social network feature information used in training the kernel; and

identify one or more annotations for at least one test content item other than the plurality of content items used to train the kernel, the identifying using the trained kernel and the content item'"'"'s content feature information to identify the one or more annotations.

22. The computer readable non-transitory storage medium of claim 21, the trained kernel modeling relationships among content, semantic and social network features.

23. The computer readable non-transitory storage medium of claim 21, the content item'"'"'s social network information identifying each individual represented in the content item.

24. The computer readable non-transitory storage medium of claim 21, the content item'"'"'s semantic information identifying one or more annotations associated with the content item.

25. The computer readable non-transitory storage medium of claim 21, the instructions further comprising instructions to:

train a content kernel using at least the content feature information associated with each content item of the plurality of content items in the training set;

train a semantic kernel using at least the semantic feature information associated with each content item of the plurality of content items in the training set;

train a social kernel using at least the social network feature information associated with each content item of the plurality of content items in the training set; and

the instructions to train the kernel jointly modeling content, semantic and social network information further comprising instructions to train the kernel using the content, semantic and social networking kernels.

26. The computer readable non-transitory storage medium of claim 25, the kernel jointly modeling content, semantic and social network information comprising the content, semantic and social network kernels and a weighting for each of the content, semantic and social network kernels,

27. The computer readable non-transitory storage medium of claim 25, the social network kernel comprising a network of nodes and edges, each node corresponding to an individual represented in at least one content item of the plurality and each edge indicating a relationship between two individuals represented together in at least one content item of the plurality of content items.

28. The computer readable non-transitory storage medium of claim 27, the social network kernel further comprising a weight associated with each edge, the weight identifying a strength of the relationship between the two individuals.

29. The computer readable non-transitory storage medium of claim 28, the weight representing a number of content items in which the two individuals are represented together.

30. The computer readable non-transitory storage medium of claim 21, the instructions further comprising instructions to:

identify at least one relationship between at least two individuals using the kernel jointly modeling content, semantic and social network information.