Visual recognition using social links

• US 10,204,090 B2
• Filed: 07/17/2017
• Issued: 02/12/2019
• Est. Priority Date: 03/17/2014
• Status: Active Grant

• Current Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Original Assignee: Oath Inc. (Verizon Communications Inc.)
• Inventors: Li, Jia, Kong, Xiangnan
• Primary Examiner(s): Zuberi, Mohammed H
• Application Number:
  US15/651,822
• Publication Number:
  US 20180004719A1
• Time in Patent Office: 575 Days
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 18/29 Graphical models, e.g. Bayesian networks
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06V 10/84 using probabilistic graphical models from image or video features, e.g. Markov models or Bayesian networks
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video

Abstract

System, method and architecture for providing improved visual recognition by modeling visual content, semantic content and an implicit social network representing individuals depicted in a collection of content, such as visual images, photographs, etc., which network may be determined based on co-occurrences of individuals represented by the content, and/or other data linking the individuals. In accordance with one or more embodiments, using images as an example, a relationship structure may comprise an implicit structure, or network, determined from co-occurrences of individuals in the images. A kernel jointly modeling content, semantic and social network information may be built and used in automatic image annotation and/or determination of relationships between individuals, for example.

Claims

1. A method comprising:

training, by at least one computing device, a machine-modeled kernel jointly modeling content, semantic information and social network information, the training comprising building the kernel using a content kernel trained using a training set comprising content feature information, a semantic kernel trained using semantic feature information of the training data set and a social network kernel trained using social network feature information of the training data set, an implicit social network determined from a plurality of content items is used in determining the social network feature information used in training the kernel;

identifying, by the at least one computing device and using a number of content items other than the plurality of content items, a plurality of individuals depicted in the number of content items using the kernel;

identifying, by the at least one computing device, a number of relationships, each relationship being between two individuals, of the plurality of individuals, identified in a same content item, of the number of content items, using the kernel; and

representing, by the at least one computing device, each relationship, of the number of identified relationships, in an electronic social network comprising a plurality of nodes and a plurality of connections, each identified relationship being represented as a connection between a pair of individuals, of the plurality of individuals, identified in the same content item using the kernel and each individual of the pair being represented as a node of the plurality of nodes.

2. The method of claim 1, further comprising identifying at least one community comprising a number of individuals, of the plurality of individuals, identified as being connected using the kernel.

3. The method of claim 2, further comprising identifying a connection between at least two communities of individuals identified using the kernel, identification of the connection between the at least two communities being based on an occurrence of one or more individuals belonging to each community of the at least two communities.

4. The method of claim 2, at least one interest being assigned to each community of the at least one community.

5. The method of claim 1, further comprising determining, for the connection between the pair of individuals, a connection strength for the pair of individuals, determination of the connection strength being based on how many of the number of content items the pair of individuals is depicted in the same content item.

6. The method of claim 1, further comprising

serving, by the at least one computing device and to a client computing device of a user via an electronic communications network, content, the serving of the content to the user computing device resulting in the content being output by the user computing device.

7. The method of claim 1, the trained kernel modeling relationships among content, semantic and social network features.

8. The method of claim 1, the content item'"'"'s social network information identifying each individual represented in the content item.

9. The method of claim 1, the content item'"'"'s semantic information identifying one or more annotations associated with the content item.

10. The method of claim 1, further comprising:

training, by at least one computing device, the content kernel using at least the content feature information associated with each content item of the plurality of content items in the training set;

training, by at least one computing device, the semantic kernel using at least the semantic feature information associated with each content item of the plurality of content items in the training set; and

training, by at least one computing device, the social kernel using at least the social network feature information associated with each content item of the plurality of content items in the training set.

11. The method of claim 1, the kernel jointly modeling content, semantic and social network information comprising the content, semantic and social network kernels and a weighting for each of the content, semantic and social network kernels.

12. A non-transitory computer-readable storage medium tangibly encoded with computer-executable instructions that when executed by a processor associated with a computing device perform a method comprising:

training a machine-modeled kernel jointly modeling content, semantic information and social network information, the training comprising building the kernel using a content kernel trained using a training set comprising content feature information, a semantic kernel trained using semantic feature information of the training data set and a social network kernel trained using social network feature information of the training data set, an implicit social network determined from a plurality of content items is used in determining the social network feature information used in training the kernel;

identifying, using a number of content items other than the plurality of content items, a plurality of individuals depicted in the number of content items using the kernel;

identifying a number of relationships, each relationship being between two individuals, of the plurality of individuals, identified in a same content item, of the number of content items, using the kernel; and

representing each relationship, of the number of identified relationships, in an electronic social network comprising a plurality of nodes and a plurality of connections, each identified relationship being represented as a connection between a pair of individuals, of the plurality of individuals, identified in the same content item using the kernel and each individual of the pair being represented as a node of the plurality of nodes.

13. The non-transitory computer-readable storage medium of claim 12, further comprising identifying at least one community comprising a number of individuals, of the plurality of individuals, identified as being connected using the kernel.

14. The non-transitory computer-readable storage medium of claim 13, further comprising identifying a connection between at least two communities of individuals identified using the kernel, identification of the connection between the at least two communities being based on an occurrence of one or more individuals belonging to each community of the at least two communities.

15. The non-transitory computer-readable storage medium of claim 13, at least one interest being assigned to each community of the at least one community.

16. The non-transitory computer-readable storage medium of claim 12, further comprising determining, for the connection between the pair of individuals, a connection strength for the pair of individuals, determination of the connection strength being based on how many of the number of content items the pair of individuals is depicted in the same content item.

17. The non-transitory computer-readable storage medium of claim 12, further comprising

serving, to a client computing device of a user via an electronic communications network, content, the serving of the content to the user computing device resulting in the content being output by the user computing device.

18. The non-transitory computer-readable storage medium of claim 12, further comprising:

training the content kernel using at least the content feature information associated with each content item of the plurality of content items in the training set;

training the semantic kernel using at least the semantic feature information associated with each content item of the plurality of content items in the training set; and

training the social kernel using at least the social network feature information associated with each content item of the plurality of content items in the training set.

19. The non-transitory computer-readable storage medium of claim 12, the kernel jointly modeling content, semantic and social network information comprising the content, semantic and social network kernels and a weighting for each of the content, semantic and social network kernels.

20. A computing device comprising:

a processor;

a non-transitory storage medium for tangibly storing thereon program logic for execution by the processor, the program logic comprising;

training logic executed by the processor for training a machine-modeled kernel jointly modeling content, semantic information and social network information, the training comprising building the kernel using a content kernel trained using a training set comprising content feature information, a semantic kernel trained using semantic feature information of the training data set and a social network kernel trained using social network feature information of the training data set, an implicit social network determined from a plurality of content items is used in determining the social network feature information used in training the kernel;

identifying logic executed by the processor for identifying, using a number of content items other than the plurality of content items, a plurality of individuals depicted in the number of content items using the kernel;

identifying logic executed by the processor for identifying a number of relationships, each relationship being between two individuals, of the plurality of individuals, identified in a same content item, of the number of content items, using the kernel; and

representing logic executed by the processor for representing each relationship, of the number of identified relationships, in an electronic social network comprising a plurality of nodes and a plurality of connections, each identified relationship being represented as a connection between a pair of individuals, of the plurality of individuals, identified in the same content item using the kernel and each individual of the pair being represented as a node of the plurality of nodes.