System and methods for analysis of user-associated images to generate non-user generated labels and utilization of the generated labels

• US 10,460,174 B2
• Filed: 07/22/2015
• Issued: 10/29/2019
• Est. Priority Date: 07/22/2014
• Status: Active Grant

• Current Assignee: Hong Kong University of Science and Technology
• Original Assignee: Hong Kong University of Science and Technology
• Inventors: She, James Pei Man, Cheung, Pak Ming
• Primary Examiner(s): Chu, Randolph I
• Application Number:
  US15/327,822
• Publication Number:
  US 20170206420A1
• Time in Patent Office: 1,560 Days
• Field of Search: None
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/29 Graphical models, e.g. Bayesian networks
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data
• CPC Class Codes: G06Q 50/01 Social networking
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video

Abstract

A system for annotating user images with non-user generated labels corresponding to a plurality of users and generating user profiles includes: one or more processor-readable media, configured to store processor-executable instructions for image analysis and user profile generation; and one or more processors, configured to, based on execution of, the stored processor-executable instructions: annotate user images with non-user-generated labels, wherein the non-user-generated labels correspond to visual features of the user images and generate user profiles based on non-user-generated tags associated with images corresponding to each respective user.

Claims

1. A system for using images generated or shared by a plurality of entities to output associations between entities of the plurality of entities, comprising:

one or more processor-readable media, configured to store processor-executable instructions; and

one or more processors, configured, based on execution of the stored processor-executable instructions, to facilitate;

extracting visual features of the images generated or shared by the plurality of entities, wherein each image is generated or shared by a respective entity of the plurality of entities;

annotating the images generated or shared by the plurality of entities with non-user-generated labels based on the extracted visual features, wherein the non-user-generated labels are machine-generated based on a machine analysis of the extracted visual features;

generating a label-based representation for a first entity based on a plurality of non-user-generated labels associated with images generated or shared by the first entity;

generating a label-based representation for a second entity based on a plurality of non-user-generated labels associated with images generated or shared by the second entity;

comparing the label-based representation for the first entity and the label-based representation for the second entity to determine whether or not they are similar;

determining an association between the first entity and the second entity in response to determining that the label-based representation for the first entity and the label-based representation for the second entity are similar; and

outputting the determined association between the first entity and the second entity.

2. The system according to claim 1, wherein the images comprise at least one of the following:

images generated by users from user-related devices or smart devices via image taking, drawing, screen capturing, receiving, liking, sharing, and/or saving;

images generated by users from cameras of user-related devices or smart devices, wherein the user-related devices or smart devices include;

smartphones, wearable devices, and/or human control devices;

images taken by cameras and stored in one or more digital formats; and

/or

images captured from videos.

3. The system according to claim 1, wherein the one or more processors are further configured to facilitate:

converting the visual features of the images into visual words representing the extracted visual features;

encoding each image with at least one feature vector based on the visual words and the extracted visual features corresponding to the visual words associated with the image;

clustering images into multiple groups based on the feature vectors corresponding to the images; and

generating at least one non-user-generated label corresponding to each group of the multiple groups.

4. The system according to claim 3, wherein clustering the images into multiple groups is further based on the images having similar elements;

and

wherein each group of the multiple groups is represented by the at least one non-user-generated label corresponding to the group.

5. The system according to claim 3, wherein clustering the images into multiple groups is further based on pre-definitions of the multiple groups;

and

wherein each group of the multiple groups is represented by at least one unique non-user-generated label obtained based on the images.

6. The system according to claim 1, wherein the visual features comprise color, edges, textures, region, objects, faces, and/or other unique feature descriptors based on Scale-invariant feature transform (SIFT), GIST or other computer vision, image/signal processing and machine learning techniques.

7. The system according to claim 1, wherein extracting the visual features utilizes at least one of the following:

a Harris Affine detector, a Maximally Stable External Regions detector, a Kadir-Brady saliency detector, or a Scale-invariant feature transform (SIFT).

8. The system according to claim 7, wherein a unique visual word is generated from a set of features as a feature representation.

9. The system according to claim 8, wherein each visual word is represented by combining corresponding features of the images.

10. The system according to claim 3, wherein each feature vector corresponds to a number of occurrences of each visual word in each image.

11. The system according to claim 3, wherein clustering the images into multiple groups is further based on similarities between feature vectors of the images being clustered, such that images that are more similar are more likely to be in the same group.

12. The system according to claim 4, wherein the similarity calculation is based on cosine similarity.

13. The system according to claim 1, wherein the label-based representation for each respective entity is a distribution of non-user-generated labels for each respective entity indicative of a number of occurrences of unique non-user-generated labels associated with the images corresponding to the respective entity.

14. The system according to claim 1, wherein the label-based representation for each respective entity is determined based on a transform function corresponding to multiple labels.

15. The system according to claim 1, wherein outputting the determined association between the first entity and the second entity further comprises:

identifying or recommending a connection between the first and second entit.

16. The system according to claim 15, wherein the identification or recommendation of the connection is conducted without utilizing social graphs or user-annotated tags.

17. The system according to claim 15, wherein one or more social graphs or user-annotated tags are used to enhance the results of the identifying or recommending.

18. The system according to claim 1, wherein outputting the determined association between the first entity and the second entity further comprises:

identifying or recommending a user community for the first entity or the second entity.

19. The system according to claim 1, wherein outputting the determined association between the first entity and the second entity further comprises:

identifying a characteristic of the first entity or the second entity.

23. The system according to claim 2, wherein the one or more processors are further configured to facilitate:

clustering extracted visual features into groups of visual features;

wherein visual words are based on the groups of visual features.

24. The system according to claim 23, wherein each visual word is a mean vector corresponding to each group of visual features.

25. The system according to claim 19, wherein the identified characteristic of the respective entity is the respective entity'"'"'s gender.

26. The system according to claim 1, wherein the images reflect one or more interests of the respective entities;

and

wherein outputting the determined association between the first entity and the second entity is based on the first entity and the second entity having one or more similar interests.

20. One or more non-transitory processor-readable media having processor-executable instructions stored thereon for using images generated or shared by a plurality of entities to output associations between entities of the plurality of entities, wherein the processor-executable instructions are executable by one or more processors to facilitate the following steps:

extracting visual features of the images generated or shared by the plurality of entities, wherein each image is generated or shared by a respective entity of the plurality of entities;

annotating the images generated or shared by the plurality of entities with non-user-generated labels based on the extracted visual features, wherein the non-user-generated labels are machine-generated based on a machine analysis of the extracted visual features;

generating a label-based representation for a first entity based on a plurality of non-user-generated labels associated with images generated or shared by the first entity;

generating a label-based representation for a second entity based on a plurality of non-user-generated labels associated with images generated or shared by the second entity;

comparing the label-based representation for the first entity and the label-based representation for the second entity to determine whether or not they are similar;

determining an association between the first entity and the second entity in response to determining that the label-based representation for the first entity and the label-based representation for the second entity are similar; and

outputting the determined association between the first entity and the second entity.

21. The one or more non-transitory processor-readable media according to claim 20, wherein the processor-executable instructions are further executable by the one or more processors to facilitate the following steps:

converting the visual features of the images into visual words representing the extracted visual features;

encoding each image with at least one feature vector based on the visual words and the extracted visual features corresponding to the visual words associated with the image;

clustering images into multiple groups based on feature vectors corresponding to the images; and

generating at least one non-user-generated label corresponding to each group of the multiple groups.

22. A method for using images generated or shared by a plurality of entities to output associations between entities of the plurality of entities, comprising:

extracting, by one or more processors, visual features of the images generated or shared by the plurality of entities, wherein each image is generated or shared by a respective entity of the plurality of entities;

annotating, by the one or more processors, the images generated or shared by the plurality of entities with non-user-generated labels based on the extracted visual features, wherein the non-user-generated labels are machine-generated based on a machine analysis of the extracted visual features;

generating, by the one or more processors, a label-based representation for a first entity based on a plurality of non-user-generated labels associated with images generated or shared by the first entity;

generating, by the one or more processors, a label-based representation for a second entity based on a plurality of non-user-generated labels associated with images generated or shared by the second entity;

comparing, by the one or more processors, the label-based representation for the first entity and the label-based representation for the second entity to determine whether or not they are similar;

determining, by the one or more processors, an association between the first entity and the second entity in response to determining that the label-based representation for the first entity and the label-based representation for the second entity are similar; and

outputting, by the one or more processors, the determined association between the first entity and the second entity.