System, method and apparatus for computer aided association of relevant images with text

• US 9,129,216 B1
• Filed: 07/15/2013
• Issued: 09/08/2015
• Est. Priority Date: 07/15/2013
• Status: Active Grant

• Current Assignee: Xdroid KFT
• Original Assignee: Xdroid KFT
• Inventors: Andras, Kocsor
• Primary Examiner(s): Holmes, Michael B
• Application Number:
  US13/941,634
• Time in Patent Office: 785 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: G06Q 10/101 Collaborative creation, e.g. joint development of products or services

Abstract

Methods, systems, and apparatus, including computer program products are disclosed for the association of relevant images with text and presentation of image recommendations to users. In an aspect, text, image and user/usage information is processed in order to extract features therefrom and create three-component feature vectors called triplets. One or more reference databases store the triplets and a modeling component builds a model to learn and recommend images based on the triplets stored. A reference database may be initially populated with information from publically available image/text for use by the modeling component. Using the model, an Illustration Index is calculated for each image in a collection for a given text. Images are ranked by their Illustration Index and provided as recommendations for use with the text. User interactions with images provides the system with personalized feedback, adding new associations/triplets to the reference database to further refine the model.

Claims

1. A computer implemented method, using a processor of a server, for recommending images, the method comprising:

receiving content comprising text,

processing the content to extract data comprising text features, wherein the text features form a text feature vector;

receiving a candidate set of image suggestions, the candidate set comprising images that may be submitted to a user taken from one or more collections of images, and for each image in the candidate set of image suggestions, processing the image to extract data comprising image features, wherein the image features form an image feature vector,

receiving user information and processing the information to extract data comprising user features, wherein the user features form a user feature vector;

storing the text feature vector, image feature vector, and user feature vector as a triplet in a reference database; and

applying means for machine learning using said triplets to learn an association function to calculate an Illustration Index, comprising a score that measures the level of association of the image suggestion to the content;

using the triplets stored in the reference database to generate one or more suggested images to be used with one or a plurality of subsequent text content input by a user in a user device; and

displaying the one or more suggested images on a user device.

2. The method as in claim 1, wherein the one or more collections of images are images stored on the user device.

3. The method as in claim 1, wherein one or more collections of images are images obtained from publically available image content.

4. The method as in claim 1, wherein the user feature vector is a null vector.

5. The method as in claim 1, wherein the reference database is initially populated with sets of triplets.

6. The method as in claim 5, wherein the sets of triplets comprise text feature vectors and image feature vectors extracted from available illustrated textual content.

7. The method as in claim 1, wherein the reference database further comprises means to grow.

8. The method as in claim 7 wherein the reference database grows to include additional triplets based on data comprising a user'"'"'s decision to select an image, the user'"'"'s selection forming a determination that the image is associated with the text.

9. The method as in claim 8, wherein the image selected by the user is selected from one or more collections of images, wherein at least one or more collections of images comprise images uploaded from the user device.

10. The method as in claim 7, wherein the reference database grows to include additional triplets based on data comprising a user'"'"'s decision to skip one or more image suggestions, the skip forming a determination that none of the suggested images were adequately associated with the text.

11. The method as in claim 1, wherein processing the text further comprises means for aggregating text feature vectors when the text comprises separate text elements, the separate text elements comprising a [title], an [abstract, lead], or a [body or body parts].

12. The method of claim 1, wherein the final set of image suggestions are ordered based on the Illustration Index.

13. The method as in claim 1, wherein the one or more collections of images comprise thematized collections of images.

14. A system comprising:

a computing device having memory storing instructions, and a processing device to execute the instructions, wherein the instructions cause the computing device to;

receive content comprising text,

process the content to extract data comprising text features, wherein the text features form a text feature vector;

receive a candidate set of image suggestions, the candidate set comprising images that may be submitted to a user taken from one or more collections of images, and for each image in the candidate set of image suggestions, processing the image to extract data comprising image features, wherein the image features form an image feature vector,

receive user information and processing the information to extract data comprising user features, wherein the user features form a user feature vector;

store the text feature vector, image feature vector, and user feature vector as a triplet in a reference database; and

apply means for machine learning using said triplets to learn an association function to calculate an Illustration Index, comprising a score that measures the level of association of the image suggestion to the content.

use the triplets stored in the reference database to generate one or more suggested images to be used with subsequent text content input by a user in a user device; and

a display device for displaying the one or more suggested images on a user device.

15. The system as in claim 14, wherein the system further comprises means for suggesting images for use by a user to illustrate text, wherein the text further comprises text used in social media or text message content.

16. A computer implemented method performed by a data processing apparatus, comprising

receiving, in response to input by a user in a user device, content comprising text;

receiving a candidate set of image suggestions, the candidate set comprising images that may be submitted to a user taken from one or more collections of images;

for each of the image suggestions in the candidate set of image suggestions determining whether the image suggestion satisfies an association criterion that defines the image suggestion as being associated with the content, the determining comprising;

determining an association function for calculating an Illustration Index, comprising a score that measures the level of association of the image suggestion to the content;

in response to determining that the image suggestion satisfies the association criterion, including the image suggestion in a final set of image suggestions; and

;

in response to determining that the image suggestion does not satisfy the association criterion, excluding the image suggestion from the final set of image suggestions; and

providing the final set of image suggestions for display on the user device.

17. The method of claim 16, wherein the final set of image suggestions are ordered based on the Illustration Index.

18. The method of claim 16, wherein the candidate set of images is generated based on machine learning methods making use of triplets stored in a reference database, the triplets comprising:

a text feature vector comprising data extracted from the content,

an image feature vector comprising data extracted from the image;

a user feature vector comprising data extracted from one or a combination of a plurality of the following;

the user identity;

image selections made by the user, the image selections forming a determination as to the association between an image and a given text;

images skipped by the user, the skip forming a determination that the image is not associated with a given text;

or

demographic information of the user.

19. The method as in claim 16, wherein the final set of image selections further comprises random images within a predetermined distance from one or more images which satisfy the association criterion.

20. A system comprising:

a. A feature extraction component coupled with a modeling component, wherein the modeling component is further coupled to an associator component, selector component, and a reference database component, wherein;

i. the feature extraction component further comprises;

means for extracting text features from text wherein the text features form a text feature vector;

means for extracting image features from images wherein the image features form an image feature vector;

means for extracting user features comprising data extracted from at least a user'"'"'s behavior; and

means for representing the text feature vector, image feature vector, and the user feature vector as a triplet comprising a three component vector;

ii. the modeling component further comprises means for machine learning using said triplets as learning samples, wherein the modeling component learns an association function and the association function outputs an Illustration Index;

iii. the associator component further comprises means for processing the association function to calculate the Illustration Index and rank images to be suggested to the user as associated with a given text;

iv. the selector component further comprises means for processing user behavior to create new triplets;

v. the reference database component further comprises means for storing said triplets in a reference database, the reference database configured to store a plurality of triplets and receive new triplets from the selector component from time to time;

b. wherein the feature extraction component, modeling component, associator component, selector component, and reference database component are in communication with one another and coupled with at least one processing device and memory;

c. wherein text input into a user device is;

i. processed by the feature extraction component along with images taken from one or more collections of images;

ii. the associator component calculates the Illustration Index for each image in a candidate set of images based on the output of an association function learned by the modeling component; and

d. a final set of image suggestions are displayed on the user device; and

e. wherein in response to a user'"'"'s decision to select or skip an image, the selector component updates the reference database component with a new triplet to be used by the modeling component for learning the association function.