System, method and apparatus for computer aided association of relevant images with text
First Claim
1. A computer implemented method, using a processor of a server, for recommending images, the method comprising:
- receiving content comprising text,processing the content to extract data comprising text features, wherein the text features form a text feature vector;
receiving a candidate set of image suggestions, the candidate set comprising images that may be submitted to a user taken from one or more collections of images, and for each image in the candidate set of image suggestions, processing the image to extract data comprising image features, wherein the image features form an image feature vector,receiving user information and processing the information to extract data comprising user features, wherein the user features form a user feature vector;
storing the text feature vector, image feature vector, and user feature vector as a triplet in a reference database; and
applying means for machine learning using said triplets to learn an association function to calculate an Illustration Index, comprising a score that measures the level of association of the image suggestion to the content;
using the triplets stored in the reference database to generate one or more suggested images to be used with one or a plurality of subsequent text content input by a user in a user device; and
displaying the one or more suggested images on a user device.
0 Assignments
0 Petitions
Accused Products
Abstract
Methods, systems, and apparatus, including computer program products are disclosed for the association of relevant images with text and presentation of image recommendations to users. In an aspect, text, image and user/usage information is processed in order to extract features therefrom and create three-component feature vectors called triplets. One or more reference databases store the triplets and a modeling component builds a model to learn and recommend images based on the triplets stored. A reference database may be initially populated with information from publically available image/text for use by the modeling component. Using the model, an Illustration Index is calculated for each image in a collection for a given text. Images are ranked by their Illustration Index and provided as recommendations for use with the text. User interactions with images provides the system with personalized feedback, adding new associations/triplets to the reference database to further refine the model.
-
Citations
20 Claims
-
1. A computer implemented method, using a processor of a server, for recommending images, the method comprising:
-
receiving content comprising text, processing the content to extract data comprising text features, wherein the text features form a text feature vector; receiving a candidate set of image suggestions, the candidate set comprising images that may be submitted to a user taken from one or more collections of images, and for each image in the candidate set of image suggestions, processing the image to extract data comprising image features, wherein the image features form an image feature vector, receiving user information and processing the information to extract data comprising user features, wherein the user features form a user feature vector; storing the text feature vector, image feature vector, and user feature vector as a triplet in a reference database; and applying means for machine learning using said triplets to learn an association function to calculate an Illustration Index, comprising a score that measures the level of association of the image suggestion to the content; using the triplets stored in the reference database to generate one or more suggested images to be used with one or a plurality of subsequent text content input by a user in a user device; and displaying the one or more suggested images on a user device. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13)
-
-
14. A system comprising:
-
a computing device having memory storing instructions, and a processing device to execute the instructions, wherein the instructions cause the computing device to; receive content comprising text, process the content to extract data comprising text features, wherein the text features form a text feature vector; receive a candidate set of image suggestions, the candidate set comprising images that may be submitted to a user taken from one or more collections of images, and for each image in the candidate set of image suggestions, processing the image to extract data comprising image features, wherein the image features form an image feature vector, receive user information and processing the information to extract data comprising user features, wherein the user features form a user feature vector; store the text feature vector, image feature vector, and user feature vector as a triplet in a reference database; and apply means for machine learning using said triplets to learn an association function to calculate an Illustration Index, comprising a score that measures the level of association of the image suggestion to the content. use the triplets stored in the reference database to generate one or more suggested images to be used with subsequent text content input by a user in a user device; and a display device for displaying the one or more suggested images on a user device. - View Dependent Claims (15)
-
-
16. A computer implemented method performed by a data processing apparatus, comprising
receiving, in response to input by a user in a user device, content comprising text; -
receiving a candidate set of image suggestions, the candidate set comprising images that may be submitted to a user taken from one or more collections of images; for each of the image suggestions in the candidate set of image suggestions determining whether the image suggestion satisfies an association criterion that defines the image suggestion as being associated with the content, the determining comprising; determining an association function for calculating an Illustration Index, comprising a score that measures the level of association of the image suggestion to the content; in response to determining that the image suggestion satisfies the association criterion, including the image suggestion in a final set of image suggestions; and
;in response to determining that the image suggestion does not satisfy the association criterion, excluding the image suggestion from the final set of image suggestions; and providing the final set of image suggestions for display on the user device. - View Dependent Claims (17, 18, 19)
-
-
20. A system comprising:
-
a. A feature extraction component coupled with a modeling component, wherein the modeling component is further coupled to an associator component, selector component, and a reference database component, wherein; i. the feature extraction component further comprises; means for extracting text features from text wherein the text features form a text feature vector; means for extracting image features from images wherein the image features form an image feature vector; means for extracting user features comprising data extracted from at least a user'"'"'s behavior; and means for representing the text feature vector, image feature vector, and the user feature vector as a triplet comprising a three component vector; ii. the modeling component further comprises means for machine learning using said triplets as learning samples, wherein the modeling component learns an association function and the association function outputs an Illustration Index; iii. the associator component further comprises means for processing the association function to calculate the Illustration Index and rank images to be suggested to the user as associated with a given text; iv. the selector component further comprises means for processing user behavior to create new triplets; v. the reference database component further comprises means for storing said triplets in a reference database, the reference database configured to store a plurality of triplets and receive new triplets from the selector component from time to time; b. wherein the feature extraction component, modeling component, associator component, selector component, and reference database component are in communication with one another and coupled with at least one processing device and memory; c. wherein text input into a user device is; i. processed by the feature extraction component along with images taken from one or more collections of images; ii. the associator component calculates the Illustration Index for each image in a candidate set of images based on the output of an association function learned by the modeling component; and d. a final set of image suggestions are displayed on the user device; and e. wherein in response to a user'"'"'s decision to select or skip an image, the selector component updates the reference database component with a new triplet to be used by the modeling component for learning the association function.
-
Specification