Predicting interesting things and concepts in content

• US 10,217,058 B2
• Filed: 01/30/2014
• Issued: 02/26/2019
• Est. Priority Date: 01/30/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Gamon, Michael, Pantel, Patrick, Mukherjee, Arjun
• Primary Examiner(s): Hill, Stanley K.
• Application Number:
  US14/168,936
• Publication Number:
  US 20150213361A1
• Time in Patent Office: 1,853 Days
• Field of Search: None
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

An “Engagement Predictor” provides various techniques for predicting whether things and concepts (i.e., “nuggets”) in content will be engaging or interesting to a user in arbitrary content being consumed by the user. More specifically, the Engagement Predictor provides a notion of interestingness, i.e., an interestingness score, of a nugget on a page that is grounded in observable behavior during content consumption. This interestingness score is determined by evaluating arbitrary documents using a learned transition model. Training of the transition model combines web browsing log data and latent semantic features in training data (i.e., source and destination documents) automatically derived by a Joint Topic Transition (JTT) Model. The interestingness scores are then used for highlighting one or more nuggets, inserting one or more hyperlinks relating to one or more nuggets, importing content relating to one or more nuggets, predicting user clicks, etc.

Claims

1. A computer-implemented process performed by a computing device, comprising:

receiving arbitrary content selected by a particular user and rendering that content to a display device;

automatically identifying and extracting a plurality of candidate items from the arbitrary content, the candidate items comprising any combination of one or more words, phrases, sentences, images, audio, hyperlinks, and topics in the arbitrary content;

predicting, for each of the plurality of the candidate items, a measure of interest with respect to the particular user, the measure of interest for each of the candidate items representing a statistical measure of a likelihood that the particular user will be interested in additional content related to the corresponding candidate item;

the predicted measure of interest provided by any combination of one or more of a machine-learned topic model and a machine-learned transition model,

the machine-learned topic model generated by applying machine-learning to automatically evaluate content of samples of source and destination documents using a generative model that posits a distribution over joint latent transition topics for each possible transition between the source and destination topics,

the machine-learned transition model generated by applying machine-learning to train on observed transitions between pairs of source content and destination content, and any combination of features extracted from those pairs and joint latent transition topics for each possible transition between the source and destination documents;

ranking the candidate items based on the predicted measure of interest for each candidate item;

selecting a predetermined number of the highest ranked candidate items;

associating additional content with the selected candidate items; and

modifying the arbitrary content by automatically arranging and displaying the associated additional content on the display device in conjunction with the arbitrary content, thereby enabling the user to interact with both the arbitrary content and the associated additional content.

2. The computer-implemented process of claim 1, wherein a transition model predicts the measure of interest for each of the candidate items, and wherein the transition model is automatically generated by process actions comprising:

receiving a collection of data points representing transitions between pairs of source content and destination content;

extracting a plurality of features from each source content;

extracting a plurality of features from each destination content; and

the transition model configured to statistically explain why the transitions between pairs of source content and destination content were made.

3. The computer-implemented process of claim 2 wherein a generative model of the source and destination content that posits a distribution over joint latent features for each possible transition of the source and destination content is used to automatically learn the features that are then extracted from the source and destination content.

4. The computer-implemented process of claim 3 wherein one or more of the features learned by the generative model are latent semantic topics derived from the source and destination content.

5. The computer-implemented process of claim 2 wherein one or more of the extracted features are manually specified prior to being extracted.

6. The computer-implemented process of claim 1 wherein the measures of interest are applied to learn per-user profiles for personalizing one or more applications to corresponding users.

7. The computer-implemented process of claim 2 wherein the pairs of source content and destination content are identified from browser log data of a plurality of users.

8. A system for augmenting user selected documents, comprising:

a general purpose computing device; and

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to;

automatically extract a plurality of candidate items from a user-selected document, the candidate items comprising any combination of one or more words, phrases, sentences, images, audio, hyperlinks, and topics in the user-selected document;

predict a separate user-specific interestingness score for each of the candidate items, the interestingness score for each of the candidate items representing a statistical measure of a likelihood that the user will be interested in additional content related to the corresponding candidate item;

the predicted user-specific interestingness score provided by any combination of one or more of a machine-learned topic model and a machine-learned transition model,

the machine-learned topic model generated by applying machine-learning to automatically evaluate content of samples of source and destination documents using a generative model that posits a distribution over joint latent transition topics for each possible transition between the source and destination topics,

the machine-learned transition model generated by applying machine-learning to train on observed transitions between pairs of source content and destination content, and any combination of features extracted from those pairs and joint latent transition topics for each possible transition between the source and destination documents;

sort the candidate items based on the corresponding interestingness scores;

select a predetermined number of the top sorted candidate items;

for each of the selected candidate items, associate the related additional content with the corresponding selected candidate items; and

modify the user-selected document by automatically arranging and displaying the associated related additional content on a display device in conjunction with the user-selected document, thereby enabling the user to interact with both the selected document and the corresponding associated related additional content.

9. The system of claim 8 further comprising program modules that direct the computing device to generate a transition model that predicts the interestingness score for each of the candidate items, and wherein the transition model is generated by:

receiving browser log data including transitions between pairs of source content and destination content;

automatically learn a plurality of features from a generative model of the source content and the destination content, the generative model representing each possible transition between the source content and the destination content;

extracting information corresponding to the learned features from the source content and the destination content; and

training on the extracted information and the learned features to automatically learn the transition model, the transition model statistically explaining why each possible transition between the source content and the destination content was made.

10. The computing device of claim 9 wherein one or more of the features learned by the generative model are latent semantic topics derived from the source and destination content.

11. The computing device of claim 9 wherein one or more of the features extracted from the source content and destination content are manually specified.

12. The computing device of claim 8 further directed by the program modules of the computer program to learn, based on the interestingness scores, user-specific profiles for personalizing one or more applications.

13. The computing device of claim 8 wherein the associated related additional content includes one or more advertisements.

14. A computer-readable storage device having computer executable instructions stored therein, said instructions causing a computing device to execute a method comprising:

rendering, on a display device, a user selected document;

automatically extracting a plurality of candidate items from a user selected document, the candidate items comprising any combination of one or more words, phrases, sentences, images, audio, hyperlinks, and topics in the user selected document;

predicting a measure of interest for each of a plurality of the candidate items with respect to the user, the measure of interest for each of the candidate items representing a statistical measure of a likelihood that the user will be interested in additional content related to the corresponding candidate item;

the predicted measure of interest provided by any combination of one or more of a machine-learned topic model and a machine-learned transition model,

the machine-learned topic model generated by applying machine-learning to automatically evaluate content of samples of source and destination documents using a generative model that posits a distribution over joint latent transition topics for each possible transition between the source and destination topics,

the transition model generated by applying machine-learning to train on observed transitions between pairs of source content and destination content, and any combination of features extracted from those pairs and joint latent transition topics for each possible transition between the source and destination documents;

ranking the candidate items based on the predicted measure of interest for each candidate item;

responsive to the predicted measures of interest, selecting one or more of the highest ranked candidate items;

associating additional content with the selected candidate items; and

modifying the user-selected document by automatically arranging and displaying the associated additional related content on the display device in conjunction with the user-selected document.

15. The computer-readable storage device of claim 14, the prediction of the measure of interest for each of the plurality items further comprising computer executable instructions comprising:

receiving transition data for a plurality of pairs of source content and destination content;

automatically learn a plurality of features from each of a plurality of possible transitions between the source content and the destination content;

extracting information corresponding to the learned features from the source content and the destination content; and

training the transition model on the extracted information and the corresponding learned features such that the transition model statistically explains why the transitions were made based on the extracted information and the corresponding learned features.

16. The computer-readable storage device of claim 15 wherein one or more of the learned features are latent semantic topics derived from the source and destination content.

17. The computer-readable storage device of claim 15 wherein one or more of the features extracted from the source content and destination content are manually specified.

18. The computer-readable storage device of claim 14 further comprising learning, from the predicted measures of interest, user-specific profiles for personalizing one or more applications.

19. The computer-readable storage device of claim 14 wherein one or more of the items of the additional related content includes one or more advertisements.

20. The computer-readable storage device of claim 15 wherein the pairs of source content and destination content are identified from browser log data.