Building and using predictive models of current and future surprises

• US 7,519,564 B2
• Filed: 06/30/2005
• Issued: 04/14/2009
• Est. Priority Date: 11/16/2004
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Horvitz, Eric J.
• Primary Examiner(s): Vincent; David R
• Assistant Examiner(s): Fernandez Rivas; Omar F
• Application Number:
  US11/172,581
• Publication Number:
  US 20060106743A1
• Time in Patent Office: 1,384 Days
• Field of Search: 706/8, 706/11, 706/12, 706 14- 21, 706 45- 48, 706 50- 55, 709/224, 709/235, 709204-207, 709/223, 707/1, 707 3- 6, 701/1, 701/10, 701/26, 701/27, 701/29, 701200-210
• US Class Current: 706/12
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/2411 based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/025 Extracting rules from data
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G08G 1/0104 Measuring and analyzing of parameters relative to traffic conditions

Abstract

Methods are described for identifying events that would be considered surprising by people and identifying how and when to transmit information to a user about situations that they would likely find surprising. Additionally, the methods of identifying surprising situations can be used to build a case library of surprising events, joined with a set of observations before the surprising events occurred. Statistical machine learning methods can be applied with data from the case library to build models that can predict when a user will likely be surprised at future times. One or more models of context-sensitive expectations of people, a view of the current world, and methods for recording streams or events before surprises occur, and for building predictive models from a case library of surprises and such historical observations can be employed. The models of current and future surprises can be coupled with display and alerting machinery.

Claims

1. A system that predicts and outputs events identified as being surprising to a person, comprising a memory having stored therein computer executable components and a processor that executes the following computer executable components:

an interface component that receives contextual and historical data;

a predictive model component that utilizes the contextual and historical data to predict an event and outputs the prediction if the prediction corresponds to one or more definitions of surprise, wherein the prediction corresponds to one or more definitions of surprise based on a probability of occurrence of the event, and wherein the predictive model component comprises;

a robust predictive model that generates a prediction of the event based on interdependencies between variables associated with the contextual and historical data, wherein the interdependencies are not contemplated by the person; and

a user expectancy model that utilizes the contextual and historical data to generate a prediction of the event based on, at least in part, machine learning and a case library that includes a plurality of surprising events and observations associated with the plurality of surprising events;

a difference analyzer component that calculates a measure of difference between the prediction made by the robust predictive model and the prediction made by the user expectancy model to determine whether an event is surprising; and

an alerting component that alerts the person of the surprising event upon the determination that the event is surprising.

2. The system of claim 1, the predictive model component further comprising at least one of:

calculating a probability of correctness associated with the prediction and outputting the prediction if the probability of correctness is above a threshold;

or

annotating the prediction with an indication of confidence.

3. The system of claim 1, wherein the one or more definitions of surprise are a function of frequency of occurrence of events that can be predicted by the predictive model component.

4. The system of claim 1, wherein the contextual data comprises data received by at least one of sensors, data collection components, or input received from the person.

5. The system of claim 1, further comprising a utility analyzer that calculates a measure of utility of outputting the predicted event to the person;

wherein the measure of utility indicates a measure of relevance in outputting the predicted event to the person based on at least one of historical data associated with the person, contextual data associated with the person, or feedback provided by the person; and

wherein the predicted event is output if the measure of relevance in outputting the predicted event to the person is above a threshold.

6. The system of claim 1, wherein the alerting component utilizes alerting rules to determine a manner in which to alert the person of the surprising event, wherein the alerting rules consider at least one of a type of device carried by the person or time of day.

7. The system of claim 1, further comprising:

a feedback component that receives from the person input relating to an event; and

an updating component that updates the one or more definitions of surprise based upon the input relating to the event received from the person.

8. The system of claim 1, further comprising a suggestion component that outputs suggestions of actions the person can perform in response to the predicted event.

9. The system of claim 1, wherein the contextual data comprises at least one of status of a variable, duration of the status of the variable, a day of week, a time of day, a time and duration of an event, whether an event is within a defined range, whether a day under consideration is a holiday, a weather condition, a current traffic condition, a previous traffic condition, a stock market condition, a sales condition, or a delay condition.

10. The system of claim 1, wherein the predictive model component stores the one or more definitions of surprise in the case library.

11. The system of claim 1, wherein the alerting component alerts the person of a time in the future that the person can expect traffic volumes associated with a traffic route of interest to be surprisingly heavier or lighter.

12. The system of claim 1, wherein the alerting component alerts the person to begin a trip earlier or later within constraints of arriving at a location around a time of interest.

13. The system of claim 1, wherein the alerting component alerts the person of a predicted surprising rise or drop in value of an entity associated with the person'"'"'s financial interest.

14. A computer-implemented method for recognizing surprising events and predicting future surprising events, comprising:

providing a predictive model;

associating definitions of surprising events with the predictive model;

providing the predictive model with contextual and historical data;

predicting an event as a function of the contextual and historical data; and

determining that the predicted event corresponds to one or more of the definitions of surprising events by calculating a measure of difference between a prediction made by a robust predictive model and a prediction made by a user expectancy model;

wherein the robust predictive model predicts events that are surprising to a person based on interdependencies between variables associated with the contextual and historical data, and wherein the interdependencies are not contemplated by the person; and

wherein the user expectancy model predicts events that are surprising to the person based on, at least in part, machine learning and a case library that includes a plurality of surprising events and observations associated with the plurality of surprising events; and

outputting the predicted event to the person if the predicted event corresponds to one or more of the definitions of surprising events.

15. The method of claim 14, further comprising:

building the case library of the definitions of surprising events

building the predictive model as one or more statistical models that perform surprise forecasting from the case library; and

employing the predictive model in real-time by monitoring evidence and inferring and outputting to the person a probability that the person will be alerted of the predicted event in the future.

16. The method of claim 15, further comprising constructing the one or more statistical models by way of at least one of the following machine learning methods:

Bayesian networks, dynamic Bayesian networks, continuous time Bayesian networks, support vector machines, neural network models, Hidden Markov Models, Markov decision processes, or particle filtering methods.

17. The method of claim 14, further comprising:

calculating a measure of utility associated with the predicted event, wherein the measure of utility is calculated based on at least one of historical data associated with the person, contextual data associated with the person, or feedback provided by the person; and

outputting the predicted event if the measure of utility is above a threshold.

18. The method of claim 14, further comprising:

receiving a first prediction of a first event from a user-expectancy model, wherein the user-expectancy model generates the first prediction based on at least one marginal statistical model defined by a subset of observations that are contemplated by the person considering contextual and historical data;

receiving a second prediction of a second event from a robust predictive model, wherein the robust predictive model utilizes the contextual and historical data to generate the second prediction based on interdependencies between variables that are not contemplated by the person; and

outputting the second prediction of the second event as a surprising event based on, at least in part, a comparison between the first prediction and the second prediction.

19. The method of claim 14, wherein the contextual data comprises at least one of status of a variable, duration of the status of the variable, a day of week, a time of day, a time and duration of an event, whether an event is within a defined range, whether a day under consideration is a holiday, a weather condition, a current traffic condition, a previous traffic condition, a stock market condition, a sales condition, or a delay condition.

20. A predictive system comprising a memory having stored therein computer executable components and a processor that executes the following computer executable components:

means for receiving contextual and historical data;

means for predicting an event;

means for defining events surprising to a person; and

means for correlating the predicted event with the defined surprising events to determine that the predicted event is a surprising event, wherein the surprising event is determined by measuring a difference between a prediction made by a robust predictive model and a prediction made by a user expectancy model;

wherein the prediction made by the robust predictive model is based on, at least in part, interdependencies between variables associated with the contextual and historical data, and wherein the interdependencies are not contemplated by the person; and

wherein the prediction made by the user expectancy model is based on, at least in part, machine learning and a case library that includes a plurality of surprising events and observations associated with the plurality of surprising events; and

means for outputting the predicted event.

21. The predictive system of claim 20, further comprising:

means for storing the defined surprising events, combined with observations that occur prior to each defined surprising event and a label defining a type of the defined surprising event; and

means for constructing models that compute a probability of different types of surprises occurring in the future.