Methods and architecture for cross-device activity monitoring, reasoning, and visualization for providing status and forecasts of a users' presence and availability

• US 7,233,933 B2
• Filed: 06/30/2003
• Issued: 06/19/2007
• Est. Priority Date: 06/28/2001
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Horvitz, Eric J., Koch, Paul, Apacible, Johnson T., Kadie, Carl M.
• Primary Examiner(s): Knight; Anthony
• Assistant Examiner(s): Holmes; Michael B.
• Application Number:
  US10/609,972
• Publication Number:
  US 20040003042A1
• Time in Patent Office: 1,450 Days
• Field of Search: 706/21, 706 1- 11, 705/10, 705/28, 705/7, 379 9- 20
• US Class Current: 706/21
• CPC Class Codes: G06Q 10/109 Time management, e.g. calendars, reminders, meetings or time accounting

Abstract

The present invention relates to a system and methodology to facilitate collaboration and communications between entities such as between automated applications, parties to a communication and/or combinations thereof. The systems and methods of the present invention include a service that supports collaboration and communication by learning predictive models that provide forecasts of one or more aspects of a users'"'"' presence and availability. Presence forecasts include a user'"'"'s current or future locations at different levels of location precision and usage of different devices or applications. Availability assessments include inferences about the cost of interrupting a user in different ways and a user'"'"'s current or future access to one or more communication channels. The predictive models are constructed from data collected by considering user activity and proximity from multiple devices, in addition to analysis of the content of users'"'"' calendars, the time of day, and day of week, for example. Various applications are provided that employ the presence and availability information supplied by the models in order to facilitate collaboration and communications between entities.

Claims

1. A computer implemented system to facilitate communications and collaboration between entities, comprising the following computer executable components:

a learning component that is trained and constructs models according to activity patterns of one or more users, the models are adjusted on an interruption workbench operable on a computer;

a video camera that is employed to record a subject'"'"'s activities and overall office context;

a component providing a first phase of model building relating to event and context capture; and

a forecasting component that employs the learning component and models to generate predictions relating to states of the one or more users.

2. The system of claim 1, the forecasting component receives one or more queries regarding an identified user'"'"'s states of presence or availability and generates one or more predictions relating to current or future states of the identified user.

3. The system of claim 2, the queries and one or more returned states are generated and received by one or more automated applications or authorized people.

4. The system of claim 3, the queries are originated by the applications or the authorized people in order to obtain answers regarding at least one of presence, availability, location, communications capability, device availability of an identified user or users.

5. The system of claim 1, the forecasting component determines complementary information relating to how long a person is expected to be absent, or how long a person may be unavailable.

6. The system of claim 2, the presence states include at least one of a time until a user will arrive at or leave a location, a time until the user will be at a location for at least time t, a time until the user will have access to a device, a time until the user will review an e-mail or other message, a time until the user will finish a conversation in progress, a likelihood the user will attend a meeting, an expected cost of interruption over time.

7. The system of claim 2, the learning component includes one or more learning models for reasoning about the presence states, the learning models can include at least one of a statistical model, a mathematical model, a Bayesian dependency model, a naï

ve Bayesian classifier, a Support Vector Machine (SVMs), a neural network and a Hidden Markov Model.

8. The system of claim 1, the learning component is trained from a user event data store that aggregates data from a plurality of different data sources associated with the one or more users.

9. The system of claim 8, the data sources include data acquisition components that record or log user event data, the data sources include at least one of a cell phone, an accelerometer, an acoustical activity recorded by microphone, a Global Positioning System (GPS), an electronic calendar, time information, a vision monitoring device, a wireless device, and a computer desktop activity.

10. The system of claim 1, the forecasting component is implemented as at least one of a server, a server farm, a client application, a web service, and an automated application that provides answers to automated systems or authorized people.

11. The system of claim 9, the data acquisition components include a signal-processing layer that enables users to configure and define parameters of audio and video sources utilized to define a user'"'"'s presence.

12. The system of claim 1, the forecasting component constructs a set of cases from an event database that is consistent with a query to enable custom-tailoring of a formulation and discretization of variables representing temporal relationships among landmarks as transitions between periods of absence and presence and appointment stan and end times, as defined by the query.

13. The system of claim 12, the event database logs periods of presence and absence as events in an event log.

14. The system of claim 13, the events are annotated by source devices that are defined by respective capabilities and locations.

15. The system of claim 14, further comprising tagging of events by specific devices, indexed by capabilities to enable a system to forecast a probability distribution over the time until the user will have access to different types of devices.

16. The system of claim 14, further comprising the source devices being assigned to fixed locations enabling a forecast of a user'"'"'s location.

17. The system of claim 1, further comprising an event system to monitor histories of a user'"'"'s interaction with computing systems, including applications that are running on a system, applications that are now in focus or that have just gone out of focus.

18. The system of claim 1, the forecasting component identifies when a user is checking email or reviewing a notification.

19. The system of claim 18, the forecasting component forecasts a time until a user will likely review a communication, given how much time has passed since the user last reviewed the communication.

20. The system of claim 1, the forecasting component determines a time until a user will engage an application or cease using the application.

21. The system of claim 1, the forecasting component predicts a time when a current conversation is likely to end.

22. A computer readable medium having computer readable instructions stored thereon for implementing the learning component and the forecasting component of claim 1.

23. The system of claim 1, at least one of the learning component and the forecasting component is associated with at least one of a communications system, a notification system, a messaging system, a priorities system, an automated meeting or interactive communications schedulers or re-schedulers, a smart caching system, an audio system, a calendaring system, a scheduling system, an automated maintenance system, an automated agent, a video system, a digital assistant, and a user tracking system in order to provide useful information to message senders or systems regarding the likelihood of establishing contact and making informed decisions based upon a user'"'"'s expected presence and availability.

25. The method of claim 23, further comprising providing forecasts according to the equation p(t_e|E, ξ

), where t_e is a time until an event of interest occurs, and evidence E includes proximal activity context, a time of day, a day of week, and multiple attributes representing the nature of active calendar items under consideration.

27. The method of claim 23, further comprising forecasting about how long it will be before the user who is present will leave their office, or, will be away for at least some time t, the proximal activity context is taken as the time since the user transitioned from absent to present.

24. A computer implemented method to provide presence forecasts, comprising the following computer implemented acts:

determining a proximal activity context that represents one or more salient recent transitions among landmark states, based on a query; and

constructing a model to predict future states that are dependent on timing of a more recent landmark state and an Expected Cost of Interruption (ECI) for a user at present or future times.

26. The method of claim 24, further comprising predicting a time until the user who has been absent will return to their office, or return to their office and remain for at least some time t, the proximal activity context is the period of time since the user transitioned from present to absent.

28. A method for building a forecasting model, comprising:

analyzing multiple distinctions associated with meetings and integrating observations into an analysis of presence and availability;

building models that relate multiple attributes of an appointment to the likelihood that the meetings will be attended, and

building models that determine an interruptability of the meetings.

29. The method of claim 28, further comprising inferring a location of meetings when locations are not clear from location strings listed with appointments.

30. The method of claim 28, further comprising building models of attendance by automatically accessing appointment properties from appointments from a commercially available server.

31. The method of claim 28, further comprising creating a draft training set of appointments and their properties, and marking an attendance field for respective appointments with guesses made via a set of heuristics about attendance.

32. The method of claim 31, further comprising employing attendance heuristics to consider an extension of desktop activity into a portion of a scheduled meeting as evidence that a meeting was not attended, and considering a lack of activity during a meeting as evidence that a meeting was attended.

33. The method of claim 31, further comprising providing forecasts that are taken with respect to attendance as a draft data set and making available tools that enable users to refine the draft data set with hand labeling of attendance.

34. The method of claim 28, further comprising generating a form that displays appointments in order of their occurrence, and displaying an attendance field containing forecasts about attendance.

35. The method of claim 34, further comprising adding assessments of a physical location of meetings, and adding how interruptible the meetings are by specifying whether meetings are low, medium, or high interruptability.

36. The method of claim 28, further comprising building models in view of a meeting date and time, a meeting duration, a subject a location, an organizer, a number and nature of the invitees, a role of the use, a response status of the user, whether the meeting is recurrent or not and whether the time is marked as busy or free on the user'"'"'s calendar.

37. The method of claim 28, further comprising exchanging information relating to an Expected Cost of Interruption (ECI) for a user at present or future times.

38. The method of claim 37, further comprising providing users with a facility to associate a dollar-value cost of interruption for an associated interruptability level.

39. The method of claim 37, further comprising computing an Expected Cost of Interruption (ECI) according to the following equation;

$\mathrm{ECI}=p<br><br>\left(A^m|E,\mathrm{\xi <br><br>}\right)<br><br>\underset{i}{\sum <br><br>}<br><br>p<br><br>\left(c_i^m|E,\mathrm{\xi <br><br>}\right)<br><br>c_i^m+\left(1-p<br><br>\left(A^m|E,\mathrm{\xi <br><br>}\right)\right)<br><br>c^d$

where A^m is an event of attending a meeting, c_t^m is a cost of interruption associated with interruptability value i, c^d is a default cost for the time period under consideration, and E represents observations about calendar attributes, a proximal context, day of week, and time of day.

40. A computer executable system to facilitate communications and collaboration between entities, comprising:

computer implemented means for learning activity patterns from one or more user events to build a model, the model is adjusted on an interruption workbench operable on a computer;

computer implemented means for employing a video camera to record a users activities and overall office context;

computer implemented means for providing a first phase of model building relating to event and context capture; and

computer implemented means for generating predictions relating to future user states based upon the model and a query from an automated application.

41. A graphical user interface for employment with a computer implemented communications and collaboration system, comprising the following computer implemented components:

one or more display objects associated with a graphical user interface that indicate predicted presences status of one or more users;

a form that displays appointments in order of their occurrence, and displaying an attendance field containing forecasts about attendance of the one or more users; and

at least one input to query the predicted presence status.

42. The graphical user interface of claim 41, the input further comprising at least one of a real time selection, an offline selection, a forecast selection, an assumptions selection, and a time selection.

43. The graphical user interface of claim 42, the forecast selection further comprising an online selection, an e-mail review selection, a telephone call selection, an office presence selection, an online at home selection, a video conference capable selection, a full screen available selection, a multiple monitors available selection, a type of phone available selection, a net meeting available selection, and a cost of interruption selection.

44. The graphical user interface of claim 41, further comprising display objects that depict graphical or numerical results of a predicted amount of time until a user is able to communicate via a given communications medium.

45. The graphical user interface of claim 44, further comprising displaying clues as to possible whereabouts of a user.

46. The graphical user interface of claim 41, further comprising generating a graph depicting a probability of a user'"'"'s presence.

47. The graphical user interface of claim 41, further comprising providing an interface to enable setting of one or more profiles for presence analysis.

48. The graphical user interface of claim 41, the interface enables setting an interruption cost associated with the one or more profiles.

49. The graphical user interface of claim 48, the interface enables setting different periods of time, days, and calendar periods for a respective profile.

50. A computer implemented system to facilitate communications and collaboration between entities, comprising the following computer executable components:

a tool that constructs models according to activity patterns of one or more users, the models are adjusted on an interruption workbench operable on a computer;

a video camera that is employed to record a subject'"'"'s activities and overall office context;

a component providing a first phase of model building relating to event and context capture; and

a forecasting component that employs the models to generate predictions relating to states of the one or more users.

51. The system of claim 50, the models employ a dynamic Bayesian model for reasoning about a user'"'"'s attentional focus.

52. The system of claim 50, the states include at least one of high-focus solo activity, medium focus solo activity, low focus solo activity, conversation in office, presentation, driving, private personal time, and sleeping.

53. The system of claim 50, the models infer a cost of different types of interruption given that a user is of a particular attentional state by processing a utility, u(D_i,A_j), capturing the cost of a user in attentional state A_j being disrupted by task or communication event D_t.

54. The system of claim 53, the cost is assessed as a willingness to pay to avoid a disruption in dollars.

55. The system of claim 53, the models compute an expected cost of interruption (ECI) by summing over the utilities, weighted by a likelihood of each state of attention, conditioned on a stream of incoming sensory information.

56. The system of claim 55, the ECI is computed as:

$\mathrm{ECI}=\underset{j}{\sum <br><br>}<br><br>p<br><br>\left(A_i|E\right)<br><br>u<br><br>\left(D_i,A_j\right)$

wherein p(A_j|E) is the probability of an attentional state, conditioned on an evidence stream E.

57. The system of claim 50, further comprising a control panel for an event system having event classes and graphical display for processing acoustical and visual information.

58. The system of claim 57, further comprising a component to capture a visual pose that is processed with a Bayesian head tracking system and a component to process ambient acoustical activity.

59. The system of claim 57, further comprising a component to monitor a user'"'"'s calendar that is automatically examined.

60. The system of claim 50, further comprising a client event system that provides an abstraction tool for coalescing patterns of low-level system events into higher-level events.

61. The system of claim 60, the low-level events are captured as an application being used, including whether a user is typing and clicking with a mouse, the higher-level events include pattern of switching among applications and indications of task completion.

62. The system of claim 59, monitoring the calendar further comprises whether a meeting is in progress, the length of time until the meeting is over, and the location of the meeting.

63. The system of claim 50, further comprising an event whiteboard to capture and share a state of low-level and higher-level events considered by a system.

64. The system of claim 63, the events include details about a birth and death of applications, an application currently “

on top” and

being interacted with, and events that capture usage patterns.

65. The system of claim 64, further comprising one or more alerting displays generated from a notification system employing models of expected cost of interruption.

66. The system of claim 65, the alerting displays are coupled with an audio herald.

67. The system of claim 50, further comprising learning of models that are employed to predict the state of interruptability of users in office settings, based on a logging of events from perceptual sensors and desktop activity.

68. The system of claim 50, the interruption workbench provides for event capture, annotation of segments of time, building models and testing statistical models of interruptability.

69. The system of claim 68, the interruption workbench includes an event logging system that synchronizes events monitored with a training session with scenes from a digital videotape, facilitating labeling of segments of time and associating them with events.

70. The system of claim 50, further comprising a component providing a second phase for model construction relating to tagging and assessment.

71. The system of claim 70, further comprising a component to allow subjects to tag periods of time viewed on a videotape as high, medium, and low interruptability.

72. The system of claim 70, further comprising a component to allow subjects to define a scale and build models that reason directly about a probability distribution over real-valued values, representing the costs of interruption.

73. The system of claim 56, further comprising computing the expected costs of interruption of different classes of interruption, for each disruption under consideration, an expected cost of interruptability is computed by substituting the likelihood of different states of interruptability, p(I_i|E), for explicit states of attention as follows:

$\mathrm{ECI}=\underset{j}{\sum <br><br>}<br><br>p<br><br>\left(I_i|E\right)<br><br>u<br><br>\left(D_i,I_j\right).$

74. The system of claim 50, the models operate on variables representing attentional forecasts about future states of interruptability.

75. The system of claim 74, future states of interruptability capture inferences about probability distributions over times until a low, medium, or high state of interruptability will be reached, and more specialized variables representing the times until states of interruptability will be achieved that will persist for different amounts of time.

76. The system of claim 75, the future states are employed for deliberating about if, when, and how to mediate communications.

77. A computer implemented system to facilitate communications and collaboration between entities, comprising the following computer executable components:

a data collection component that collects information about a user'"'"'s presence and absence at different locations and/or changes over time relating to a user'"'"'s access to different devices and/or communication channels;

a learning component that is trained and build models according to activity patterns of one or more users, the models are adjusted on an interruption workbench operable on a computer,

a video camera that is employed to record a subject'"'"'s activities and overall office context;

a component providing a first phase of model building relating to event and context capture; and

a forecasting component that employs the learning component and models to generate predictions relating to states of the one or more users for specific queries and/or situations about a user'"'"'s current or future availability and presence.

78. The system of claim 77, the information is collected in a relational database.

79. The system of claim 77, the information includes multiple dimensions of a user'"'"'s activities across multiple devices, and appointment status, as encoded in a calendar.

80. The system of claim 78, the information includes start and stop times of different interactions and appointment status that encoded as distinct dimensions in the database.

81. The system of claim 78, further comprising static and dynamically constructed predictive models that are determined by making queries for information across multiple dimensions of the database.

82. The system of claim 81, the queries are created dynamically, based on a current situation including at least one of a time of day, a day of a week, and a current transition status of a user for key transitions.

83. The system of claim 81, the models include determining a predictive goal associated with a desired forecast including a time until a communication channel will become available if not currently available.

84. The system of claim 81, the models include at least one of a time series model, an ARIMA model, a dynamic Bayesian network, and a Continuous Time Bayesian Network.

85. The system of claim 84, the Bayesian network is learned dynamically by acquiring a set of matching cases from a database, via appropriate querying of the database, and then employing a statistical analysis of the cases.

86. The system of claim 77, further comprising a component that determines specific types of sub-goals per a contacting user'"'"'s interest in current status and in forecasts of presences and availabilities.

87. The system of claim 86, the sub-goals relate to at least one of a location, an interruptability, and an availability of communication channels.

88. The system of claim 87, the communications channels can be reasoned about directly or can be derived as information by linking channels with location.

89. The system of claim 88, the information about channels linked to locations is stored in a location and device schema.

90. The system of claim 89, the schema includes information relating to when a device is added, a device type, a channel, and a location.

91. The system of claim 90, further comprising a user interface that interacts with the schema for adding and removing devices, locations, and channels.

92. The system of claim 90, the location is automatically determined from at least one of an 802.11 signal and a GPS signal.

93. The system of claim 77, further comprising a component to generate forecasts that are relayed to users that are granted privileges to view different aspects or levels of detail about another user'"'"'s current presence at a location, interruptability, a user'"'"'s access of one or more communication channels, or about forecasts of future presence and availability.

94. The system of claim 93, further comprising a component for selectively revealing different types of forecasts or current status.

95. The system of claim 94, wherein selectively revealing includes an abstraction of information to less precise information in a time domain.

96. The system of claim 94, wherein selectively revealing includes different people having different privileges including relationship information in an org chart, or by dynamic information.

97. The system of claim 96, the information is provided to a communication agent regarding possible actions to take.

98. The system of claim 77, further comprising at least one of:

a component to automatically reply to messages employing user forecasting information;

a component to automatically share user forecasting information with others; and

a component to automatically attempt to reschedule a communication based upon user forecasting information.