Systems and methods for constructing and using models of memorability in computing and communications applications

• US 20040153445A1
• Filed: 02/25/2003
• Published: 08/05/2004
• Est. Priority Date: 02/04/2003
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Dumais, Susan T., Horvitz, Eric J., Cutrell, Edward B., Koch, Paul B., Ringel, Meredith J.
• Application Number:
  US10/374,436
• Publication Number:
  US 20040153445A1
• Time in Patent Office: Days
• US Class Current: 707/3
• CPC Class Codes: G06F 16/148 File search processing
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G16H 10/20 for electronic clinical trials or questionnaires

Abstract

One or more models of memorability are provided that facilitate various computer-based applications including those centering on the storage, retrieval, and processing of information, applications that remind people about items they risk not recalling or overlooking, and facilitating communications of reminders. In one application, the models are used to help compose and navigate large personal stores of information about a user'"'"'s activities, communications, images, and other content. In another application, views of files in directories are extended with the addition of memory landmarks, and a means for controlling the number of landmarks provided via changing a threshold on inferred memorability. Another application centers on the use of models of memorability to select subsets of images from larger sets representing events, for display in a slide show or ambient photo display. In another application, a system is provided that facilitates computer-based searching for information by providing for the design and analysis of timeline visualizations in connection with displaying results to queries based at least in part on an index of content. A query is received by a query component (which can be part of search engine that provides a unified index of information a user has been exposed to). The query component parses the query into portions relevant to effecting a meaningful search in accordance with the subject invention. The query component can access and populate a data store which may include information searched for. A landmark component receives and/or accesses information from the query component as well as the data store, and anchors public and/or personal landmark events to search results-related information.

Claims

1. A system that facilitates computer-based searching, comprising:

a query component that receives information related to a search for information; and

a landmark component that employs content-based landmark information to facilitate the search for information, the landmark information corresponding to contextual information related to event(s) memorable to an originator of the search.

2. The system of claim 1 providing timeline visualizations in connection with displaying results to the search based at least in part on an index of personal content.

3. The system of claim 1 further comprising a search engine that provides a unified index of information to which a user has been exposed.

4. The system of claim 3, the information comprising at least one of:

web pages, email, documents, pictures, and audio.

5. The system of claim 2, results of searches are presented with an overview-plus-detail timeline visualization.

6. The system of claim 5, further providing a summary view that shows distribution of search hits over time.

7. The system of claim 5, further providing a detailed view that allows for inspection of individual search results.

8. The system of claim 7, annotating returned items with icons and/or short descriptions.

9. The system of claim 1, the landmark component extending a basic time view by adding public landmarks and/or personal landmarks.

10. The system of claim 1, employing contextual information to support searching through content.

11. The system of claim 1, anchoring timeline-based presentations of search with public and/or personal landmark events.

12. The system of claim 1, further comprising an indexing component that can index text and/or metadata of items that a user has been exposed to so as to facilitate a fast and easy manner to search over content.

13. A computer readable medium having stored thereon the components of claim 1.

14. A method that facilitates computer-based searching, comprising:

receiving information related to a search for information;

employing content-based landmark information to facilitate the search for information, the landmark information corresponding to contextual information related to event(s) memorable to an originator of the search; and

providing a timeline visualization of search results based at least in part upon an index of a subset of the contextual information.

15. The method of claim 14 further comprising employing one or memorability models to determine the landmark information.

16. The method of claim 15, the memorability models include at least one of a voting model, a heuristic model, a rules model, a statistical model, an inference model, and a complimentary model.

17. The method of claim 16, the complimentary model is based upon patterns of forgetfulness.

18. The method of claim 14 further comprising employing the landmark information in a browser interface that associates one or more events relating to the landmark information to one or more items that are retrievable by the browser.

19. A system that facilitates computer-based searching, comprising:

means for receiving information related to a search for information;

means for employing content-based landmark information to facilitate the search for information, the landmark information corresponding to contextual information related to event(s) memorable to an originator of the search; and

means for providing a timeline visualization of search results based at least in part upon an index of a subset of the contextual information.

20. A system employing memorability models, comprising:

one or more memorability models that automatically capture an ability of people to recognize events as landmarks in time; and

an application that employs the memorability models to facilitate processing of information in accordance with the events.

21. The system of claim 20, the memorability models include procedures and policies for assigning a measure of memorability to events that can be employed by various computer-based applications to aid users in processing, receiving, and/or communicating information.

22. The system of claim 21, the events can include at least one of appointments, annotations in a user'"'"'s calendar, holidays, news stories over time, and images.

23. The system of claim 20, the memorability models are employed to provide a personalized index containing landmarks in time, the index is employed in at least one application relating to browsing directories of information and in reviewing results of a search engine.

24. The system of claim 20, the memorability models can include at least one of voting models, heuristic models, rules models, statistical models, and complimentary models that are based on patterns.

25. The system of claim 24, the voting models automatically poll a set of users in order to score the memorability of public events.

26. The system of claim 25, the score is based on scalar measures of memorability that include at least one of salience of news stories taken from a corpus of news stories and querying a set of people to assign a value.

27. The system of claim 24, the heuristic models utilize properties of messages and create informal policies that assign scores or deterministic categories of memorability based on functions of the properties.

28. The system of claim 27, further comprising a heuristic function that analyzes the increasing duration of events on a calendar as positively influencing the memorability of the events.

29. The system of claim 28, the heuristic function is applied to which images or subsets of images from a set of images serve as the most memorable of sets of images taken at the event based one or more properties of the images.

30. The system of claim 29, the properties include at least one of a composition of objects in a scene, a color histogram, faces recognized, features involving the sequence and temporal relationships among pictures, a picture associated with short inter-picture intervals, a capturing of excitement of a photographer about an aspect of the events, and properties that indicate that a user'"'"'s activity with regard to the image.

31. The system of claim 30, the user'"'"'s activity includes examining or displaying the image with longer or shorter dwell time, editing the image, cropping the image, and renaming the image.

32. The system of claim 30, further comprising automated analysis of image quality including focus and orientation.

33. The system of claim 24, the rules models include rules for automatically assigning measures of memorability to news stories that include properties relating to at least one of the number of news stories, persistence in the media, number of casualties, the dollar value of the loss associated with the news story, features capturing dimensions of surprise or atypical, and the proximity to the user of the event.

34. The system of claim 33, the statistical models employ machine learning methods that provide models which predict the memorability of items, the statistical models include the use of Bayesian learning, which can generate at least one of Bayesian dependency models (such as Bayesian networks), naï

ve Bayesian classifiers, and Support Vector Machines (SVMs).

35. The system of claim 24, further comprising a trainer component that takes explicit examples of landmark items or items that are forgotten.

36. The system of claim 35, the trainer is supplied with examples identified through implicit training.

37. The system of claim 24, the complimentary models describe the use of variants of memorability which are focused on inferring the likelihood that users will not recall a forthcoming event.

38. The system of claim 37, the complimentary models utilize inferences in applications to highlight in a selective manner the information that a user is likely to forget in a visually salient manner, or to change the timing or alerting of information in accordance with the likelihood that the information will not be remembered.

39. The system of claim 37, the complimentary models are combined with messaging and reminding systems including context-sensitive costs and benefits of transmitting information and alerting a user about information that is possibly forgotten.

40. The system of claim 20, further comprising a threshold adjustment allowing landmark events from a user'"'"'s calendar to be displayed that have a higher likelihood than a threshold of being memorable, per the setting of the adjustment.

41. The system of claim 40, further comprising a display that progressively lightens events with progressively lower likelihoods of being a landmark.

42. The system of claim 41, further comprising a step that assigns intensity as a function of membership of an event within different ranges of likelihood of being a landmark.

43. The system of claim 20, further comprising a training interface that fetches a file of a user'"'"'s calendar appointments over the years and allows the user to indicate whether appointments serve as memory landmarks.

44. The system of claim 43, the training interface further comprises a train button that creates a statistical classifier that takes multiple properties of events on a user'"'"'s calendar and predicts the likelihood that each event is a landmark event.

45. The system of claim 44, the likelihood is based on the following expression:

p(memory landmark |E1 . . . En), wherein p is a probability and E1 . . . En is evidence relating to one or more event properties.

46. The system of claim 20, further comprising an inference model to process memorability variables including at least one of, whether or not peers are at a meeting, the day of week, the time of day, the duration of a meeting, whether the meeting is recurrent, the time set for early reminding about a meeting, the role of a user, did the meeting come via an alias or from a person, how many attendees are at the meeting, are a user'"'"'s direct reports, manager, or manager'"'"'s manager at the meeting, who is the organizer of the meeting, the subject of the meeting, the location of the meeting, and how did the user respond to the meeting request.

47. The system of claim 46, further comprising processing at least one of “

organizer atypia,”

“

location atypia,” and

“

attendees atypia”

that are computed from a user'"'"'s appointment store and capture the rarity or “

atypia”

of properties of an event or appointment.

48. The system of claim 47, further comprising discretizing typicality for a Location, an Organizer, and an Attendee into states based on ranges of frequency.

49. The system of claim 20, further comprising one or more controls that are selected by users for controlling how and when events are displayed.

50. A method for applying memorability information, comprising:

automatically labeling events or items with numerical or categorical labels according to a measure of the likelihood that an item will be recalled, recognized as a landmark, or be most representative of an event or time; and

applying the labeling to information-management applications.

51. The method of claim 50, further comprising employing mathematical functions that assign a scalar measure of salience of events or items as being recalled, recognized as landmarks, or most representative of events or times.

52. The method of claim 51, further comprising at least one of:

applying statistical models of memorability via machine learning methods that are trained implicitly or with an explicit training system;

collecting information about a sample of memorable or non-memorable events or items that provides real-time inference or classification about the likelihood that an event or items as being recalled, recognized as landmarks, or be most representative of events or times; and

providing a probability distribution over different degrees of the event or item.

53. The method of claim 50, further comprising automatically filtering a stream of heterogeneous events and content, so as to selectively store events for log of lifetime events.

54. The method of claim 50, further comprising hierarchically browsing a log of heterogeneous events and content or browsing data at different levels of temporal precision.

55. The method of claim 50, further comprising employing representative landmarks and memorability to selectively choose pictures for an ambient display of pictures drawn from a picture library.

56. The method of claim 50, further comprising employing representative memory landmarks and memorability to selectively choose a set of pictures in a slide show over time or at different points in time about one or more events, under constraints in the total number of slides that a user desires to show.

57. The method of claim 50, further comprising employing representative memory landmarks and memorability to selectively choose a set of items to characterize or summarize the contents of a corpus of items.

58. The method of claim 57, the items include at least one of an image, a photo library, a thumbnails of graphics or photo images displayed on files, items, or folders of documents.

59. The method of claim 50, the information-management applications are applied to at least one of a memorability application, relating to will an item be recalled and understood, a memorable landmark relating to will an item be viewed as a milestone in time, and a representative landmark relating to is the item representative of a period of time, event, or sequence of events.

60. A method for determining reminders, comprising:

automatically training models from data; and

performing inference about items that are potentially forgotten.

61. The method of claim 60, further comprising:

inferring a likelihood that an item will be forgotten; and

performing a cost-benefit analysis of an expected value of reminding a user about the item.

62. The method of claim 60, further comprising performing expected-utility decision making about if and when to come forward to remind a user about something that they are likely to forget given an item type and context in view of a cost of an interruption.

63. The method of claim 60, further comprising controlling of alerting about reminders in desktop applications or mobile devices via the incorporation of the disruptiveness and the cost of a transmission.

64. The method of claim 60, further comprising automatically assisting patients with various cognitive deficits that may lead to memory aberrancies.

65. The method of claim 64, further comprising automatically predicting the likelihood that a patient with Alzheimer'"'"'s disease is at a particular stage of the illness.

66. The method of claim 65, further comprising at least one of automatically providing audiovisual cues to users and automatically providing ideal reminders.