Storing and recalling information to augment human memories

• US 9,443,037 B2
• Filed: 07/19/2006
• Issued: 09/13/2016
• Est. Priority Date: 12/15/1999
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Abbott, Kenneth H. III, Newell, Dan, Robarts, James O.
• Primary Examiner(s): Reyes, Mariela
• Assistant Examiner(s): MINCEY, JERMAINE A
• Application Number:
  US11/490,540
• Publication Number:
  US 20070043459A1
• Time in Patent Office: 3,709 Days
• Field of Search: 707/104.1
• US Class Current: 1/1
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/487 using geographical or spatial information, e.g. location
• CPC Class Codes: G06F 16/907 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99948 Application of database or data structure, e.g. distributed, multimedia, or image

Abstract

A system for computer-based storing of information about a current state so that later recall of the information can augment human memories. In particular, when information about a current event of interest is to be stored, a variety of current state information of different types (e.g., video, audio, and textual information) about the environment and about a user can be acquired via sensors and other input devices. The variety of state information can then be associated together as a group and stored for later retrieval. Other information can also be associated with the group, such as one or more recall tags that facilitate later retrieval of the group, or one or more annotations to provide contextual information when the other state information is later retrieved and presented to the user. When information about a past event is to be recalled, one or more identifying recall tags can be received that are used to identify one or more state information groups that match the identifying tags. Some or all of the previously-acquired state information for the identified state information groups can then be presented to the user on appropriate output devices. Other information, such as annotations, can also be presented to the user in order to describe the state information and thus assist the user'"'"'s recollection of the previous state when the information was stored.

Claims

1. A method in a computer for augmenting a memory of a user, the computer having a plurality of stored groups of information such that each stored group of information describes an environment external to the user and in which the user is physically present at the time of an event involving the user, the stored information including at least one tag and descriptive information about the environment and recorded information sensed about the environment at the time of the event, the event operating as a trigger to automatically create an indexed record of the environment and the at least one tag comprising human memory recall information, the method comprising:

receiving an indication of a tag representative of an event within the environment, the indicated tag comprising human memory recall information;

determining whether one of the stored groups of information has a tag matching the indicated tag; and

when one of the stored groups of information is determined to have a matching tag, augmenting the memory of the user about the environment by,

presenting to the user the recorded information sensed about the environment at the time of the event from the determined group of information, the recorded information being the information sensed about the environment at the time of a prior event and stored in a stored group of information determined to have the matching tag; and

presenting to the user the descriptive information about the environment at the time of the event from the determined group of information.

2. The method of claim 1 wherein the indication of the tag is received from the user.

3. The method of claim 1 wherein:

the method further comprises processing inputs to the computer to detect a subsequent physical event, the physical event occurring within a subsequent physical environment in which the user is present, the physical event involving the user; and

the processing comprises constructing a model of current conditions of the user based on the inputs such that the indication of the tag is generated automatically without human intervention.

4. The method of claim 3, wherein the subsequent event comprises meeting a person.

5. The method of claim 1 wherein the indicated tag is textual information.

6. The method of claim 1 wherein the indicated tag is audio information.

7. The method of claim 1 wherein the indicated tag is video information.

8. The method of claim 1 including, before the determining of whether one of the stored groups of information has a tag matching the indicated tag, receiving one or more indications of one or more additional tags, and wherein the determining includes matching at least one of the additional indicated tags to at least one of the stored groups of information.

9. The method of claim 8 wherein at least one of the stored groups of information includes multiple tags, and wherein the determining further includes identifying one of the stored groups of information whose multiple tags match the indicated tag and the additional indicated tags.

10. The method of claim 1 wherein:

the event is a physical event; and

stored groups of information of the plurality of stored groups each comprise recorded audio and visual data representing the environment at the time of the physical event.

11. The method of claim 1 including, after the determining, sending information from the determined group to another module.

12. The method of claim 4, wherein the descriptive information comprises information about the person.

13. The method of claim 1 wherein the determined one group of information includes sensed information about a physiological state of the user at a previous time when the recorded information for the determined group was recorded.

14. A computer-implemented method for augmenting a memory of a user about an environment external to the user, the method comprising:

receiving an indication of an event, the event within the environment, the environment being a physical environment in which the user is present, and the event occurring in the environment and involving the user at a first time;

in response to receiving the indication of the event, storing information about a current state of the environment by;

recording sensed audio or visual information regarding the environment;

receiving an indication of a tag related to the environment, the tag comprising human memory recall information; and

associating the indicated tag with the recorded environment information, thereby using the event as a trigger to automatically create an indexed record of a state of the environment;

at a second time, subsequent to the first time, in response to receiving an input comprising human memory recall information identifying the tag, presenting to the user the recorded environment information in audio or visual form to augment the memory of the user about the environment.

15. The method of claim 14 wherein the recorded information is obtained with at least one video recorder coupled to the computer.

16. The method of claim 14 wherein:

the storing of the information about the current state is in response to an indication from the user; and

the storing of the information about the current state of the environment comprises storing an indication of descriptive information about the environment at the time of the event.

17. The method of claim 14 wherein the storing of the information about the current state is in response to an indication generated automatically without human intervention.

18. The method of claim 14 including, after the storing of the information about the current state, presenting to the user the stored information about the current state in response to a received indication matching the tag.

19. A computer-readable medium whose contents cause a computer to augment a memory of a user, the computer having access to multiple stored groups of information that each has at least one associated tag and associated descriptive information about an environment outside the computer at the time of a physical event involving the user and associated recorded information for that environment, the physical event operating as a trigger to automatically create an indexed record of the environment and the at least one associated tag comprising human memory recall information, by performing a method comprising:

receiving an indication of a tag representative of a subsequent physical event involving the user, the indicated tag comprising human memory recall information;

determining one of the stored groups of information that has an associated tag matching the indicated tag, the determined group comprising information about the environment at the time of a prior physical event involving the user, and the matching being on similarity of the subsequent physical event to the prior physical event; and

augmenting the memory of the user about the prior event by presenting information for the environment to the user that is associated with the determined group of information, the presented information including the recorded information for the environment and the descriptive information about the environment.

20. A computer-readable medium whose contents cause a computer to augment a memory of a user about an environment involving the user within an environment in which the user is physically present, by performing a method comprising:

storing information about a current state of the environment by;

detecting an event, the event within the environment and the event perceptible to the user;

recording information regarding the environment contemporaneous with the event, the recorded information comprising information detected with a sensor input device in the environment in which the user is physically present;

receiving an indication of a tag that characterizes the event, the indicated tag comprising human memory recall information;

receiving an indication of descriptive information about the environment; and

associating the indicated tag and the indicated descriptive information with the recorded environment information, thereby using the event as a trigger to automatically create an indexed record of the environment, so that the recorded environment information and the descriptive information are adapted for later presentation to the user to augment the memory of the user relating to the event.