Storing and recalling information to augment human memories

• US 6,549,915 B2
• Filed: 06/06/2001
• Issued: 04/15/2003
• Est. Priority Date: 12/15/1999
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Tangis Corporation
• Inventors: Newell, Dan, Abbott, Kenneth H. III, Robarts, James O.
• Primary Examiner(s): Mizrahi, Diane D.
• Application Number:
  US09/876,814
• Publication Number:
  US 20020032689A1
• Time in Patent Office: 678 Days
• Field of Search: 707/10, 707/104.1, 707/1, 712/23, 329/341, 341/51
• US Class Current: 1/1
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/487 using geographical or spatial information, e.g. location
• CPC Class Codes: G06F 16/907 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99948 Application of database or data structure, e.g. distributed, multimedia, or image

Abstract

A system for computer-based storing of information about a current state so that later recall of the information can augment human memories. In particular, when information about a current event of interest is to be stored, a variety of current state information of different types (e.g., video, audio, and textual information) about the environment and about a user can be acquired via sensors and other input devices. The variety of state information can then be associated together as a group and stored for later retrieval. Other information can also be associated with the group, such as one or more recall tags that facilitate later retrieval of the group, or one or more annotations to provide contextual information when the other state information is later retrieved and presented to the user. When information about a past event is to be recalled, one or more identifying recall tags can be received that are used to identify one or more state information groups that match the identifying tags. Some or all of the previously-acquired state information for the identified state information groups can then be presented to the user on appropriate output devices. Other information, such as annotations, can also be presented to the user in order to describe the state information and thus assist the user'"'"'s recollection of the previous state when the information was stored.

Claims

1. A method in a computer for assisting a user to recall information about an environment of the user at a previous time, the computer having a plurality of stored groups of information each describing an environment of the user at a previous time and each having an associated recall tag to assist in retrieval of that stored group of information, each stored group of information including descriptive information about the environment and information recorded from the environment at the previous time by at least one computer input device, the method comprising:

receiving an indication of a recall tag for use in retrieval of a stored group of information, the indication received by an executing State Recall Module;

determining whether one of the stored groups of information has an associated recall tag matching the indicated recall tag, the determining performed under control of the State Recall Module; and

when one of the stored groups of information is determined to have a matching recall tag, assisting the user to recall information about the environment at the previous time when the stored information for the determined one group of information was recorded by, under the control of the State Recall Module,

presenting to the user the recorded information from the determined one group of information; and

presenting to the user the descriptive information from the determined one group of information.

2. The method of claim 1 wherein the indication of the tag is received from the user.

3. The method of claim 1 wherein the indication of the tag is generated automatically without human intervention.

4. The method of claim 1 including, before the receiving of the indication of the tag, generating a stored group of information about a current state of the environment in response to a received indication.

5. The method of claim 1 wherein the tag is textual information.

6. The method of claim 1 wherein the tag is audio information.

7. The method of claim 1 wherein the tag is video information.

8. The method of claim 1 wherein the determined group of information includes sensed information about a state of the user at the previous time when the information was recorded for the determined group.

9. A method in a computer for assisting a user in recalling a previous state, the computer having a plurality of stored state fragments each describing a previous time and having an associated topic related to the described previous time, each state fragment including information recorded at the previous time for that state fragment by at least one computer input sensor and descriptive information about the recorded information, the method comprising:

receiving from the user an indication of a topic related to a previous time, the indication received by a State Recall Module of the computer;

determining whether one of the stored state fragments has an associated topic matching the indicated topic, the determining performed by the State Recall Module; and

when one of the stored state fragments is determined to have a matching associated topic, assisting the user to recall information about the previous time described by the determined one state fragment by, under control of the State Recall Module,

presenting to the user the recorded information of the one state fragment; and

presenting to the user the descriptive information of the one state fragment.

10. The method of claim 9 including, before the receiving of the indication of the topic, generating the one state fragment in response to an indication received from the user.

11. The method of claim 9 wherein the one state fragment includes sensed information about an environment of the user at the previous time described by the one state fragment.

12. The method of claim 9 wherein the one state fragment includes information about a state of the computer at the previous time described by the one state fragment.

13. A method in a computer for assisting a user in recalling a past state, the computer having a plurality of stored state fragments each describing a previous time, each state fragment having information recorded at the previous time for that state fragment by at least one computer input sensor and having an associated topic related to the recorded information, the method comprising:

monitoring the user to collect a group of information about a current state of the user;

monitoring an environment of the user to collect a group of information about a current state of the environment; and

without user intervention,

determining based on the collected groups of information a topic of interest to the user, the determining performed by an executing User Modeling Module; and

assisting the user in recalling a past state having information related to the determined topic by, under control of an executing State Recall Module,

identifying a stored state fragment with an associated topic that matches the determined topic; and

presenting to the user the recorded information of the identified state fragment.

14. The method of claim 13 including, before the identifying of the stored state fragment, generating the stored state fragment in response to an indication generated automatically without user intervention.

15. The method of claim 13 wherein the identified state fragment includes information about a state of the computer at the previous time described by the identified state fragment.

16. The method of claim 13 wherein the identified state fragment additionally has descriptive information about the recorded information, and including presenting to the user the descriptive information of the identified state fragment.

17. A method in a computer for assisting a user in recalling a previous state, the computer having a plurality of stored groups of information each describing a previous state and each having a recall tag, each group of information having information recorded from an environment external to the user by at least one computer input device and having information recorded from the user by at least one computer input device, the method comprising:

receiving an indication of a recall tag for use in retrieval of a stored group of information, the indication received by an executing State Recall Module;

determining whether one of the stored groups of information has a recall tag matching the indicated recall tag, the determining performed under control of the State Recall Module; and

when one of the stored groups of information is determined to have a matching recall tag, assisting the user in recalling the previous state described by the one group of information by, under control of the State Recall Module,

presenting to the user the recorded environment information from the one group of information; and

presenting to the user the recorded user information from the one group of information.

18. The method of claim 17 wherein the recorded information from the user includes sensed information about the user.

19. The method of claim 17 wherein the recorded information from the user includes information input by the user.

20. The method of claim 17 wherein the one stored group of information additionally has an annotation, and including presenting to the user the annotation.

21. A computer-readable medium whose contents cause a computer to provide information to a user about an environment of the user at a previous time, the computer having a plurality of stored groups of information each describing an environment of the user at a previous time and each having an associated recall tag, each group of information having descriptive information about the environment and having information recorded from the environment at the previous time by at least one computer input device, by performing a method comprising:

receiving an indication of a recall tag related to a previous time, the indication received by a State Recall Module of the computer;

determining whether one of the stored groups of information has an associated recall tag matching the indicated recall tag, the determining performed under control of the State Recall Module; and

when one of the stored groups of information is determined to have a matching recall tag, providing information to the user about the environment at the previous time described by the determined one group of information by, under control of the State Recall Module,

presenting to the user the recorded information from the determined one group of information; and

presenting to the user the descriptive information from the determined one group of information.

22. The computer-readable medium of claim 21 wherein the tag is audio information.

23. The computer-readable medium of claim 21 wherein the tag is video information.

24. The computer-readable medium of claim 21 wherein the determined group of information includes sensed information about the user at the previous time when information was recorded for the determined group of information.

25. The computer-readable medium of claim 21 wherein the determined group of information includes information about a state of the computer at the previous time when information was recorded for the determined group of information.

26. A computer system for reminding a user of a prior environment external to the user, comprising:

a storage means having a plurality of stored groups of information each describing an environment of the user at a previous time and each having an associated recall tag to assist in retrieval of that stored group of information, each group of information having descriptive information about the environment and having information recorded from the environment at the previous time by at least one computer input device; and

a state recall module that receives an indication of a recall tag, that determines whether one of the stored groups of information has an associated recall tag matching the indicated recall tag, and that, when one of the stored groups of information is determined to have a matching recall tag, reminds the user of the environment at the previous time described by the determined group of information by presenting to the user the recorded information from the determined group of information and by presenting to the user the descriptive information from the determined group of information.

27. The computer system of claim 26 including a user modeling module that monitors the user and an environment of the user to collect a group of information about a current state, and that determines based on the monitoring when to augment the memory of the user about the environment at a previous time.