Storing and recalling information to augment human memories

• US 20030154476A1
• Filed: 02/21/2003
• Published: 08/14/2003
• Est. Priority Date: 12/15/1999
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Robarts, James O., Newell, Dan, Abbott, Kenneth H. III
• Granted Patent: US 7,155,456 B2
• Time in Patent Office: Days
• US Class Current: 725/37
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/487 using geographical or spatial information, e.g. location
• CPC Class Codes: G06F 16/907 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99948 Application of database or data structure, e.g. distributed, multimedia, or image

Abstract

A system for computer-based storing of information about a current state so that later recall of the information can augment human memories In particular, when information about a current event of interest is to be stored, a variety of current state information of different types (e.g., video, audio, and textual information) about the environment and about a user can be acquired via sensors and other input devices. The variety of state information can then be associated together as a group and stored for later retrieval. Other information can also be associated with the group, such as one or more recall tags that facilitate later retrieval of the group, or one or more annotations to provide contextual information when the other state information is later retrieved and presented to the user. When information about a past event is to be recalled, one or more identifying recall tags can be received that are used to identify one or more state information groups that match the identifying tags. Some or all of the previously-acquired state information for the identified state information groups can then be presented to the user on appropriate output devices. Other information, such as annotations, can also be presented to the user in order to describe the state information and thus assist the user'"'"'s recollection of the previous state when the information was stored.

Claims

1. A method in a computer to augment memories of a user, the computer having input devices capable of recording audio and video information from the environment and having output devices capable of presenting audio and video information, the method comprising:

receiving a plurality of indications from the user, each received indication to store a state fragment having information about a current state of an environment of the user;

for each of the plurality of received indications, storing a state fragment by,

recording video information from the environment using at least one of the input devices;

recording at least one audio recall tag from the user using at least one of the input devices, the recall tag related to a subject of the recorded video information;

recording at least one audio annotation from the user using at least one of the input devices, the annotation providing descriptive information about the recorded video information; and

creating a stored state fragment having the recorded video information, the recorded audio recall tags, and the recorded audio annotations; and

when the user indicates that one of the stored state fragments is to be recalled, recalling a stored state fragment by,

receiving from the user an indication of a subject;

comparing the indicated subject to the recall tags of the stored state fragments; and

when the indicated subject matches a recall tag of one of the stored state fragments,

presenting to the user on at least one of the output devices the recorded video information of the one stored state fragment; and

presenting to the user on at least one of the output devices the recorded audio annotations of the one stored state fragment, so that the user can recall video and annotation information for a previous state of an environment of the user.

2. The method of claim 1 wherein the indicated subject matches a recall tag of multiple of the stored state fragments, and wherein the presenting of the recorded video information and of the recorded audio annotation is performed for each of the multiple stored state fragments.

3. The method of claim 1 wherein the indicated subject matches a recall tag of multiple of the stored state fragments, and wherein the presenting of the recorded video information and of the recorded audio annotation is performed for a stored state fragment having a highest degree of matching.

4. The method of claim 1 including:

for each of a plurality of indications generated automatically by the computer without human intervention, storing a state fragment by,

recording video information from the environment using at least one of the input devices;

receiving a computer-generated recall tag related to a subject of the recorded video information;

receiving a computer-generated annotation providing descriptive information about the recorded video information; and

creating a stored state fragment having the recorded video information, received recall tag, and received annotation.

5. The method of claim 1 including:

when the computer automatically indicates without human intervention that one of the stored state fragments is to be recalled, recalling a stored state fragment by,

receiving a computer-generated indication of a subject;

comparing the indicated subject to the recall tags of the stored state fragments; and

when the indicated subject matches a recall tag of one of the stored state fragments,

presenting to the user on at least one of the output devices the recorded video information of the stored state fragment having the matching recall tag; and

presenting to the user on at least one of the output devices the recorded audio annotations of the stored state fragment having the matching recall tag.

6. The method of claim 1 including deleting the one stored state fragment after the recalling of that state fragment.

7. The method of claim 1 including, after the storing of the one state fragment and before the recalling of that state fragment, manipulating the information stored in that state fragment.

8. The method of claim 1 including, after the storing of the state fragments, indexing the stored state fragments based on the recall tags to facilitate later recall of a state fragment matching a specified recall tag.

9. A computer-implemented method for augmenting a memory of a user about an environment external to the user, the method comprising:

storing information about a current state of the environment by,

recording information from the environment with at least one input device of the computer;

receiving an indication of a tag related to the environment;

receiving an indication of descriptive information about the environment; and

associating the indicated tag and the indicated descriptive information with the recorded environment information,

so that if the tag is later supplied, the recorded environment information and the descriptive information can be presented to augment the memory of the user of the state of the environment at the time of the storing.

10. The method of claim 9 wherein the storing of the information about the current state is in response to an indication from the user.

11. The method of claim 9 wherein the storing of the information about the current state is in response to an indication generated automatically without human intervention.

12. The method of claim 9 including, after the storing of the information about the current state, presenting to the user the stored information about the current state in response to a received indication matching the tag.

13. The method of claim 9 wherein the tag is textual information.

14. The method of claim 9 wherein the tag is audio information.

15. The method of claim 9 wherein the tag is video information.

16. The method of claim 9 wherein the stored information about the current state includes sensed information about a current state of the user.

17. The method of claim 9 wherein the stored information about the current state includes input information received from the user.

18. The method of claim 9 wherein the stored information about the current state includes information about a current state of the computer.

19. The method of claim 9 including storing information about a plurality of states of the environment.

20. A method in a computer for augmenting a memory of a user, the method comprising:

in response to each of a plurality of indications from the user, storing a group of information about a current state by,

receiving from the user an indication of at least one input sensor of the computer;

recording information about the current state with the indicated input sensors;

receiving from the user an indication of a topic related to the current state;

receiving from the user an indication of information describing the current state; and

associating the indicated topic and the indicated descriptive information with the recorded information; and

indexing the stored groups of information in order to facilitate later retrieval of a stored group of information.

21. The method of claim 20 including:

receiving information from the user to identify one of the stored groups of information;

identifying the one of the stored groups by using the received information and the indexing; and

presenting to the user information from the identified stored group.

22. The method of claim 20 wherein the recorded information about the current state includes sensed information about an environment of the user.

23. The method of claim 20 wherein the stored groups of information include information about a current state of the computer.

24. A method in a computer for augmenting a memory of a user, the method comprising:

monitoring the user to collect a group of information about a current state of the user;

monitoring an environment of the user to collect a group of information about a current state of the environment; and

without user intervention,

determining based on the collected groups of information a level of user interest in at least one of the current state of the user and the current state of the environment; and

when the level of user interest in at least one of the current states is determined to exceed a threshold, storing information about at least one of the current states by,

recording information about at least one of the current states with at least one input sensor of the computer;

determining a topic related to the recorded information; and

associating the determined topic with the recorded information,

so that the stored information can later be presented to the user to augment the memory of the user of the current states at the time of the recording.

25. The method of claim 24 including:

after the storing of the information,

receiving information that identifies the stored information; and

presenting to the user the stored information.

26. The method of claim 24 wherein the determined topic is a recall tag to be later used in identifying the stored information.

27. The method of claim 24 including, without user intervention, determining information describing the recorded information and associating the determined descriptive information with the determined topic and the recorded information.

28. A method in a computer for augmenting a memory of a user, the computer having a plurality of stored groups of information each having a tag, each group of information describing an environment of the user at a previous time, each group of information having descriptive information about the environment and having information recorded from the environment at the previous time by at least one computer input device, the method comprising:

receiving an indication of a tag;

determining whether one of the stored groups of information has a tag matching the indicated tag; and

when one of the stored groups of information is determined to have a matching tag, augmenting the memory of the user about the environment at the previous time when the information for the determined group of information was recorded by,

presenting to the user the recorded information from the determined group of information; and

presenting to the user the descriptive information from the determined group of information.

29. The method of claim 28 wherein the indication of the tag is received from the user.

30. The method of claim 28 wherein the indication of the tag is generated automatically without human intervention.

31. The method of claim 28 including, before the receiving of the indication of the tag, generating a stored group of information about a current state of the environment in response to a received indication.

32. The method of claim 28 wherein the tag is textual information.

33. The method of claim 28 wherein the tag is audio information.

34. The method of claim 28 wherein the tag is video information.

35. The method of claim 28 wherein the determined group of information includes sensed information about a state of the user at the previous time when the information was recorded for the determined group.

36. A method in a computer for augmenting a memory of a user, the computer having a plurality of stored state fragments each describing a previous time, each state fragment having information recorded at the previous time for that state fragment by at least one computer input sensor, an associated topic related to the recorded information, and descriptive information about the recorded information, the method comprising:

receiving from the user an indication of a topic;

determining whether one of the stored state fragments has an associated topic matching the indicated topic; and

when one of the stored state fragments is determined to have a matching associated topic, augmenting the memory of the user about the previous time described by the determined state fragment by,

presenting to the user the recorded information of the one state fragment; and

presenting to the user the descriptive information of the one state fragment.

37. The method of claim 36 including, before the receiving of the indication of the topic, generating the one state fragment in response to an indication received from the user.

38. The method of claim 36 wherein the one state fragment includes sensed information about an environment of the user at the previous time described by the one state fragment.

39. The method of claim 36 wherein the one state fragment includes information about a state of the computer at the previous time described by the one state fragment.

40. A method in a computer for augmenting a memory of a user, the computer having a plurality of stored state fragments each describing a previous time, each state fragment having information recorded at the previous time for that state fragment by at least one computer input sensor and having an associated topic related to the recorded information, the method comprising:

monitoring the user to collect a group of information about a current state of the user;

monitoring an environment of the user to collect a group of information about a current state of the environment; and

without user intervention,

determining based on the collected groups of information a topic of interest to the user; and

augmenting the memory of the user about a past state having information related to the determined topic by,

identifying a stored state fragment with an associated topic that matches the determined topic; and

presenting to the user the recorded information of the identified state fragment.

41. The method of claim 40 including, before the identifying of the stored state fragment, generating the stored state fragment in response to an indication generated automatically without user intervention.

42. The method of claim 40 wherein the identified state fragment includes information about a state of the computer at the previous time described by the identified state fragment.

43. The method of claim 40 wherein the identified state fragment additionally has descriptive information about the recorded information, and including presenting to the user the descriptive information of the identified state fragment

44. A computer-implemented method for augmenting a memory of a user, the method comprising:

storing information about a current state by,

recording information from an environment external to the user with at least one input device of the computer;

recording information from the user with at least one input device of the computer;

receiving an indication of a tag related to the current state, the indicated tag to be used for later retrieval of the recorded environment information and the recorded user information; and

associating the indicated tag with the recorded environment information and the recorded user information,

so that if the tag is later supplied, the recorded environment information and the recorded user information can be used to augment the memory of the user of the state at the time of the storing.

45. The method of claim 44 wherein the recorded information from the user includes sensed information about the user.

46. The method of claim 44 wherein the recorded information from the user includes information input by the user.

47. The method of claim 44 including associating a received annotation with the stored information.

48. A method in a computer for augmenting a memory of a user, the computer having a plurality of stored groups of information each having a tag, each group of information describing a previous state, each group of information having information recorded from an environment external to the user by at least one computer input device and having information recorded from the user by at least one computer input device, the method comprising:

receiving an indication of a tag;

determining whether one of the stored groups of information has a tag matching the indicated tag; and

when one of the stored groups of information is determined to have a matching tag, augmenting the memory of the user about the previous state described by the one group of information by,

presenting to the user the recorded environment information from the one group of information; and

presenting to the user the recorded user information from the one group of information.

49. The method of claim 48 wherein the recorded information from the user includes sensed information about the user.

50. The method of claim 48 wherein the recorded information from the user includes information input by the user.

51. The method of claim 48 wherein the one stored group of information additionally has an annotation, and including presenting to the user the annotation.

52. A method for augmenting memory of a user of a wearable computer having a plurality of input devices, the method comprising:

receiving a plurality of indications to store information about a current state;

for each of the plurality of received indications, storing a state fragment by,

recording sensed information about an environment external to the user from one of the input sensor devices;

recording sensed information about the user from one of the input sensor devices; and

associating the recorded environment information and the recorded user information as the stored state fragment; and

augmenting the memory of the user of a past state by,

receiving an indication of a topic of interest to the user;

identifying at least one stored state fragment having information related to the topic of interest; and

presenting to the user the recorded environment information and the recorded user information associated with the identified state fragment.

53. A computer-readable medium whose contents cause a computer to augment a memory of a user about an environment external to the user, by:

storing information about a current state of the environment by,

recording information from the environment with at least one input device of the computer;

receiving an indication of a tag related to the environment;

receiving an indication of descriptive information about the environment; and

associating the indicated tag and the indicated descriptive information with the recorded environment information,

so that if the tag is later supplied, the recorded environment information and the descriptive information can be presented to augment the memory of the user of the state of the environment at the time of the storing.

54. The computer-readable medium of claim 53 wherein the tag is audio information.

55. The computer-readable medium of claim 53 wherein the tag is video information.

56. The computer-readable medium of claim 53 wherein the stored information about the current state includes sensed information about a current state of the user.

57. The computer-readable medium of claim 53 wherein the stored information about the current state includes information about a current state of the computer.

58. A computer-readable medium whose contents cause a computer to augment a memory of a user, the computer having a plurality of stored groups of information each having a tag, each group of information describing an environment of the user at a previous time, each group of information having descriptive information about the environment and having information recorded from the environment at the previous time by at least one computer input device, by:

receiving an indication of a tag;

determining whether one of the stored groups of information has a tag matching the indicated tag; and

when one of the stored groups of information is determined to have a matching tag, augmenting the memory of the user about the environment at the previous time described by the determined group of information by,

presenting to the user the recorded information from the determined group of information; and

presenting to the user the descriptive information from the determined group of information.

59. The computer-readable medium of claim 58 wherein the tag is audio information.

60. The computer-readable medium of claim 58 wherein the tag is video information.

61. The computer-readable medium of claim 58 wherein the determined group of information includes sensed information about the user at the previous time when information was recorded for the determined group of information.

62. The computer-readable medium of claim 58 wherein the determined group of information includes information about a state of the computer at the previous time when information was recorded for the determined group of information.

63. A computer system for augmenting a memory of a user about an environment external to the user, comprising:

a state storage module that stores information about a current state of the environment by recording information from the environment with at least one input device of the computer, by receiving an indication of a tag related to the environment, by receiving an indication of descriptive information about the environment, and by associating the indicated tag and the indicated descriptive information with the recorded environment information; and

a storage means for storing the information about the current state of the environment.

64. The computer system of claim 63 including a state recall module that receives information from the user to identify one of the stored groups of information, and that presents to the user information from the identified one of the stored groups.

65. The computer system of claim 63 including a user modeling module that monitors the user and an environment of the user to collect a group of information about a current state, and that determines based on the monitoring when to store information about the current state of the environment.

66. A computer system for augmenting a memory of a user about an environment external to the user, comprising:

a storage means having a plurality of stored groups of information each having a tag, each group of information describing an environment of the user at a previous time, each group of information having descriptive information about the environment and having information recorded from the environment at the previous time by at least one computer input device; and

a state recall module that receives an indication of a tag, that determines whether one of the stored groups of information has a tag matching the indicated tag, and that, when one of the stored groups of information is determined to have a matching tag, augments the memory of the user about the environment at the previous time described by the determined group of information by presenting to the user the recorded information from the determined group of information and by presenting to the user the descriptive information from the determined group of information.

67. The computer system of claim 66 including a user modeling module that monitors the user and an environment of the user to collect a group of information about a current state, and that determines based on the monitoring when to augment the memory of the user about the environment at a previous time.

68. A data structure stored in memory of a computer for use in augmenting a memory of a user, the data structure describing a state of the user at a previous time, comprising:

information about an environment external to the user sensed by an input sensor device of the computer at the previous time;

information from the user received by an input device of the computer at the previous time; and

a tag for identifying the data structure,

so that if the tag is supplied, the environment information and the user information can be presented to augment the memory of the user of the state at the previous time.

69. The data structure of claim 68 including an annotation that describes the environment information and the user information.

70. The data structure of claim 68 including information about a state of the computer at the previous time.

71. The data structure of claim 68 wherein the user information includes sensed information about the user at the previous time.

72. The data structure of claim 68 wherein the user information includes information input by the user at the previous time.

73. A generated data signal transmitted via a data transmission medium, the data signal comprising:

information about an environment external to the user sensed by an input sensor device of the computer at the previous time;

information from the user received by an input device of the computer at the previous time; and

a tag for identifying the data signal,

so that if the tag is supplied, the environment information and the user information can be presented to augment the memory of the user of the previous time.

74. The data signal of claim 73 including an annotation that describes the environment information and the user information.

75. The data signal of claim 73 including information about a state of the computer at the previous time.