Storing and recalling information to augment human memories

• US 7,155,456 B2
• Filed: 02/21/2003
• Issued: 12/26/2006
• Est. Priority Date: 12/15/1999
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Tangis Corporation
• Inventors: Robarts, James O., Newell, Dan, Abbott, Kenneth H. III
• Primary Examiner(s): MIZRAHI, DIANE D
• Application Number:
  US10/371,436
• Publication Number:
  US 20030154476A1
• Time in Patent Office: 1,404 Days
• Field of Search: 707/10, 707/104.1
• US Class Current: 1/1
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/487 using geographical or spatial information, e.g. location
• CPC Class Codes: G06F 16/907 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99948 Application of database or data structure, e.g. distributed, multimedia, or image

Abstract

A system for computer-based storing of information about a current state so that later recall of the information can augment human memories In particular, when information about a current event of interest is to be stored, a variety of current state information of different types (e.g., video, audio, and textual information) about the environment and about a user can be acquired via sensors and other input devices. The variety of state information can then be associated together as a group and stored for later retrieval. Other information can also be associated with the group, such as one or more recall tags that facilitate later retrieval of the group, or one or more annotations to provide contextual information when the other state information is later retrieved and presented to the user. When information about a past event is to be recalled, one or more identifying recall tags can be received that are used to identify one or more state information groups that match the identifying tags. Some or all of the previously-acquired state information for the identified state information groups can then be presented to the user on appropriate output devices. Other information, such as annotations, can also be presented to the user in order to describe the state information and thus assist the user'"'"'s recollection of the previous state when the information was stored.

Claims

1. A method in a computer for augmenting a memory of a user, the computer-implemented method comprising:

receiving an indication of a tag;

determining whether one of a plurality of stored groups of information has a tag matching the indicated tag, each stored group of information has a tag matching the indicated tag, each stored group of information describing an environment experienced by the user at a previous time and including at least one tag and descriptive information about the environment and recorded information for the environment, the recorded information for each stored group including information recorded by an input device of the computer at the previous time for that stored group; and

when one of the stored groups of information is determined to have a matching tag, augmenting the memory of the user about the environment described by that determined group of information by,

presenting to the user the recorded information from the determined group of information; and

presenting to the user the descriptive information from the determined group of information.

2. The method of claim 1 wherein the indication of the tag is received from the user.

3. The method of claim 1 wherein the indication of the tag is generated automatically without human intervention.

4. The method of claim 1 including, before the receiving of the indication of the tag, generating a stored group of information about a current state of the environment in response to a received indication.

5. The method of claim 1 wherein the indicated tag is textual information.

6. The method of claim 1 wherein the indicated tag is audio information.

7. The method of claim 1 wherein the indicated tag is video information.

8. The method of claim 1 wherein the determined one group of information includes sensed information about a state of the user at a previous time when the recorded information for the determined group was recorded.

9. The method of claim 1 including, before the determining of whether one of the stored groups of information has a tag matching the indicated tag, receiving one or more indications of one or more additional tags, and wherein the determining includes matching at least one of the additional indicated tags to at least one of the stored groups of information.

10. The method of claim 9 wherein at least one of the stored groups of information includes multiple tags, and wherein the determining further includes identifying one of the stored groups of information whose multiple tags match the indicated tag and the additional indicated tags.

11. The method of claim 1 including receiving one or more indications of additional tags.

12. The method of claim 11 wherein the determining further includes determining for each of the additional indicated tags that the one stored group of information has a tag that matches the additional indicated tag.

13. The method of claim 11 wherein the determining further includes determining for each of at least one of the additional tags that the one stored group of information has a tag that matches the additional indicated tag.

14. The method of claim 11 including determining one or more additional stored groups of information distinct from the one stored group of information that have tags matching one or more of the additional indicated tags.

15. The method of claim 14 including, after the determining of the additional stored groups of information, selecting at least one of the additional stored group of information with which to augment the memory of the user.

16. The method of claim 1 including, after the determining, sending information from the determined group to another module.

17. The method of claim 16 wherein the another module is a software module on the computer.

18. The method of claim 16 wherein the another module is a software module on a remote computer.

19. The method of claim 16 wherein the another module is an application.

20. The method of claim 16 including receiving of the information by the another module, and wherein the another module performs an action in response to the receiving.

21. The method of claim 1 including, after the determining, sending the determined group to another module.

22. A method for a computing system to augment a memory of a user about an environment, the computer-implemented method comprising;

storing information about a current state of an environment external to a user of the computing system by,

recording information regarding the environment using at least one device providing input information to the computing system;

receiving from the user an indication of a recall tag related to the environment;

receiving an indication of descriptive information about the environment; and

associating the indicated recall tag and the indicated descriptive information with the recorded environment information in such a manner that the recall tag can later be used to automatically retrieve both the indicated descriptive information and the recorded environment information,

so that if the tag is later supplied, the recorded environment information and the descriptive information can be presented to augment the memory of the user of the environment.

23. The method of claim 22 wherein the storing of the information about the current state is in response to an indication from the user.

24. The method of claim 22 wherein the storing of the information about the current state is in response to an indication generated automatically without human intervention.

25. The method of claim 22 including, after the storing of the information about the current state, presenting to the user the stored information about the current state in response to a received indication matching the tag.

26. The method of claim 22 wherein the tag is textual information.

27. The method of claim 22 wherein the tag is audio information.

28. The method of claim 22 wherein the tag is video information.

29. The method of claim 22 wherein the stored information about the current state includes sensed information about a current state of the user.

30. The method of claim 22 wherein the stored information about the current state includes input information received from the user.

31. The method of claim 22 wherein the stored information about the current state includes information about a current state of the computer.

32. The method of claim 22 wherein the indication of the descriptive information about the environment is generated under control of the user.

33. The method of claim 22 wherein the indication of descriptive information about the environment is generated automatically without human intervention.

34. The method of claim 22 wherein the storing of information about a current state of the environment includes,

receiving an indication of additional tags related to the environment;

and

associating the additional indicated tags and the indicated descriptive information with the recorded environment information,

so that if the one or more tags is later supplied, the recorded environment information and the descriptive information can be presented to augment the memory of the user of the environment.

35. The method of claim 22 wherein the recorded information is obtained with at least one input device of a remote computing device.

36. The method of claim 22 wherein the indication of descriptive information about the environment is received from another software module.

37. The method of claim 22 wherein the descriptive information about the environment is received from another software module.

38. The method of claim 22 wherein the descriptive information has an associated type of output device, and wherein presentation of the descriptive information includes presenting it on an output device of an associated type.

39. A method in a computer for augmenting a memory of a user, the computer-implemented method comprising:

receiving an indication of one or more tags;

receiving an indication of one or more criteria;

determining whether one or more of a plurality of stored groups of information has tags matching the indicated tags and meets the indicated criteria, each stored group of information describing an environment of a user that is external to the computer and including at least one tag and descriptive information about the environment and recorded information for the environment, the recorded information for each stored group including information recorded from the external environment by a device providing input information to the computer; and

when one or more of the stored groups of information is determined to have matching tags and to have met the criteria, augmenting the memory of the user about the environment by presenting to the user indications of the determined one or more groups of information.

40. The method of claim 39 wherein the computer is a body-supported computing device.

41. The method of claim 39 wherein the indications include the descriptive information from the determined one or more groups of information.

42. The method of claim 39 wherein the indications include the recorded information from the determined one or more groups of information.

43. The method of claim 39 wherein the indicated criteria includes an indication of a range of time, and wherein meeting the criteria includes for each of the determined one or more groups of information determining that the recorded information was recorded within the indicated range of time.

44. The method of claim 39 wherein the indicated criteria includes an indication of a type of data, and wherein meeting the criteria includes for each of the determined one or more groups of information determining that the recorded information is of the indicated type.

45. The method of claim 39 wherein the indicated criteria includes an indication of a location, and wherein meeting the criteria includes for each of the determined one or more groups of information determining that the recorded information was recorded at the indicated location.

46. The method of claim 39 including, after one or more of the stored groups of information is determined to have matching tags and to have met the criteria, selecting one of the multiple of the stored groups of information with which to augment the memory of the user.

47. The method of claim 46 wherein the selecting is done under control of the user.

48. The method of claim 46 wherein the selecting is done automatically without user intervention.

49. The method of claim 39 wherein the determining includes matching a subset of the indicated tags and indicated criteria.

50. The method of claim 39 wherein the selecting includes approximate matching of the indicated tags and indicated criteria.

51. A method in a computer for augmenting a memory of a user, the computer-implemented method comprising:

receiving an indication of one or more tags, or of one or more non-tag criteria, or of one or more tags and one or more non-tag criteria;

determining one of a plurality of stored groups of information that, if one or more tags are indicated, has tags matching the indicated tags, and that, if one or more non-tag criteria are indicated, has met the indicated criteria, each stored group of information describing an environment external to the computer and including at least one tag and descriptive information about the environment and recorded information for the environment, the recorded information for each stored group including information recorded from the external environment by a device providing input information to the computer; and

augmenting the memory of the user about the environment described by the determined stored group of information by providing to another computer at least some of the determined group of information.

52. A computer-implemented method for augmenting a memory of a user, the computer communicating with another computer the method comprising:

sending to the another computer an indication of one or more tags, or an indication of one or more non-tag criteria, or an indication of one or more tags and of one or more non-tag criteria;

receiving from the another computer one or more of a plurality of stored groups of information that are determined to, if one or more tags are sent, have tags matching the sent tags, and to, if one or more non-tag criteria are sent, have met the indicated criteria, each stored group of information describing an external environment and including at least one tag and descriptive information about the environment and recorded information for the environment, the recorded information for each stored group including information recorded from the external environment by a device providing input information to the another computer; and

augmenting the memory of the user about an environment by presenting to the user indications of the determined one or more groups of information.

53. A computer-readable medium whose contents cause a computer to augment a memory of a user, by performing a computer-implemented method comprising:

receiving an indication of a tag;

determining one of the stored groups of information that has an associated tag matching the indicated tag, each stored group of information describing an environment of a user that is external to the computer and including at least one tag and descriptive information about the environment and recorded information for the environment, the recorded information for each stored group including information recorded from the external environment by a device providing input information to the computer; and

augmenting the memory of the user about an environment by presenting information for the environment to the user that is associated with the determined group of information, the presented information including the recorded information for the environment and the descriptive information about the environment.

54. A computer-readable medium whose contents cause a computer to augment a memory of a user about an environment, by performing a computer-implemented method comprising:

storing information about a current state of an external environment experienced by a user of the computer by,

recording information regarding the environment using at least one device providing input information to the computer;

receiving an indication of a tag related to the environment;

receiving an indication of descriptive information about the environment; and

associating the indicated tag and the indicated descriptive information with the recorded environment information,

so that the recorded environment information and the descriptive information can later be presented to augment the memory of the user of the environment.