Real-time annotator

• US 8,131,750 B2
• Filed: 12/28/2007
• Issued: 03/06/2012
• Est. Priority Date: 12/28/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Bathiche, Steven N., Guday, Shai, Russell, Zachary Lewis, Multerer, Boyd Cannon, Whitten, Jon Marcus Randall, Wilson, Andrew David, MacLaurin, Matthew B.
• Primary Examiner(s): CHANNAVAJJALA, SRIRAMA T
• Application Number:
  US11/966,914
• Publication Number:
  US 20090171901A1
• Time in Patent Office: 1,530 Days
• Field of Search: 345/618, 345/625, 345/634, 715230-233, 704250-251, 348/239, 348/231.5, 707693-694, 707/749, 707/758, 707/795, 707/796, 707/805, 707821-922, 707705-706, 707/725, 707/769, 707/777, 713/168, 382/100, 382/115, 706/45, 706/61, 600/323, 600/301
• US Class Current: 707/769
• CPC Class Codes: G06F 16/24573 using data annotations, e.g. user-defined metadata
• CPC Class Codes: Y10S 707/913 Multimedia

Abstract

Systems (and corresponding methodologies) that annotate experience data in real-time are provided. The real-time annotated experience data can be employed in accordance with augmented reality systems which are capable of overlaying virtual data upon real-world data. The system employs ‘smart-tags’ that are capable of identifying data that relates to and/or is associated with the real-world scenarios and situations.

Claims

1. A system comprising:

a processing unit;

a memory coupled to the processing unit;

an interface component maintained on the memory and executed on the processing unit to monitor parameters and contextual factors related to an experience;

a context analyzer component that establishes the parameters and the contextual factors based on at least on a user'"'"'s state of mind associated with the experience, to facilitate establishing tags for the parameters and the contextual factors;

a tag generator that applies a tag to the parameters based on detecting the user'"'"'s state of mind, wherein the tag includes a smart tag describing the contextual factors associated with the user'"'"'s state of mind associated with the experience;

a tag identifier component that associates a plurality of tags related to the user'"'"'s state of mind associated with the experience to real-time elements, wherein the plurality of tags includes the smart tag;

a retrieval component that accesses tagged data related to the user'"'"'s state of mind;

an annotation component that annotates the experience in real-time with the accessed data to provide memory recall information to assist a user based on the user'"'"'s state of mind associated with the experience; and

the smart tag that presents real-time annotation inside at least one of a lens viewer in glasses or a small ear phone.

2. The system of claim 1, wherein the tag identifier component employs at least one of facial recognition, pattern recognition, speech recognition, audio analysis or text analysis to establish the plurality of tags.

3. The system of claim 1, wherein the annotation component renders the annotated experience via at least one of a visual or audio device.

4. The system of claim 1, further comprising an annotation system that tags the data related to a plurality of user experiences in view of contextual factors.

5. The system of claim 4, further comprising the interface component that receives the data related to elements.

6. The system of claim 5, further comprising an information gather component that collects the data related to the elements via a sensory mechanism.

7. The system of claim 6, wherein the sensory mechanism is at least one of a physiological or environmental sensor.

8. The system of claim 6, further comprising a query component that employs a query to supplement the collected data with data from a network-based source.

9. The system of claim 6, further comprising:

an analysis component that determines criteria related to the collected data; and

the tag generator component that generates the tags based upon a subset of the criteria.

10. The system of claim 9, further comprising:

a pattern analyzer component that identifies at least one of identity of an individual or object; and

the tag generator component that generates the tags based upon a subset of the criteria.

11. The system of claim 9, further comprising:

a sound analyzer component that identifies at least one of identity of an individual or object; and

the tag generator component that generates the tags based upon a subset of the criteria.

12. The system of claim 9, further comprising:

the context analyzer component that establishes context that further includes at least one of an activity context or environment context; and

the tag generator component that generates the tags based upon a subset of the context.

13. The system of claim 1, further comprising a machine learning and reasoning component that employs at least one of a probabilistic and a statistical-based analysis that infers an action that a user desires to be automatically performed.

14. A computer-implemented method comprising:

monitoring, by a processing unit of a computer, an element of an experience, wherein the element includes activity data;

automatically detecting, by the processing unit, a user action related to the element of the experience in real-time;

analyzing the element of the experience by identifying content;

based on the analysis, generating a plurality of tags that describe the element of the experience in real-time;

employing the analysis to identify the plurality of tags;

annotating the element in real-time with the activity data identified by a subset of the tags, the subset of tags include information related to an acquaintance, a social connection with the acquaintance, and a like or a dislike of the acquaintance; and

presenting real-time annotation of the element with the activity data identified by the subset of tags to a user in a mash-up a pop-up, or an overlay, the overlay includes overlaying virtual data over the experience in real-time.

15. The computer-implemented method of claim 14, wherein the act of analyzing includes at least one of analyzing a pattern, analyzing a sound, or analyzing a context of the element.

16. The computer-implemented method of claim 14, further comprising retrieving the activity data identified by the subset of the tags from a network-based source.

17. The computer-implemented method of claim 14, further comprising:

monitoring a disparate set of elements;

identifying data related to a subset of the disparate set of elements; and

tagging the data based upon content in view of context.

18. A system comprising:

a processing unit;

a memory coupled to the processing unit;

an annotation component maintained on the memory and executed on the processing unit to determine when to annotate real-time data by an automatic classifier system;

an interface component for monitoring experiences of a user;

a pattern analyzer component for analyzing elements of a subset of the experiences;

a context analyzer component for establishing the elements based on a user'"'"'s state of mind in response to the experiences;

a tag generator component for generating tags associated to the elements based upon the analysis and the user'"'"'s state of mind;

the tag generator component further comprising indexing data related to the elements as a function of the tags;

the tag generator component further comprising storing the indexed data;

the pattern analyzer component further comprising annotating the indexed data based on the user'"'"'s state of mind with the real-time data; and

the annotation component presenting the real time annotation through at least one of a lens viewer in glasses or on a small ear phone.

19. The system of claim 18, the annotation component further comprising annotating an experience of a disparate user in real-time based upon the indexed data.

20. The system of claim 19, wherein:

the interface component further comprises monitoring experiences of the disparate user; and

the pattern analyzer component further comprises analyzing the experiences, wherein the analysis is employed to locate or retrieve the indexed data.