Event augmentation with real-time information

• US 9,330,499 B2
• Filed: 05/20/2011
• Issued: 05/03/2016
• Est. Priority Date: 05/20/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Perez, Kathryn Stone, Tardif, John A.
• Primary Examiner(s): Du, Haixia
• Application Number:
  US13/112,919
• Publication Number:
  US 20120293548A1
• Time in Patent Office: 1,810 Days
• Field of Search: 345/633
• US Class Current: 1/1
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2219/004 Annotating, labelling

Abstract

A system and method to present a user wearing a head mounted display with supplemental information when viewing a live event. A user wearing an at least partially see-through, head mounted display views the live event while simultaneously receiving information on objects, including people, within the user'"'"'s field of view, while wearing the head mounted display. The information is presented in a position in the head mounted display which does not interfere with the user'"'"'s enjoyment of the live event.

Claims

1. A computational machine-system implemented method providing supplemental information to a user with a see-through head mounted display and viewing a live event, the supplemental information supplementing live action occurrences within the viewed live event, the method comprising:

receiving an indication from the see-through head mounted display that the user is attending the live event and gazing at an in-view portion of the live event through the see-through head mounted display, the live event having a finite event duration;

determining at least one of likely next fields of view to be provided as in-view portions of the live event gazed at through the see-through head mounted display and identifying real objects expected to be present within the determined at least one of likely next fields of view during the event duration;

retrieving corresponding supplemental information describing at least a portion of the real objects expected to be present in the determined at least one of likely next fields of view;

determining virtual display elements within which the retrieved corresponding supplemental information are potentially to be presented to the user by way of the see-through head mounted display if the determined at least one of likely next fields of view becomes a current see-through view;

determining one or more relative positions within the determined at least one of likely next fields of view for the determined virtual display elements to be presented to the user; and

in response to at least one of the identified real objects being present in a current see through view in accordance with one of the at least one of likely next fields of view, rendering at least part of the retrieved corresponding supplemental information in a corresponding one or more of the determined virtual display elements, where the rendered- to virtual display elements are positioned at their respectively determined relative positions within the see-through head mounted display, each rendered-to virtual display element being finally rendered at at least one of a position proximate to its corresponding identified real object that is present in the current see through that field of view and a position corresponding to a cue indicating presence of a scripted part in the live event.

2. The method of claim 1 wherein said rendering of the supplemental information in the see-through head mounted display includes providing information regarding at least one of the identified real objects and/or an action in the live event in the field of view of the user, the rendering including determining a display location for the supplemental information in the head mounted display.

3. The method of claim 1 wherein the step of receiving an indication includes detecting the user'"'"'s location at the live event during the event duration.

4. The method of claim 1 wherein the step of retrieving includes retrieving supplemental information regarding scripted events likely to occur at scripted times in the live event.

5. The method of claim 1 wherein the step of retrieving includes retrieving supplemental information describing real objects likely to be in the field of view of the user at a future portion of the live event.

6. The method of claim 1 wherein the step of rendering the supplemental information includes transmitting the retrieved corresponding supplemental information and display positions for the display elements within a display field in the see-through head mounted display concurrently with allowing see-through viewing of the real objects.

7. The method of claim 1 wherein the method further includes a step of tracking a location of the user and an orientation of the user'"'"'s head.

8. The method of claim 1 further including repeating said retrieving, determining elements, and rendering steps for each movement of the user which changes the user'"'"'s field of view.

9. An apparatus configured for supplementing user enjoyment of a live event having a finite event duration, comprising:

a see-through head mounted display wearable by a user and having a display and a field of view;

a processor and code instructing the processor to perform steps of;

registering attendance of the user at the live event with an information provider;

detecting a current field of view of a real world scene directly viewable by the user and an apparatus position;

identifying real objects in the current field of view for which supplemental information is available during the event duration;

retrieving at least first supplemental information for the display based on the current field of view including supplemental information associated with the identified real objects in the current field of view;

retrieving second supplemental information describing other real objects likely to be in a possible future field of view of the see-through head mounted display at a corresponding future portion of the live event;

determining virtual display elements that are to host display of the first supplemental information and thereby to display the first supplemental information in the display of the head mounted display;

determining positions within the current field of view for the determined virtual display elements that are to be presented to the user the determined positions being configured to allow a see-through view of the real world object associated with supplemental information hosted in the virtual display elements;

responsive to the current field of view of the scene and the apparatus position being unchanged, displaying the first supplemental information in the display within the head mounted display while the user is viewing the real world objects in a see-through viewing manner during the live event with the current field of view of the scene and the apparatus position; and

repeating at least the detecting and displaying steps based on one or more of the following;

upon movement of the user to define a new field of view;

upon a change in an occurrence in the live event to update information about a real object; and

upon a change in the real objects present in the user'"'"'s field of view.

10. The apparatus of claim 9 further including code instructing the processor to retrieve additional supplemental information for the same field of view.

11. The apparatus of claim 10 further including code associating the additional supplemental information with the real objects in the scene of the user.

12. The apparatus of claim 9 wherein the supplemental information is scripted supplemental information regarding a scripted event.

13. The apparatus of claim 9 wherein the supplemental information is unscripted supplemental information provided for a semi-random live event.

14. The apparatus of claim 9 wherein the determining includes providing additional data concerning known occurrences likely to occur at a future time within the live event.

15. The apparatus of claim 9 further including code to provide a user interface to the display to activate a trigger to display the supplemental information when the object associated with the supplemental information is in the user'"'"'s field of view.

16. The apparatus of claim 9 wherein the code determining display elements responds to input from human director.

17. The apparatus of claim 9 wherein the code determining display elements includes rules associating the supplemental information with actions and the objects in the live event.

18. A computational machine-system implemented method of presenting event-supplementing information to a user viewing real objects of a live event with a see-through head mounted display device, the live event having a finite event duration, the method comprising:

for at least one live event having a finite event duration;

providing supplemental information relevant to real objects within the user'"'"'s potential field of view;

receiving an indication from the see-through head mounted display device of the user being in attendance at the live event with access to view the real objects directly in the field of view of the user;

detecting a start of the live event;

monitoring the real objects and actions in the live event for the event duration;

receiving information from the see-through head mounted display device of a position, orientation and see-through field of view of the user during the event duration;

determining real objects in the field of view of the user as being directly viewable by gazing through the see-through head mounted display device during the event duration;

transmitting first supplemental information regarding the real objects and actions in the see-through field of view of the user to the see-through head mounted display device, the first supplemental information being displayable responsive to an event cue in the live event;

transmitting second supplemental information describing real objects likely to be in a future possible see-through field of view of the user at a future portion of the live event;

determining and transmitting positions within the field of view for unobtrusively rendering virtual display elements that are to be presented to the user as hosting the first supplemental information, the determined positions being configured to allow a see-through view of the real world object associated with the first supplemental information in the display element;

repeating the receiving information,

determining, and transmitting based on any of the following;

upon movement of the user to define a new field of view;

upon a change in an occurrence in the live event to update information about a real object; and

upon a change in the real objects in the user'"'"'s field of view.

19. The computer implemented method of claim 18 further including selecting a sub-set of the first supplemental information associated with the real objects in a display field.