PERSONAL AUDIO/VISUAL SYSTEM

• US 20130083003A1
• Filed: 09/30/2011
• Published: 04/04/2013
• Est. Priority Date: 09/30/2011
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Perez, Kathryn Stone, Latta, Stephen G., Vaught, Benjamin I., Geisner, Kevin A., Kipman, Alex Aben-Athar, Karr, Jennifer A., Sugden, Ben J.
• Application Number:
  US13/250,878
• Publication Number:
  US 20130083003A1
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: A63B 24/0062 Monitoring athletic performances, e.g. for determining the work of a user on an exercise apparatus, the completed jogging or cycling distance
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/216 using geographical information, e.g. location of the game device or player using GPS
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/422 automatically for the purpose of assisting the player, e.g. automatic braking in a driving game
• CPC Class Codes: A63F 13/5375 for graphically or textually suggesting an action, e.g. by displaying an arrow indicating a turn in a driving game
• CPC Class Codes: A63F 13/54 involving acoustic signals, e.g. for simulating revolutions per minute [RPM] dependent engine sounds in a driving game or reverberation against a virtual wall
• CPC Class Codes: A63F 13/55 Controlling game characters or game objects based on the game progress
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/79 involving player-related data, e.g. identities, accounts, preferences or play histories
• CPC Class Codes: A63F 2300/203 Image generating hardware
• CPC Class Codes: A63F 2300/301 using an additional display connected to the game console, e.g. on the controller
• CPC Class Codes: A63F 2300/303 for displaying additional data, e.g. simulating a Head Up Display
• CPC Class Codes: A63F 2300/6676 by dedicated player input
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 : Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 : Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0304 : Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 11/20 : Drawing from basic elements, e.g. lines or circles
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/0012 : Biomedical image inspection
• CPC Class Codes: G06T 7/20 : Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 40/20 : Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

The technology described herein incudes a see-through, near-eye, mixed reality display device for providing customized experiences for a user. The system can be used in various entertainment, sports, shopping and theme-park situations to provide a mixed reality experience.

Claims

1. A method for presenting a personalized experience using a personal A/V apparatus, comprising:

automatically determining a three dimensional location of the personal A/V apparatus, the personal A/V apparatus includes one or more sensors and a see-through display;

automatically determining an orientation of the personal A/V apparatus;

automatically determining a gaze of a user looking through the see-through display of the personal A/V apparatus;

automatically determining a three dimensional location of a movable object in the field of view of the user through the see-through display, the determining of the three dimensional location of the movable object is performed using the one or more sensors;

transmitting the three dimensional location of the personal A/V apparatus, the orientation, the gaze and the three dimensional location of the movable object to a server system;

accessing weather data at the server system and automatically determining the effects of weather on the movement of the movable object;

accessing course data at the server system;

accessing the user'"'"'s profile at the server system, the user'"'"'s profile including information about the user'"'"'s skill and past performance;

automatically determining a recommend action on the movable object base on the three dimensional location of the movable object, the weather data and the course data;

automatically adjusting the recommendation based on the user'"'"'s skill and past performance;

transmitting the adjusted recommendation to the personal A/V apparatus; and

displaying the adjusted recommendation in the see-through display of the personal A/V apparatus.

2. The method of claim 1, further comprising:

automatically tracking the movable object after the user performs the adjusted recommendation.

3. The method of claim 2, further comprising:

automatically updating the user'"'"'s profile to include data based on the tracking of the movable object.

4. The method of claim 1, wherein:

the displaying the adjusted recommendation in the see-through display of the personal A/V apparatus includes graphically showing in the see-through display how to address the movable object.

5. The method of claim 4, wherein:

the graphically showing in the see-through display how to address the movable object includes rendering a virtual graphic in the see-through display in relation to the position of the movable object in the see-through display, the user is able to view the real world movable object through the see-through display.

6. The method of claim 1, further comprising:

accessing data of another player playing the same course as the user; and

graphically depicting in the see-through display the another player playing the same course synchronized with the user.

7. The method of claim 1, wherein:

the movable object is a golf ball; and

the adjusted recommendation includes an indication of a club.

8. The method of claim 1, further comprising:

receiving a request for help;

automatically determining the current situation with respect to the user, the course and the movable object;

displaying rules about the current situation in the see-through display; and

displaying an explanation about the current situation in the see-through display.

9. An apparatus for presenting a personalized experience, comprising:

a personal A/V apparatus that includes a see-through, near-eye, augmented reality display that is worn by a user; and

one or more servers in wireless communication with the personal A/V apparatus, the one or more servers automatically determine that the user is within an attraction, the one or more servers access a user profile for the user and identify one or more enhancement packages for the attraction that match parameters of the user profile, the one or more servers filter out enhancement packages that have already been experienced by the user and choose one of the remaining enhancement packages;

the chosen enhancement package is implemented by the personal A/V apparatus sensing data about its location and orientation, the personal A/V apparatus sensing data about gaze of the user, the one or more servers determining a graphic to add to the see-through, near-eye, augmented reality display, and the personal A/V apparatus rendering the determined graphic in the see-through, near-eye, augmented reality display, the one or more servers automatically determine that the user has completed the attraction and terminate the chosen enhancement package in response to determining that the user has completed the attraction.

10. The apparatus of claim 9, wherein:

the one or more servers includes a local Supplemental Information Provider data processing system and a central server that is remote from and in communication with the Supplemental Information Provider; and

the personal A/V apparatus automatically connects to the Supplemental Information Provider when in proximity to the attraction.

11. The apparatus of claim 10, wherein:

the attraction is a ride at an amusement park;

the local Supplemental Information Provider data processing system and the central server are positioned at the amusement park; and

the graphic is themed to the ride.

12. The apparatus of claim 9, further comprising:

one or more sensors positioned in proximity to the one or more servers, the one or more sensors automatically determine that the user is within the attraction.

13. The apparatus of claim 9, wherein:

the one or more servers automatically determine that the personal A/V apparatus is waiting at the attraction and provide content to the user via the personal A/V apparatus while the A/V apparatus is waiting at the attraction, the one or more servers determine that the personal A/V apparatus is no longer waiting at the attraction and then stop providing the content to the user via the personal A/V apparatus, the personal A/V apparatus will pause the content in response to a companion personal A/V apparatus.

14. A method for presenting a personalized experience using a personal A/V apparatus, comprising:

accessing a data structure indicating storage locations and items to be stored at each of the locations;

automatically determining a current location of a personal A/V apparatus;

automatically determining that the current location is a storage location based on the data structure;

identifying items to be stored at the current location based on the data structure;

automatically sensing presence of a plurality of items;

automatically identifying items from the data structure that are missing from the current location based on the automatically sensing presence of the plurality of items and the data structure;

creating a list, adding the items that are missing from the current location to a list, and storing the list;

displaying the list in the personal A/V apparatus; and

ordering the items that are missing from the current location.

15. The method of claim 14, wherein:

the ordering the items that are missing includes the personal A/V apparatus automatically and electronically ordering the items that are missing from one or more online sellers.

16. The method of claim 14, wherein:

automatically sensing presence of the plurality of items includes automatically recognizing one or more items from an image.

17. The method of claim 16, further comprising:

determining the orientation of the personal A/V apparatus; and

determining the gaze of a user of the personal A/V apparatus, the determining that the current location is a storage location includes using the gaze and orientation to determine that the area being viewed by the user is the storage location.

18. The method of claim 14, wherein:

the storage location is a food storage location and the plurality of items are food items; and

the method further comprises automatically identifying that the user has a food allergy to one of the plurality of food items and graphically identifying in a see-through optical system of the personal A/V apparatus the one of the plurality of food items and that the user has a food allergy to the one of the plurality of food items.

19. The method of claim 14 wherein:

the storage location is a food storage location and the plurality of items are food items; and

the method further comprises automatically identifying a recipe for which all of the ingredients are in possession of a user of the personal A/V, graphically identifying a food item in a field of view of the user looking through a see-through optical system of the personal A/V apparatus that is in the recipe and graphically reporting the recipe in the see-through optical system of the personal A/V apparatus.

20. The method of claim 19, further comprising:

automatically identifying that the user has a food allergy to one of the plurality of food items; and

graphically identifying in a see-through optical system of the personal A/V apparatus the one of the plurality of food items and that the user has a food allergy to the one of the plurality of food items, the automatically sensing presence of the plurality of items includes automatically recognizing one or more items from an image.