Exercising applications for personal audio/visual system

• US 8,847,988 B2
• Filed: 03/30/2012
• Issued: 09/30/2014
• Est. Priority Date: 09/30/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Geisner, Kevin A., Perez, Kathryn Stone, Latta, Stephen G., Sugden, Ben J., Vaught, Benjamin I., Kipman, Alex Aben-Athar
• Primary Examiner(s): Hoang, Phi
• Assistant Examiner(s): Akhavannik, Mohammad H
• Application Number:
  US13/436,499
• Publication Number:
  US 20130083009A1
• Time in Patent Office: 914 Days
• Field of Search: 345/633
• US Class Current: 345/633
• CPC Class Codes: A63B 71/00 Games or sports accessories not covered in groups A63B1/00 - A63B69/00 starting appliances A63K3/02
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/216 using geographical information, e.g. location of the game device or player using GPS
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/352 involving special game server arrangements, e.g. regional servers connected to a national server or a plurality of servers managing partitions of the game world
• CPC Class Codes: A63F 13/52 involving aspects of the displayed game scene
• CPC Class Codes: A63F 13/533 for prompting the player, e.g. by displaying a game menu
• CPC Class Codes: A63F 13/5375 for graphically or textually suggesting an action, e.g. by displaying an arrow indicating a turn in a driving game
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 13/79 involving player-related data, e.g. identities, accounts, preferences or play histories
• CPC Class Codes: A63F 13/816 Athletics, e.g. track-and-field sports
• CPC Class Codes: A63F 2300/1012 involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/205 for detecting the geographical location of the game platform
• CPC Class Codes: A63F 2300/208 for storing personal settings or data of the player
• CPC Class Codes: A63F 2300/301 using an additional display connected to the game console, e.g. on the controller
• CPC Class Codes: A63F 2300/305 for providing a graphical or textual hint to the player
• CPC Class Codes: A63F 2300/308 Details of the user interface
• CPC Class Codes: A63F 2300/513 server hierarchy, e.g. local, regional, national or dedicated for different tasks, e.g. authenticating, billing
• CPC Class Codes: A63F 2300/535 : for monitoring, e.g. of user parameters, terminal parameters, application parameters, network parameters
• CPC Class Codes: A63F 2300/5553 : user representation in the game field, e.g. avatar
• CPC Class Codes: A63F 2300/5573 : player location
• CPC Class Codes: A63F 2300/66 : for rendering three dimensional images
• CPC Class Codes: A63F 2300/695 : Imported photos, e.g. of the player
• CPC Class Codes: A63F 2300/8005 : Athletics
• CPC Class Codes: A63F 2300/8082 : Virtual reality
• CPC Class Codes: G02B 2027/014 : comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 : Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 : slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0101 : characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G02B 27/017 : Head mounted
• CPC Class Codes: G02B 27/0172 : characterised by optical features
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 2320/068 : for control of viewing angle adjustment
• CPC Class Codes: G09G 2320/0693 : Calibration of display systems
• CPC Class Codes: G09G 2380/00 : Specific applications
• CPC Class Codes: G09G 3/003 : to produce spatial visual effects
• CPC Class Codes: G16H 20/30 : relating to physical therapies or activities, e.g. physiotherapy, acupressure or exercising

Abstract

The technology described herein includes a see-through, near-eye, mixed reality display device for providing customized experiences for a user. The personal A/V apparatus serves as an exercise program that is always with the user, provides motivation for the user, visually tells the user how to exercise, and lets the user exercise with other people who are not present.

Claims

1. A method for presenting a personalized experience using a personal see-through A/V apparatus, comprising:

accessing a location of the personal see-through A/V apparatus;

automatically determining a first exercise routine for a user based on the location;

accessing data for another person for a second exercise routine different from the first exercise routine;

estimating a hypothetical performance of how the other person would perform the first exercise routine based on the data; and

presenting a virtual image in the personal see-through A/V apparatus based on the first exercise routine, including presenting a virtual image of someone performing the first exercise routine based on the hypothetical performance so that the user can see the virtual image inserted into a real scene viewed through the personal see-through A/V apparatus as the user performs the first exercise routine.

2. The method of claim 1, wherein:

the hypothetical performance is based on past performance of the second exercise routine.

3. The method of claim 1, wherein the presenting a virtual image in the personal see-through A/V apparatus based on the first exercise routine includes:

augmenting scenery on a route of the first exercise routine so that the user can see additional scenery inserted into real scenery viewed through the personal see-through A/V apparatus.

4. The method of claim 1, further comprising recording data for user actions for the user who is wearing the personal see-through A/V apparatus for a period of time in which the user is not exercising, wherein:

the automatically determining a first exercise routine for a user further includes;

accessing a fitness goal for the user for the period of time including the time during which the user actions were recorded;

determining the exercise routine based on the recorded user actions for the user to meet the fitness goal.

5. The method of claim 4, wherein the recording data for the user wearing the personal see-through A/V apparatus for a period of time in which the user is not exercising includes:

tracking physical activity of the user for at least a portion of the period of time.

6. The method of claim 4, wherein the recording data for the user wearing the personal see-through A/V apparatus for a period of time in which the user is not exercising includes:

tracking biometric data for the user for at least a portion of the period of time.

7. The method of claim 1,

wherein the accessing data for another person for a second exercise routine different from the first exercise routine includes accessing real time exercise data for the other person at a location that is remote from the personal see-through A/V apparatus, wherein:

the presenting a virtual image in the personal see-through A/V apparatus includes presenting a virtual image based on the exercise data so that the user wearing the personal see-through A/V apparatus can see a representation of the other person appear to exercise in the same 3D coordinate system with the user.

8. The method of claim 1, wherein presenting a virtual image in the personal see-through A/V apparatus based on the first exercise routine includes rendering an image representing a person indicating the person'"'"'s performance synchronized in time and space with the user, the rendering being performed on the personal see-through A/V apparatus so that the user can see the virtual image inserted into a real scene viewed through the personal see through A/V apparatus.

9. A method for presenting an experience using a see-through A/V apparatus, comprising:

automatically determining a three dimensional position of the see-through A/V apparatus, the see through A/V apparatus allows a user wearing the apparatus to view a real scene;

determining a first course of action based on the determined three dimensional position;

identifying data for another user for a second course of action that is different from the first course of action;

estimating a hypothetical performance of how the other user would perform on the first course of action based on the data for the other user for the second course of action; and

rendering an image representing the other user indicating the other user'"'"'s hypothetical performance at the same time and three dimensional location as the wearer, the rendering being performed on the see-through A/V apparatus so that the wearer can see the image inserted as a virtual image into the real scene.

10. The method of claim 9, wherein the identifying data for another user includes accesses data for a past performance of the second course of action.

11. The method of claim 9, wherein the identifying data for another user includes accessing real time data for someone at a location that is remote from the see-through A/V apparatus.

12. The method of claim 9, further comprising:

tracking the user with a camera to generate camera data;

analyzing the camera data; and

providing feedback to the user in the A/V apparatus based on the analysis of the camera data.

13. A personal see-through A/V apparatus, comprising:

a see-through, near-eye, augmented reality display that is worn by a user;

one or more sensors;

processing logic in communication with the one or more sensors and the augmented reality display, the processing logic;

accesses a location of the personal see-through A/V apparatus;

automatically determines a first exercise routine for the user based on the location;

accesses data for another person for a second exercise routine different from the first exercise routine;

estimates a hypothetical performance of how the other person would perform the first exercise routine based on the data; and

presents a virtual image in the see-through, near-eye, augmented reality display based on the first exercise routine, the processing logic presents a virtual image of someone performing the first exercise routine based on the hypothetical performance so that the user can see the virtual image inserted into a real scene viewed through the see-through, near-eye, augmented reality display as the user performs the first exercise routine.

14. The apparatus of claim 13, wherein the hypothetical performance is based on past performance of the second exercise routine.

15. The apparatus of claim 13, wherein when the processing logic accesses data for another person for the second exercise routine different from the first exercise routine the processing logic accesses real time exercise data for the other person at a location that is remote from the personal see-through A/V apparatus, wherein:

when the processing logic presents the virtual image in the personal see-through A/V apparatus the processing logic presents a virtual image based on the exercise data so that the user wearing the personal see-through A/V apparatus can see a representation of the other person appear to exercise in the same 3D coordinate system with the user.

16. The apparatus of claim 13, wherein when the processing logic presents a virtual image in the personal see-through A/V apparatus based on the first exercise routine the processing logic renders an image representing a person indicating the person'"'"'s performance synchronized in time and space with the user, the rendering being performed in the see-through, near-eye, augmented reality display so that the user can see the virtual image inserted into a real scene viewed through the see-through, near-eye, augmented reality display.

17. The apparatus of claim 13, wherein the processing logic:

tracks actions of the user while the user is not exercising;

accesses an activity goal for the user for a period of time including the time during which the user actions were tracked;

determines an exercise routine for the user based on the tracked user actions for the user to meet the activity goal;

presents a signal in the see-through, near-eye, augmented reality display based on the exercise routine to help the user meet the activity goal.

18. The apparatus of claim 17, wherein when the processing logic presents a signal in the see-through, near-eye, augmented reality display based on the exercise routine to help the user meet the activity goal, the processing logic:

renders a virtual object in the see-through, near-eye, augmented reality display such that it appears at a different location along an exercise route each time the user takes the exercise route, and challenges the user to find the virtual object.

19. The apparatus of claim 13, wherein the processing logic is further configured to determine the exercise routine for the user based, at least in part, on the location.

20. The apparatus of claim 13, wherein the processing logic includes:

one or more servers in communication with the see-through, near-eye, augmented reality display.