Exercising application for personal audio/visual system

• US 9,355,583 B2
• Filed: 08/28/2014
• Issued: 05/31/2016
• Est. Priority Date: 09/30/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Geisner, Kevin A., Perez, Kathryn Stone, Latta, Stephen G., Sugden, Ben J., Vaught, Benjamin I., Kipman, Alex Aben-Athar
• Primary Examiner(s): Wu, Xiao
• Assistant Examiner(s): Akhavannik, Mohammad H
• Application Number:
  US14/472,267
• Publication Number:
  US 20150049114A1
• Time in Patent Office: 642 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: A63B 71/00 Games or sports accessories not covered in groups A63B1/00 - A63B69/00 starting appliances A63K3/02
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/216 using geographical information, e.g. location of the game device or player using GPS
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/352 involving special game server arrangements, e.g. regional servers connected to a national server or a plurality of servers managing partitions of the game world
• CPC Class Codes: A63F 13/52 involving aspects of the displayed game scene
• CPC Class Codes: A63F 13/533 for prompting the player, e.g. by displaying a game menu
• CPC Class Codes: A63F 13/5375 for graphically or textually suggesting an action, e.g. by displaying an arrow indicating a turn in a driving game
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 13/79 involving player-related data, e.g. identities, accounts, preferences or play histories
• CPC Class Codes: A63F 13/816 Athletics, e.g. track-and-field sports
• CPC Class Codes: A63F 2300/1012 involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/205 for detecting the geographical location of the game platform
• CPC Class Codes: A63F 2300/208 for storing personal settings or data of the player
• CPC Class Codes: A63F 2300/301 using an additional display connected to the game console, e.g. on the controller
• CPC Class Codes: A63F 2300/305 for providing a graphical or textual hint to the player
• CPC Class Codes: A63F 2300/308 Details of the user interface
• CPC Class Codes: A63F 2300/513 server hierarchy, e.g. local, regional, national or dedicated for different tasks, e.g. authenticating, billing
• CPC Class Codes: A63F 2300/535 : for monitoring, e.g. of user parameters, terminal parameters, application parameters, network parameters
• CPC Class Codes: A63F 2300/5553 : user representation in the game field, e.g. avatar
• CPC Class Codes: A63F 2300/5573 : player location
• CPC Class Codes: A63F 2300/66 : for rendering three dimensional images
• CPC Class Codes: A63F 2300/695 : Imported photos, e.g. of the player
• CPC Class Codes: A63F 2300/8005 : Athletics
• CPC Class Codes: A63F 2300/8082 : Virtual reality
• CPC Class Codes: G02B 2027/014 : comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 : Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 : slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0101 : characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G02B 27/017 : Head mounted
• CPC Class Codes: G02B 27/0172 : characterised by optical features
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 2320/068 : for control of viewing angle adjustment
• CPC Class Codes: G09G 2320/0693 : Calibration of display systems
• CPC Class Codes: G09G 2380/00 : Specific applications
• CPC Class Codes: G09G 3/003 : to produce spatial visual effects
• CPC Class Codes: G16H 20/30 : relating to physical therapies or activities, e.g. physiotherapy, acupressure or exercising

Abstract

The technology described herein includes a see-through, near-eye, mixed reality display device for providing customized experiences for a user. The personal A/V apparatus serves as an exercise program that is always with the user, provides motivation for the user, visually tells the user how to exercise, and lets the user exercise with other people who are not present.

Claims

1. A method for presenting a personalized experience using a personal see-through A/V apparatus, comprising:

accessing a first exercise routine for a first person;

accessing data for a second person for a second exercise routine different from the first exercise routine, wherein the second exercise routine is associated with a location that is remote from the personal see-through A/V apparatus;

estimating a performance of how the second person would perform the first exercise routine based on the data; and

presenting a virtual image of someone performing the first exercise routine based on the estimated performance so that the first person can see the virtual image inserted into a real scene viewed through the personal see-through A/V apparatus as the first person performs the first exercise routine.

2. The method of claim 1, wherein:

the estimated performance is based on past performance of the second exercise routine.

3. The method of claim 1, wherein:

the estimated performance is based on a live performance of the second exercise routine.

4. The method of claim 1, wherein the presenting a virtual image of someone performing the first exercise routine based on the estimated performance includes:

presenting an avatar of the second person that integrates the second person into an environment of the first person.

5. The method of claim 4, wherein the second person is exercising at the remote location from the personal see-through A/V apparatus.

6. The method of claim 1, wherein the accessing data for the second person for the second exercise routine different from the first exercise routine includes:

accessing real time exercise data for the second person at the location that is remote from the personal see-through A/V apparatus.

7. The method of claim 6, wherein the presenting a virtual image of someone performing the first exercise routine based on the estimated performance includes:

presenting the virtual image based on the real time exercise data so that the first person can see a representation of the second person appear to exercise in the same 3D coordinate system with the first person.

8. The method of claim 1, wherein the presenting the virtual image of someone performing the first exercise routine based on the estimated performance includes:

rendering an avatar representing the second person'"'"'s performance synchronized in time and space with the first person, the rendering being performed on the personal see-through A/V apparatus so that the first person can see the avatar inserted into a real scene viewed through the personal see-through A/V apparatus.

9. A system, comprising:

a see-through, near-eye, augmented reality display;

a sensor;

processing logic in communication with the sensor and the augmented reality display, wherein the processing logic is configured to;

access a first exercise routine associated with a first location of the see-through, near-eye, augmented reality display, wherein a first person is associated with the see-through, near-eye, augmented reality display;

access data for a second person for a second exercise routine different from the first exercise routine, the second exercise routine associated with a second location that is remote from the first location;

estimate a performance of how the second person would perform the first exercise routine based on the data; and

present a virtual image of someone performing the first exercise routine based on the estimated performance so that the first person can see the virtual image inserted into a real scene viewed through the augmented reality display when the see-through, near-eye, augmented reality display is at the first location associated with the first exercise routine.

10. The system of claim 9, wherein the processing logic is configured to receive as input the first exercise routine.

11. The system of claim 9, wherein the first exercise routine is a first route and the second exercise routine is a second route.

12. The system of claim 9, wherein the processing logic is configured to estimate the performance based on a live performance of the second exercise routine by the second person.

13. The system of claim 9, wherein the processing logic is configured to:

access real time exercise data for the second person at the second location that is remote from first location; and

estimate the performance of how the second person would perform the first exercise routine based on the real time exercise data.

14. The system of claim 9, wherein the processing logic comprises a server in communication with the see-through, near-eye, augmented reality display.

15. A personal see-through A/V apparatus, comprising:

a see-through, near-eye, augmented reality display;

a sensor; and

a processor in communication with the sensor and the augmented reality display, wherein the processor is configured to;

access a first course of action for a first person, wherein the first course of action is for a present location of the personal see-through A/V apparatus;

access data for a second person for a second course of action different from the first course of action, the second course of action is for a remote physical location from the personal see-through A/V apparatus;

estimate a performance of how the second person would perform the first course of action based on the accessed data; and

present a virtual image of someone performing the first course of action based on the estimated performance so that the first person can see the virtual image inserted into a real scene viewed through the personal see-through A/V apparatus when the personal see-through A/V apparatus is at the present location.

16. The personal see-through A/V apparatus of claim 15, wherein the processor is configured to estimate the performance based on a live performance of the second course of action.

17. The personal see-through A/V apparatus of claim 16, wherein the processor is configured to present the virtual image based on the live performance so that the first person can see a representation of the second person appear to exercise in the same 3D coordinate system with the first person.

18. The personal see-through A/V apparatus of claim 15, wherein the processor is configured to render an image representing the second person'"'"'s performance synchronized in time and space with the first person, wherein the processor is configured to render on the personal see-through A/V apparatus so that the first person can see the virtual image inserted into a real scene viewed through the personal see through A/V apparatus.

19. The personal see-through A/V apparatus of claim 15, wherein the virtual image is based on real time video data.

20. The personal see-through A/V apparatus of claim 15, wherein to estimate the performance the processor is configured to:

access 3D positional data for the second person; and

determine speed and direction from the 3D positional data.