Enhancing a sport using an augmented reality display

• US 9,345,957 B2
• Filed: 09/28/2012
• Issued: 05/24/2016
• Est. Priority Date: 09/30/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Geisner, Kevin A., Perez, Kathryn Stone, Latta, Stephen G., Sugden, Ben J., Vaught, Benjamin I., Kipman, Alex Aben-Athar, Clavin, John
• Primary Examiner(s): Hoang, Phi
• Assistant Examiner(s): Le, Michael
• Application Number:
  US13/631,511
• Publication Number:
  US 20130095924A1
• Time in Patent Office: 1,334 Days
• Field of Search: 345/633, 345/634, 345/419, 345/156, 345/8, 345/103, 345/107, 463/32, 382/103, 382/107
• US Class Current: 1/1
• CPC Class Codes: A63F 13/21 characterised by their sensors, purposes or types
• CPC Class Codes: A63F 13/80 Special adaptations for executing a specific game genre or game mode
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/204 the platform being a handheld device
• CPC Class Codes: A63F 2300/301 using an additional display connected to the game console, e.g. on the controller
• CPC Class Codes: A63F 2300/6676 by dedicated player input
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09B 19/0038 Sports
• CPC Class Codes: G16H 20/30 relating to physical therapies or activities, e.g. physiotherapy, acupressure or exercising
• CPC Class Codes: G16H 40/63 for local operation
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes

Abstract

Technology is described for providing a personalized sport performance experience with three dimensional (3D) virtual data displayed by a near-eye, augmented reality display of a personal audiovisual (A/V) apparatus. A physical movement recommendation is determined for the user performing a sport based on skills data for the user for the sport, physical characteristics of the user, and 3D space positions for at least one or more sport objects. 3D virtual data depicting one or more visual guides for assisting the user in performing the physical movement recommendation may be displayed from a user perspective associated with a display field of view of the near-eye AR display. An avatar may also be displayed by the near-eye AR display performing a sport. The avatar may perform the sport interactively with the user or be displayed performing a prior performance of an individual represented by the avatar.

Claims

1. A method for providing a personalized sport performance experience with three dimensional (3D) virtual data being displayed by a near-eye, augmented reality (AR) display of a personal audiovisual (A/V) apparatus comprising:

identifying a physical location which the personal A/V apparatus is within based on location data detected by the personal A/V apparatus;

identifying one or more 3D space positions of at least one or more sport objects in a sport performance area associated with the physical location based on a three dimensional mapping of objects in the sport performance area;

accessing a memory for physical characteristics including one or more physical dimensions of a user and skills data for a sport stored for the user in user profile data;

determining a physical movement recommendation by a processor for the user performing the sport based on the skills data for the sport, the physical characteristics including one or more physical dimensions of the user, and 3D space positions for at least the one or more sport objects; and

displaying three-dimensional (3D) virtual data depicting one or more visual guides from a displaying three-dimensional (3D) virtual data depicting one or more visual guides from a user perspective in a display field of view of the near-eye AR display, the one or more visual guides including image data of one or more of the user'"'"'s body parts of execution of the physical movement recommendation displayed from the perspective of the user during execution of the physical movement recommendation by the user and which displayed visual guides can be followed by one or more actual body parts of the user by placement over the visual guides in the display field of view during the execution of the physical movement recommendation.

2. The method of claim 1 wherein the one or more visual guides include image data of one or more body parts moving in relation to a sport object for the sport from the user perspective associated with the display field of view.

3. The method of claim 2 wherein the physical characteristics of the user include physical dimensions of the user and a strength rating.

4. The method of claim 1 further comprising:

tracking an actual physical movement of the user in performing the sport based on image and depth data from one or more capture devices in the physical location.

5. The method of claim 4 further comprising:

tracking motion of a real sport object set in motion by the actual physical movement of the user based on sensor data; and

displaying virtual data illustrating the motion of the real sport object.

6. The method of claim 4 further comprising:

displaying 3D virtual data illustrating motion of a virtual sport object responsive to the actual physical movement of the user from the user perspective associated with the display field of view.

7. The method of claim 1 further comprising:

determining the physical movement recommendation by the processor for the user performing the sport based on environmental conditions in the sport performance area.

8. The method of claim 1 wherein the one or more sport objects include a virtual object and a real object.

9. The method of claim 1 further comprising:

displaying a virtual sport performance area for the user performing the sport; and

determining the physical movement recommendation by the processor for the user based on one or more features in the virtual sport performance area.

10. The method of claim 1 wherein the one or more visual guides including image data of a reference model of execution of the physical movement recommendation comprises image data of one or more body parts used during execution of the physical movement recommendation.

11. The method of claim 1 wherein the one or more visual guides including image data of a reference model of execution of the physical movement recommendation comprises image data of one or more of the user'"'"'s own body parts which the user can place his or her one or more actual body parts over and follow during execution of the physical movement recommendation.

12. A portable personal audiovisual (A/V) apparatus for providing a personalized sport performance experience with three dimensional (3D) virtual data comprising:

a near-eye, augmented reality display having a display field of view and being supported by a near-eye support structure;

one or more processors having access to memory storing physical characteristics including one or more physical dimensions of a user wearing the personal A/V apparatus and skills data for the user for one or more sports;

a natural user interface including at least one image capture device communicatively coupled to the one or more processors and being supported by the near-eye support structure for tracking a physical sport movement being performed by a user;

the one or more processors executing software for determining a physical sport movement recommendation having a sport objective score for each physical sport movement based on a probability of success in achieving at least one sport objective performed by the user based on a sport being performed and weighted based on a current context of a session of the performance, the physical characteristics including one or more physical dimensions of the user wearing the personal A/V apparatus, skills data for the user for the sport being played and one or more objects in a sport performance area in which the user is performing the sport; and

the one or more processors controlling the near-eye, augmented reality display of the personal A/V apparatus for displaying three-dimensional (3D) virtual data depicting one or more visual guides from a user perspective in the display field of view of the near-eye AR display, the one or more visual guides including image data of a reference model of execution of the physical sport movement recommendation displayed from the perspective of the user during execution of the physical sport movement recommendation by the user and which displayed visual guides can be followed by one or more actual body parts of the user in the display field of view during the execution of the physical movement recommendation.

13. The apparatus of claim 12 further comprising:

a location sensing unit for determining a physical location of the user based on non-image sensor data.

14. The apparatus of claim 12 further comprising:

the one or more processors executing software for storing sport performance data representing the physical sport movement in user profile data;

the one or more processors executing software for analyzing the sport performance data in accordance with execution criteria for the tracked physical sport movement and environmental conditions of the sport performance area;

assigning a skill level for the physical sport movement based on execution criteria; and

the one or more processors updating skills data for the sport for the user in the memory based on the assigned skill level for the physical sport movement.

15. The apparatus of claim 14 further comprising:

a microphone of the personal A/V apparatus for capturing a sound made in a physical location by a real sport object,

the one or more processors being communicatively coupled to receive audio data representing the sound from the microphone and executing a sound recognition engine for identifying the sound as a sound related to the physical sport movement; and

the one or more processors comparing the sound related to the physical movement against execution sound criteria for assigning a sound factor; and

the one or more processors assigning the skill level for the physical sport movement based on execution criteria which includes the sound factor.

16. The apparatus of claim 14 further comprising:

the one or more processors determining audio parameters for a contact sound for a virtual sport object contacted in the physical sport movement based on the sports performance data for the physical sport movement;

an audio engine for generating the contact sound related to the physical sport movement in accordance with the audio parameters determined for the contact sound;

the one or more processors executing the audio engine for playing the contact sound with the determined audio parameters through at least one audio output device of the personal A/V apparatus.

17. The apparatus of claim 12 wherein the one or more visual guides including image data of a reference model of execution of the physical movement recommendation comprises image data of one or more body parts used during execution of the physical movement recommendation.

18. The apparatus of claim 12, wherein

a sport objective score is assigned for each physical sport movement of the user based on a probability of success in achieving the at least one sport objective,

a user execution score is assigned for each physical sport movement performed by the user based on the skills data, and

the physical sport movement recommendation score for each physical movement performed by the user is based on a respective sport objective score and user execution score.

19. One or more processor readable storage devices comprising instructions which cause one or more processors to execute a method for providing a personalized sport performance experience with three dimensional (3D) virtual data being displayed by a near-eye, augmented reality (AR) display of a personal audiovisual (A/V) apparatus comprising:

identifying a physical location which the personal A/V apparatus is within based on location data detected by the personal A/V apparatus;

identifying one or more 3D space positions of at least one or more sport objects in a sport performance area associated with the physical location based on a three dimensional mapping of objects in the sport performance area;

accessing a memory for physical characteristics of a user and skills data for a sport stored for the user in user profile data;

determining a physical movement recommendation having a physical sport movement recommendation score satisfying a selection criteria based on a highest combined score for a sport objective score and a user execution score by a processor for the user performing the sport based on the skills data for the sport, the physical characteristics including one or more physical dimensions of the user, and 3D space positions for at least the one or more sport objects; and

displaying three-dimensional (3D) virtual data depicting one or more visual guides from a user perspective in a display field of view of the near-eye AR display, the one or more visual guides including image data of one or more of the user'"'"'s body parts of execution of the physical movement recommendation displayed from the perspective of the user during execution of the physical movement recommendation by the user and which displayed visual guides can be followed by one or more actual body parts of the user by placement over the visual guides in the display field of view during the execution of the physical movement recommendation.

20. The one or more processor readable storage devices of claim 19 wherein the one or more visual guides including image data of a reference model of execution of the physical movement recommendation comprises image data of one or more body parts used during execution of the physical movement recommendation.