ENHANCING A SPORT USING AN AUGMENTED REALITY DISPLAY

• US 20130095924A1
• Filed: 09/28/2012
• Published: 04/18/2013
• Est. Priority Date: 09/30/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Geisner, Kevin A., Perez, Kathryn Stone, Latta, Stephen G., Sugden, Ben J., Vaught, Benjamin I., Kipman, Alex Aben-Athar, Clavin, John
• Granted Patent: US 9,345,957 B2
• Time in Patent Office: Days
• US Class Current: 463/32
• CPC Class Codes: A63F 13/21 characterised by their sensors, purposes or types
• CPC Class Codes: A63F 13/80 Special adaptations for executing a specific game genre or game mode
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/204 the platform being a handheld device
• CPC Class Codes: A63F 2300/301 using an additional display connected to the game console, e.g. on the controller
• CPC Class Codes: A63F 2300/6676 by dedicated player input
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09B 19/0038 Sports
• CPC Class Codes: G16H 20/30 relating to physical therapies or activities, e.g. physiotherapy, acupressure or exercising
• CPC Class Codes: G16H 40/63 for local operation
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes

Abstract

Technology is described for providing a personalized sport performance experience with three dimensional (3D) virtual data displayed by a near-eye, augmented reality display of a personal audiovisual (A/V) apparatus. A physical movement recommendation is determined for the user performing a sport based on skills data for the user for the sport, physical characteristics of the user, and 3D space positions for at least one or more sport objects. 3D virtual data depicting one or more visual guides for assisting the user in performing the physical movement recommendation may be displayed from a user perspective associated with a display field of view of the near-eye AR display. An avatar may also be displayed by the near-eye AR display performing a sport. The avatar may perform the sport interactively with the user or be displayed performing a prior performance of an individual represented by the avatar.

Claims

1. A method for providing a personalized sport performance experience with three dimensional (3D) virtual data being displayed by a near-eye, augmented reality (AR) display of a personal audiovisual (A/V) apparatus comprising:

automatically identifying a physical location which the personal A/V apparatus is within based on location data detected by the personal A/V apparatus;

automatically identifying one or more 3D space positions of at least one or more sport objects in a sport performance area associated with the physical location based on a three dimensional mapping of objects in the sport performance area;

accessing a memory for physical characteristics of a user and skills data for a sport stored for the user in user profile data;

determining a physical movement recommendation by a processor for the user performing the sport based on the skills data for the sport, the physical characteristics of the user, and 3D space positions for at least the one or more sport objects; and

displaying three-dimensional (3D) virtual data depicting one or more visual guides for assisting the user in performing the physical movement recommendation from a user perspective associated with a display field of view of the near-eye AR display.

2. The method of claim 1 wherein the one or more visual guides include image data of one or more body parts moving in relation to a sport object for the sport from the user perspective associated with the display field of view.

3. The method of claim 2 wherein the physical characteristics of the user include physical dimensions of the user and a strength rating.

4. The method of claim 1 further comprising:

tracking an actual physical movement of the user in performing the sport based on image and depth data from one or more capture devices in the physical location.

5. The method of claim 4 further comprising:

tracking motion of a real sport object set in motion by the actual physical movement of the user based on sensor data; and

displaying virtual data illustrating the motion of the real sport object.

6. The method of claim 4 further comprising:

displaying 3D virtual data illustrating motion of a virtual sport object responsive to the actual physical movement of the user from the user perspective associated with the display field of view.

7. The method of claim 1 further comprising:

determining the physical movement recommendation by the processor for the user performing the sport based on environmental conditions in the sport performance area.

8. The method of claim 1 wherein the one or more sport objects include a virtual object and a real object.

9. The method of claim 1 further comprising:

displaying a virtual sport performance area for the user performing the sport; and

determining the physical movement recommendation by the processor for the user based on one or more features in the virtual sport performance area.

10. A portable personal audiovisual (A/V) apparatus for providing a personalized sport performance experience with three dimensional (3D) virtual data comprising:

a near-eye, augmented reality display having a display field of view and being supported by a near-eye support structure;

one or more processors having access to memory storing physical characteristics of a user wearing the personal A/V apparatus and skills data for the user for one or more sports;

a natural user interface including at least one image capture device communicatively coupled to the one or more processors and being supported by the near-eye support structure for tracking a physical sport movement being performed by a user;

the one or more processors executing software for determining a physical sport movement recommendation based on a sport being performed, skills data for the user for the sport being played and one or more objects in a sport performance area in which the user is performing the sport; and

the one or more processors controlling the near-eye, augmented reality display of the personal A/V apparatus for displaying 3D virtual data for assisting the user in performing the physical sport movement recommendation from a user perspective associated with the display field of view.

11. The apparatus of claim 10 wherein the near-eye, augmented reality display is an optical see-through, augmented reality display.

12. The apparatus of claim 10 further comprising:

a location sensing unit for determining a physical location of the user based on non-image sensor data.

13. The apparatus of claim 10 further comprising:

the one or more processors executing software for storing sport performance data representing the physical sport movement in user profile data;

the one or more processors executing software for analyzing the sport performance data in accordance with execution criteria for the tracked physical sport movement and environmental conditions of the sport performance area;

assigning a skill level for the physical sport movement based on execution criteria; and

the one or more processors updating skills data for the sport for the user in the memory based on the assigned skill level for the physical sport movement.

14. The apparatus of claim 13 further comprising:

a microphone of the personal A/V apparatus for capturing a sound made in a physical location by a real sport object,

the one or more processors being communicatively coupled to receive audio data representing the sound from the microphone and executing a sound recognition engine for identifying the sound as a sound related to the physical sport movement; and

the one or more processors comparing the sound related to the physical movement against execution sound criteria for assigning a sound factor; and

the one or more processors assigning the skill level for the physical sport movement based on execution criteria which includes the sound factor.

15. The apparatus of claim 13 further comprising:

the one or more processors determining audio parameters for a contact sound for a virtual sport object contacted in the physical sport movement based on the sports performance data for the physical sport movement;

an audio engine for generating the contact sound related to the physical sport movement in accordance with the audio parameters determined for the contact sound;

the one or more processors executing the audio engine for playing the contact sound with the determined audio parameters through at least one audio output device of the personal A/V apparatus.

16. One or more processor readable storage devices comprising instructions which cause one or more processors to execute a method for providing a personalized sport performance experience with three dimensional (3D) virtual data displayed by a near-eye, augmented reality (AR) display of a portable personal audiovisual (A/V) apparatus, the method comprising:

receiving data indicating selection of an avatar for display performing a sport during a sport performance session;

accessing one or more memories by a processor of the personal A/V apparatus for sport performance data for the avatar; and

displaying 3D virtual data of the avatar performing the sport in a context of the sport performance session based on the sport performance data for the avatar from a user perspective associated with a display field of view of the near-eye, (AR) display of the personal A/V apparatus.

17. The one or more processor readable storage devices of claim 16 further comprising:

the avatar represents the user, and the 3D virtual data is a prior sport performance of the user.

18. The one or more processor readable storage devices of claim 16 further comprising:

the avatar represents a person other than the user, and the 3D virtual data is a prior sport performance by the person other than the user.

19. The one or more processor readable storage devices of claim 18 further comprising the performance location of the user is a real physical location for performing the sport.

20. The one or more processor readable storage devices of claim 16 wherein displaying 3D virtual data of the avatar performing the sport from the user perspective further comprises displaying 3D virtual data of the avatar responding to a physical sport movement of the user.