Sport training on augmented/virtual reality devices by measuring hand-eye coordination-based measurements

• US 10,617,933 B2
• Filed: 08/14/2017
• Issued: 04/14/2020
• Est. Priority Date: 08/14/2017
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Vijil, Enara C., Banerjee, Dipyaman, Dey, Kuntal
• Primary Examiner(s): Liddle, Jay Trent
• Application Number:
  US15/676,499
• Publication Number:
  US 20190046859A1
• Time in Patent Office: 974 Days
• CPC Class Codes: A63B 2024/0012 Comparing movements or motion sequences with a registered reference
• CPC Class Codes: A63B 2071/063 Spoken or verbal instructions
• CPC Class Codes: A63B 2071/0666 worn on the head or face, e.g. combined with goggles or glasses
• CPC Class Codes: A63B 2102/16 Table tennis
• CPC Class Codes: A63B 2220/40 Acceleration
• CPC Class Codes: A63B 2220/803 Motion sensors
• CPC Class Codes: A63B 2220/805 Optical or opto-electronic sensors
• CPC Class Codes: A63B 2220/836 Sensors arranged on the body of the user
• CPC Class Codes: A63B 2230/06 heartbeat rate only
• CPC Class Codes: A63B 24/0006 Computerised comparison for qualitative assessment of motion sequences or the course of a movement
• CPC Class Codes: A63B 69/0015 for cricket
• CPC Class Codes: A63B 69/0017 for badminton
• CPC Class Codes: A63B 69/38 for tennis A63B61/006, A63B69/0073 and A63B69/0097 take precedence
• CPC Class Codes: A63B 71/0622 Visual, audio or audio-visual systems for entertaining, instructing or motivating the user
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G09B 19/0038 Sports
• CPC Class Codes: G09B 5/06 with both visual and audible presentation of the material to be studied
• CPC Class Codes: G09B 9/00 Simulators for teaching or training purposes for the use of weapons F41; computing aspects G06; protocols for games, networked simulations or virtual reality H04L67/131

Abstract

A method, computer system, and computer program product for sport training on an augmented reality device or a virtual reality device is provided. The embodiment may include capturing a plurality of user movement data using one or more sensors. The embodiment may also include measuring a user body and eye gaze position based on the plurality of captured user movement data. The embodiment may further include calculating a body position difference by comparing the measured user body and eye gaze position with an expert-specified body position sequence. The embodiment may also include determining a body position quality threshold is not satisfied based on the calculated body position difference. The embodiment may further include generating an instruction based on the compared calculated body position difference.

Claims

1. A processor-implemented method for sport training on an augmented reality device or a virtual reality device, the method comprising:

capturing, by a processor, a plurality of user movement data using one or more sensors while a user engages in a simulation of a physical activity using the augmented reality device or the virtual reality device, wherein the plurality of captured user movement data comprises user hand position data, user eye gaze tracking data, user head position data, user arm position data, user leg position data, and user hip position data;

measuring a user body and eye gaze position based on the plurality of captured user movement data;

calculating a body position difference by comparing the measured user body and eye gaze position with a preconfigured body position sequence, wherein the calculated body position difference comprises a numerical value for each portion of the user body that varies from the preconfigured body position sequence;

determining a body position quality threshold is not satisfied based on the calculated body position difference;

generating an instruction based on the compared calculated body position difference, wherein the generated instruction informs the user that the plurality of captured user movement data differs from the preconfigured body position sequence and an aggregate value of the difference between different body positions; and

displaying the generated instruction as a real-time written message on a display screen of the augmented reality device or the virtual reality device.

2. The method of claim 1, wherein determining the body position quality threshold is not satisfied further comprises:

incrementing a counter by one unit.

3. The method of claim 2, further comprising:

determining an observation cycle threshold is satisfied based on the incremented counter.

4. The method of claim 1, further comprising:

presenting the generated instruction to the user.

5. The method of claim 4, wherein the generated instruction is a verbal instruction presented to the user through a speaker with a corresponding written instruction displayed on a graphical user interface.

6. The method of claim 1, wherein the one or more sensors is selected from a group consisting of a gyroscope, an accelerometer, a magnetometer, an eye gaze tracker, or a heart rate monitor.

7. The method of claim 1, further comprising:

filtering the generated instruction based on a plurality of user preferences.

8. The method of claim 1, wherein calculating the body position difference further comprises:

calculating a series of correlations, wherein the series of correlations are a correlation of a plurality of user eye gaze positions and a plurality of expert eye gaze positions, a correlation of a plurality of user hand and body movements and a plurality of expert hand and body movements, and a correlation of a plurality of expert hand and body movements and a plurality of expert eye gaze positions with a plurality of user hand and body movements and a plurality of user eye gaze positions.

9. A computer system for sport training on an augmented reality device or a virtual reality device, the computer system comprising:

one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage media, and program instructions stored on at least one of the one or more tangible storage media for execution by at least one of the one or more processors via at least one of the one or more memories, wherein the computer system is capable of performing a method comprising;

capturing a plurality of user movement data using one or more sensors while a user engages in a simulation of a physical activity using the augmented reality device or the virtual reality device, wherein the plurality of captured user movement data comprises user hand position data, user eye gaze tracking data, user head position data, user arm position data, user leg position data, and user hip position data;

measuring a user body and eye gaze position based on the plurality of captured user movement data;

calculating a body position difference by comparing the measured user body and eye gaze position with a preconfigured body position sequence, wherein the calculated body position difference comprises a numerical value for each portion of the user body that varies from the preconfigured body position sequence;

determining a body position quality threshold is not satisfied based on the calculated body position difference;

generating an instruction based on the compared calculated body position difference, wherein the generated instruction informs the user that the plurality of captured user movement data differs from the preconfigured body position sequence and an aggregate value of the difference between different body positions; and

displaying the generated instruction as a real-time written message on a display screen of the augmented reality device or the virtual reality device.

10. The computer system of claim 9, wherein determining the body position quality threshold is not satisfied further comprises:

incrementing a counter by one unit.

11. The computer system of claim 10, further comprising:

determining an observation cycle threshold is satisfied based on the incremented counter.

12. The computer system of claim 9, further comprising:

presenting the generated instruction to the user.

13. The computer system of claim 12, wherein the generated instruction is a verbal instruction presented to the user through a speaker with a corresponding written instruction displayed on a graphical user interface.

14. The computer system of claim 9, wherein the one or more sensors is selected from a group consisting of a gyroscope, an accelerometer, a magnetometer, an eye gaze tracker, or a heart rate monitor.

15. The computer system of claim 9, further comprising:

filtering the generated instruction based on a plurality of user preferences.

16. The computer system of claim 9, wherein calculating the body position difference further comprises:

calculating a series of correlations, wherein the series of correlations are a correlation of a plurality of user eye gaze positions and a plurality of expert eye gaze positions, a correlation of a plurality of user hand and body movements and a plurality of expert hand and body movements, and a correlation of a plurality of expert hand and body movements and a plurality of expert eye gaze positions with a plurality of user hand and body movements and a plurality of user eye gaze positions.

17. A computer program product for sport training on an augmented reality device or a virtual reality device, the computer program product comprising:

one or more non-transitory computer-readable tangible storage media and program instructions stored on at least one of the one or more non-transitory tangible storage media, the program instructions executable by a processor of a computer to perform a method, the method comprising;

capturing a plurality of user movement data using one or more sensors while a user engages in a simulation of a physical activity using the augmented reality device or the virtual reality device, wherein the plurality of captured user movement data comprises user hand position data, user eye gaze tracking data, user head position data, user arm position data, user leg position data, and user hip position data;

measuring a user body and eye gaze position based on the plurality of captured user movement data;

calculating a body position difference by comparing the measured user body and eye gaze position with a preconfigured body position sequence, wherein the calculated body position difference comprises a numerical value for each portion of the user body that varies from the preconfigured body position sequence;

determining a body position quality threshold is not satisfied based on the calculated body position difference;

generating an instruction based on the compared calculated body position difference, wherein the generated instruction informs the user that the plurality of captured user movement data differs from the preconfigured body position sequence and an aggregate value of the difference between different body positions; and

displaying the generated instruction as a real-time written message on a display screen of the augmented reality device or the virtual reality device.

18. The computer program product of claim 17, wherein determining the body position quality threshold is not satisfied further comprises:

incrementing a counter by one unit.

19. The computer program product of claim 18, further comprising:

determining an observation cycle threshold is satisfied based on the incremented counter.

20. The computer program product of claim 17, further comprising:

presenting the generated instruction to the user.