Interactive exercise and training system and method

• US 10,705,597 B1
• Filed: 12/17/2019
• Issued: 07/07/2020
• Est. Priority Date: 12/17/2019
• Status: Active Grant

• Current Assignee: Liteboxer Technologies, Inc.
• Original Assignee: Liteboxer Technologies, Inc.
• Inventors: Morin, Jeffrey W., Rollins, Andrew J.
• Primary Examiner(s): Eurice, Michael J
• Application Number:
  US16/717,588
• Time in Patent Office: 203 Days
• CPC Class Codes: A63B 2024/0096 using performance related parameters for controlling electronic or video games or avatars
• CPC Class Codes: A63B 2071/0625 Emitting sound, noise or music
• CPC Class Codes: A63B 2071/0636 3D visualisation
• CPC Class Codes: A63B 2071/0655 Tactile feedback
• CPC Class Codes: A63B 2071/0666 worn on the head or face, e.g. combined with goggles or glasses
• CPC Class Codes: A63B 2071/0694 Visual indication, e.g. Indicia
• CPC Class Codes: A63B 2220/20 Distances or displacements
• CPC Class Codes: A63B 2220/40 Acceleration
• CPC Class Codes: A63B 2220/80 Special sensors, transducers or devices therefor
• CPC Class Codes: A63B 2220/806 Video cameras
• CPC Class Codes: A63B 2220/836 Sensors arranged on the body of the user
• CPC Class Codes: A63B 2225/02 Testing, calibrating or measuring of equipment
• CPC Class Codes: A63B 2244/102 Boxing
• CPC Class Codes: A63B 24/0003 Analysing the course of a movement or motion sequences during an exercise or trainings sequence, e.g. swing for golf or tennis
• CPC Class Codes: A63B 24/0062 Monitoring athletic performances, e.g. for determining the work of a user on an exercise apparatus, the completed jogging or cycling distance
• CPC Class Codes: A63B 69/32 with indicating devices
• CPC Class Codes: A63B 69/322 to instruct user where to strike
• CPC Class Codes: A63B 71/0622 Visual, audio or audio-visual systems for entertaining, instructing or motivating the user
• CPC Class Codes: A63B 71/0669 Score-keepers or score display devices
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/212 : using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/213 : comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/428 : involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/833 : Hand-to-hand fighting, e.g. martial arts competition A63F13/837 takes precedence
• CPC Class Codes: G02B 2027/0187 : slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 : Head mounted
• CPC Class Codes: G02B 27/0172 : characterised by optical features
• CPC Class Codes: G02B 27/0179 : Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G06F 3/011 : Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 : Head tracking input arrangements
• CPC Class Codes: G06F 3/014 : Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/0304 : Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 : with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 : Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A virtual reality (VR) system comprising a head-mounted display (HMD) and handheld controller set is enhanced to provide a more realistic end user VR experience, e.g., for boxing or other interactive training. In this approach, and in lieu of simply establishing a boundary area for the VR experience, the user also maps a position of a real-world object into a reference frame of the VR environment. This mapping is facilitated using the handheld controller itself, e.g., as positioned in a backwards-facing manner on the user'"'"'s forearm. The real-world object is then simulated in the 3D VR environment as rendered by the VR HMD, and the user interacts with the real-world object (or its simulation) to provide a more enjoyable and useful interactive experience.

Claims

1. A method of spatially projecting a real-world object into a three-dimensional (3D) virtual space, comprising:

receiving a first control signal from a controller positioned on a user'"'"'s appendage, the controller having an associated virtual reality (VR) head-mounted display (HMD);

based on the first control signal, calibrating the controller to identify an endpoint position of the user'"'"'s appendage with respect to a reference frame in a VR environment;

using the controller as calibrated, receiving one or more second control signals mapping given points of the real-world object surface to points in the reference frame of the VR environment, the given points comprising a surface of the real-world object;

projecting the surface into the 3D virtual space rendered by the VR HMD; and

thereafter, and responsive to interaction between the controller endpoint position and the surface, tracking a user'"'"'s simulated interaction with the real-world object;

wherein the second control signals are generated by touching the controller with respect to first, second and third distinct points on the surface, and storing the location of each such distinct point.

2. The method as described in claim 1 wherein the first, second and third distinct points are traced as a triangle along a plane corresponding to the surface.

3. The method as described in claim 1 wherein the first control signal is received from one of:

the controller backwards-positioned on the user'"'"'s forearm, and the controller positioned on the user'"'"'s leg.

4. The method as described in claim 1 wherein calibrating the controller to identify the endpoint position comprises determining a distance from the controller to a fixed physical location over a set of frames, identifying a frame with a lowest distance, and setting that distance at the endpoint position.

5. The method as described in claim 1 wherein the spatially-projected 3D object is overlaid onto real-time video of an associated operating environment.

6. A method of spatially projecting a real-world object into a three-dimensional (3D) virtual space, comprising:

receiving a first control signal from a controller positioned on a user'"'"'s appendage, the controller having an associated virtual reality (VR) head-mounted display (HMD);

based on the first control signal, calibrating the controller to identify an endpoint position of the user'"'"'s appendage with respect to a reference frame in a VR environment;

using the controller as calibrated, receiving one or more second control signals mapping given points of the real-world object surface to points in the reference frame of the VR environment, the given points comprising a surface of the real-world object;

projecting the surface into the 3D virtual space rendered by the VR HMD; and

thereafter, and responsive to interaction between the controller endpoint position and the surface, tracking a user'"'"'s simulated interaction with the real-world object;

wherein the real-world object is a punching surface or a wall.

7. The method as described in claim 6 wherein the first control signal is received from one of:

the controller backwards-positioned on the user'"'"'s forearm, and the controller positioned on the user'"'"'s leg.

8. The method as described in claim 6 wherein calibrating the controller to identify the endpoint position comprises determining a distance from the controller to a fixed physical location over a set of frames, identifying a frame with a lowest distance, and setting that distance at the endpoint position.

9. The method as described in claim 6 wherein the spatially-projected 3D object is overlaid onto real-time video of an associated operating environment.

10. The method as described in claim 6 wherein the punching surface comprises a set of individual targets.

11. The method as described in claim 10 wherein the set of individual targets are configured in a starburst arrangement around a center point.

12. A method of spatially projecting a real-world object into a three-dimensional (3D) virtual space, comprising:

receiving a first control signal from a controller positioned on a user'"'"'s appendage, the controller having an associated virtual reality (VR) head-mounted display (HMD);

based on the first control signal, calibrating the controller to identify an endpoint position of the user'"'"'s appendage with respect to a reference frame in a VR environment;

using the controller as calibrated, receiving one or more second control signals mapping given points of the real-world object surface to points in the reference frame of the VR environment, the given points comprising a surface of the real-world object;

projecting the surface into the 3D virtual space rendered by the VR HMD;

simulating a virtual opponent within the VR environment, the opponent holding an element configured for striking, the element aligned with the surface; and

thereafter, and responsive to interaction between the controller endpoint position and the surface, tracking a user'"'"'s simulated interaction with the real-world object.

13. The method as described in claim 12 wherein the first control signal is received from one of:

the controller backwards-positioned on the user'"'"'s forearm, and the controller positioned on the user'"'"'s leg.

14. The method as described in claim 12 wherein calibrating the controller to identify the endpoint position comprises determining a distance from the controller to a fixed physical location over a set of frames, identifying a frame with a lowest distance, and setting that distance at the endpoint position.

15. The method as described in claim 12 wherein the spatially-projected 3D object is overlaid onto real-time video of an associated operating environment.

16. A method of spatially projecting a real-world object into a three-dimensional (3D) virtual space, comprising:

receiving a first control signal from a controller positioned on a user'"'"'s appendage, the controller having an associated virtual reality (VR) head-mounted display (HMD);

based on the first control signal, calibrating the controller to identify an endpoint position of the user'"'"'s appendage with respect to a reference frame in a VR environment;

receiving additional control signals defining a boundary area for the VR environment;

using the controller as calibrated, receiving one or more second control signals mapping given points of the real-world object surface to points in the reference frame of the VR environment, the given points comprising a surface of the real-world object;

projecting the surface into the 3D virtual space rendered by the VR HMD; and

thereafter, and responsive to interaction between the controller endpoint position and the surface, tracking a user'"'"'s simulated interaction with the real-world object.

17. The method as described in claim 16 wherein the user'"'"'s simulated interaction with the real-world object occurs within the boundary area.

18. The method as described in claim 16 wherein the first control signal is received from one of:

the controller backwards-positioned on the user'"'"'s forearm, and the controller positioned on the user'"'"'s leg.

19. The method as described in claim 16 wherein calibrating the controller to identify the endpoint position comprises determining a distance from the controller to a fixed physical location over a set of frames, identifying a frame with a lowest distance, and setting that distance at the endpoint position.

20. The method as described in claim 16 wherein the spatially-projected 3D object is overlaid onto real-time video of an associated operating environment.