Motion mirroring system that incorporates virtual environment constraints

• US 10,786,728 B2
• Filed: 05/23/2017
• Issued: 09/29/2020
• Est. Priority Date: 05/23/2017
• Status: Active Grant

• Current Assignee: Newlight Capital, LLC
• Original Assignee: Blast Motion Incorporated
• Inventors: Haas, Juergen, Bose, Bhaskar, Bentley, Michael, Kaps, Ryan
• Primary Examiner(s): Liddle, Jay Trent
• Assistant Examiner(s): Hsu, Ryan
• Application Number:
  US15/602,853
• Publication Number:
  US 20180339223A1
• Time in Patent Office: 1,225 Days
• Field of Search: 463 3, 463 2, 473151
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/235 using a wireless connection, e.g. infrared or piconet
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/525 Changing parameters of virtual cameras
• CPC Class Codes: A63F 13/812 Ball games, e.g. soccer or baseball
• CPC Class Codes: A63F 13/833 Hand-to-hand fighting, e.g. martial arts competition A63F13/837 takes precedence
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A system that mirrors motion of a physical object by displaying a virtual object moving in a virtual environment. The mirroring display may be used for example for feedback, coaching, or for playing virtual games. Motion of the physical object is measured by motion sensors that may for example include an accelerometer, a gyroscope, and a magnetometer. Sensor data is transmitted to a computer that calculates the position and orientation of the physical object and generates a corresponding position and orientation of the virtual object. The computer may correct or adjust the calculations using sensor data redundancies. The virtual environment may include constraints on the position, orientation, or motion of the virtual object. These constraints may be used to compensate for accumulating errors in position and orientation. The system may for example use proportional error feedback to adjust position and orientation based on sensor redundancies and virtual environment constraints.

Claims

1. A motion mirroring system that incorporates virtual environment constraints, comprising

a motion capture element configured to couple with a moveable object located in a real environment, wherein said motion capture element comprises

a sensor configured to capture one or more values associated with an orientation, position, velocity, acceleration, angular velocity, and angular acceleration of said motion capture element;

a first communication interface; and

,

a microprocessor coupled with said sensor and said first communication interface, wherein said microprocessor is configured to

collect data that comprises said one or more values from said sensor;

transmit said data via said first communication interface;

a computer that comprises

a display;

a memory that contains

a virtual environment state comprising a position and orientation of a virtual object in a virtual environment, wherein said virtual object represents said moveable object in said virtual environment, and wherein said virtual environment state mirrors motion of said moveable object in said real environment with motion of said virtual object in said virtual environment; and

,

one or more constraints on said position and orientation of said virtual object in said virtual environment;

a second communication interface configured to communicate with said first communication interface to obtain said data;

wherein said computer is configured to

receive said data via said second communication interface;

calculate a position and orientation of said moveable object in said real environment from said data;

apply one or more corrections to said position and orientation of said moveable object in said real environment based on one or more redundancies in said data,

wherein said sensor comprises a 3-axis accelerometer, a 3-axis rate gyroscope, and a 3-axis magnetometer, and

wherein said one or more redundancies in said data is obtained from said 3-axis rate gyroscope combined with said 3-axis accelerometer and said a 3-axis magnetometer;

transform said position and orientation of said moveable object in said real environment into said position and orientation of said virtual object in said virtual environment after said apply said one or more corrections;

apply said one or more constraints in said virtual environment to compensate for accumulating errors in said position or orientation;

determine whether said position and orientation of said virtual object in said virtual environment satisfies said one or more constraints;

when said position and orientation of said virtual object in said virtual environment does not satisfy one or more of said one or more constraints, modify said position and orientation of said virtual object in said virtual environment to satisfy said one or more constraints to compensate for an error in said position and orientation not fully corrected by said one or more corrections as an error correction applied after said one or more constraints are applied in said virtual environment,

wherein said error correction is in addition to said one or more corrections,

wherein said error correction comprises continuous error correction or periodic proportional error correction, and

wherein said continuous error correction or said periodic proportional error correction is based on a type of constraint of said one or more constraints, such that said type of constraint determines whether said continuous error correction is applied or whether said periodic proportional error correction is applied; and

, display said virtual environment and said virtual object on said display.

2. The system of claim 1, wherein said moveable object is one or more of a piece of equipment, an article of clothing, or a body part of a person.

3. The system of claim 1, wherein said redundancies in said data comprise

a calculation of said orientation of said moveable object in said real environment by integration of angular velocity data from said 3-axis rate gyroscope;

a measurement of said orientation of said moveable object in said real environment relative to an Earth magnetic field from said 3-axis magnetometer; and

,

a measurement of said orientation of said moveable object in said real environment relative to an Earth gravitational field from said 3-axis accelerometer, when said data indicates that said moveable object is substantially stationary.

4. The system of claim 3, wherein

said calculate a position and orientation of said moveable object in said real environment from said data comprises said integration of angular velocity data from said 3-axis rate gyroscope;

and,

said apply one or more corrections to said position and orientation of said moveable object in said real environment comprises one or more of

calculate a first rotational error between

a predicted magnetic field vector based on said integration of angular velocity data, and

a measured magnetic field vector from said 3-axis magnetometer;

apply a first fraction of said first rotational error to said orientation of said moveable object in said real environment;

calculate a second rotational error between

a predicted gravitational field vector based on said integration of angular velocity data, and

a measured gravitational field vector from said 3-axis accelerometer;

apply a second fraction of said second rotational error to said orientation of said moveable object in said real environment.

5. The system of claim 1, wherein

said virtual environment comprises a virtual camera that forms an image of said virtual environment;

said display said virtual environment and said virtual object on said display transmits said image of said virtual environment to said display; and

,

a position and orientation of said virtual camera in said virtual environment is modifiable by a user.

6. The system of claim 1, wherein a change in said position and orientation of said moveable object in said real environment results in a corresponding change within a predefined time period of said position and orientation of said virtual object in said virtual environment shown on said display.

7. The system of claim 1, wherein said one or more constraints comprise a region in said virtual environment that said virtual object must be located within.

8. The system of claim 1, wherein said one or more constraints comprise a location in said virtual environment that said virtual object must be located near.

9. The system of claim 8, wherein said modify said position and orientation of said virtual object in said virtual environment to satisfy said one or more constraints comprises

calculate an error vector between said position of said virtual object in said virtual environment and said location;

and,

add a fraction of said error vector to said position of said virtual object in said virtual environment.

10. The system of claim 1, wherein said one or more constraints comprise one or more of a maximum speed, a minimum speed, a maximum acceleration, a minimum acceleration, a maximum angular velocity, a minimum angular velocity, a maximum angular acceleration, and a minimum angular acceleration of said virtual object.

11. The system of claim 1, wherein said one or more constraints comprise one or more virtual barriers that said virtual object cannot pass through.

12. The system of claim 11, wherein said one or more virtual barriers comprise a ground surface.

13. The system of claim 1, wherein

said moveable object is a piece of sporting equipment used in a sports activity;

said virtual environment is a virtual game for said sports activity; and

,

said virtual object in said virtual environment is a representation of said piece of sporting equipment.

14. The system of claim 13, wherein said sports activity is one or more of golf, tennis, badminton, racquetball, table tennis, squash, baseball, softball, cricket, hockey, field hockey, croquet, football, rugby, Australian rules football, soccer, volleyball, water polo, polo, basketball, lacrosse, billiards, horseshoes, shuffleboard, handball, bocce, bowling, dodgeball, kick ball, track and field events, curling, martial arts, boxing, archery, pistol shooting, rifle shooting, ice skating, gymnastics, surfing, skateboarding, snowboarding, skiing, windsurfing, roller blading, bicycling, and racing.

15. The system of claim 13, wherein

said virtual object is used to contact and launch a virtual game piece in said virtual game;

and,

said computer calculates a launch velocity of said game piece based on a motion of said virtual object at a time of impact between said virtual object and said virtual game piece.

16. The system of claim 13, wherein said computer relocates said virtual object to a new position in said virtual environment in response to one or more events in said virtual game.

17. The system of claim 16, wherein said relocates said virtual object to a new position occurs gradually to prevent a user from observing a discontinuous change in position on said display.

18. The system of claim 16, wherein

said virtual game is a virtual golf game;

said virtual object is a virtual golf club; and

,

said relocates said virtual object to a new position comprises moves said virtual golf club to a location of a virtual golf ball after a shot, or to a starting position for a new hole after completion of a previous hole.

19. The system of claim 18, wherein said moves said virtual golf club further comprises modifies an orientation of said virtual golf club to aim at a hole from said new position.