Tracking arm movements to generate inputs for computer systems

• US 10,379,613 B2
• Filed: 10/18/2017
• Issued: 08/13/2019
• Est. Priority Date: 05/16/2017
• Status: Active Grant

• Current Assignee: Finchxr Limited
• Original Assignee: Finch Technologies Ltd.
• Inventors: Erivantcev, Viktor Vladimirovich, Kulchurin, Rustam Rafikovich, Lobanov, Alexander Sergeevich, Sergeev, Iakov Evgenevich, Kartashov, Alexey Ivanovich
• Primary Examiner(s): Lee, Jr., Kenneth B
• Application Number:
  US15/787,555
• Publication Number:
  US 20180335855A1
• Time in Patent Office: 664 Days
• Field of Search: None
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/235 using a wireless connection, e.g. infrared or piconet
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors

Abstract

A system including: a first sensor module having an inertial measurement unit and attached to an upper arm of a user, the first sensor module generating first motion data identifying an orientation of the upper arm; a second sensor module having an inertial measurement unit and attached to a hand of the user, the second sensor module generating second motion data identifying an orientation of the hand; and a computing device coupled to the first sensor module and the second sensor module through communication links, the computing device calculating, based on the orientation of the upper arm and the orientation of the hand, an orientation of a forearm connected to the hand by a wrist of the user and connected to the upper arm by an elbow joint of the user.

Claims

1. A system, comprising:

a first sensor module having an inertial measurement unit and attached to an upper arm of a user, the first sensor module generating first motion data identifying an orientation of the upper arm;

a second sensor module having an inertial measurement unit and attached to a hand of the user, the second sensor module generating second motion data identifying an orientation of the hand; and

a computing device coupled to the first sensor module and the second sensor module through communication links, the computing device calculating, based on the orientation of the upper arm and the orientation of the hand, an orientation of a forearm connected to the hand by a wrist of the user and connected to the upper arm by an elbow joint of the user, wherein the computing device calculates the orientation of the forearm by;

determining a first rotation from a lengthwise direction of the upper arm to a lengthwise direction of the hand rotating along a first axis that is perpendicular to both the lengthwise direction of the upper arm and the lengthwise direction of the hand;

reversing the first rotation along the first axis from the orientation of the hand to obtain a second rotation along the lengthwise direction of the upper arm;

projecting, on to a plane, the lengthwise direction of the hand to obtain a lengthwise direction of the forearm;

calculating a third rotation from the lengthwise direction of the upper arm to the lengthwise direction of the forearm along a second axis perpendicular to the plane; and

determining, by the computing device, the orientation of the forearm as a combination of an initial orientation aligned with the upper arm rotated according to the second rotation along the lengthwise direction of the upper arm and then rotated according to the third rotation along the second axis.

2. The system of claim 1, wherein each of the first sensor module and the second sensor module further includes a communication device for a communication link with the computing device.

3. The system of claim 1, wherein each of the inertial measurement unit of the first sensor module and the inertial measurement unit of the second sensor module includes a micro-electromechanical system (MEMS) gyroscope.

4. The system of claim 3, wherein each of the inertial measurement unit of the first sensor module and the inertial measurement unit of the second sensor module further includes a magnetometer and a MEMS accelerometer.

5. The system of claim 1, wherein the orientation of the forearm is calculated without an inertial measurement unit attached to the forearm of the user.

6. The system of claim 1, wherein the orientation of the forearm is calculated without a sensor module on the forearm of the user.

7. The system of claim 1, wherein the second axis is parallel to a direction from an inner side of the upper arm to an outer side of the upper arm;

and the plane is parallel with the lengthwise direction of the upper arm and a direction from a back side of the upper arm to a front side of the upper arm.

8. A method, comprising:

receiving, from a first sensor module attached to an upper arm of a user, first motion data identifying an orientation of the upper arm;

receiving, from a second sensor module attached to a hand of the user, second motion data identifying an orientation of the hand;

calculating, based on the orientation of the upper arm and the orientation of the hand, an orientation of a forearm connected to the hand by a wrist of the user and connected to the upper arm by an elbow joint of the user;

wherein the orientation of the forearm is calculated to be equal to a first orientation resulting from;

rotating, from a second orientation aligned with the orientation of the upper arm, about a lengthwise direction of the forearm, by a first angle to arrive at third orientation; and

rotating, from the third orientation, about an axis parallel to a direction from a back side of the upper arm to a front side of the upper arm, by a second angle to arrive at the first orientation; and

determining a position of the hand in a three dimensional space based on the orientation of the upper arm, and the orientation of the forearm, the orientation of the hand, and a skeleton model of the upper arm, the forearm, and the hand.

9. The method of claim 8, wherein the first angle is a rotation of the hand about a lengthwise direction of the hand.

10. The method of claim 9, wherein the second angle is a rotation, projected in the plane, of the lengthwise direction of the hand from the lengthwise direction of the upper arm in the plane.

11. The method of claim 10, wherein the plane is perpendicular to a direction pointing from an inner side of the upper arm and an outer side of the upper arm.

12. The method of claim 11, wherein the plane is parallel to the lengthwise direction of the upper arm.

13. The method of claim 12, wherein the plane is further parallel to a direction pointing from a back side of the upper arm to a front side of the upper arm.

14. A non-transitory computer storage medium storing instructions which, when executed by a computing device, instructs the computing device to perform a method, the method comprising:

receiving, by the computing device from a first sensor module attached to an upper arm of a user, first motion data identifying an orientation of the upper arm;

receiving, by the computing device from a second sensor module attached to a hand of the user, second motion data identifying an orientation of the hand; and

calculating, by the computing device based on the orientation of the upper arm and the orientation of the hand, an orientation of a forearm connected to the hand by a wrist of the user and connected to the upper arm by an elbow joint of the user, wherein the calculating comprises;

calculating, by the computing device, an orientation of the hand relative to the upper arm;

determining, by the computing device, a first rotation of a lengthwise direction of the hand from a lengthwise direction of the upper arm along a closest arc to the lengthwise direction of the hand in the orientation of the hand relative to the upper arm;

subtracting, by the computing device, the first rotation from the orientation of the hand relative to the upper arm to obtain a second rotation of the hand along the lengthwise direction of the upper arm;

projecting, on to a plane by the computing device, the lengthwise direction of the hand in the orientation of the hand relative to the upper arm to determine a lengthwise direction of the forearm relative to the upper arm;

computing, by the computing device, a closest arc rotation from the lengthwise direction of the upper arm to the lengthwise direction of the forearm relative to the upper arm; and

determining, by the computing device, the orientation of the forearm based on combination of a rotation from an orientation aligned with the upper arm, along the lengthwise direction of the upper arm according to the second rotation of the hand along the lengthwise direction of the upper arm and a rotation within the plane according to the closest arc rotation from the lengthwise direction of the upper arm to the lengthwise direction of the forearm relative to the upper arm.

15. The non-transitory computer storage medium of claim 14, wherein the plane is perpendicular to a direction pointing from an inner side of the upper arm and an outer side of the upper arm.

16. The non-transitory computer storage medium of claim 15, wherein the plane is parallel to the lengthwise direction of the upper arm.

17. The non-transitory computer storage medium of claim 16, wherein the plane is further parallel to a direction pointing from a back side of the upper arm to a front side of the upper arm.