Tracking finger movements to generate inputs for computer systems

• US 10,534,431 B2
• Filed: 10/24/2017
• Issued: 01/14/2020
• Est. Priority Date: 05/16/2017
• Status: Active Grant

• Current Assignee: Finchxr Limited
• Original Assignee: Finch Technologies Ltd.
• Inventors: Erivantcev, Viktor Vladimirovich, Kulchurin, Rustam Rafikovich, Lobanov, Alexander Sergeevich, Sergeev, Iakov Evgenevich, Kartashov, Alexey Ivanovich
• Primary Examiner(s): Lam, Vinh T
• Application Number:
  US15/792,255
• Publication Number:
  US 20180335843A1
• Time in Patent Office: 812 Days
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/235 using a wireless connection, e.g. infrared or piconet
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors

Abstract

A system including: a first sensor module having an inertial measurement unit and attached to a palm of a hand of a user; a second sensor module having an inertial measurement unit and attached to a first bone of a finger (e.g., a middle or proximal phalange bone) on the palm; and a computing device coupled to the first sensor module and the second sensor module to calculate, based on the orientation of the palm and the orientation of the first bone, orientations of the second bones of the finger (e.g., a distal or proximal phalange bone, a metacarpal bone of the thumb) that have no separately attached inertial measurement unit, according to a predetermined ratio of rotation from a reference orientation along a same axis of rotation.

Claims

1. A system, comprising:

a first sensor module having an inertial measurement unit and attached to a palm of a hand of a user, the first sensor module generating first motion data identifying an orientation of the palm of the user, wherein the hand has a finger including a first bone and second bones;

a second sensor module having an inertial measurement unit and attached to the first bone of the finger, the second sensor module generating second motion data identifying an orientation of the first bone of the finger; and

a computing device coupled to the first sensor module and the second sensor module through one or more communication link, the computing device calculating, based on the orientation of the palm and the orientation of the first bone, orientations of the second bones of the finger,

wherein the orientations of the second bones of the finger are calculated to have rotated along a same axis from a reference orientation as the orientation of the first bone of the finger.

2. The system of claim 1, wherein the finger is a thumb;

the first bone is a proximal phalange bone of the thumb, and the second bones are a metacarpal bone and a distal phalange bone of the thumb.

3. The system of claim 1, wherein the finger is one of:

an index finger, a middle finger, a ring finger, and a little finger;

the first bone is a middle phalange of the finger; and

the second bones are proximal and distal phalange bones of the finger.

4. The system of claim 1, wherein each of the inertial measurement unit of the first sensor module and the inertial measurement of the second sensor module includes a micro-electromechanical system (MEMS) gyroscope.

5. The system of claim 4, wherein each of the inertial measurement unit of the first sensor module and the inertial measurement of the second sensor module further includes a magnetometer and a MEMS accelerometer.

6. The system of claim 1, wherein the orientations of the second bones of the finger are calculated without an inertial measurement unit attached to the second bones of the finger.

7. The system of claim 1, wherein the computing device calculates the orientation of the forearm by:

determining a first rotation angle, rotating from a predetermined reference direction relative to the palm to a lengthwise direction of the first bone along an axis that is perpendicular to both the predetermined reference direction and the lengthwise direction of first bone;

determining second rotation angles, rotating from the predetermined reference direction along the axis, based on a predetermined ratio among the first rotation and the second rotations; and

rotating the orientation of the first bone according to the second rotations to obtain orientations of the second bones.

8. The system of claim 7, wherein the orientation of the first bone and the orientations of the second bones corresponding to rotation an orientation aligned with the reference direction and by the first rotation angle and the second rotation angles respectively along the axis.

9. A method, comprising:

receiving, from a first sensor module attached to a palm of a hand of a user, first motion data identifying an orientation of the palm, the hand having a finger including a first bone and second bones connected by two finger joints;

receiving, from a second sensor module attached to the first bone of the finger, second motion data identifying an orientation of the first bone of the finger;

calculating, based on the orientation of the palm and the orientation of the first bone of the finger, orientations of the second bones of the finger; and

determining a position of the finger in a three dimensional space based on the orientation of the first bone of the finger, the orientations of the second bones of the finger, and a skeleton model of the hand,

wherein the orientations of the second bones of the finger are calculated to have rotated along a same axis from a reference orientation as the orientation of the first bone of the finger.

10. The method of claim 9, wherein the orientation of the first bone and the orientation of the second bones correspond to rotating a reference orientation along a same axis by a first angle and second angles.

11. The method of claim 10, wherein the orientations of the second bones of the finger are calculated based on a predetermined ratio among the first angle and the second angles.

12. The method of claim 11, wherein the axis is perpendicular to a lengthwise direction of the first bone and a reference direction predetermined relative to the palm.

13. The method of claim 12, wherein the finger is not a thumb;

and the reference direction is parallel to the palm and parallel to a direction pointing from a wrist of the hand and the palm.

14. The method of claim 13, wherein the finger is a thumb;

and the reference direction is not parallel to the palm.

15. The method of claim 11, wherein the finger is a middle finger of the hand;

and the method further comprises;

calculating orientations of bones of a ring finger of the hand based on the orientation of the first bone of the middle finger.

16. A non-transitory computer storage medium storing instructions which, when executed by a computing device, instructs the computing device to perform a method, the method comprising:

receiving, from a first sensor module attached to a palm of a hand of a user, first motion data identifying an orientation of the palm, the hand having a finger including a first bone and second bones connected by two finger joints,

receiving, from a second sensor module attached to the first bone of the finger, second motion data identifying an orientation of the first bone of the finger;

calculating, based on the orientation of the palm and the orientation of the first bone of the finger, orientations of the second bones of the finger; and

determining a position of the finger in a three dimensional space based on the orientation of the first bone of the finger, the orientations of the second bones of the finger, and a skeleton model of the hand,

wherein the orientations of the second bones of the finger are calculated to have rotated along a same axis from a reference orientation as the orientation of the first bone of the finger.

17. The non-transitory computer storage medium of claim 16 wherein the orientations of the second bones of the finger are calculated based on a predetermined ratio of rotation angles from the reference orientation for the first bone of the finger and the second bones of the finger.

18. The non-transitory computer storage medium of claim 17, wherein the axis is perpendicular to a lengthwise direction of the first bone in the orientation of the first bone and a reference direction predetermined relative to the palm.

19. The non-transitory computer storage medium of claim 18, wherein when the finger is a thumb, the reference direction is not parallel to the palm;

and when the finger is not a thumb, the reference direction is parallel to the palm.