Wearable sensor for tracking articulated body-parts

• US 10,234,941 B2
• Filed: 10/04/2012
• Issued: 03/19/2019
• Est. Priority Date: 10/04/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Kim, David, Izadi, Shahram, Hilliges, Otmar, Butler, David Alexander, Hodges, Stephen, Olivier, Patrick Luke, Chen, Jiawen, Oikonomidis, Iason
• Primary Examiner(s): Faragalla, Michael A
• Assistant Examiner(s): Shah, Sujit
• Application Number:
  US13/644,701
• Publication Number:
  US 20140098018A1
• Time in Patent Office: 2,357 Days
• Field of Search: 345156-184, 345345, 382103, 361679
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042

Abstract

A wearable sensor for tracking articulated body parts is described such as a wrist-worn device which enables 3D tracking of fingers and optionally also the arm and hand without the need to wear a glove or markers on the hand. In an embodiment a camera captures images of an articulated part of a body of a wearer of the device and an articulated model of the body part is tracked in real time to enable gesture-based control of a separate computing device such as a smart phone, laptop computer or other computing device. In examples the device has a structured illumination source and a diffuse illumination source for illuminating the articulated body part. In some examples an inertial measurement unit is also included in the sensor to enable tracking of the arm and hand

Claims

1. A wearable sensing device for gesture-based control of a computing device, the wearable sensing device comprising:

a camera that captures images of an articulated body part of a user;

a structured illumination source comprising a laser line projector positioned at a specified distance from the camera to illuminate the articulated body part with a known structured illumination pattern comprising a laser line;

a tracking module configured to;

receive the captured images of the articulated body part,

receive the laser line, a laser image, and a known baseline distance relative to the camera,

receive a kinematic model of the articulated body part, and

track the kinematic model of the articulated body part based on the captured images by calculating 3D positions of segments of the laser line of the captured images, including by performing binarization of the laser image, segmenting laser line segments, using a digit separation process to separate any merged laser line segments, and obtaining the 3D position of each laser line segment by triangulation to define a 3D point for each digit relative to the camera; and

a communication interface that sends the tracked kinematic model of the articulated body part to the computing device to control the computing device in accordance with a gesture associated with the tracked kinematic model of the articulated body part.

2. A device as claimed in claim 1 wherein a bridge portion of the device is sized and shaped to be worn on an anterior side of a wrist of a user to track the kinematic model of the articulated body part, the kinematic model of the articulated body part being a hand of the user associated with the wrist.

3. A device as claimed in claim 1 comprising the structured illumination source positioned on a bridge arm at the specified distance from the camera to illuminate the articulated body part with the known structured illumination pattern.

4. A device as claimed in claim 1 comprising at least one diffuse illumination source to illuminate the articulated body part with diffuse illumination.

5. A device as claimed in claim 1 wherein the articulated body part is a hand and wherein the kinematic model comprises, for each finger of the hand, three bones connected in series by three joints, each joint having a joint angle being an angle between a line projecting longitudinally along a first bone entering the joint from the palm end towards the fingertip, and a second bone leaving the joint, and wherein the articulated model is tracked by determining a first one of the joint angles from at least some of the captured images and determining the other two joint angles in dependence on the first joint angle.

6. A device as claimed in claim 5 wherein the at least some of the captured images are images of the hand illuminated with structured illumination of the known structured illumination pattern.

7. A device as claimed in claim 1 wherein the processor is arranged to track the kinematic model by calculating at least two angles of the kinematic model independently of one another from the captured images.

8. A device as claimed in claim 1 wherein the laser line projector illuminates the articulated body part with a single laser line toward an articulated body part of the user to capture laser images.

9. A device as claimed in claim 1 comprising at least one infra-red light emitting diode to diffusely illuminate the articulated body part.

10. A device as claimed in claim 3, further comprising the bridge arm physically coupled to the user, the bridge arm arranged to lie on an inner forearm of the user, wherein the bridge arm has a pair of slots through which a fixing strap may be threaded to enable the laser line projector and the camera to be fastened to a wrist of the user.

11. A device as claimed in claim 1, wherein the tracking module uses the captured images to track the kinematic model of the articulated body part in real time without a need for markers to be worn on the articulated body part.

12. A method of tracking a 3D articulated model of an articulated body part comprising:

receiving light emitting diode (LED) images of a hand and associated depth information from a sensing device worn on a wrist associated with the hand, the sensing device comprising a structured illumination source positioned at a specified distance from a camera to illuminate the hand with a known structured illumination pattern comprising a laser line, the associated depth information including a known baseline distance relative to the camera;

performing background subtraction of the LED images of the hand;

tracking a kinematic model of the hand based on the received LED images by calculating 3D positions of segments of the laser line of the received LED images, including by performing binarization of the LED images, segmenting laser line segments, using a digit separation process to separate any merged laser line segments, and obtaining the 3D position of each laser line segment by triangulation to define a 3D point for each digit relative to the camera; and

determining a gesture of the hand based on the tracked kinematic model.

13. A method as claimed in claim 12 comprising calculating a 2D position of an end effector of the hand from the LED images of the hand illuminated with diffuse illumination.

14. A method as claimed in claim 12 comprising computing, for each image element of one of the LED images of the hand illuminated with diffuse illumination, an estimated distance of a surface depicted by that image element from the camera on the basis of intensity of the image element, the diffuse illumination provided by at least one source, each source having identical characteristics;

computing from the estimated distances a quantity which is invariant to the distance of the surfaces depicted by the image elements, and carrying out template matching over the quantity.

15. A method as claimed in claim 12 comprising calculating a depth of an unknown location on the hand from the LED images of the hand illuminated with structured illumination from the structured illumination source and using the depth in the process of tracking the kinematic model of the hand.

16. A method as claimed in claim 12 comprising calculating at least two angles by searching possible combinations of values of the two angles which fit data from the received LED images of the hand.

17. A method as claimed in claim 12, further comprising calculating at least two angles in a single chain of angles of the kinematic model of the hand independently of one another from the received LED images and associated depth information.

18. A tracking system for tracking a 3D articulated model of an articulated body part comprising:

an input/output controller that receives light emitting diode (LED) images and corresponding depth data from a sensing device configured to be worn on a body comprising the articulated body part, the sensing device comprising;

a camera that captures LED images of the articulated body part, and

a structured illumination source comprising a laser line projector affixed to a bridge arm positioned at a specified distance from the camera to illuminate the articulated body part with a known structured illumination pattern comprising a laser line;

a processor arranged to;

perform background subtraction of the LED images of the articulated body part, and

receive a kinematic model of the articulated body part; and

a tracking module configured to;

receive the laser line, the LED images, and the specified distance from the camera;

track the kinematic model of the articulated body part by calculating 3D positions of segments of the laser line of the captured images, including by performing binarization of the laser image, segmenting laser line segments, using a digit separation process to separate any merged laser line segments, and obtaining the 3D position of each laser line segment by triangulation to define a 3D point for each digit relative to the camera, and

determine a gesture of the articulated body part associated with the tracked kinematic model.

19. A tracking system as claimed in claim 18, wherein the tracking module further comprises determining at least two angles of the kinematic model of the articulated body part independently of one another from the received LED images and the corresponding depth data.

20. A tracking system as claimed in claim 18 wherein the tracking module is at least partially implemented using hardware logic selected from any one or more of:

a field-programmable gate array, a program-specific integrated circuit, a program-specific standard product, a system-on-a-chip, a complex programmable logic device.