WEARABLE SENSOR FOR TRACKING ARTICULATED BODY-PARTS

• US 20140098018A1
• Filed: 10/04/2012
• Published: 04/10/2014
• Est. Priority Date: 10/04/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Kim, David, Izadi, Shahram, Hilliges, Otmar, Butler, David Alexander, Hodges, Stephen, Chen, Jiawen, Olivier, Patrick Luke, Oikonomidis, Iason
• Granted Patent: US 10,234,941 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042

Abstract

A wearable sensor for tracking articulated body parts is described such as a wrist-worn device which enables 3D tracking of fingers and optionally also the arm and hand without the need to wear a glove or markers on the hand. In an embodiment a camera captures images of an articulated part of a body of a wearer of the device and an articulated model of the body part is tracked in real time to enable gesture-based control of a separate computing device such as a smart phone, laptop computer or other computing device. In examples the device has a structured illumination source and a diffuse illumination source for illuminating the articulated body part. In some examples an inertial measurement unit is also included in the sensor to enable tracking of the arm and hand

Claims

1. A wearable sensing device for gesture-based control of a computing device, the wearable sensing device comprising:

a camera to capture images of an articulated part of a body of a wearer of the sensing device;

a tracking module arranged to use the captured images to track a 3D articulated model of the articulated body part in real time without the need for markers to be worn on the articulated body part;

a communication interface arranged to send the tracked 3D articulated model to the computing device in order to control the computing device according to 3D articulation of the articulated body part.

2. A device as claimed in claim 1 sized and shaped to be worn on an anterior side of a user'"'"'s wrist to track a 3D articulated model of the user'"'"'s hand.

3. A device as claimed in claim 1 comprising a structured illumination source positioned at a specified distance from the camera to illuminate the articulated body part with a known structured illumination pattern.

4. A device as claimed in claim 1 comprising at least one diffuse illumination source to illuminate the articulated body part with diffuse illumination.

5. A device as claimed in claim 1 wherein the articulated body part is a hand and wherein the articulated model comprises, for each finger of the hand, three bones connected in series by three joints, each joint having a joint angle being an angle between a line projecting longitudinally along a first bone entering the joint from the palm end towards the fingertip, and a second bone leaving the joint, and wherein the articulated model is tracked by determining a first one of the joint angles from at least some of the captured images and determining the other two joint angles in dependence on the first joint angle.

6. A device as claimed in claim 5 wherein the at least some of the captured images are images of the hand illuminated with structured illumination of a known pattern.

7. A device as claimed in claim 1 wherein the processor is arranged to track the 3D articulated model by calculating at least two angles of the articulated model independently of one another from the images.

8. A device as claimed in claim 1 comprising a laser line projector arranged to illuminate the articulated body part with a laser line.

9. A device as claimed in claim 1 comprising at least one infra-red light emitting diode to diffusely illuminate the articulated body part.

10. A device as claimed in claim 1 comprising an inertial measurement unit and wherein the tracking module tracks a 3D pose of a body part on which the device is worn.

12. A method as claimed in claim 10 comprising calculating a 2D position of an end effector of the articulated body part from the images of the articulated body part illuminated with diffuse illumination.

14. A method as claimed in claim 10 comprising calculating a depth of an unknown location on the articulated body part from the images of the articulated body part illuminated with structured illumination and using the depth in the process of tracking the articulated model.

15. A method as claimed in claim 10 comprising mapping image areas, of the images of the articulated body part illuminated with diffuse illumination, to digits, where the articulated body part is a hand, by finding vertical edges in the images, identifying a hand contour from the vertical edges, identifying valleys in the hand contour and finding paths which follow edges of the valleys.

16. A method as claimed in claim 10 comprising calculating the at least two angles by searching possible combinations of values of the two angles which fit data from the received images.

17. A method as claimed in claim 10 at least partially carried out using hardware logic.

19. A tracking system as claimed in claim 17 which is integral with the sensing device.

20. A tracking system as claimed in claim 17 wherein the tracking module is at least partially implemented using hardware logic selected from any one or more of:

a field-programmable gate array, a program-specific integrated circuit, a program-specific standard product, a system-on-a-chip, a complex programmable logic device.

11. A method of tracking a 3D articulated model of an articulated body part comprising:

receiving images from a sensing device worn on a body comprising the articulated body part, the images comprising images of the articulated body part illuminated with structured illumination and images of the articulated body part illuminated with diffuse illumination;

tracking the 3D articulated model by calculating, at a processor, at least two angles in a single chain of angles of the articulated model independently of one another from the received images.

13. A method as claimed in claim 11 comprising computing, for each image element of one of the images of the articulated body part illuminated with diffuse illumination, an estimated distance of a surface depicted by that image element from the camera on the basis of intensity of the image element;

computing from the estimated distances a quantity which is invariant to the distance of the surfaces depicted by the image elements, and carrying out template matching over the quantity.

18. A tracking system for tracking a 3D articulated model of an articulated body part comprising:

an input to receive images from a sensing device worn on a body comprising the articulated body part, the images comprising images of the articulated body part illuminated with structured illumination and images of the articulated body part illuminated with diffuse illumination; and

a tracking module arranged to track the 3D articulated model by calculating at least two angles of the articulated model independently of one another from the received images.