Motion and gesture input from a wearable device

• US 9,939,899 B2
• Filed: 12/17/2015
• Issued: 04/10/2018
• Est. Priority Date: 09/25/2015
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Allec, Nicholas Paul Joseph, Mu, Xiaoyi
• Primary Examiner(s): HICKS, CHARLES V
• Application Number:
  US14/973,573
• Publication Number:
  US 20170090567A1
• Time in Patent Office: 845 Days
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0308 comprising a plurality of distinctive and separately oriented light emitters or reflectors associated to the pointing device, e.g. remote cursor controller with distinct and separately oriented LEDs at the tip whose radiations are captured by a photo-detector associated to the screen
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

This disclosure relates to detecting hand gesture input using an electronic device, such as a wearable device strapped to a wrist. The device can have multiple photodiodes, each sensing light at a different position on a surface of the device that faces skin of a user. Examples of the disclosure detect hand gestures by recognizing patterns in sensor data that are characteristic of each hand gesture, as the tissue expands and contracts and anatomical features in the tissue move during the gesture.

Claims

1. A method of detecting hand gestures performed by a user of an electronic device including a plurality of photodiodes, the method comprising:

during a first period in which the user performs a first hand gesture, collecting first sensor data from the plurality of photodiodes, the first sensor data indicative of tissue movement caused by the first hand gesture;

during a second period in which the user performs a second hand gesture, collecting second sensor data from the plurality of photodiodes, the second sensor data indicative of tissue movement caused by the second hand gesture;

calculating first optical signal characteristics based on the first sensor data and second optical signal characteristics based on the second sensor data;

assigning some or all of the first optical signal characteristics to a first cluster of optical signal characteristics;

assigning some or all of the second optical signal characteristics to a second cluster of optical signal characteristics;

during a third period, collecting third sensor data from the plurality of photodiodes;

calculating third optical signal characteristics based on the third sensor data;

determining whether the third optical signal characteristics belong to the first cluster, the second cluster, or a third cluster;

in accordance with a determination that the third optical signal characteristics belong to the first cluster, determining that the user has performed the first hand gesture;

in accordance with a determination that the third optical signal characteristics belong to the second cluster, determining that the user has performed the second hand gesture; and

in accordance with a determination that the third optical signal characteristics belong to the third cluster, determining that the user has not performed the first hand gesture or the second hand gesture,

wherein each cluster is formed by the respective optical signal characteristics having proximate values.

2. The method of claim 1, the method further comprising:

comparing the first cluster to the second cluster;

based on comparing the first cluster to the second cluster, determining there is more of the first optical signal characteristics assigned to the first cluster than to the second cluster; and

in accordance with the determination that there is more of the first optical signal characteristics assigned to the first cluster than to the second cluster, assigning the first cluster to the first hand gesture.

3. The method of claim 2, the method further comprising:

based on comparing the first cluster to the second cluster, determining there is more of the second optical signal characteristics assigned to the second cluster than to the first cluster; and

in accordance with the determination that there is more of the second optical signal characteristics assigned to the second cluster than to the first cluster, assigning the second cluster to the second hand gesture.

4. The method of claim 1, the method further comprising:

generating a first template for the first hand gesture and a second template for the second hand gesture; and

comparing the third optical signal characteristics to the first and second templates.

5. The method of claim 4, the method further comprising:

calculating first mean optical signal characteristics for the first cluster, wherein the first template is generated based on the first mean optical signal characteristics for the first cluster; and

calculating second mean optical signal characteristics for the second cluster, wherein the second template is generated based on the second mean optical signal characteristics for the second cluster.

6. The method of claim 5, wherein comparing the third optical signal characteristics to the first and second templates includes:

calculating a first distance from the third optical signal characteristics to the first template and calculating a second distance from the third optical signal characteristics to the second template;

in accordance with a determination that the first distance is shorter than the second distance, determining that the third optical signal characteristics belong to the first cluster; and

in accordance with a determination that the second distance is shorter than the first distance, determining that the third optical signal characteristics belong to the second cluster.

7. The method of claim 4, wherein generating the first template for the first hand gesture includes storing some or all of the first sensor data as the first template, and generating the second template for the second hand gesture includes storing some or all of the second sensor data as the second template.

8. The method of claim 1, wherein calculating the first optical signal characteristics includes calculating at least one of:

an amplitude difference between a peak and a trough of the first sensor data, a time difference between a peak and a trough of the first sensor data, a maximum amplitude of the first sensor data, a period between peaks of the first sensor data, and a phase of the first sensor data.

9. The method of claim 1, the method further comprising:

filtering heart rate frequencies from the first sensor data before calculating the first optical signal characteristics based on the first sensor data.

10. The method of claim 1, the method further comprising:

during the first period in which the user performs the first hand gesture, further collecting the first sensor data from at least one of a force sensor and an accelerometer.

11. The method of claim 1, wherein collecting the first sensor data from the plurality of photodiodes includes collecting a first channel of infrared light and collecting a second channel of green light.

12. The method of claim 1, the method further comprising:

during a fourth period in which the user does not perform the first or second hand gesture, collecting fourth sensor data from the plurality of photodiodes;

calculating fourth optical signal characteristics based on the fourth sensor data; and

assigning some or all of the fourth optical signal characteristics to the third cluster of optical signal characteristics.

13. The method of claim 1, wherein assigning the first and second optical signal characteristics to the first and second clusters is performed using a k-means clustering algorithm.

14. The method of claim 13, wherein the k-means clustering algorithm is also applied to the third optical signal characteristics and determining whether the third optical signal characteristics belong to the first cluster, the second cluster, or the third cluster is based on the k-means clustering algorithm.

15. A non-transitory computer readable medium, the computer readable medium containing instructions, that, when executed, perform a method comprising:

during a first period in which a user performs a first hand gesture, collecting first sensor data from a plurality of photodiodes, the first sensor data indicative of tissue movement caused by the first hand gesture;

during a second period in which the user performs a second hand gesture, collecting second sensor data from the plurality of photodiodes, the second sensor data indicative of tissue movement caused by the second hand gesture;

calculating first optical signal characteristics based on the first sensor data and second optical signal characteristics based on the second sensor data;

assigning some or all of the first optical signal characteristics to a first cluster of optical signal characteristics;

assigning some or all of the second optical signal characteristics to a second cluster of optical signal characteristics;

during a third period, collecting third sensor data from the plurality of photodiodes;

calculating third optical signal characteristics based on the third sensor data;

determining whether the third optical signal characteristics belong to the first cluster, the second cluster, or a third cluster;

in accordance with a determination that the third optical signal characteristics belong to the first cluster, determining that the user has performed the first hand gesture;

in accordance with a determination that the third optical signal characteristics belong to the second cluster, determining that the user has performed the second hand gesture; and

in accordance with a determination that the third optical signal characteristics belong to the third cluster, determining that the user has not performed the first hand gesture or the second hand gesture,

wherein each cluster is formed by the respective optical signal characteristics having proximate values.

16. The non-transitory computer readable medium of claim 15, the method further comprising:

comparing the first cluster to the second cluster;

based on comparing the first cluster to the second cluster, determining there is more of the first optical signal characteristics assigned to the first cluster than to the second cluster; and

in accordance with the determination that there is more of the first optical signal characteristics assigned to the first cluster than to the second cluster, assigning the first cluster to the first hand gesture.

17. The non-transitory computer readable medium of claim 16, the method further comprising:

based on comparing the first cluster to the second cluster, determining there is more of the second optical signal characteristics assigned to the second cluster than to the first cluster; and

in accordance with the determination that there is more of the second optical signal characteristics assigned to the second cluster than to the first cluster, assigning the second cluster to the second hand gesture.

18. The non-transitory computer readable medium of claim 15, the method further comprising:

generating a first template for the first hand gesture and a second template for the second hand gesture; and

comparing the third optical signal characteristics to the first and second templates.

19. The non-transitory computer readable medium of claim 18, the method further comprising:

calculating first mean optical signal characteristics for the first cluster, wherein the first template is generated based on the first mean optical signal characteristics for the first cluster; and

calculating second mean optical signal characteristics for the second cluster, wherein the second template is generated based on the second mean optical signal characteristics for the second cluster.

20. An electronic device comprising:

one or more processors;

memory;

a plurality of photodiodes; and

one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, which when executed by the one or more processors, cause the electronic device to perform a method comprising;

during a first period in which a user performs a first hand gesture, collecting first sensor data from the plurality of photodiodes, the first sensor data indicative of tissue movement caused by the first hand gesture;

during a second period in which the user performs a second hand gesture, collecting second sensor data from the plurality of photodiodes, the second sensor data indicative of tissue movement caused by the second hand gesture;

calculating first optical signal characteristics based on the first sensor data and second optical signal characteristics based on the second sensor data;

assigning some or all of the first optical signal characteristics to a first cluster of optical signal characteristics;

assigning some or all of the second optical signal characteristics to a second cluster of optical signal characteristics;

during a third period, collecting third sensor data from the plurality of photodiodes;

calculating third optical signal characteristics based on the third sensor data;

determining whether the third optical signal characteristics belong to the first cluster, the second cluster, or a third cluster;

in accordance with a determination that the third optical signal characteristics belong to the first cluster, determining that the user has performed the first hand gesture;

in accordance with a determination that the third optical signal characteristics belong to the second cluster, determining that the user has performed the second hand gesture; and

in accordance with a determination that the third optical signal characteristics belong to the third cluster, determining that the user has not performed the first hand gesture or the second hand gesture,

wherein each cluster is formed by the respective optical signal characteristics having proximate values.