Identifying gestures using motion data

• US 9,996,109 B2
• Filed: 08/14/2015
• Issued: 06/12/2018
• Est. Priority Date: 08/16/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Carceroni, Rodrigo, Ozkan, Derya, Shah, Suril, Sanketi, Pannag Raghunath
• Primary Examiner(s): Amadiz, Rodney
• Application Number:
  US14/826,437
• Publication Number:
  US 20160048161A1
• Time in Patent Office: 1,033 Days
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 1/3206 Monitoring of events, devices or parameters that trigger a change in power modality
• CPC Class Codes: G06F 1/3265 Power saving in display device
• CPC Class Codes: G06F 2200/1637 Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

In one example, a method includes determining, by a processor (104) of a wearable computing device (102) and based on motion data generated by a motion sensor (106) of the wearable computing device, one or more strokes. In this example, the method also includes generating, by the processor and based on the motion data, a respective attribute vector for each respective stroke from the one or more strokes and classifying, by the processor and based on the respective attribute vector, each respective stroke from the one or more strokes into at least one category. In this example, the method also includes determining, by the processor and based on a gesture library and the at least one category for each stroke from the one or more strokes, a gesture. In this example, the method also includes performing, by the wearable device and based on the gesture, an action.

Claims

1. A method comprising:

determining, by a processor of a wearable computing device and based on motion data generated by a motion sensor of the wearable computing device, one or more strokes, wherein the one or more strokes are determined by segmenting the motion data based on the features of the motion data;

generating, by the processor and based on the motion data, a respective attribute vector for each respective stroke from the one or more strokes;

classifying, by the processor and based on the respective attribute vector, each respective stroke from the one or more strokes into at least one category;

determining, by the processor and based on a gesture library and the at least one category for each stroke from the one or more strokes, a gesture; and

performing, by the wearable device and based on the gesture, an action.

2. The method of claim 1, further comprising:

generating task-specific motion data by at least converting the motion data into a task-specific coordinate system; and

determining, based on a task-specific coordinate system and the task-specific motion data, an orientation of the wearable device relative to a gravity vector,

wherein determining the gesture is further based on the orientation of the wearable device.

3. The method of claim 2, wherein:

the wearable computing device is operating in a first power mode while the motion sensor generates the motion data,

the orientation of the wearable device indicates that a user of the wearable device is viewing a display of the wearable device,

the gesture comprises a lift-and-look gesture, and

performing the action comprises transitioning, by the wearable device, a display of the wearable device from operating in the first power mode to operating in a second power mode, wherein the display uses less power while operating in the first power more than while operating in the second power mode.

4. The method of claim 1, further comprising:

generating, by the motion sensor of the wearable computing device, the motion data;

responsive to determining, by a processor of the motion sensor that is different than the processor of the wearable computing device, that the motion data satisfies one or more thresholds, outputting the motion data to the processor of the wearable computing device.

5. The method of claim 4, wherein the wearable computing device includes an application processor that is different than the processor of the wearable computing device and the processor of the motion sensor.

6. The method of claim 1, wherein classifying each stroke from the plurality of stokes comprises classifying each stroke into a binary classification using support vector machines.

7. The method of claim 1, wherein classifying each stroke from the plurality of strokes comprises:

classifying, by the processor and based on a distance function between feature vectors of each stroke from the plurality of strokes, each stroke from the plurality of strokes into the at least one category.

8. The method of claim 1, wherein classifying each stroke from the plurality of strokes comprises:

classifying, by the processor and using deep belief networks, each stroke from the plurality of strokes.

9. The method of claim 1, wherein the motion data includes a sequence of values, and wherein determining the one or more strokes comprises:

determining a local minimum value within a neighborhood of the sequence of values; and

responsive to determining that the local minimum value corresponds to a curvature of the motion data;

determining that the local minimum value is a starting boundary of a stroke of the one or more strokes;

evaluating subsequent values within the sequence of values to determine a value corresponding to an edge of the motion data; and

determining that the value corresponding to the edge is an ending boundary of the stroke.

10. The method of claim 9, wherein the stroke is a first stroke of the one or more strokes, wherein the edge is a first edge, the method further comprising:

responsive to determining that the value that corresponds to the first edge is an ending boundary of the stroke;

evaluating subsequent values within the sequence of values to determine a value corresponding to a second edge of the motion data; and

determining that the value corresponding to the second edge is a starting boundary of a second stroke.

11. A wearable computing device comprising:

a motion sensor;

one or more processors configured to;

determine, based on motion data generated by a motion sensor of the wearable computing device, one or more strokes, wherein, to determine the one or more strokes, the one or more processors are configured to segment the motion data based on the features of the motion data;

generate, based on the motion data, a respective attribute vector for each respective stroke from the one or more strokes;

classify, based on the respective attribute vector, each respective stroke from the one or more strokes into at least one category;

determine, based on a gesture library and the at least one category for each stroke from the one or more strokes, a gesture; and

perform, based on the gesture, an action.

12. The wearable computing device of claim 11, wherein the one or more processors are configured to:

generate task-specific motion data by at least converting the motion data into a task-specific coordinate system;

determine, based on a task-specific coordinate system and the task-specific motion data, an orientation of the wearable device relative to a gravity vector; and

determine the gesture further based on the orientation of the wearable device.

13. The wearable computing device of claim 12, wherein:

the wearable computing device is operating in a first power mode while the motion sensor generates the motion data,

the orientation of the wearable device indicates that a user of the wearable device is viewing a display of the wearable device,

the gesture comprises a lift-and-look gesture, and

the one or more processors are configured to perform the action by at least transitioning, a display of the wearable device from operating in the first power mode to operating in a second power mode, wherein the display uses less power while operating in the first power more than while operating in the second power mode.

14. The wearable computing device of claim 11, wherein the motion sensor is associated with a processor of the one or more processors, and wherein the processor associated with the motion sensor is configured to:

receive the motion data generated by the motion sensor of the wearable computing device; and

responsive to determining that the motion data satisfies one or more thresholds, output the motion data to another processor of the one or more processors of wearable computing device.

15. The wearable computing device of claim 14, wherein the wearable computing device comprises an application processor that is different than the other processor of the wearable computing device and the processor associated with the motion sensor.

16. The wearable computing device of claim 11, wherein the one or more processors are configured to classify each stroke from the plurality of stokes by at least classifying each stroke into a binary classification using support vector machines.

17. The wearable computing device of claim 11, wherein the one or more processors are configured to classify each stroke from the plurality of strokes by at least:

classifying, based on a distance function between feature vectors of each stroke from the plurality of strokes, each stroke from the plurality of strokes into the at least one category.

18. The wearable computing device of claim 11, wherein the motion data includes a sequence of values, and wherein the one or more processors are configured to determine the one or more motion strokes by at least:

determining a local minimum value within a neighborhood of the sequence of values; and

responsive to determining that the local minimum value corresponds to a curvature of the motion data;

determining that the local minimum value is a starting boundary of a stroke of the one or more strokes;

evaluating subsequent values within the sequence of values to determine a value corresponding to an edge of the motion data; and

determining that the value corresponding to the edge is an ending boundary of the stroke.

19. The wearable computing device of claim 18, wherein the stroke is a first stroke of the one or more strokes, wherein the edge is a first edge, and wherein the one or more processors are configured to:

responsive to determining that the value that corresponds to the first edge is an ending boundary of the stroke;

evaluate subsequent values within the sequence of values to determine a value corresponding to a second edge of the motion data; and

determine that the value corresponding to the second edge is a starting boundary of a second stroke.

20. A non-transitory computer-readable storage medium storing instructions that, when executed, cause at least one processor of a wearable computing device to:

determine, based on motion data generated by a motion sensor of the wearable computing device, one or more strokes, wherein the one or more strokes are determined by segmenting the motion data based on the features of the motion data;

generate, based on the motion data, a respective attribute vector for each respective stroke from the one or more strokes;

classify, based on the respective attribute vector, each respective stroke from the one or more strokes into at least one category;

determine, based on a gesture library and the at least one category for each stroke from the one or more strokes, a gesture; and

perform an action based on the gesture.