Motion pattern classification and gesture recognition

• US 10,209,778 B2
• Filed: 08/17/2015
• Issued: 02/19/2019
• Est. Priority Date: 06/03/2011
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Moore, Christopher, Tu, Xiaoyuan, Vieta, William Matthew
• Primary Examiner(s): Rosario, Nelson
• Application Number:
  US14/828,422
• Publication Number:
  US 20150355721A1
• Time in Patent Office: 1,282 Days
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Methods, program products, and systems for gesture classification and recognition are disclosed. In general, in one aspect, a system can determine multiple motion patterns for a same user action (e.g., picking up a mobile device from a table) from empirical training data. The system can collect the training data from one or more mobile devices. The training data can include multiple series of motion sensor readings for a specified gesture. Each series of motion sensor readings can correspond to a particular way a user performs the gesture. Using clustering techniques, the system can extract one or more motion patterns from the training data. The system can send the motion patterns to mobile devices as prototypes for gesture recognition.

Claims

1. A method performed by a mobile device, comprising:

receiving, at the mobile device, a plurality of motion patterns, each of the motion patterns including a timed series of motion pattern vectors and being associated with a motion pattern time and a sphere of influence;

receiving a plurality of motion sensor readings from a motion sensor of the mobile device, the motion sensor readings including a plurality of motion reading vectors, each motion reading vector corresponding to a motion reading time;

selecting, using a time window and from the motion sensor readings, a time series of motion reading vectors;

calculating a motion deviation distance between the selected time series of motion reading vectors and each of the motion patterns, including applying dynamic time warping based on the motion pattern times of the motion pattern and the motion reading times of the series of motion reading vectors;

sliding the time window along a time dimension on the received motion sensor readings, and performing the selecting and calculating operations for each sliding time window until, according to a calculated motion deviation distance, the selected time series of motion reading vectors is located within the sphere of influence of a motion pattern, which is then designated as a matching motion pattern; and

performing a specified task on the mobile device based on the matching motion pattern.

2. The method of claim 1, wherein each motion pattern vector includes at least one of:

a linear acceleration value on each of a plurality of motion axes;

an angular rate value on each of a plurality of motion axes;

magnetometer data;

or

gravimeter data.

3. The method of claim 2, wherein each motion reading vector includes at least one of:

an angular rate value or a linear acceleration value on each of the axes as measured by the motion sensor, the motion sensor including at least one of an accelerometer or a gyroscope;

or

a transformed acceleration value that is calculated based on one or more acceleration values as measured by the motion sensor.

4. The method of claim 1, wherein calculating the motion deviation distance between the selected time series of motion reading vectors and each of the motion patterns comprises:

calculating a vector distance between;

each motion reading vector in the selected time series of motion reading vectors, and

each motion pattern vectors in the motion pattern; and

calculating the motion deviation distance based on each vector distance.

5. The method of claim 4, wherein calculating the motion deviation distance based on each vector distance comprises:

identifying a series of vector distances ordered according to the motion pattern times and the motion reading times; and

designating a measurement of the vector distances in the identified series as the motion deviation distance.

6. The method of claim 5, wherein the measurement comprises at least one of a sum or a weighted sum.

7. The method of claim 5, wherein identifying the series of vector distances comprises identifying a series of vector distances wherein the measurement of the series of vector distances has a smallest value.

8. The method of claim 4, wherein the vector distances include at least one of a Euclidean distance between a motion pattern vector and a motion reading vector or a Manhattan distance between a motion pattern vector and a motion reading vector.

9. The method of claim 1, wherein performing the specified task on the mobile device includes at least one of:

changing a configuration of the mobile device;

providing a user interface for display or removing a user interface from display on the mobile device;

launching or terminating an application program on the mobile device;

or

initiating or terminating a communication between the mobile device and another device.

10. The method of claim 9, wherein changing the configuration of the mobile device includes changing an input mode of the mobile device between a touch screen input mode and a voice input mode.

11. A non-transitory, computer-readable storage medium having instructions stored thereon, that when executed by one or more processors, cause the one or more processors to perform operations comprising:

receiving, at a mobile device, a plurality of motion patterns, each of the motion patterns including a timed series of motion pattern vectors and being associated with a motion pattern time and a sphere of influence;

receiving a plurality of motion sensor readings from a motion sensor of the mobile device, the motion sensor readings including a plurality of motion reading vectors, each motion reading vector corresponding to a motion reading time;

selecting, using a time window and from the motion sensor readings, a time series of motion reading vectors;

calculating a motion deviation distance between the selected time series of motion reading vectors and each of the motion patterns, including applying dynamic time warping based on the motion pattern times of the motion pattern and the motion reading times of the series of motion reading vectors;

sliding the time window along a time dimension on the received motion sensor readings, and performing the selecting and calculating operations for each sliding time window until, according to a calculated motion deviation distance, the selected time series of motion reading vectors is located within the sphere of influence of a motion pattern, which is then designated as a matching motion pattern; and

performing a specified task on the mobile device based on the matching motion pattern.

12. The non-transitory, computer-readable storage medium of claim 11, wherein each motion pattern vector includes at least one of:

a linear acceleration value on each of a plurality of motion axes;

an angular rate value on each of a plurality of motion axes;

magnetometer data;

or

gravimeter data.

13. The non-transitory, computer-readable storage medium of claim 12, wherein each motion reading vector includes at least one of:

an angular rate value or a linear acceleration value on each of the axes as measured by the motion sensor, the motion sensor including at least one of an accelerometer or a gyroscope;

or

a transformed acceleration value that is calculated based on one or more acceleration values as measured by the motion sensor.

14. The non-transitory, computer-readable storage medium of claim 11, wherein calculating the motion deviation distance between the selected time series of motion reading vectors and each of the motion patterns comprises:

calculating a vector distance between;

each motion reading vector in the selected time series of motion reading vectors, and

each motion pattern vectors in the motion pattern; and

calculating the motion deviation distance based on each vector distance.

15. The non-transitory, computer-readable storage medium of claim 14, wherein calculating the motion deviation distance based on each vector distance comprises:

identifying a series of vector distances ordered according to the motion pattern times and the motion reading times; and

designating a measurement of the vector distances in the identified series as the motion deviation distance.

16. The non-transitory, computer-readable storage medium of claim 15, wherein the measurement comprises at least one of a sum or a weighted sum.

17. The non-transitory, computer-readable storage medium of claim 15, wherein identifying the series of vector distances comprises identifying a series of vector distances wherein the measurement of the series of vector distances has a smallest value.

18. The non-transitory, computer-readable storage medium of claim 14, wherein the vector distances include at least one of a Euclidean distance between a motion pattern vector and a motion reading vector or a Manhattan distance between a motion pattern vector and a motion reading vector.

19. The non-transitory, computer-readable storage medium of claim 11, wherein performing the specified task on the mobile device includes at least one of:

changing a configuration of the mobile device;

providing a user interface for display or removing a user interface from display on the mobile device;

launching or terminating an application program on the mobile device;

or

initiating or terminating a communication between the mobile device and another device.

20. The non-transitory, computer-readable storage medium of claim 19, wherein changing the configuration of the mobile device includes changing an input mode of the mobile device between a touch screen input mode and a voice input mode.

21. A system, comprising:

a mobile device configured to perform operations comprising;

receiving, at the mobile device, a plurality of motion patterns, each of the motion patterns including a timed series of motion pattern vectors and being associated with a motion pattern time and a sphere of influence;

receiving a plurality of motion sensor readings from a motion sensor of the mobile device, the motion sensor readings including a plurality of motion reading vectors, each motion reading vector corresponding to a motion reading time;

selecting, using a time window and from the motion sensor readings, a time series of motion reading vectors;

calculating a motion deviation distance between the selected time series of motion reading vectors and each of the motion patterns, including applying dynamic time warping based on the motion pattern times of the motion pattern and the motion reading times of the series of motion reading vectors;

sliding the time window along a time dimension on the received motion sensor readings, and performing the selecting and calculating operations for each sliding time window until, according to a calculated motion deviation distance, the selected time series of motion reading vectors is located within the sphere of influence of a motion pattern, which is then designated as a matching motion pattern; and

performing a specified task on the mobile device based on the matching motion pattern.

22. The system of claim 21, wherein each motion pattern vector includes at least one of:

a linear acceleration value on each of a plurality of motion axes;

an angular rate value on each of a plurality of motion axes;

magnetometer data;

or

gravimeter data.

23. The system of claim 22, wherein each motion reading vector includes at least one of:

an angular rate value or a linear acceleration value on each of the axes as measured by the motion sensor, the motion sensor including at least one of an accelerometer or a gyroscope;

or

a transformed acceleration value that is calculated based on one or more acceleration values as measured by the motion sensor.

24. The system of claim 21, wherein calculating the motion deviation distance between the selected time series of motion reading vectors and each of the motion patterns comprises:

calculating a vector distance between;

each motion reading vector in the selected time series of motion reading vectors, and

each motion pattern vectors in the motion pattern; and

calculating the motion deviation distance based on each vector distance.

25. The system of claim 24, wherein calculating the motion deviation distance based on each vector distance comprises:

identifying a series of vector distances ordered according to the motion pattern times and the motion reading times; and

designating a measurement of the vector distances in the identified series as the motion deviation distance.

26. The system of claim 25, wherein the measurement comprises at least one of a sum or a weighted sum.

27. The system of claim 25, wherein identifying the series of vector distances comprises identifying a series of vector distances wherein the measurement of the series of vector distances has a smallest value.

28. The system of claim 24, wherein the vector distances include at least one of a Euclidean distance between a motion pattern vector and a motion reading vector or a Manhattan distance between a motion pattern vector and a motion reading vector.

29. The system of claim 21, wherein performing the specified task on the mobile device includes at least one of:

changing a configuration of the mobile device;

providing a user interface for display or removing a user interface from display on the mobile device;

launching or terminating an application program on the mobile device;

or

initiating or terminating a communication between the mobile device and another device.

30. The system of claim 29, wherein changing the configuration of the mobile device includes changing an input mode of the mobile device between a touch screen input mode and a voice input mode.