Behavioral recognition on mobile devices

• US 8,922,485 B1
• Filed: 12/17/2010
• Issued: 12/30/2014
• Est. Priority Date: 12/18/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Lloyd, Matthew
• Primary Examiner(s): Bukowski, Kenneth
• Application Number:
  US12/972,206
• Time in Patent Office: 1,474 Days
• US Class Current: 345/158
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0487 using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: H04M 1/6505 storing speech in digital form
• CPC Class Codes: H04M 2250/12 including a sensor for measuring a physical value, e.g. temperature or motion telephones including GPS signal receivers H04M2250/10

Abstract

The disclosure generally relates to identifying gestures on mobile devices for use in connection with speech detection. A gesture may be identified using at least one Bayesian network, such as a Hidden Markov Model. Each Bayesian network corresponds to one of a plurality of different gesture types, which may include different gestures or similar gestures performed at different speeds, for example. Each Bayesian network includes states corresponding to partitioned segments in an accelerometer time series for the associated gesture type. Segmenting the accelerometer timeseries allows the Bayesian network to account for the speed at which the user makes the gesture, minimizing any effect of the user'"'"'s speed in performing the gesture on identifying the gesture type. An action may be taken based on the identified gesture type. For example, speech recording may start or stop and/or a notification regarding speech recording may be provided depending on the gesture identified.

Claims

1. A computer-implemented method, comprising:

receiving, by a processor included in a mobile device, accelerometer values for a period of time during which the mobile device completes a gesture;

partitioning the accelerometer values into a plurality of distinct segments;

detecting a movement of the mobile device based on detecting a set of acceleration values of the mobile device;

identifying that the mobile device completed the gesture based on applying the movement of the mobile device to a first Bayesian network and a second Bayesian network, wherein;

the first Bayesian network includes a first plurality of states, wherein each respective state in the first plurality of states corresponds to a respective one of the plurality of distinct segments, and wherein the first Bayesian network corresponds to the gesture being performed at a first speed, and

the second Bayesian network includes a second plurality of states, wherein each respective state in the second plurality of states corresponds to a respective one of the plurality of distinct segments, and wherein the second Bayesian network corresponds to the gesture being performed at a second speed that is slower than the first speed; and

taking an action in response to identifying that the mobile device completed the gesture.

2. The computer-implemented method of claim 1, wherein the one or more of the first and second Bayesian networks comprises a Hidden Markov Model.

3. The computer-implemented method of claim 1, wherein the action taken in response to identifying that the mobile device completed the gesture comprises recording speech from the person.

4. The computer-implemented method of claim 1, wherein the action taken in response to identifying that the mobile device completed the gesture comprises ceasing recording speech from the person.

5. The computer-implemented method of claim 1, wherein the action taken in response to identifying that the mobile device completed the gesture comprises providing feedback to a person, the feedback comprising at least one of auditory feedback, tactile feedback, and visual feedback.

6. The computer-implemented method of claim 5, wherein the feedback indicates that speech recording is commencing.

7. The computer-implemented method of claim 5, wherein the feedback indicates that speech recording is ceasing.

8. The computer-implemented method of claim 1, wherein the set of acceleration values uses a spherical polar coordinate system.

9. The computer-implemented method of claim 1, wherein each state of the first and second Bayesian networks is associated with at least one observation for the gesture associated with the Bayesian network, the at least one observation comprising at least one of an endpointer state, a proximity vector, and an audio vector.

10. The computer-implemented method of claim 1, further comprising training the first and second Bayesian networks based on real movement trajectories during multiple iterations of the gesture.

11. The computer-implemented method of claim 10, wherein the training the first and second Bayesian networks comprises aligning accelerometer timeseries data for each real movement trajectory.

12. The computer-implemented method of claim 11, wherein the step of aligning the accelerometer timeseries data for each real movement trajectory comprises trimming at least one of the accelerometer timeseries.

13. The computer-implemented method of claim 1, wherein detecting the movement of the mobile device is further based on camera data corresponding to motion of lips of a user of the mobile device.

14. The computer-implemented method of claim 1, wherein detecting the movement of the mobile device is further based on detecting a set of positional angles of the mobile device, and receiving proximity data from a proximity sensor included in the mobile device.

15. A non-transitory computer-readable medium having computer-readable program code embodied therein that when executed by a processor, cause a computer system to perform the steps of:

receiving accelerometer values for a period of time during which a mobile device completes a gesture;

partitioning the accelerometer values into a plurality of distinct segments;

detecting a movement of the mobile device based on detecting a set of acceleration values of the mobile device;

identifying that the mobile device completed the gesture based on applying the movement of the mobile device to a first Bayesian network and a second Bayesian network, wherein;

the first Bayesian network includes a first plurality of states, wherein each respective state in the first plurality of states corresponds to a respective one of the plurality of distinct segments, and wherein the first Bayesian network corresponds to the gesture being performed at a first speed, and

the second Bayesian network includes a second plurality of states, wherein each respective state in the second plurality of states corresponds to a respective one of the plurality of distinct segments, and wherein the second Bayesian network corresponds to the gesture being performed at a second speed that is slower than the first speed; and

taking an action in to identifying that the mobile device completed the gesture.

16. The computer-readable medium of claim 15, wherein the one or more of the first and second Bayesian networks comprises a Hidden Markov Model.

17. The computer-readable medium of claim 15, wherein the action taken in response to identifying that the mobile device completed the gesture comprises recording speech from the person.

18. The computer-readable medium of claim 15, wherein the action taken in response to identifying that the mobile device completed the gesture comprises ceasing recording speech from the person.

19. The computer-readable medium of claim 15, wherein the action taken in response to identifying that the mobile device completed the gesture comprises providing feedback to a person, the feedback comprising at least one of auditory feedback, tactile feedback, and visual feedback.

20. The computer-readable medium of claim 19, wherein the feedback indicates that speech recording is commencing.

21. The computer-readable medium of claim 19, wherein the feedback indicates that speech recording is ceasing.

22. The computer-readable medium of claim 15, wherein the set of acceleration values uses a spherical polar coordinate system.

23. The computer-readable medium of claim 15, wherein each state of the first and second Bayesian networks is associated with at least one observation for the gesture associated with the Bayesian network, the at least one observation comprising at least one of an endpointer state, a proximity vector, and an audio vector.

24. The computer-readable medium of claim 15, further comprising training the first and second Bayesian networks based on real movement trajectories during multiple iterations of the gesture.

25. The computer-readable medium of claim 24, wherein the training the first and second Bayesian networks comprises aligning accelerometer timeseries data for each real movement trajectory.

26. The computer-readable medium of claim 25, wherein the step of aligning the accelerometer timeseries data for each real movement trajectory comprises trimming at least one of the accelerometer timeseries.

27. The computer-readable medium of claim 15, wherein detecting the movement of the mobile device is further based on camera data corresponding to motion of lips of a user of the mobile device.

28. The computer-readable medium of claim 15, wherein detecting the movement of the mobile device is further based on detecting a set of positional angles of the mobile device, and receiving proximity data from a proximity sensor included in the mobile device.