Methods and apparatus for learning sensor data patterns for gesture-based input

• US 9,582,080 B1
• Filed: 06/25/2015
• Issued: 02/28/2017
• Est. Priority Date: 06/25/2014
• Status: Active Grant

• Current Assignee: Bosch Sensortec GmbH (Robert Bosch GmbH)
• Original Assignee: Rithmio, Inc.
• Inventors: Tilton, Adam K., Ghiotto, Shane T., Mehta, Prashant G.
• Primary Examiner(s): NGUYEN, JENNIFER T
• Application Number:
  US14/751,073
• Time in Patent Office: 614 Days
• Field of Search: 345173-178, 345156-157, 715/863
• US Class Current: 1/1
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/047 Pattern matching networks; Rete networks
• CPC Class Codes: G06N 7/00 Computing arrangements based on specific mathematical models
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: G06V 40/25 Recognition of walking or running movements, e.g. gait recognition
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/144 Training of HMMs
• CPC Class Codes: G10L 25/03 characterised by the type of extracted parameters

Abstract

Methods and systems for learning, recognition, classification and analysis of real-world cyclic patterns using a model having n oscillators, with primary frequency ω₁, ω₂, . . . , ω_n. The state of the oscillators is evolved over time using sensor observations, which are also used to determine the sensor characteristics, or the sensor observation functions. Once trained, a set of activity detection filters may be used to classify sensor data as being associated with an activity.

Claims

1. A method for generating a gesture-detection (GD) filter comprising:

receiving sensor data captured by at least one sensor during a gesture, the at least one sensor selected from the group consisting of (i) a motion sensor that is attached to a user, (ii) a motion sensor that is attached to a user equipment, (iii) an audio sensor, and (iv) an electrode array attached to a head of the user, the sensor data comprising a plurality of sequential sensor measurements;

determining a set of activity frequencies associated with the sensor data received during the gesture;

iteratively calculating;

(i) a prediction error based on a difference between at least one of the plurality of sequential sensor measurements and an observation prediction ĥ

_t from an observation model, the observation model operating on phases in a set of phases associated with the set of activity frequencies;

(ii) an updated set of estimated activity phases based on the prediction error and the set of activity frequencies;

(iii) an updated observation function of the observation model based on the prediction error; and

generating the GD filter based at least in part of the set of activity frequencies and the updated observation function.

2. The method of claim 1 further comprising:

determining a convergence of the observation function.

3. The method of claim 2 wherein generating the GD filter based at least in part of the set of activity frequencies and the updated observation function is carried out in response to the determined convergence of the observation function.

4. The method of claim 1 wherein generating the GD filter based at least in part of the set of activity frequencies and the updated observation function is carried out in response to receiving a generate-filter command.

5. The method of claim 1 wherein the gesture is a physical gesture.

6. The method of claim 5 wherein the physical gesture is selected from the group consisting of (i) a single-hand gesture, (ii) a two-hand gesture, (iii) a head gesture, (iv) a full body gesture, and (v) a user-equipment assisted gesture.

7. The method of claim 1 wherein the gesture is an audible gesture.

8. The method of claim 7 wherein the audible gesture is selected from the group consisting of (i) a spoken word, (ii) a spoken phrase, (iii) a vocal sound, (iv) a vocal sound pattern, (v) a tone of voice, (vi) a tonal vocal pattern, and (vii) a non-vocal user-produced sound.

9. The method of claim 1 wherein the gesture is a mental gesture.

10. The method of claim 9 wherein the mental gesture is selected from the group consisting of (i) a thought of a user, and (ii) a thought pattern of the user.

11. The method of claim 1 wherein the at least one sensor comprises an accelerometer.

12. The method of claim 1 wherein the at least one sensor comprises a gyroscope.

13. The method of claim 1 wherein the at least one sensor comprises a microphone.

14. The method of claim 1, wherein the gesture is associated with a computer task.

15. The method of claim 14 wherein the computer task is a computer task selected from the group consisting of (i) opening an application, (ii) closing an application, (iii) minimizing an application, (iv) navigating a user interface, (v) a multimedia-playback control, (vi) a text input.

16. The method of claim 1, wherein the gesture is associated with a robotic-system task.

17. The method of claim 16 wherein the robotic system task is a robotic system task selected from the group consisting of (i) activating a servo-motor to rotate in a certain direction, (ii) activating a servo-motor to rotate through a certain angle, (iii) activating a servo-motor to rotate at a certain speed, (iv) activating a linear actuator to move in a certain direction, (v) activating a linear actuator to move a certain distance, (vi) activating a linear actuator to move at a certain speed, (vii) activating a motor to rotate in a certain direction, and (viii) activating a motor to rotate at a certain speed.

18. The method of claim 1 further comprising:

uploading the generated GD filter to a cloud database of GD filters.

19. The method of claim 1 further comprising:

comparing the generated GD filter with a set of previously generated and named GD filters;

determining a most-likely name for the generated GD filter based on the comparison; and

naming the generated GD filter with the determined most-likely name.

20. An apparatus comprising a non-transitory computer-readable medium having instructions stored thereon that when executed cause a processor to:

receive sensor data captured by at least one sensor during a gesture, the at least one sensor selected from the group consisting of (i) a motion sensor that is attached to a user, (ii) a motion sensor that is attached to a user equipment, and (iii) an audio sensor, the sensor data comprising a plurality of sequential sensor measurements;

determine a set of activity frequencies associated with the sensor data received during the gesture;

iteratively calculate;

(i) a prediction error based on a difference between at least one of the plurality of sequential sensor measurements and an observation prediction ĥ

_t from an observation model, the observation model operating on phases in a set of phases associated with the set of activity frequencies;

(ii) an updated set of estimated activity phases based on the prediction error and the set of activity frequencies;

(iii) an updated observation function of the observation model based on the prediction error; and

generate the GD filter based at least in part of the set of activity frequencies and the updated observation function.