Audio-based gesture detection

• US 9,494,683 B1
• Filed: 06/18/2013
• Issued: 11/15/2016
• Est. Priority Date: 06/18/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Sadek, Ramy Sammy
• Primary Examiner(s): Ratcliffe, Luke
• Assistant Examiner(s): Baghdasaryan, Hovhannes
• Application Number:
  US13/920,975
• Time in Patent Office: 1,246 Days
• US Class Current: 1/1
• CPC Class Codes: G01S 15/04 Systems determining presence of a target
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path

Abstract

An audio device may be configured to detect proximity of a user hand based on analysis of audio signals captured by the audio device. For example, the audio device may be configured to perform acoustic echo cancellation based on dynamically calculated filter coefficients. The coefficients may be analyzed to determine proximity of the user hand. The proximity of the hand may be evaluated over time to detect user hand gestures.

Claims

1. An audio device, comprising:

a speaker configured to generate output audio based at least in part on an output audio signal;

a microphone configured to produce an input audio signal, wherein the input audio signal contains one or more acoustic echoes of the output audio;

an acoustic echo cancellation element configured to;

receive the input audio signal;

dynamically update a sequence of acoustic echo cancellation (AEC) coefficients to estimate the one or more acoustic echoes of the output audio, wherein a first one of the AEC coefficients corresponds to a first distance from the audio device and a second one of the AEC coefficients corresponds to a second distance from the audio device; and

a gesture detection element configured to;

receive, from the acoustic echo cancellation element, the sequence of AEC coefficients;

detect a first proximity of an object to the audio device based at least in part on relative positions of the AEC coefficients within the sequence and amplitudes of a least a portion of the AEC coefficients in the sequence within a first set of the sequence of AEC coefficients corresponding to a first time;

detect a second proximity of the object to the audio device based at least in part on relative positions of the AEC coefficients within the sequence and amplitudes of the least a portion of the AEC coefficients in the sequence within a second set of the sequence of AEC coefficients corresponding to a second time; and

declare a gesture event when the first proximity of the object to the audio device and the second proximity of the object to the audio device indicate that the object is near the audio device and moving toward the audio device.

2. The audio device of claim 1, wherein the audio device is configured to mute and unmute the output audio in response to a mute gesture event.

3. The audio device of claim 1, wherein the gesture detection element detects the first proximity and the second proximity by identifying AEC coefficients from the first and second sets that correspond to an acoustic reflection of the output audio.

4. The audio device of claim 1, wherein the gesture detection element is further configured to detect motion of the object based at least in part on the sequence of AEC coefficients and to generate the gesture event based at least in part on the motion of the object.

5. The audio device of claim 1, wherein the acoustic echo cancellation element comprises an adaptive finite impulse response filter.

6. The audio device of claim 1, further comprising a plurality of microphones.

7. A method comprising:

receiving, from a microphone, an input audio signal that includes an acoustic echo generated, at least in part, on acoustic coupling between the microphone and a speaker and a position of an object;

updating a sequence of acoustic echo cancellation (AEC) parameters to cancel at least a portion of the acoustic echo in the input audio signal, wherein the AEC parameters include a first parameter related to a first distance from the object and a second parameter related to a second distance from the object; and

evaluating the AEC parameters to determine proximity of the object relative to one or more of the microphone or the speaker, wherein evaluating the AEC parameters includes detecting a positional change of one or more of the AEC parameters within the sequence.

8. The method of claim 7, wherein the parameters comprise a dynamically updated sequence of AEC filter parameters, and wherein the evaluating comprises identifying one or more of the AEC filter parameters that correspond to the acoustic echo.

9. The method of claim 7, further comprising declaring a user input event based at least in part on the determined proximity of the object.

10. The method of claim 7, further comprising muting or unmuting an audio device based at least in part on the determined proximity of the object.

11. The method of claim 7, further comprising controlling an audio device based at least in part on the determined proximity of the object.

12. The method of claim 7, wherein the cancelling uses an adaptive filter and wherein the AEC parameters comprise AEC coefficients that are dynamically updated by the adaptive filter.

13. The method of claim 7, further comprising repeatedly evaluating the proximity of the object to determine motion of the object.

14. The method of claim 7, further comprising:

repeatedly evaluating the proximity of the object to determine motion of the object; and

declaring a user input event based at least in part on the determined motion of the object.

15. One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform acts comprising:

causing output audio to be produced;

causing an input audio signal to be produced that contains one or more components of the output audio; and

comparing the input audio signal to the output audio to detect proximity of an object relative to one or more of a microphone or speaker based, at least in part, on detection of a flanging of the input audio signal relative to the output audio.

16. The one or more non-transitory computer-readable media of claim 15, wherein the comparing comprises adjusting one or more parameters to cancel at least a portion of acoustic echo in the input audio signal.

17. The one or more non-transitory computer-readable media of claim 15, wherein the comparing comprises estimating an audio echo component of the input audio signal, wherein the audio echo component is based at least in part on an acoustic reflection of the output audio by the object.

18. The one or more non-transitory computer-readable media of claim 15, further comprising generating flanged signals based at least in part on the output audio and comparing the flanged signals to the input audio signal.

19. The one or more non-transitory computer-readable media of claim 15, the acts further comprising declaring a user interface event in response to the detected proximity of the object.

20. The one or more non-transitory computer-readable media of claim 15, the acts further comprising controlling an audio device based at least in part on the detected proximity of the object.