Voice controlled assistant with non-verbal user input

• US 9,864,576 B1
• Filed: 09/09/2013
• Issued: 01/09/2018
• Est. Priority Date: 09/09/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Liu, Baiyang, Secker-Walker, Hugh Evan
• Primary Examiner(s): Nguyen, An T
• Application Number:
  US14/021,824
• Time in Patent Office: 1,583 Days
• CPC Class Codes: G06F 1/1688 the I/O peripheral being integrated loudspeakers
• CPC Class Codes: G06F 1/169 the I/O peripheral being an integrated pointing device, e.g. trackball in the palm rest area, mini-joystick integrated between keyboard keys, touch pads or touch stripes G06F1/1643 takes precedence; constructional details of pointing devices G06F3/033
• CPC Class Codes: G06F 1/181 Enclosures for portable computers G06F1/1613
• CPC Class Codes: G06F 1/3215 Monitoring of peripheral devices
• CPC Class Codes: G06F 1/325 Power saving in peripheral device
• CPC Class Codes: G06F 2200/1636 Sensing arrangement for detection of a tap gesture on the housing
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback

Abstract

A voice controlled assistant having a housing to hold one or more microphones, one or more speakers, and various computing components. The voice controlled assistant facilitates transactions and other functions primarily through verbal interactions with a user. In some situations, a transaction may require entry of a code, which the user may wish to enter in a non-verbal way. The voice controlled assistant is configured to analyze an audio signal to detect user interactions with the surface of the voice controlled assistant and to interpret the detected interactions as entry of the code.

Claims

1. A device comprising:

a housing comprising at least one side and a top surface, the top surface comprising a first portion associated with a first zone and a second portion associated with a second zone;

one or more processors,

memory;

one or more speakers configured to output an audible prompt to enter an input via one or more taps on the top surface of the device;

one or more microphones configured to capture audio and generate an audio signal based on the captured audio, the audio signal representing a series of taps, the series of taps comprising, in order, a first tap on the top surface of the housing and a second tap on the top surface of the housing, the first tap being at a first location in the first portion and the second tap being at a second location in the second portion;

a pattern recognition module, stored in the memory and executable by the one or more processors to analyze the audio signal generated by the one or more microphones to;

detect, within the audio signal, the first tap and the second tap,

determine the first location of the first tap and the second location of the second tap,

compare a tap sequence corresponding to the first tap at the first location and the second tap at the second tap location with a predetermined tap sequence associated with the input, and

determine that the tap sequence corresponds to the predetermined tap sequence; and

a light indicator configured to emit light from the housing proximate the top surface of the housing, the light indicator being controlled to provide;

at least in part in response to determining the first location by analyzing the audio signal, a first visual cue at the first zone at a first time, and at least in part in response to determining the second location by analyzing the audio signal, a second visual cue at the second zone at a second time subsequent to the first time.

2. The device of claim 1, wherein the predetermined tap sequence corresponds to a preset code that has been preset by the user, by another user, or by a manufacturer of the device.

3. The device of claim 1, further comprising a control module stored in the memory and executable by the one or more processors to perform one or more actions at least partly in response to the evaluating.

4. The device of claim 3, wherein the one or more actions include at least one of:

increasing a volume at which the one or more speakers output the sound;

decreasing the volume at which the one or more speakers output the sound;

muting the one or more speakers;

powering off the device;

putting the device to sleep;

confirming a transaction;

authorizing a transaction;

or

adjusting at least one setting of the device.

5. The device of claim 1, wherein the pattern recognition module is further executable by the one or more processors to perform a spectral analysis on the audio signal to detect the series of taps.

19. The device as recited in claim 1, wherein the pattern recognition module determines the location of each tap by analyzing the audio signal containing the tap to determine from various signal components provided by the one or more microphones, a direction from which the audio originated.

20. The device as recited in claim 1, wherein the light indicator comprises multiple light segments that are independently controllable to exhibit at least one of on/off illumination states, different colors, sequences, or visual effect.

25. The device as recited in claim 1, wherein the light indicator is further configured to illuminate the first zone after receipt of the entirety of the series of taps and to cease illuminating the first zone prior to illuminating the second zone.

6. A method comprising:

under control of a computing device configured with executable instructions,

outputting an audible prompt to enter a code via a series of taps on a surface of the computing device;

illuminating a first visual designation on the surface of the computing device and a second visual designation on the surface of the computing device, the first visual designation identifying a first zone on the surface and the second visual designation identifying a second zone on the surface;

generating an audio signal from sound captured from an environment by one or more microphones;

identifying a first tap within the audio signal, the first tap corresponding to a first contact with the surface;

based at least in part on characteristics of the audio signal;

determining a first location of the first tap;

determining that the first location corresponds to the first zone; and

illuminating a first visual cue at the first zone;

identifying a second tap within the audio signal, the second tap corresponding to a second contact with the surface subsequent to the first tap;

based at least in part on characteristics of the audio signal;

determining a second location of the second tap;

determining that the second location corresponds to the second zone; and

illuminating a second visual cue at the second zone subsequent to the first visual cue; and

interpreting the first tap and the second tap correspond to the predetermined series.

7. The method as recited in claim 6, further comprising illuminating the first zone and the second zone on the surface of the computing device in an order in which the first and second taps are identified.

8. The method as recited in claim 6, wherein:

the first location corresponds to a particular portion of the housing of the computing device; and

the user input comprises a value associated with the particular portion of the housing.

9. The method as recited in claim 6, further comprising

evaluating the user input;

and

performing one or more functions in response to the evaluating, the one or more functions being selected from a group of functions comprising operation-based functions, application-driven functions, communication-related functions, security-related functions, and entertainment functions.

10. The method as recited in claim 6, wherein the audible prompt prompts the user to enter a preset code, the method further comprising

determining whether the user input corresponds to the preset code;

and

permitting a function to be performed if the user input corresponds to the preset code.

11. The method as recited in claim 6, further comprising at least one of:

evaluating whether the user input corresponds to a preset code;

or

transmitting the user input to a remote system for evaluating whether the user input corresponds to the preset code.

21. The method as recited in claim 6, wherein the determining the location of the tap comprises analyzing the audio signal containing the tap to determine from various signal components provided by the one or more microphones, a direction from which the sound originated to identify a location associated the tap.

22. The method as recited in claim 6, wherein illuminating one or more zones comprises controlling individual light elements, of a light indicator with multiple light elements, to provide a different visible appearance of the individual light elements.

23. The method as recited in claim 6, wherein illuminating one or more zones comprises actively controlling individual light emitting diodes (LEDs), from among multiple LEDs mounted on the computing device, to exhibit at least one of different colors, sequences, or visual effect.

12. A device comprising:

one or more microphones configured to capture audio and generate an audio signal based on the captured audio;

one or more speakers configured to output audio;

one or more processors; and

one or more computer-readable media having computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising;

causing to be output, via the one or more speakers, an audible prompt to enter information via one or more contacts with a surface of the device;

detecting a first tap within the audio signal, the first tap corresponding to a first contact with the device after the audible prompt;

based on characteristics of the audio signal;

determining a first location on the device associated with the first tap; and

illuminating a first indicator zone on the device from among multiple indicator zones, the first indicator zone being proximate the first location;

interpreting the tap as a user input, a value of the user input based at least in part on the location; and

upon receiving a second tap at a second location on the device, illuminating a second indicator zone from the multiple indicator zones, the second indicator zone being proximate the second location.

13. The device of claim 12, further comprising a light indicator comprising multiple zones and configured to emit light, the emitted light to illuminate one or more zones of the multiple zones at least partly in response to the user providing the tap.

14. The device of claim 13, wherein the tap is within one of the zones and the user input corresponds to an input value associated with the zone tapped.

15. The device of claim 13, wherein each of the one or more zones corresponds to an alpha-numerical value.

16. The device of claim 12, further comprising a haptic mechanism configured to output a touch-perceptive sensation at least partly in response to detecting the tap.

17. The device of claim 12, the operations further comprising performing one or more functions in response to identifying that the tap, the one or more functions selected from a group of functions comprising operation-based functions, application-driven functions, communication-related functions, security-related functions, and entertainment functions.

18. The device of claim 12, wherein the location is determined using at least one of:

passive acoustic location techniques;

beamforming techniques;

acoustic triangulation;

or

time-of-arrival location techniques.

24. The device as recited in claim 12, wherein determining a location comprises, for each tap, analyzing the audio signal containing the tap to determine from various signal components provided by the one or more microphones, a direction from which the sound originated.