Response endpoint selection

• US 9,271,111 B2
• Filed: 12/14/2012
• Issued: 02/23/2016
• Est. Priority Date: 12/14/2012
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Blanksteen, Scott Ian
• Primary Examiner(s): Barry, Lance L
• Application Number:
  US13/715,741
• Publication Number:
  US 20140172953A1
• Time in Patent Office: 1,166 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: H04L 51/07 characterised by the inclusion of specific contents
• CPC Class Codes: H04L 51/10 Multimedia information
• CPC Class Codes: H04L 51/18 Commands or executable codes
• CPC Class Codes: H04L 51/21 Monitoring or handling of messages
• CPC Class Codes: H04L 67/14 Session management for real-time applications in data packet communications networks H04L65/1066
• CPC Class Codes: H04L 67/306 User profiles
• CPC Class Codes: H04L 67/535 Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34
• CPC Class Codes: H04W 4/02 Services making use of location information
• CPC Class Codes: H04W 4/029 Location-based management or tracking services
• CPC Class Codes: H04W 4/33 for indoor environments, e.g. buildings

Abstract

A computing system has multiple endpoint computing devices in local environments to receive verbal requests from various users and a central or remote system to process the requests. The remote system generates responses and uses a variety of techniques to determine where and when to return responses audibly to the users. For each request, the remote system understands who is making the request, determines when to provide the response to the user, ascertains where the user is when it is time to deliver the response, discovers which of the endpoint devices are available to deliver the response, and evaluates which of the available devices is best to deliver the response. The system then delivers the response to the best endpoint device for audible emission or other form of presentation to the user.

Claims

1. A computing system comprising:

a remote computing system;

multiple endpoint devices located in various locations local to one or more users, a first endpoint device comprising;

one or more processors;

computer-readable storage media storing computer-executable instructions;

at least one microphone to receive audio input from a user, the audio input containing a user request; and

an interface to transmit the user request to the remote computing system;

the remote computing system comprises one or more executable modules configured to produce a response to the user request, to determine when to deliver the response, to select a second endpoint device that is available to provide the response to the user, and to send the response to the second endpoint device; and

the second endpoint device comprising;

one or more processors;

computer-readable storage media storing computer-executable instructions;

a camera to capture images; and

an interface to send the captured images to the remote computing system for ascertaining the location of the user and receive the response from the remote computing system; and

at least one speaker to output the response in audio form to the user.

2. The computing system as recited in claim 1, wherein the user request is selected from a group of requests comprising reminders, timers, alarms, calendar entries, directions, instructions, and reservations.

3. The computing system as recited in claim 1, wherein the remote computing system is configured to determine when to deliver the response by at least one of performing natural language understanding processing on the user request, using information from a calendar application, using information from a user profile associated with the user, or using information about events in an activity history associated with the user.

4. The computing system as recited in claim 1, wherein the first endpoint device further comprises a speech recognition module maintained in the one or more computer-readable storage media and executed by the one or more processors to convert a signal from the microphone representing the audio input of the user into text.

5. The computing system as recited in claim 1, wherein the one or more modules of the remote computing system are further configured to ascertain a location of the user prior to selecting the second endpoint device that is available at the location to provide the response to the user.

6. The computing system as recited in claim 1, further comprising a third endpoint device, wherein the one or more modules of the remote computing system are further configured to choose between the second and third endpoint devices to provide the response to the user.

7. The computing system as recited in claim 1, wherein the remote computing system is further configured to ascertain the location of the user by receiving audio data from one or more of the endpoint devices.

8. The computing system as recited in claim 1, wherein the remote computing system is further configured to ascertain the location of the user by reviewing at least one of a calendar associated with the user or an activity history of the user.

9. The computing system as recited in claim 1, wherein the remote computing system is configured to select the second endpoint device by evaluating the one or more of the endpoint devices using at least one analysis comprising:

a distance analysis to determine a distance of an endpoint device from the user;

a background analysis to determine a volume of background noise of an endpoint device;

a signal-to-noise ratio (SNR) analysis to determine an SNR at an endpoint device with respect to the user and background noise sources;

an echo analysis to determine echo characteristics of an environment in which an endpoint device resides;

a Doppler analysis to determine Doppler characteristics of audio input from the user relative to an endpoint device; and

an environment analysis to determine a number of people proximal to an endpoint device.

10. A computer-implemented method comprising:

under control of one or more computer systems configured with executable instructions,

receiving, from a first computing device, a request initiated by a first user;

processing the request to generate a response;

selecting a second computing device to deliver the response, the second computing device associated with a second user different from the first user; and

delivering the response to the selected second computing device.

11. The computer-implemented method as recited in claim 10, wherein receiving the request comprises receiving audio input indicative of voice entry by the first user into the first computing device and delivering the response comprises sending audio data for audio output to the second user by the second computing device different from the first computing device.

12. The computer-implemented method as recited in claim 10, wherein selecting the second computing device to deliver the response comprises ascertaining a location of the second user to receive the response and selecting the second computing device from among multiple computing devices available at the location.

13. The computer-implemented method as recited in claim 12, wherein ascertaining a location of the second user comprises at least one of:

polling one or more optical devices for visual confirmation of the second user;

polling one or more audio devices for voice confirmation of the second user;

locating an electronic device associated with the second user;

or

reviewing a calendar associated with the second user.

14. The computer-implemented method as recited in claim 10, wherein selecting the second computing device comprises at least one of:

analyzing proximity of the second computing device to the second user;

analyzing volume of background noise of the second computing device;

analyzing signal-to-noise ratio of the second computing device with respect to the second user and background noise sources;

analyzing echo characteristics of an environment in which the second computing device resides;

analyzing Doppler characteristics of audio input from the second user relative to the second computing device;

or

analyzing a number of people proximal to the second computing device.

15. The computer-implemented method as recited in claim 10, further comprising determining a time to return the response.

16. The computer-implemented method as recited in claim 10, further comprising determining a time to return the response by, in part, performing natural language understanding on the request.

17. A computer-implemented method comprising:

under control of one or more computer systems configured with executable instructions,

receiving, from a first computing device, a message for delivery from a first user;

determining a location of a second user that is different from the first user;

selecting a second computing device; and

delivering the message to the selected second computing device for presentation to the second user.

18. The computer-implemented method as recited in claim 17, further comprising determining a time to deliver the message to the second user.

19. The computer-implemented method as recited in claim 17, wherein determining a location of the second user comprises at least one of:

polling one or more optical devices for visual confirmation of the second user;

polling one or more audio devices for voice confirmation of the second user;

locating an electronic device associated with the second user;

or

reviewing a calendar associated with the second user.

20. The computer-implemented method as recited in claim 17, wherein selecting the second computing device comprises determining multiple computing devices available at the location and choosing the second computing device from among the multiple computing devices available at the location.

21. The computer-implemented method as recited in claim 17, further comprising repeating the determining, the selecting, and the delivering to resend the message to the second user.