Determining user location with remote controller

• US 9,430,931 B1
• Filed: 06/18/2014
• Issued: 08/30/2016
• Est. Priority Date: 06/18/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Sadek, Ramy Sammy, Liu, Yue, Sjoberg, Robert Warren, Flenniken, Robert Ramsey
• Primary Examiner(s): Negron, Daniell L
• Application Number:
  US14/308,601
• Time in Patent Office: 804 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G08B 21/24 Reminder alarms, e.g. anti-loss alarms

Abstract

An audio device may be configured to work in conjunction with a handheld remote controller to receive voice commands from a user. The audio device may have multiple local microphones that are used for sound source localization, to determine the position of the user. A remote audio signal may be received from the remote controller and used in conjunction with local microphone signals generated by the local microphones to aid in determining the position of the user. The last known position of the user may be recorded whenever the user speaks into the remote controller. When the user is unable to find the remote controller, the audio device may direct the user toward the last known position of the user.

Claims

1. A device comprising:

a communications interface configured to receive a remote audio signal from a remote controller that is held and spoken into by a user, wherein the remote audio signal represents speech of the user;

an array of microphones configured to produce multiple local audio signals representing the speech of the user, wherein the microphones of the array have a fixed spatial arrangement within the device, and wherein the remote controller is at a variable position relative to the device;

a sound source localization service configured to analyze the local audio signals and the remote audio signal to determine one or more positional coordinates corresponding to a position of the user relative to the device when the user speaks into the remote controller;

a tracking service configured to record the one or more positional coordinates as an indication of a last determined location of the remote controller relative to the device; and

a notification service configured to provide information to the user regarding the last determined location of the remote controller based at least in part on the one or more positional coordinates.

2. The device of claim 1, wherein the one or more positional coordinates comprise one or more of:

a direction coordinate;

a distance coordinate;

or

a Cartesian coordinate.

3. The device of claim 1, wherein the notification service is configured to provide the information regarding the last determined location of the remote controller by verbally guiding the user.

4. The device of claim 1, further comprising a visual indicator capable of indicating different directions, wherein the notification service is configured to provide the information regarding the last determined location of the remote controller by visually indicating one of the different directions with the visual indicator.

5. The device of claim 1, wherein the sound source localization service comprises an audio beamformer that is configured to perform actions comprising:

processing the local audio signals to produce multiple directional audio signals that emphasize sound from different directions, respectively; and

comparing the remote audio signal to each of the directional audio signals to determine which directional audio signal has the strongest presence of the speech of the user.

6. The device of claim 1, wherein the sound source localization service is further configured to compare a strength of the speech of the user at the remote controller to a strength of the speech of the user at the device to determine a distance of the remote controller from the device.

7. A method, comprising:

receiving, at a device, a remote audio signal from a remote controller that is held and spoken into by a user, wherein the remote audio signal represents speech of the user;

receiving one or more local microphone audio signals from an array of microphones on the device, the one or more local microphone audio signals represent the speech of the user;

analyzing the one or more local microphone audio signals based at least in part on the remote audio signal to determine a position of the user relative to the device;

recording the determined position of the user as a last determined location of the remote controller relative to the device; and

notifying the user regarding the last determined location of the remote controller.

8. The method of claim 7, wherein determining the position of the user comprises determining one or more of:

a direction coordinate;

a distance coordinate;

or

a Cartesian coordinate.

9. The method of claim 7, wherein the remote audio signal is received using a Bluetooth interface.

10. The method of claim 7, wherein the analyzing comprises performing audio beamforming.

11. The method of claim 7, wherein the analyzing comprises comparing a first strength of the speech of the user in the remote audio signal to a second strength of the speech of the user in the one or more local microphone audio signals.

12. The method of claim 7, wherein the notifying comprises verbally guiding the user.

13. The method of claim 7, wherein the notifying comprises visually guiding the user.

14. The method of claim 7, wherein the notifying comprises identifying landmarks that are near the last known location of the remote controller.

15. The method of claim 7, wherein the one or more local microphone audio signals are produced by microphones at different microphone locations in the array;

and wherein analyzing the one or more microphone audio signals comprises;

identifying the speech of the user in each of the one or more microphone audio signals based at least in part on the remote audio signal; and

evaluating differences in arrival times of the speech of the user at each of the microphones based at least in part on the identified speech of the user in the one or more microphone audio signals.

16. The method of claim 7, wherein analyzing the one or more microphone audio signals comprises:

processing the one or more microphone audio signals to produce multiple directional signals that emphasize sound originating from different directions, respectively; and

comparing the remote audio signal to each of the directional audio signals to determine which directional audio signal has the strongest presence of the speech of the user.

17. The method of claim 7, wherein:

the remote controller is held at a first distance from a mouth of the user;

the one or more microphone audio signals are received from one or more microphones that are at least a second distance from the mouth of the user; and

the second distance is greater than the first distance.

18. A method comprising:

receiving, by a device within an environment, a reference audio signal from a handheld device;

receiving one or more microphone audio signals that contain audio from a source proximate to the handheld device, wherein the one or more microphone audio signals are received from an array of microphones having a fixed spatial arrangement within the device; and

performing sound source localization on the one or more microphone audio signals based at least in part on the reference audio signal to determine a position of the handheld device within the environment.

19. The method of claim 18, wherein the reference audio signal corresponds to an identifying sound that is emitted by the handheld device and the audio from a source proximate to the handheld device comprises audio emitted by the handheld device.

20. The method of claim 18, wherein determining the position of the handheld device comprises determining one or more one or more of:

a direction coordinate;

a distance coordinate;

or

a Cartesian coordinate.

21. The method of claim 18, wherein performing the sound source localization comprises:

identifying user speech in each of the one or more microphone audio signals based at least in part on the reference audio signal; and

evaluating differences in arrival times of the user speech at each of the microphones in the array of microphones based at least in part on the identified user speech in the one or more microphone audio signals.

22. The method of claim 18, wherein performing the sound source localization comprises:

processing the one or more microphone audio signals to produce multiple directional signals that emphasize sound originating from different directions, respectively; and

comparing the reference audio signal to each of the directional audio signals to determine which directional audio signal has the strongest presence of the audio.

23. The method of claim 18, wherein:

the handheld device is held at a first distance from a mouth of a user;

the one or more microphones are at least a second distance from the mouth of the user; and

the second distance is greater than the first distance.