Device arbitration for listening devices

• US 9,812,126 B2
• Filed: 04/01/2015
• Issued: 11/07/2017
• Est. Priority Date: 11/28/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Khan, Yaser, Uzelac, Aleksandar, Hwang, Daniel J., Paolantonio, Sergio, Kam, Jenny, Kannan, Vishwac Sena, Mooney, II, Dennis James, Brush, Alice Jane Bernheim
• Primary Examiner(s): Leland, III, Edwin S
• Application Number:
  US14/676,647
• Publication Number:
  US 20160155443A1
• Time in Patent Office: 951 Days
• Field of Search: 704275
• CPC Class Codes: G06F 1/3203 Power management, i.e. event-based initiation of a power-saving mode
• CPC Class Codes: G06F 1/3209 Monitoring remote activity, e.g. over telephone lines or network connections
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command

Abstract

An electronic device in a topology of interconnected electronic devices can listen for a wake phrase and voice commands. The device can control when and how it responds so that a single device responds to voice commands. Per-task device preferences can be stored for a user. If a preferred device is not available, the task can still be performed on a device that has appropriate capabilities. Machine learning can determine a user'"'"'s preferences. Power conservation and effective user interaction can result.

Claims

1. A method of controlling which electronic device out of a topology of interconnected electronic devices responds to a wake phrase, the method comprising:

in an electronic device of the topology of interconnected electronic devices configured to listen for voice commands, receiving a wake phrase from a microphone of the electronic device;

recognizing the wake phrase; and

responsive to recognizing the wake phrase, waking the electronic device and controlling which one electronic device out of the topology of interconnected electronic devices plays an audio prompt that indicates an active listening state in response to the wake phrase, wherein the controlling invokes response arbitration rules selecting a single electronic device out of the topology of interconnected electronic devices as a primary device, wherein selection of the primary device is based on a determination that the single electronic device has been used most recently, of the interconnected electronic devices, prior to the wake phrase being received from the microphone, wherein the determination is based on recorded activity indicating physical activity near or at the electronic device.

2. The method of claim 1 wherein:

the response arbitration rules specify rules of succession for devices when a primary device is not available, the rules of succession indicating a fallback list of alternative devices.

3. The method of claim 2 wherein

an electronic device is designated as a device on the fallback list of alternative devices based on the frequency of usage.

4. The method of claim 1 wherein:

an electronic device having a second most recent physical activity prior to receipt of the wake phrase is chosen in response to determining that the primary device is not available.

5. The method of claim 1 further comprising:

determining whether the wake phrase was spoken by an authorized user, wherein the wake phrase is not recognized when not spoken by an authorized user.

6. The method of claim 1 wherein:

the recognizing is performed by an auxiliary processor of the electronic device while the electronic device is in a standby state.

7. The method of claim 6 wherein:

the controlling is performed by a main processor of the electronic device after the electronic device has transitioned out of the standby state; and

the electronic device consumes less power when in the standby state.

8. The method of claim 7 wherein:

waking the electronic device activates the main processor from an inactive state.

9. The method of claim 1 further comprising:

receiving a voice command with the microphone of the electronic device;

recognizing a task in the voice command received with the microphone; and

responsive to recognizing the task, controlling which one electronic device out of the topology of interconnected electronic devices performs the recognized task, wherein the controlling invokes task arbitration rules selecting a single electronic device out of the topology of interconnected electronic devices according to user device preference or stored capabilities of the interconnected electronic devices.

10. The method of claim 9 wherein:

the task arbitration rules specify rules of succession for devices when a preferred device for the task is not available.

11. The method of claim 9 further comprising:

recognizing repeated, consistent usage of a particular device for a particular task;

setting the user preference to indicate the particular device as associated with the particular task before listening;

wherein the recognized task comprises the particular task, and controlling which one electronic device performs the task comprises choosing the particular device for the particular task based on the user preference.

12. The method of claim 9 further comprising:

determining via the stored capabilities of the interconnected electronic devices that the recognized task is not performable at the electronic device; and

selecting an other electronic device to perform the recognized task.

13. The method of claim 12 further comprising:

handing the recognized task off to an other electronic device.

14. The method of claim 12 further comprising:

sending a wake command to the other electronic device.

15. The method of claim 12 wherein:

the selected single electronic device did not hear the voice command.

16. An electronic device configured to recognize a wake phrase when in a low-power standby state in an environment comprising a topology of interconnected electronic devices, the electronic device comprising:

a plurality of hardware sensors comprising a microphone;

a speaker; and

a stored device listening state, wherein the stored device listening state transitions from the low-power standby state upon recognition of the wake phrase as received by the microphone, and wherein the electronic device is configured to control which one electronic device out of the topology of interconnected electronic devices transitions to an active listening state and plays an audio prompt that indicates the active listening state in response to the wake phrase according to an electronic device, of the interconnected electronic devices, being used most recently prior to the wake phrase being received by the microphone, wherein the electronic device being used most recently is determined based on recorded activity indicating physical activity near or at the electronic device.

17. The electronic device of claim 16 wherein the electronic device further comprises:

an auxiliary voice recognition subsystem configured to recognize the wake phrase and transition the electronic device out of the low-power standby state; and

a main voice recognition subsystem configured to control which one electronic device out of the topology of interconnected electronic devices transitions to an active listening state.

18. The electronic device of claim 16 further comprising:

a voice recognizer configured to recognize a voice command received by the microphone of the electronic device, wherein the voice command comprises a task; and

a task arbiter configured to control which one electronic device out of the topology of interconnected electronic devices performs the task according to user preference or capabilities of the interconnected electronic devices.

19. At least one computer memory containing machine-executable instructions causing an electronic device to perform a method controlling which electronic device out of topology of interconnected electronic devices responds to a wake phrase, the method comprising:

receiving a wake phrase of a virtual persona from a microphone of the electronic device;

recognizing the wake phrase of the virtual persona in the electronic device;

responsive to recognizing the wake phrase of the virtual persona, waking the electronic device and controlling which one electronic device out of the topology of interconnected electronic devices plays an audio prompt that indicates a listening state in response to the wake phrase, wherein the controlling invokes arbitration rules selecting a single device out of the topology of interconnected electronic devices when a user preference indicating a primary device designation for the interconnected electronic devices indicates a primary device that is not available, wherein the single device is determined to be a device used most recently prior to receiving the wake phrase based on recorded activity representing physical activity near or at the single device, wherein the single device is different from the electronic device;

receiving a command phrase from the microphone of the electronic device;

recognizing a task in the command phrase;

via a timeout, determining that a preferred device for the task is not available;

responsive to determining that a preferred device for the task is not available, determining a fallback device for the task; and

handing the task off to the fallback device.