Application focus in speech-based systems

• US 9,552,816 B2
• Filed: 12/19/2014
• Issued: 01/24/2017
• Est. Priority Date: 12/19/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: VanLund, Peter Spalding, Piersol, Kurt Wesley, Meyers, James David, Simpson, Jacob Michael, Gundeti, Vikram Kumar, Thomas, David Robert, Miles, Andrew Christopher
• Primary Examiner(s): Baker, Charlotte M
• Application Number:
  US14/578,056
• Publication Number:
  US 20160180853A1
• Time in Patent Office: 767 Days
• Field of Search: 704/275, 704/235, 704/270.1, 704/E15.047, 707/722, 707/740, 725/58, 709/203, 455/414.1, 381/93, 381/121, 381/318, 381/83, 381/95, 381/96, 379/88.16, 379/88.17
• US Class Current: 1/1
• CPC Class Codes: G06F 9/5011 the resources being hardware resources other than CPUs, Servers and Terminals
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 17/22 Interactive procedures; Man-machine interfaces
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/228 of application context

Abstract

A speech-based system includes an audio device in a user premises and a network-based service that supports use of the audio device by multiple applications. The audio device may be directed to play audio content such as music, audio books, etc. The audio device may also be directed to interact with a user through speech. The network-based service monitors event messages received from the audio device to determine which of the multiple applications currently has speech focus. When receiving speech from a user, the service first offers the corresponding meaning to the application, if any, that currently has primary speech focus. If there is no application that currently has primary speech focus, or if the application having primary speech focus is not able to respond to the meaning, the service then offers the user meaning to the application that currently has secondary speech focus.

Claims

1. A system, comprising:

a command service configured to;

communicate with multiple applications, communicate with an audio device, and send a command to the audio device to perform an activity for an audio application that provides audio content to be played by the audio device, wherein the command specifies an application identifier corresponding to the audio application;

control logic configured to perform acts comprising;

receiving an event message from the audio device regarding sound played by the audio device, wherein the event message specifies the application identifier corresponding to the audio application;

if the event message indicates that the sound played by the audio device is part of a speech interaction with a user, designating the audio application as being primarily active;

if the event message indicates that the sound played by the audio device is not part of a speech interaction with a user, designating the audio application as being secondarily active;

a speech recognition service configured to receive an audio signal from the audio device and to recognize user speech in the audio signal;

a language understanding service configured to determine a meaning of the user speech;

the control logic being configured to perform further actions comprising;

if there is a primarily active application among the multiple applications, requesting that the primarily active application respond to the user speech by (a) performing a first action that is indicated at least in part by the meaning of the user speech or (b) generating a first speech response to the user speech; and

if there is no primarily active application among the multiple applications and if there is a secondarily active application among the multiple applications, requesting that the secondarily active application respond to the user speech by (a) performing a second action that is indicated at least in part by the meaning of the user speech or (b) generating a second speech response to the user speech.

2. The system of claim 1, wherein the event message specifies an event classification indicating whether the sound is part of a speech interaction with the user, the classification indicating that the sound comprises at least one of:

speech that is part of a user interaction;

speech that is not part of a user interaction;

audio content that is part of a user interaction;

audio content that is not part of a user interaction;

or

an audio notification given in response to detection by the audio device of a condition.

3. The system of claim 1, wherein the event message indicates that the second audio is a notification given in response to detection by the audio device of a condition, the acts further comprising designating the audio application as being primarily active.

4. The system of claim 1, the actions further comprising:

determining that no event message identifying the audio application has been received for a predefined time period; and

removing the designation of the audio application being primarily active.

5. A method, comprising:

providing a command to an audio device to perform an activity, wherein the command identifies a responsible application from among multiple applications;

receiving an event message from the audio device regarding sound presented by the audio device, the event message identifying the responsible application;

if the event message indicates that the sound is part of a user interaction, designating the responsible application as being primarily active;

receiving speech captured by the audio device;

determining a meaning of the speech; and

if there is a primarily active application among the multiple applications that can respond to the meaning, requesting the primarily active application to respond to the meaning.

6. The method of claim 5, further comprising:

if the event message does not indicate that the audio is part of a user interaction, designating the responsible application as being secondarily active; and

if there is no primarily active application among the multiple applications that can response to the meaning, requesting a secondarily active application of the multiple applications to respond to the meaning.

7. The method of claim 6, further comprising, if there is no primarily active application among the multiple applications that can response to the meaning:

determining that the secondarily active application can respond to the meaning; and

designating the secondarily active application as being primarily active.

8. The method of claim 6, further comprising:

receiving an indication from the primarily active application that the primarily active application will not respond to the meaning; and

in response to receiving the indication from the primarily active application, requesting the secondarily active application to respond to the meaning.

9. The method of claim 5, further comprising determining that the primarily active application can respond to the meaning before requesting the primarily active application to respond to the meaning.

10. The method of claim 5, wherein the classification indicates that the audio is at least one of:

speech that is part of a user interaction;

speech that is not part of a user interaction;

audio content that is part of a user interaction;

audio content that is not part of a user interaction;

or

an audio notification given in response to detection by the audio device of a condition.

11. The method of claim 10, wherein the audio notification comprises:

a background audio notification that is not part of a user interaction;

or

a foreground audio notification that is part of a user interaction.

12. The method of claim 5, wherein:

the command specifies an application identifier that identifies the responsible application; and

the event message specifies the application identifier to identify the responsible application.

13. The method of claim 5, further comprising:

determining that no event message identifying the responsible application has been received for a predefined time period; and

removing the designation of the responsible application as being primarily active.

14. A method, comprising:

receiving a first event message from a device regarding a first action performed by the device, the event message identifying a first responsible application from among multiple applications, wherein each of the multiple applications can respond to one or more meanings expressed by user speech;

determining that the first action is part of a user interaction;

designating the first responsible application as being primarily active;

identifying a first meaning of first user speech; and

determining that there is a primarily active application among the multiple applications that can respond to the first meaning; and

selecting the primarily active application to respond to the first meaning.

15. The method of claim 14, further comprising:

receiving a second event message from the device regarding a second action performed by the device, the second event message identifying a second responsible application from among the multiple applications;

determining that the second action is not part of a user interaction;

designating the second responsible application as being secondarily active;

determining a second meaning of second user speech;

determining that there is no primarily active application among the multiple applications that can respond to the second meaning; and

selecting the secondarily active application to respond to the second meaning.

16. The method of claim 15, further comprising:

determining a third meaning of third user speech;

determining that the primarily active application will not respond to the third meaning; and

requesting the secondarily active application to respond to the third meaning.

17. The method of claim 15, further comprising:

determining a third meaning of third user speech;

receiving an indication from the primarily active application that the primarily active application will not respond to the third meaning; and

requesting the secondarily active application to respond to the third meaning.

18. The method of claim 14, wherein the event message indicates a classification of the audio, the classification indicating that the audio is:

speech that is part of a user interaction;

speech that is not part of a user interaction;

audio content that is part of a user interaction;

audio content that is not part of a user interaction;

or

an audio notification given in response to detection by the audio device of a condition.

19. The method of claim 18, wherein the audio notification comprises:

a background audio notification that is not part of a user interaction;

or

a foreground audio notification that is part of a user interaction.

20. The method of claim 14, wherein the first event message specifies an application identifier that identifies the first responsible application.