Management layer for multiple intelligent personal assistant services

• US 10,748,531 B2
• Filed: 04/13/2017
• Issued: 08/18/2020
• Est. Priority Date: 04/13/2017
• Status: Active Grant

• Current Assignee: Harman International Industries Incorporated (Samsung Electronics Co. Ltd.)
• Original Assignee: Harman International Industries Incorporated (Samsung Electronics Co. Ltd.)
• Inventors: Kim, Seon Man
• Primary Examiner(s): Kim, Jonathan C
• Application Number:
  US15/487,347
• Publication Number:
  US 20180301147A1
• Time in Patent Office: 1,223 Days
• Field of Search: None
• CPC Class Codes: G06F 16/685 using automatically derived transcript of audio data, e.g. lyrics speech recognition G10L15/00
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/08 Speech classification or search
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command

Abstract

Performing speech recognition in a multi-device system includes receiving a first audio signal that is generated by a first microphone in response to a verbal utterance, and a second audio signal that is generated by a second microphone in response to the verbal utterance; dividing the first audio signal into a first sequence of temporal segments; dividing the second audio signal into a second sequence of temporal segments; comparing a sound energy level associated with a first temporal segment of the first sequence to a sound energy level associated with a first temporal segment of the second sequence; based on the comparing, selecting, as a first temporal segment of a speech recognition audio signal, one of the first temporal segment of the first sequence and the first temporal segment of the second sequence; and performing speech recognition on the speech recognition audio signal.

Claims

1. One or more non-transitory computer-readable storage media including instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:

receiving a first response to a first user input, comprising;

detecting an activation phrase within a first audio signal included in the first user input;

selecting, based on the activation phrase, a first intelligent personal assistant (IPA) that is included in a group of candidate IPAs;

extracting query content from the first audio signal;

generating a second audio signal based on the query content;

transmitting the second audio signal to the first IPA;

receiving, from the first IPA, a first response audio signal; and

storing the first response audio signal as the first response;

after storing the first response audio signal from the first IPA, receiving a second response to the first user input, comprising;

detecting a change listener command in a third audio signal, wherein the change listener command comprises a word or phrase directing the query content to an additional IPA included in the group of candidate IPAs;

selecting, based on the change listener command, a second IPA based on the change listener command;

transmitting the second audio signal to the second IPA;

receiving, from the second IPA, a second response audio signal; and

storing the second response audio signal as the second response;

determining, based on the change listener command, a playback order that provides the second response before providing the first response; and

causing, based on the playback order, the second response audio signal to be played before causing the first response audio signal to be played.

2. The one or more non-transitory computer-readable storage media of claim 1, further comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:

detecting an additional activation phrase in the first audio signal; and

based on the additional activation phrase, selecting an additional IPA that is included in the group of candidate IPAs.

3. The one or more non-transitory computer-readable storage media of claim 2, further comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the step of transmitting the second audio signal to the additional IPA.

4. The one or more non-transitory computer-readable storage media of claim 2, further comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:

extracting additional query content from the first audio signal that is associated with the additional activation phrase;

generating, based on the additional query content, a fourth audio signal that includes the additional query content; and

transmitting, to the additional IPA, the fourth audio signal.

5. A system comprising:

a memory storing an intelligent personal assistant (IPA) management application; and

one or more processors that are coupled to the memory and execute the IPA management application to;

receive a first response to a first user input, comprising;

detecting an activation phrase within a first audio signal included in the first user input;

selecting, based on the activation phrase, a first intelligent personal assistant (IPA) that is included in a group of candidate IPAs;

extracting query content from the first audio signal;

generating a second audio signal based on the query content;

transmitting the second audio signal to the first IPA;

receiving, from the first IPA, a first response audio signal; and

storing the first response audio signal as the first response;

after storing the first response audio signal from the first IPA, receive a second response to the first user input, comprising;

detecting a change listener command in a third audio signal, wherein the change listener command comprises a word or phrase directing the query content to an additional IPA included in the group of candidate IPAs;

selecting, based on the change listener command, a second IPA based on the change listener command;

transmitting the second audio signal to the second IPA;

receiving from the second IPA, a second response audio signal;

storing the second response audio signal as the second response;

determine, based on the change listener command, a playback order that provides the second response before providing the first response; and

cause, based on the playback order, the second response audio signal to be played before causing the first response audio signal to be played.

6. The system of claim 5, wherein the one or more processors further execute the IPA management application to:

detect a second change listener command in the first audio signal;

determine, based on the second change listener command, a third IPA included in the group of candidate IPAs;

extract additional query content from the first audio signal;

generate, based on the additional query content, a fourth audio signal that includes the additional query content; and

transmit, to the third IPA, the fourth audio signal.

7. The system of claim 6, wherein the additional query content includes a voice command.

8. The system of claim 6, wherein determining the third IPA based on the second change listener command comprises determining the third IPA based on at least one of:

a previously-issued voice command,

a previously-issued voice query, or

which process is currently being controlled by an IPA service.

9. The system of claim 8, wherein determining the third IPA based on the previously-issued voice command comprises determining that the third IPA is an IPA that received a previously-issued voice command via the second audio signal.

10. The system of claim 8, wherein determining the third IPA based on which process is currently being controlled by the IPA service comprises determining that the third IPA is controlling the process.

11. A method for responding to a speech input, the method comprising:

receiving a first response to a first user input, comprising;

detecting an activation phrase within a first audio signal included in the first user input;

selecting, based on the activation phrase, a first intelligent personal assistant (IPA) that is included in a group of candidate IPAs;

extracting query content from the first audio signal;

generating a second audio signal based on the query content;

transmitting the second audio signal to the first IPA;

receiving, from the first IPA, a first response audio signal; and

storing the first response audio signal as the first response;

after storing the first response audio signal from the first IPA, receiving a second response to the first user input, comprising;

detecting a change listener command in a third audio signal, wherein the change listener command comprises a word or phrase directing the query content to an additional IPA included in the group of candidate IPAs;

selecting, based on the change listener command, a second IPA based on the change listener command;

transmitting the second audio signal to the second IPA;

receiving, from the second IPA, a second response audio signal;

storing the second response audio signal as the second response;

determining, based on the change listener command, a playback order that provides the second response before providing the first response; and

causing, based on the playback order, the second response audio signal to be played before causing the first response audio signal to be played.

12. The method of claim 11, further comprising:

detecting an additional activation phrase in the first audio signal; and

based on the additional activation phrase, selecting a second IPA that is included in the group of candidate IPAs.