Conversational software agent

• US 10,192,550 B2
• Filed: 03/01/2016
• Issued: 01/29/2019
• Est. Priority Date: 03/01/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Froelich, Raymond J.
• Primary Examiner(s): Chawan, Vijay B
• Application Number:
  US15/057,682
• Publication Number:
  US 20170256257A1
• Time in Patent Office: 1,064 Days
• Field of Search: 704270, 7042701, 704275, 704231, 704236, 704233, 704251, 704257, 704248, 704253, 700259
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/0635 updating or merging of old and new templates; Mean values; Weighting
• CPC Class Codes: G10L 2015/221 Announcement of recognition results
• CPC Class Codes: G10L 2015/225 Feedback of the input speech
• CPC Class Codes: G10L 2025/783 based on threshold decision
• CPC Class Codes: G10L 25/87 Detection of discrete points within a voice signal
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

Voice input is received from a user. An ASR system generates in memory a set of words it has identified in the voice input, and update the set each time it identifies a new word in the voice input to add the new word to the set. A condition indicative of speech inactivity in the voice input is detected. A response for outputting to the user is generated based on the set of identified words, in response to the detection of the speech inactivity condition. The generated response is outputted to the user after an interval of time—commencing with the detection of the speech inactivity condition—has ended and only if no more words have been identified in the voice input by the ASR system in that interval of time.

Claims

1. A computer system comprising:

an input configured to receive voice input from a user;

an automatic speech recognition (ASR) system for identifying individual words in the voice input, the ASR system configured to generate in memory a set of one or more words it has identified in the voice input, and update the set each time it identifies a new word in the voice input by adding the new word to the set;

a speech detection module configured to detect speech activity in the voice input, prevent a speech inactivity interval from commencing until a grammatically complete sentence is detected in the voice input, cause the speech inactivity interval to commence when the grammatically complete sentence is detected in the speech input, and determine whether the ASR system has identified any more words in the voice input during the speech inactivity interval; and

a response module configured to generate a response for output based on the set of identified words, in response to the detection of an end of the speech inactivity interval, the response module configured to output the generated response after the speech inactivity interval has ended and only if the ASR system has not identified any more words in the voice input during the speech inactivity interval such that the generated response is not output if one or more words are identified in the voice input during the speech inactivity interval.

2. A computer system according to claim 1, wherein the speech detection module is configured monitor the set of identified words in the memory as it is updated by the ASR system, and detect said speech inactivity condition based on said monitoring of the identified set of words.

3. A computer system according to claim 2, further comprising a language model, wherein detecting the speech inactivity condition comprises detecting, by the speech detection module, when the set of identified words forms the grammatically complete sentence according to the language model.

4. A computer system according to claim 2, wherein detecting the speech inactivity condition comprises detecting, by the speech detection module, that no new words have been identified by the ASR system for a pre-determined duration, wherein the interval of time commences with the expiry of the pre-determined duration.

5. A computer system according to claim 1, wherein the response is an audio response for playing out to the user in audible form.

6. A computer system according to claim 1, comprising a video generation module configured to, in response to the response module determining that the ASR system has not identified any more words in the interval of time, output to the user a visual indication that the outputting of the response is about to begin.

7. A computer system according to claim 6, wherein the video generation module is configured to generate and output to the user a moving image of an avatar, wherein the visual indication is a visual action performed by the avatar.

8. A computer system according to claim 1, wherein each word of the set is stored in the memory as a string of one or more characters.

9. A computer system according to claim 1, further comprising:

a lookup module configured to receive at least one word from the set in the memory at a first time whilst updates to the set by the ASR system are still ongoing, and perform a lookup to pre-retrieve information associated with the at least one word;

wherein the response generation module is configured to access the set in the memory at a later time, the set having been updated by the ASR system at least once between the first time and the later time, the response being generated by the response module based on the set as accessed at the later time, wherein the response incorporates the information pre-retrieved by the lookup module.

10. A computer system according to claim 5, further comprising:

a response delivery module configured to begin outputting the audio response to the user when the interval of time has ended, wherein the outputting of the audio response is terminated before it has completed in response to the speech detection module detecting the start of a subsequent speech interval in the voice input.

11. A computer system according to claim 10, wherein the speech detection module is configured to detect the start of subsequent speech interval by detecting an identification of another word in the voice input by the ASR system, the speech interval commencing with the detection of the other word.

12. A computer system according to claim 1, further comprising:

a speech overload detection module configured to detect at a time during a speech interval of the voice input a speech overload condition; and

a notification module configured to output to the user, in response to said to detection, a notification of the speech overload condition, wherein the overload condition is detected based on one or more of;

a number of words that the ASR system has identified so far in that speech interval, and/or

a rate at which words are being identified by the ASR system in that speech interval;

or

a state of an AI tree being driven by the voice input.

13. A computer-implemented method comprising:

receiving voice input from a user as part of communication between the user and a data processing system;

identifying individual words in the voice input and generating in memory a set of one or more words identified in the voice input, and updating the set of one or more words when a new word in the voice input is identified to add the new word to the set;

detecting speech activity in the voice input, prevent a speech inactivity interval from commencing until a grammatically complete sentence is detected in the voice input, cause the speech inactivity interval to commence when the grammatically complete sentence is detected in the speech input, and determining whether any more words are identified in the voice input during the speech activity interval;

generating based on the set of identified words, in response to a detected speech inactivity condition, a response for outputting;

determining whether one or more additional words in the voice input during an interval of time commencing with the detection of the speech inactivity condition; and

outputting the response after said interval of time has ended if one or more words are not detected in the voice input in the interval of time such that the generated response is not if one or more words are identified in the voice input in the interval of time.

14. A method according to claim 13, wherein the voice input is received from a user device via a communication network, wherein the outputting step is performed by a software agent transmitting the response to the user device via the network so as to cause the user device to output the response to the user.

15. A method according to claim 14, wherein the voice input is received from and the response outputted to the user in real-time, thereby effecting a real-time communication event.

16. A method according to claim 13, the method being implemented in a communication system, wherein the communication system comprises a user account database storing, for each of a plurality of users of the communication system, a user identifier that uniquely identifies that user within the communication system;

wherein a user identifier of a software agent is also stored in the user account database so that the software agents appears as another user of the communication system.

17. A method according to claim 13, further comprising:

monitoring the set of identified words in the memory as it is updated, wherein the speech inactivity condition is detected based on the monitoring of the set of identified words.

18. A method according to claim 13, wherein the response is an audio response for playing out in audible form.

19. A computer-readable memory device storing instructions that, when executed by a data processing system, control the data processing system to perform operations comprising:

receiving voice input from a user as part of communication between the user and the data processing system;

identifying individual words in the voice input and generating in memory a set of one or more words identified in the voice input, and updating the set of one or more words when a new word in the voice input is identified to add the new word to the set;

detecting speech activity in the voice input, prevent a speech inactivity interval from commencing until a grammatically complete sentence is detected in the voice input, cause the speech inactivity interval to commence when the grammatically complete sentence is detected in the speech input, and determining whether any more words are identified in the voice input during the speech activity interval;

generating based on the set of identified words, in response to a detected speech inactivity condition, a response for outputting;

determining whether one or more additional words in the voice input during an interval of time commencing with the detection of the speech inactivity condition; and

outputting the response after said interval of time has ended if one or more words are not detected in the voice input in the interval of time such that the generated response is not if one or more words are identified in the voice input in the interval of time.

20. A computer-readable memory device as recited in claim 19, wherein the voice input is received from and the response outputted to the user in real-time, thereby effecting a real-time communication event.