Speech recognition

• US 10,140,986 B2
• Filed: 08/05/2016
• Issued: 11/27/2018
• Est. Priority Date: 03/01/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Froelich, Raymond J.
• Primary Examiner(s): Chawan, Vijay B
• Application Number:
  US15/229,854
• Publication Number:
  US 20170256258A1
• Time in Patent Office: 844 Days
• Field of Search: 704233, 704248, 704253, 704270, 7042701, 704231, 704236, 704257, 704 9
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 15/19 Grammatical context, e.g. disambiguation of the recognition hypotheses based on word sequence rules
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/222 Barge in, i.e. overridable guidance for interrupting prompts
• CPC Class Codes: G10L 2015/0635 updating or merging of old and new templates; Mean values; Weighting
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 25/87 Detection of discrete points within a voice signal
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

Voice input is received from a user. An ASR system generates in memory a set of words it has identified in the voice input, and update the set each time it identifies a new word in the voice input to add the new word to the set, during at least one interval of speech activity. Information is pre-retrieved whilst the speech activity interval is still ongoing, for conveying in a response to be outputted at the end of the speech activity interval.

Claims

1. A computer system comprising:

an input configured to receive voice input from a user;

an Automatic Speech Recognition (ASR) system for identifying individual words in the voice input, wherein the ASR system is configured to generate in memory during at least one speech activity interval in the voice input a word set of one or more words it has identified in the voice input, and update the word set in the memory based on identifying a new word in the voice input to add the new word to the word set;

a lookup module configured to retrieve at least one word from the word set during the speech activity interval in the voice input at a first time whilst the speech activity interval is still ongoing, and perform a lookup whilst the speech activity interval is still ongoing to pre-retrieve information associated with the at least one word, the pre-retrieved information including one or more relevant words that are contextually relevant to and different from the at least one word from the word set; and

a response generation module configured to detect an end of the speech activity interval at a second time that is subsequent to the first time, the word set having been updated by the ASR system at least once between the first time and the second time, and the response generation module further configured to generate a response for output, wherein the response conveys the pre-retrieved information from the lookup module.

2. A computer system according to claim 1, wherein the response is generated based on one or more of a confidence value assigned to one or more words of the word set by the ASR system, or a confidence value assigned to the word set by a language model of the computer system.

3. A computer system according to claim 1, wherein the pre-retrieved information is pre-retrieved from an Internet search engine.

4. A computer system according to claim 1, wherein the user is a user of a communication system and the pre-retrieved information is information about another user in the communication system.

5. A computer system according to claim 3, wherein the pre-retrieved information comprises contact details for the other user.

6. A computer system according to claim 1, comprising a speech detection module configured monitor the word set as it is updated by the ASR system, wherein the response generation module is configured to detect the end of the speech activity interval based on said monitoring of the word set.

7. A computer system according to claim 6, further comprising a language model, wherein detecting the end of the speech activity interval comprises detecting, by the speech detection module, when the word set forms a grammatically complete sentence according to the language model.

8. A computer system according to claim 6, wherein detecting the end of the speech activity interval comprises detecting, by the speech detection module, that no new words have been identified by the ASR system for a pre-determined duration.

9. A computer system according to claim 1, wherein the response is an audio response for playing out in audible form.

10. A computer system according to claim 1, comprising a video generation module configured to generate and output a moving image of an avatar configured to convey when output of the response is about to begin by performing a visual action.

11. A computer system according to claim 6, wherein the response is an audio response, the computer system further comprising:

a speech detection module;

a response delivery module configured to begin outputting the audio response to the user when the speech activity interval has ended, wherein the outputting of the audio response is terminated before it has completed in response to the speech detection module detecting a start of a subsequent speech activity interval in the voice input.

12. A computer system according to claim 11, wherein the speech detection module is configured to detect the start of subsequent speech activity interval by detecting an identification of another word in the voice input by the ASR system, the subsequent speech activity interval commencing with the detection of the other word.

13. A computer-implemented method of effecting communication between a user and an artificial intelligence software agent executed on a computer, the method comprising:

receiving at an Automatic Speech Recognition (ASR) system voice input from the user;

identifying by the ASR system individual words in the voice input;

generating a word set of one or more identified words in memory and during at least one speech activity interval in the voice input;

updating the word set in the memory based on identifying a new word in the voice input to add the new word to the word set;

retrieving during the speech activity interval in the voice input at least one word from the word set at a first time whilst the speech activity interval is ongoing;

performing a lookup whilst the speech activity interval is ongoing to pre-retrieve information associated with the at least one word, the pre-retrieved information including one or more relevant words that are contextually relevant to and different from the at least one word from the word set;

detecting an end of the speech activity interval at a second time that is subsequent to the first time, the word set having been updated at least once between the first time and the second time; and

generating a response by the software agent, wherein the response includes the pre-retrieved information.

14. A method according to claim 13, wherein the voice input is received from a user device via a communication network, the method further comprising transmitting the response to the user device via the communication network so as to cause the user device to output the response.

15. A method according to claim 14, wherein the voice input is received from and the response outputted to the user in real-time, thereby effecting a real-time communication event between the user and the software agent via the communication network.

16. A method according to claim 13, the method being implemented in a communication system, wherein the communication system comprises a user account database storing, for each of a plurality of users of the communication system, a user identifier that uniquely identifies that user within the communication system;

wherein a user identifier of the software agent is also stored in the user account database so that the software agents appears as another user of the communication system.

17. A method according to claim 13, comprising monitoring the word set in the memory as it is updated by the ASR system, the end of the speech activity interval being detected based on said monitoring of the word set.

18. A method according to claim 13, wherein detecting the end of the speech activity interval comprises detecting when the word set forms a grammatically complete sentence according to a language model.

19. A computer program product comprising an artificial intelligence software agent stored on a computer readable memory device, the software agent being configured when executed by one or more processors to perform operations comprising:

identifying individual words in voice input during a speech activity interval in the voice input to generate a word set that includes the individual words;

retrieving at least one word from the word set during the speech activity interval in the voice input and at a first time whilst the speech activity interval is ongoing;

performing a lookup whilst the speech activity interval is ongoing to pre-retrieve information associated with the at least one word, the pre-retrieved information including one or more relevant words that are contextually relevant to and different from the at least one word from the word set;

detecting an end of the speech activity interval at a second time that is subsequent to the first time, the word set having been updated at least once between the first time and the second time; and

generating a response for output, wherein the response conveys the pre-retrieved information.

20. A computer program product according to claim 19, wherein the response is output by transmitting the response to a user device via a network so as to cause the user device to output the response.