Post processing of natural language automatic speech recognition

• US 9,431,012 B2
• Filed: 04/30/2012
• Issued: 08/30/2016
• Est. Priority Date: 04/30/2012
• Status: Active Grant

• Current Assignee: Blackberry Limited
• Original Assignee: 2236008 Ontario Inc. (Blackberry Limited)
• Inventors: Fry, Darrin Kenneth
• Primary Examiner(s): He, Jialong
• Application Number:
  US13/460,462
• Publication Number:
  US 20130289988A1
• Time in Patent Office: 1,583 Days
• Field of Search: 704/270, 704/275
• US Class Current: 1/1
• CPC Class Codes: G10L 15/19 Grammatical context, e.g. disambiguation of the recognition hypotheses based on word sequence rules
• CPC Class Codes: G10L 15/20 Speech recognition techniques specially adapted for robustness in adverse environments, e.g. in noise, of stress induced speech G10L21/02 takes precedence
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 15/32 Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems

Abstract

A post-processing speech system includes a natural language-based speech recognition system that compares a spoken utterance to a natural language vocabulary that includes words used to generate a natural language speech recognition result. A master conversation module engine compares the natural language speech recognition result to domain specific words and phrases. A voting engine selects a word or a phrase from the domain specific words and phrases that is transmitted to an application control system. The application control system transmits one or more control signals that are used to control an internal or an external device or an internal or an external process.

Claims

1. A post-processing automated speech recognition system comprising:

a natural language-based speech recognition system that compares a spoken utterance to a natural language vocabulary comprising words in one or more active grammars to generate a natural language speech recognition result and a natural language confidence score;

a master conversation module engine that post processes the natural language speech recognition result by comparing the natural language speech recognition result generated by the natural language-based speech processing system to lists of words and phrases from a plurality of active sub-grammars that are mapped to a plurality of domain specific words and phrases, and generates a post processed natural language speech recognition result confidence score for each of the listed words and phrases from the plurality of active sub-grammars based on a likelihood that the respective listed words and phrases from the plurality of active sub-grammars match the natural language speech recognition result;

a voting module engine that is interfaced to the master conversation module engine and that is operable to select a word or a phrase from the list of words and phrases from the plurality of active sub-grammars, based on the post processed natural language speech recognition result confidence scores;

wherein;

when the voting module engine selects a word or phrase from the plurality of active sub-grammars based on the post processed natural language speech recognition result confidence scores, the selected word or phrase is transmitted to an application control system that transmits control signals used to control an internal or an external device or an internal or an external process;

or

when both of;

the voting module does not select a word or phrase from the plurality of active sub-grammars based on the post processed natural language speech recognition result confidence scores, and

the natural language confidence score exceeds a specified threshold,

then the master conversation module engine selects the natural language speech recognition result to be transmitted to the application control system;

where the natural language-based speech recognition system, the master conversation module engine, and the voting module engine are executed by one or more processors.

2. The post-processing automated speech recognition system of claim 1 where the natural language speech recognition result and the word or the phrase from the plurality of active sub-grammars selected by the voting module engine are represented by language agnostic indicia, objects or symbols that control one or more devices or processes.

3. The post-processing automated speech recognition system of claim 1 where a portion of the natural language-based speech recognition system is remote from the master conversation module engine and the plurality of active sub-grammars are remote from a portion of the natural language-based speech processing system.

4. The post-processing automated speech recognition system of claim 1 further comprising an ambiguity resolver module engine that resolves an ambiguity arising from the comparisons between the natural language speech recognition result generated by the natural language-based speech recognition system and the list of words and phrases from the plurality of active sub-grammars.

5. The post-processing automated speech recognition system of claim 1 further comprising a grammar-based speech recognition system that compares the spoken utterance to a grammar-based vocabulary comprising the active sub-grammars that are mapped to the plurality of domain specific words and phrases.

6. The post-processing automated speech recognition system of claim 5 further comprising an alignment engine that selects between the word or the phrase selected by the voting module engine and recognition results generated by the grammar-based speech recognition system, where the alignment engine transmits the selection to one of a plurality of application control systems.

7. The post-processing automated speech recognition system of claim 6 where the voting module engine comprises a portion of the master conversation module engine and the word or the phrase selected by the voting engine comprises the recognition result of the master conversation module engine, and where the alignment engine selects between the recognition results of the master conversation module engine and the recognition result generated by the grammar-based speech recognition system based on the natural language confidence score and a grammar-based confidence score.

8. The post-processing automated speech recognition system of claim 7 where the alignment engine selects between the recognition results of the master conversation module engine and the recognition result generated by the grammar-based speech recognition system based on an order the recognition results are received at the alignment engine.

9. The post-processing automated speech recognition system of claim 6 further comprising an application engine that controls an internal or external device or process in response to the alignment engine'"'"'s transmission of a selection result.

10. The post-processing automated speech recognition system of claim 1 further comprising a grammar-based speech processing system that generates a grammar-based speech recognition result and grammar-based confidence score synchronously with the natural language-based speech recognition system.

11. The post-processing automated speech recognition system of claim 10 where the grammar-based speech processing system and the natural language-based speech processing system process the same spoken utterance.

12. The post-processing automated speech recognition system of claim 10 where the grammar-based speech processing system and the natural language-based speech recognition system are executed by a plurality of processors that run simultaneously.

13. The post-processing automated speech recognition system of claim 10 where the grammar-based speech processing system and the natural language-based speech recognition system, and the voting module engine are executed by a plurality of parallel processors.

14. The post-processing automated speech recognition system of claim 10 where the grammar-based speech processing system and the natural language-based speech recognition system, and the voting module engine each comprise a computing thread executed by a multitasking processor.

15. The post-processing automated speech recognition system of claim 1 further comprising a grammar-based speech recognition system that generates grammar-based speech recognition results synchronously with a plurality of natural language-based speech recognition systems interfaced to the master conversation module engine.

16. A computer implemented method of automatically recognizing speech comprising:

capturing speech utterances and converting the speech utterances into frames of speech;

recognizing speech utterances by comparing the frames of speech to a list of words in active grammars;

generating a natural language-based speech recognition result and a natural language confidence score;

post processing the natural language-based speech recognition result by comparing the natural language-based speech recognition result to a domain specific vocabulary that comprises lists of words and phrases from sub-grammars that are mapped to domain specific words or phrases and generating a post process confidence score for each of the words and phrases from the sub-grammars, based on a likelihood that the words and phrases match the natural language-based recognition result; and

selecting a speech recognition result based on a voting scheme;

wherein;

a word or phrase from the sub-grammars that best matches the natural language speech recognition result based on the post process confidence scores, is selected as the speech recognition result;

or

the natural language speech recognition result is selected as the speech recognition result when both of;

a word or phrase from the sub-grammars is not selected as the speech recognition result based on the post process confidence scores, and

the natural language confidence score exceeds a specified threshold.

17. The computer implemented method of claim 16 further comprising generating multiple confidence scores that reflect a likelihood that each of a plurality of words and a plurality of phrases that comprise the sub-grammars match the speech recognition result.

18. The computer implemented method of claim 17 where the natural language-based speech recognition result and sub-grammars comprise indicia representing a plurality of words and a plurality of phrases.

19. A method of automatically recognizing speech comprising:

capturing a speech utterance;

recognizing the speech utterance by comparing the speech utterance to a list of words in active grammars by a natural language-based speech recognition system;

generating automated speech recognition results and a confidence score from the natural language-based speech recognition system;

post processing the natural language-based speech recognition results by comparing the natural language speech recognition results to a domain specific vocabulary that comprises lists of words and phrases from sub-grammars mapped to one or more application control signals that control an internal or external device or process through an application engine; and

selecting a speech recognition post processing result based on a voting scheme, wherein;

a word or phrase from the sub-grammars that best matches the natural language speech recognition result based on the post processing comparisons is selected as the speech recognition post processing result;

or

the natural language speech recognition result is selected as the speech recognition post processing result when both of;

a word or phrase from the sub-grammars is not selected as the speech recognition post processing result based on the post processing comparisons, and

the confidence score from the natural language-based speech recognition system exceeds a specified threshold.

20. The method of claim 19 further comprising executing a second selection by selecting between the speech recognition post processing result and a grammar-based speech recognition result that is generated by a grammar-based speech recognition system that processes the same speech utterance used to generate the automated speech recognition results from the natural language-based speech recognition system.