Speech processing system

• US 20020120448A1
• Filed: 11/13/2001
• Published: 08/29/2002
• Est. Priority Date: 11/20/2000
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Charlesworth, Jason Peter Andrew, Garner, Philip Neil
• Granted Patent: US 6,801,891 B2
• Time in Patent Office: Days
• US Class Current: 704/254
• CPC Class Codes: G10L 15/12 using dynamic programming techniques, e.g. dynamic time warping [DTW]
• CPC Class Codes: G10L 15/142 Hidden Markov Models [HMMs]
• CPC Class Codes: G10L 15/187 Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams

Abstract

A system is provided for decoding one or more sequences of sub-word units output by a speech recognition system into one or more representative words. The system uses a dynamic programming technique to align the sequence of sub-word units output by the recognition system with a number of dictionary sub-word unit sequences representative of dictionary words to identify the most likely word or words corresponding to the spoken input.

Claims

1. An apparatus for identifying one or more words corresponding to a sequence of sub-word units output by a recognition system in response to a rendition of the one or more words, the apparatus comprising:

first receiving means for receiving the recognised sequence of sub-word units representative of the one or more words to be identified;

second receiving means for receiving a plurality of dictionary sub-word sequences, each representative of one or more known words;

means for comparing sub-word units of the recognised sequence with sub-word units of each dictionary sequence to provide a set of comparison results;

means for combining the comparison results obtained by comparing the sub-word units of the recognised sequence with the sub-word units of the same dictionary word to provide, for each dictionary word, a measure of the similarity between the recognised sequence and the dictionary sequence; and

means for identifying said one or more words using the similarity measures provided by the combining means for all the dictionary sequences.

2. An apparatus according to claim 1, wherein said comparing means is operable to compare the recognised sequence with the plurality of dictionary sequences concurrently.

3. An apparatus according to claim 1, wherein said comparing means comprises:

means for aligning sub-word units of the recognised sequence with the sub-word units of a current dictionary sequence to form a number of aligned pairs of sub-word units; and

a sub-word comparator for comparing the sub-word units of each aligned pair to generate a comparison score representative of the similarity between the sub-word units of the aligned pair; and

wherein said combining means is operable to combine the comparison scores for all the aligned pairs of sub-word units for the current dictionary sequence to provide said measure of the similarity between the recognised sequence and the current dictionary sequence.

4. An apparatus according to claim 3, wherein said sub-word unit comparator comprises:

first comparing means for comparing, for each aligned pair, the recognised sequence sub-word unit in the aligned pair with each of a plurality of sub-word units taken from a set of predetermined sub-word units, to provide a corresponding plurality of comparison scores representative of the similarities between the recognised sequence sub-word unit and the respective sub-word units of the set;

second comparing means for comparing, for each aligned pair, the dictionary sequence sub-unit in the aligned pair with each of said plurality of sub-word units from the set to provide a further corresponding plurality of comparison scores representative of the similarities between said dictionary sequence sub-word unit and the respective sub-word units of the set;

means for combining the comparison scores obtained when comparing the recognised and dictionary sequence sub-word units in the aligned pair with the same sub-unit from the set, to generate a plurality of combined comparison scores for each aligned pair; and

means for determining said comparison score representative of the similarity between the aligned pair of sub-word units in dependence upon said plurality of intermediate comparison scores for the aligned pair.

5. An apparatus according to claim 4, wherein said first and second comparing means are operable to compare the recognised sequence sub-word unit and the dictionary sequence sub-word unit respectively with each of the sub-word units in the set of sub-word units.

6. An apparatus according to claim 4, wherein said first and second comparing means are operable to provide intermediate comparison scores which are indicative of a probability of confusing the corresponding sub-word unit taken from a set of predetermined sub-word units as the sub-word unit in the aligned pair.

7. An apparatus according to claim 6, wherein said combining means is operable to combine the comparison scores in order to multiply the probabilities of confusing the corresponding sub-word unit taken from the set as the sub-word units in the aligned pair.

8. An apparatus according to claim 7, wherein each of said sub-word units in said set of predetermined sub-word units has a predetermined probability of occurring within a sequence of sub-word units and wherein said combining means is operable to weigh each of said combined comparison scores in dependence upon the respective probability of occurrence for the sub-word unit of the set used to generate the combined comparison score.

9. An apparatus according to claim 8, wherein said combining means is operable to combine said intermediate comparison scores by calculating:

P(d_i¹|p_r)P(d²_j|p_r)P(p_r)

where d¹_i and d²_j are an aligned pair of dictionary and recognised sequence sub-word units respectively;

P(d²_j|p_r) is the intermediate comparison score output by said first comparing means and is representative of the probability of confusing set sub-word unit p_r as recognised sequence sub-word unit d²_j;

P(d¹_i|p_r) is the comparison score output by said second comparing means and is representative of the probability of confusing set sub-word unit p_r as dictionary sequence sub-word unit d¹_i; and

P(p_r) is a weight which represents the probability of set sub-word unit p_r occurring in a sequence of sub-word units.

10. An apparatus according to claim 9, wherein the confusion probabilities for the recognised sequence sub-word units and the dictionary sequence sub-word units are determined in advance and depend upon the recognition system that was used to generate the respective sub-word unit sequences.

11. An apparatus according to claim 7, wherein said intermediate comparison scores represent log probabilities and wherein said combining means is operable to multiply said probabilities by adding the respective intermediate comparison scores.

12. An apparatus according to claim 1, wherein said comparing means has a plurality of different comparison modes of operation and in that the apparatus further comprises:

means for determining if the current dictionary sequence of sub-word units was generated from an audio input or a typed input and for outputting a determination result; and

means for selecting, for the current dictionary sub-word sequence, the mode of operation of said comparing means in dependence upon said determination result.

13. An apparatus according to claim 4, wherein each of the sub-word units in said dictionary and recognised sequences of sub-word units belong to said set of predetermined sub-word units and wherein said first and second comparing means are operable to provide said comparison scores using predetermined data which relate the sub-word units in said set to each other.

14. An apparatus according to claim 13, wherein said predetermined data comprises, for each sub-word unit in the set of sub-word units, a probability for confusing that sub-word unit with each of the other sub-word units in the set of sub-word units.

15. An apparatus according to claim 1, wherein said aligning means comprises dynamic programming means for aligning said dictionary and recognised sequences of sub-word units using a dynamic programming technique.

16. An apparatus according to claim 1, wherein each of said sub-word units represents a phoneme.

19. A method according to claim 18, wherein said comparing step compares the recognised sequence with the plurality of dictionary sequences concurrently.

20. A method according to claim 18, wherein said comparing step comprises the steps of:

aligning sub-word units of the recognised sequence with the sub-word units of a current dictionary sequence to form a number of aligned pairs of sub-word units; and

a sub-word unit comparing step of comparing the sub-word units of each aligned pair to generate a comparison score representative of the similarity between the sub-word units of the aligned pair; and

wherein said combining step combines the comparison scores for all the aligned pairs of sub-word units for the current dictionary sequence to provide said measure of the similarity between the recognised sequence and the current dictionary sequence.

21. A method according to claim 20, wherein said sub-word unit comparing step comprises the steps of:

a first comparing step of comparing, for each aligned pair, the recognised sequence sub-word unit in the aligned pair with each of a plurality of sub-word units taken from a set of predetermined sub-word units, to provide a corresponding plurality of comparison scores representative of the similarities between the recognised sequence sub-word unit and the respective sub-word units of the set;

a second comparing step of comparing, for each aligned pair, the dictionary sequence sub-unit in the aligned pair with each of said plurality of sub-word units from the set to provide a further corresponding plurality of comparison scores representative of the similarities between said dictionary sequence sub-word unit and the respective sub-word units of the set;

combining the comparison scores obtained when comparing the recognised and dictionary sequence sub-word units in the aligned pair with the same sub-unit from the set, to generate a plurality of combined comparison scores for each aligned pair; and

determining said comparison score representative of the similarity between the aligned pair of sub-word units in dependence upon said plurality of intermediate comparison scores for the aligned pair.

22. A method according to claim 21, wherein said first and second comparing steps compare the recognised sequence sub-word unit and the dictionary sequence sub-word unit respectively with each of the sub-word units in the set of sub-word units.

23. A method according to claim 21, wherein said first and second comparing steps provide intermediate comparison scores which are indicative of a probability of confusing the corresponding sub-word unit taken from a set of predetermined sub-word units as the sub-word unit in the aligned pair.

24. A method according to claim 23, wherein said combining step combines the comparison scores in order to multiply the probabilities of confusing the corresponding sub-word unit taken from the set as the sub-word units in the aligned pair.

25. A method according to claim 24, wherein each of said sub-word units in said set of predetermined sub-word units has a predetermined probability of occurring within a sequence of sub-word units and wherein said combining step weighs each of said combined comparison scores in dependence upon the respective probability of occurrence for the sub-word unit of the set used to generate the combined comparison score.

26. A method according to claim 25, wherein said combining step combines said intermediate comparison scores by calculating:

P(d¹_i|p_r)P(d²_j|p_r)P(p_r)

where d¹_i and d²_j are an aligned pair of dictionary and recognised sequence sub-word units respectively;

P(d²_j|p_r) is the intermediate comparison score output by said first comparing step and is representative of the probability of confusing set sub-word unit p_r as recognised sequence sub-word unit d²_j;

P(d¹_i|p_r) is the comparison score output by said second comparing step and is representative of the probability of confusing set sub-word unit p_r as dictionary sequence sub-word unit d¹_i; and

P(p_r) is a weight which represents the probability of set sub-word unit p_r occurring in a sequence of sub-word units.

27. A method according to claim 26, wherein the confusion probabilities for the recognised sequence sub-word units and the dictionary sequence sub-word units are determined in advance and depend upon the recognition system that was used to generate the respective sub-word unit sequences.

28. A method according to claim 24, wherein said intermediate comparison scores represent log probabilities and wherein said combining step multiplies said probabilities by adding the respective intermediate comparison scores.

29. A method according to claim 18, further comprising the steps of:

determining if the current dictionary sequence of sub-word units was generated from an audio input or a typed input and for outputting a determination result; and

selecting, for the current dictionary sub-word sequence, a comparison technique employed in said comparing step in dependence upon said determination result.

30. A method according to claim 21, wherein each of the sub-word units in said dictionary and recognised sequences of sub-word units belong to said set of predetermined sub-word units and wherein said first and second comparing means are operable to provide said comparison scores using predetermined data which relate the sub-word units in said set to each other.

31. A method according to claim 30, wherein said predetermined data comprises, for each sub-word unit in the set of sub-word units, a probability for confusing that sub-word unit with each of the other sub-word units in the set of sub-word units.

32. A method according to claim 18, wherein said aligning step uses a dynamic programming technique to align said dictionary and recognised sequences of sub-word units.

33. A method according to claim 18, wherein each of said sub-word units represents a phoneme.

17. A speech recognition system comprising:

means for receiving speech signals to be recognised;

means for storing sub-word unit models;

means for comparing the received speech signals with the sub-word unit models to generate one or more sequences of sub-word units representative of the received speech signals;

a word dictionary relating sequences of sub-word units to words; and

a word decoder for processing the one or more sequences of sub-word units output by said comparing means using the word dictionary to generate one or more words corresponding to the received speech signals;

characterised in that said word decoder comprises;

first receiving means for receiving the recognised sequence of sub-word units representative of the received speech signals;

second receiving means for receiving from said word dictionary a plurality of dictionary sub-word sequences, each representative of one or more known words;

means for comparing sub-word units of the recognised sequence with sub-word units of each received dictionary sequence to provide a set of comparison results;

means for combining the comparison results obtained by comparing the sub-word units of the recognised sequence with the sub-word units of each received dictionary word to provide a respective measure of the similarity between the recognised sequence and each received dictionary sequence; and

means for identifying said one or more words using the similarity measures provided by the combining means.

18. A method of identifying one or more words corresponding to a sequence of sub-word units output by a recognition system in response to a rendition of the one or more words, the method comprising the steps of:

a first receiving step of receiving the recognised sequence of sub-word units representative of the one or more words to be identified;

a second receiving step of receiving a plurality of dictionary sub-word sequences, each representative of one or more known words;

comparing sub-word units of the recognised sequence with sub-word units of each dictionary sequence to provide a set of comparison results;

combining the comparison results obtained by comparing the sub-word units of the recognised sequence with the sub-word units of the same dictionary word to provide, for each dictionary word, a measure of the similarity between the recognised sequence and the dictionary sequence; and

identifying said one or more words using the similarity measures provided by the combining step for all the dictionary sequences.

34. A speech recognition method comprising the steps of:

receiving speech signals to be recognised;

comparing the received speech signals with stored sub-word unit models to generate one or more sequences of sub-word units representative of the received speech signals; and

processing the one or more sequences of sub-word units output by said comparing step using a stored word dictionary to generate one or more words corresponding to the received speech signals;

characterised in that said processing step comprises the steps of;

a first receiving step of receiving the recognised sequence of sub-word units representative of the received speech signal;

a second receiving step of receiving a plurality of dictionary sub-word sequences, each representative of one or more known words;

comparing sub-word units of the recognised sequence with sub-word units of each received dictionary sequence to provide a set of comparison results;

combining the comparison results obtained by comparing the sub-word units of the recognised sequence with the sub-word units of each received dictionary word to provide a respective measure of the similarity between the recognised sequence and each received dictionary sequence; and

identifying said one or more words using the similarity measures provided by the combining step.

35. A storage medium storing processor implementable instructions for controlling a processor to implement a method of identifying one or more words corresponding to a sequence of sub-word units output by a recognition system in response to a rendition of the one or more words, the processor implementable instructions comprising:

instructions for a first receiving step of receiving the recognised sequence of sub-word units representative of the one or more words to be identified;

instructions for a second receiving step of receiving a plurality of dictionary sub-word sequences, each representative of one or more known words;

instructions for comparing sub-word units of the recognised sequence with sub-word units of each received dictionary sequence to provide a set of comparison results;

instructions for combining the comparison results obtained by comparing the sub-word units of the recognised sequence with the sub-word units of each received dictionary word to provide a respective measure of the similarity between the recognised sequence and each received dictionary sequence; and

instructions for identifying said one or more words using the similarity measures provided by the combining step.

36. Processor implementable instructions for controlling a processor to implement a method of identifying one or more words corresponding to a sequence of sub-word units output by a recognition system in response to a rendition of the one or more words, the processor implementable instructions comprising:

instructions for a first receiving step of receiving the recognised sequence of sub-word units representative of the one or more words to be identified;

instructions for a second receiving step of receiving a plurality of dictionary sub-word sequences, each representative of one or more known words;

instructions for comparing sub-word units of the recognised sequence with sub-word units of each received dictionary sequence to provide a set of comparison results;

instructions for combining the comparison results obtained by comparing the sub-word units of the recognised sequence with the sub-word units of each received dictionary word to provide a respective measure of the similarity between the recognised sequence and each received dictionary sequence; and

instructions for identifying said one or more words using the similarity measures provided by the combining step.