Combined speech and handwriting recognition

• US 20050159948A1
• Filed: 12/05/2004
• Published: 07/21/2005
• Est. Priority Date: 09/05/2001
• Status: Active Grant

• Current Assignee: Cerence Operating Company (Cerence Inc.)
• Original Assignee: Voice Signal Technologies Incorporated (Microsoft Corporation)
• Inventors: Roth, Daniel L., Porter, Edward W.
• Granted Patent: US 7,467,089 B2
• Time in Patent Office: Days
• US Class Current: 704/233
• CPC Class Codes: G10L 15/19 Grammatical context, e.g. disambiguation of the recognition hypotheses based on word sequence rules
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

The invention relates to the combination of speech recognition with handwriting and/or character recognition. This includes the innovation of selecting one or more best-scoring recognition candidates as a function of recognition of both handwritten and spoken representations of a sequence of one or more words to be recognized. It also includes the innovation of using character or handwriting recognition of one or more letters to alphabetically filter speech recognition of one or more words. It also includes the innovations of using speech recognition of one or more letter-identifying words to alphabetically filter handwriting recognition, and of using speech recognition to correct handwriting recognition of one or more words.

Claims

1. A method of word recognition comprising:

receiving a handwritten representation of all or a part of a given sequence of one or more words to be recognized;

receiving a spoken representation of said sequence of one or more words;

performing handwriting recognition the handwritten representation and speech recognition upon the spoken representation and selecting one or more best scoring recognition candidates each comprised of a sequences of one or more words based on the scoring of recognition candidates against both the handwritten and spoken representations.

2. A method of word recognition comprising:

receiving a spoken representation of a given sequence of one or more words to be recognized;

receiving a filtering input consisting of handwriting or character drawing input;

using handwriting or character recognition, respectively, to define a filter representing one or more sequences of characters selected by said recognition as most likely corresponding to said filtering input; and

using a combination of said filter and speech recognition performed on said spoken representation to select one or more recognition candidates, each consisting of a sequence of one or more words, selected as a function of the closeness of their match against the spoken representation and whether or not they match one of the one or more character sequences associated said filter.

3. A method as in claim 2 wherein said filtering input consists of handwriting.

4. A method as in claim 3 wherein:

said filter represents a plurality of sequences of characters; and

said selection of recognition candidates selects a plurality of best scoring recognition candidates, different ones of which can match different sequences of characters represented by said filter

5. A method as in claim 4 wherein said plurality of character sequences represented by one filter and used in said selection of recognition candidates can be of different character length.

6. A method as in claim 3 wherein:

said filter represents only one of sequences of characters which is used for filtering; and

said selection of recognition candidates selects a plurality of best scoring recognition candidates, all of which match said one character sequence.

7. A method as in claim 2 wherein said filtering input consists of one or more separate character drawings.

8. A method as in claim 7 wherein:

said filter represents a plurality of sequences of characters; and

said selection of recognition candidates selects a plurality of best scoring recognition candidates, different ones of which can match different sequences of characters represented by said filter

9. A method as in claim 7 wherein:

said filter represents only one of sequences of characters which is used for filtering; and

said selection of recognition candidates selects a plurality of best scoring recognition candidates, all of which match said one character sequence.

10. A method as in claim 2:

further including;

receiving a spoken representation of a second sequence of one or more words to be recognized;

using speech recognition to output a corresponding sequence of one or more words into a sequential body of text;

responding to user input with the pointing device that touches a sequence of one or more words in said body of text by selecting the touched sequence as a sequence to be correction;

treating the portion of the spoken representation of said second sequence of words as said given sequence of words; and

then

receiving said filtering input;

using said handwriting or character recognition to define said filter; and

using said combination of the filter and speech recognition to select one or more recognition candidates.

11. A method of word recognition comprising:

receiving a handwritten representation of a given sequence of one or more words to be recognized;

receiving a filtering input consisting one or more utterances representing a sequence of one or more letter identifying words;

using speech recognition to define a filter representing one or more sequences of characters selected by said recognition as most likely corresponding to said filtering input; and

using a combination of said filter and handwriting recognition performed on said handwritten representation to select one or more recognition candidates, each consisting of a sequence of one or more words, selected as a function of the closeness of their match against the handwritten representation and whether or not they match one of the one or more character sequences associated said filter.

12. A method as in claim 11 wherein:

the filtering input is a sequence of continuously spoken letter identifying words; and

the speech recognition is continuous speech recognition.

13. A method as in claim 11 wherein:

the filtering input is a sequence of discretely spoken letter identifying words; and

the speech recognition is discrete speech recognition.

14. A method as in claim 11 wherein:

said filter represents a plurality of sequences of characters; and

said selection of recognition candidates selects a plurality of best scoring recognition candidates, different ones of which can match different sequences of characters represented by said filter

15. A method as in claim 14 wherein said plurality of character sequences represented by one filter and used in said selection of recognition candidates can be of different character length.

16. A method as in claim 15 wherein:

the filtering input is a sequence of continuously spoken letter names; and

the speech recognition is continuous speech recognition.

17. A method as in claim 11 wherein:

said filter represents only one of sequences of characters which is used for filtering; and

said selection of recognition candidates selects a plurality of best scoring recognition candidates, all of which match said one character sequence.

18. A method as in claim 11 further including providing a user interface which enables a user to select whether the filtering input is recognized with discrete or continuous recognition.

19. A method as in claim 11 further including providing a user interface which enables a user to select whether the filtering input is recognized in a mode which favors the recognition of letter names or of non-letter name letter identifying words.

20. A method of word recognition comprising:

receiving a handwritten representation of a given sequence of one or more words to be recognized;

performing handwriting recognition upon said handwritten representation to produce one or more best scoring recognition candidates, each of which contains one or more words selected as having a likelihood of corresponding to the one or more words of said handwritten representation;

then receiving a spoken representation of a given sequence of one or more words to be recognized;

performing speech recognition upon said spoken representation to produce one or more best scoring recognition candidates, each of which contains one or more words selected as having a likelihood of corresponding to the one or more words of said spoken representation;

using information in one of said speech recognition'"'"'s best scoring candidates to correct the prior recognition of said handwritten representation.

21. A method as in claim 20 wherein said using of speech recognition information to correct handwriting recognition includes replacing a best scoring recognition candidate produced by the handwriting recognition with a best scoring recognition candidate produced by the speech recognition.

22. A method as in claim 20 wherein said using of speech recognition information to correct handwriting recognition includes interpreting one of the recognition candidates produced by the speech recognition as a command, and performing said command in corrections of a best scoring recognition candidate produced by the handwriting recognition.