Method and system for concurrent use of two or more closely coupled communication recognition modalities

• US 20050102138A1
• Filed: 12/27/2000
• Published: 05/12/2005
• Est. Priority Date: 12/27/2000
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Mao, Xiaodong
• Granted Patent: US 7,533,014 B2
• Time in Patent Office: Days
• US Class Current: 704/231
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features

Abstract

A method and system are provided in which a speech recognition system and one or more other input modalities are run in parallel. The system is especially useful in easing the Chinese input obstruction due to the difficulty of conforming Chinese ideograms to typical processor system input devices. The system allows a user to concurrently input information using different modalities. For example, a user may input spoken words and written characters (or data through other modalities). Each modality produces a list of possible words and their probability of being the input. If the most probable possible word is the same for both modalities then it is selected as the input. If not, an acoustic profile of the most probable speech input or an average acoustic profile of the possible speech inputs is used to validate a possible word of the other modalities or present a list of more probable possibilities to the user. The user may modify the written characters (or other input) from the provided possibilities. In this way the differing input modalities do not interfere with each other, but instead complement each other.

Claims

1. A method comprising:

receiving input through a plurality of input modalities;

ascertaining an input pool for the input of each modality, the input pools containing possible words for the input, the possible words of each pool ordered based upon a probability that a possible word is the input; and

selecting a highest probability possible word if the input pools for each modality contain the same word as the highest probability possible word.

2. The method of claim 1, wherein an input modality is speech.

3. The method of claim 2, wherein an input modality is handwriting.

4. The method of claim 3, wherein handwriting is Chinese ideograms.

5. The method of claim 3, wherein the possible words are written in pinyin.

6. The method of claim 5, further comprising:

developing an acoustic profile of the highest probability possible word of the speech input pool;

developing an acoustic profile for each of the possible words of the handwriting input pool; and

selecting the highest probability possible word of the speech input pool if the highest probability possible word of the speech input pool matches an acoustic profile of a possible word of the handwriting input pool.

7. The method of claim 5, further comprising:

developing an average acoustic profile for the possible words of the speech input pool;

developing an acoustic profile for each of the possible words of the handwriting input pool and the speech input pool to create a combined input pool;

calculating a difference between the average acoustic profile and each of the acoustic profiles of the combined input pool;

reordering the possible words of the combined input pool based upon the difference, such that the possible word of the combined input pool having an acoustic profile with a least difference from the average acoustic profile is ordered first;

selecting the handwriting input pool possibility, from the combined pool, having the least difference if the difference is below a specified threshold.

displaying the reordered possible words of the combined input pool; and

receiving user modification to the handwriting input.

8. An apparatus comprising:

means for receiving input through a plurality of input modalities;

means for ascertaining an input pool for the input of each modality, the input pools containing possible words for the input, the possible words of each pool ordered based upon a probability that a possible word is the input; and

means for selecting a highest probability possible word if the input pools for each modality contain the same word as the highest probability possible word.

9. The apparatus of claim 8, wherein an input modality is speech.

10. The apparatus of claim 9, wherein an input modality is handwriting.

11. The apparatus of claim 10, wherein handwriting is Chinese ideograms.

12. The apparatus of claim 10, wherein the possible words are written in pinyin.

13. The apparatus of claim 12, further comprising:

means for developing an acoustic profile of the highest probability possible word of the speech input pool;

means for developing an acoustic profile for each of the possible words of the handwriting input pool; and

means for selecting the highest probability possible word of the speech input pool if the highest probability possible word of the speech input pool matches an acoustic profile of a possible word of the handwriting input pool.

14. The apparatus of claim 12, further comprising:

means for developing an average acoustic profile for the possible words of the speech input pool;

means for developing an acoustic profile for each of the possible words of the handwriting input pool and the speech input pool to create a combined input pool;

means for calculating a difference between the average acoustic profile and each of the acoustic profiles of the combined input pool;

means for reordering the possible words of the combined input pool based upon the difference, such that the possible word of the combined input pool having an acoustic profile with a least difference from the average acoustic profile is ordered first;

means for selecting the handwriting input pool possibility, from the combined pool, having the least difference if the difference is below a specified threshold.

means for displaying the reordered possible words of the combined input pool; and

means for receiving user modification to the handwriting input.

15. A machine-readable medium that provides executable instructions, which when executed by a digital processing system, cause the set of processors to perform a method comprising:

receiving input through a plurality of input modalities;

ascertaining an input pool for the input of each modality, the input pools containing possible words for the input, the possible words of each pool ordered based upon a probability that a possible word is the input; and

selecting a highest probability possible word if the input pools for each modality contain the same word as the highest probability possible word.

16. The machine-readable medium of claim 15, wherein an input modality is speech.

17. The machine-readable medium of claim 16, wherein an input modality is handwriting.

18. The machine-readable medium of claim 17, wherein handwriting is Chinese ideograms.

19. The machine-readable medium of claim 17, wherein the possible words are written in pinyin.

20. The machine-readable medium of claim 19, further comprising:

developing an acoustic profile of the highest probability possible word of the speech input pool;

developing an acoustic profile for each of the possible words of the handwriting input pool; and

selecting the highest probability possible word of the speech input pool if the highest probability possible word of the speech input pool matches an acoustic profile of a possible word of the handwriting input pool.

21. The machine-readable medium of claim 19, further comprising:

developing an average acoustic profile for the possible words of the speech input pool;

developing an acoustic profile for each of the possible words of the handwriting input pool and the speech input pool to create a combined input pool;

calculating a difference between the average acoustic profile and each of the acoustic profiles of the combined input pool;

reordering the possible words of the combined input pool based upon the difference, such that the possible word of the combined input pool having an acoustic profile with a least difference from the average acoustic profile is ordered first;

selecting the handwriting input pool possibility, from the combined pool, having the least difference if the difference is below a specified threshold.

displaying the reordered possible words of the combined input pool; and

receiving user modification to the handwriting input.