Computer system with integrated telephony, handwriting and speech recognition functions

• US 6,167,376 A
• Filed: 12/21/1998
• Issued: 12/26/2000
• Est. Priority Date: 12/21/1998
• Status: Expired due to Term

• Current Assignee: NetAirus Technologies LLC
• Original Assignee: Richard J. Ditzik
• Inventors: Ditzik, Richard Joseph
• Primary Examiner(s): Smits, Talivaldis I.
• Assistant Examiner(s): Nolan, Daniel A
• Application Number:
  US09/217,339
• Time in Patent Office: 736 Days
• Field of Search: 704/9, 704/207, 704/278, 704/260, 704/235, 704/275
• US Class Current: 704/235
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 2203/0382 Plural input, i.e. interface arrangements in which a plurality of input device of the same type are in communication with a PC
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06V 10/987 with the intervention of an operator
• CPC Class Codes: G06V 30/1423 the instrument generating sequences of position coordinates corresponding to handwriting preprocessing or recognising digital ink G06V30/32
• CPC Class Codes: G06V 30/274 Syntactic or semantic context, e.g. balancing

Abstract

A computer system with speech recognition systems and handwriting recognition systems are disclosed that work closely together to improve the total recognition accuracy of each system alone. Handwriting recognition system may include a pen/stylus input device and associated program functions. These systems or programs may be combined with computer and telephone functions to provide integrated applications having voice output programs, Internet access, e-mail/v-mail, and personal information manager functions. The computer system can recognize speaker-dependent and speaker-independent speech, converting this information to computer recognizable text, which may be displayed onto a display device in realtime or delayed. Speech recognition errors may be corrected via a pen input device, and the pen information may be recognized, converted to text and graphics. This data may then be displayed immediately or displayed later at a user specified time. Recognized handwritten pen information may be integrated into speech recognized text and stored in a data storage system.

Claims

1. A method of performing handwriting and speech recognition functions, the method implemented in a programmed computer comprising a processor, a data storage system, at least one user input device, and at least one display device, the method comprising the steps of:

a. providing at least one control program to control the said computer;

b. receiving speech input spoken by a user, converting the said input into computer recognizable data under control of the said computer'"'"'s control program;

c. recognizing the said speech data by identifying the best match to known words or phases of a spoken language;

d. receiving handwriting data from a user via a pen or stylus input device under control of the said computer control program;

e. recognizing the said handwriting data from said user pen input and converting it into text, graphics or both;

f. displaying said recognized speech text with any recognition errors, and displaying said converted recognized handwriting data, at roughly the same time as said speech text; and

g. altering by adding, changing or deleting said displayed text via said pen or stylus input device.

2. A method as recited in claim 1, in which the handwriting data from the said pen input device is not recognized and is displayed as electronic ink on said display, tracking the pen tip'"'"'s path.

3. A method as recited in claim 1, in which the user has the option to display said recognized handwriting data within a time delay interval, wherein the maximum delay range interval is measured beginning immediately after recognition process is completed to the time of a command given by the user.

4. A method as recited in claim 1, further comprising steps to provide computer telephony functions, wherein voice and data information is communicated to and from telephone like communication networks, and wherein incoming speech information from a remote telephone is accepted, recognized and displayed in similar manner as the local speech input.

5. A method as recited in claim 4, in which the displaying speech text and handwriting data step presents at least one screen showing speech recognition setup menus, handwriting recognition setup menus, and telephony program setup menus.

6. A computer program, residing on a computer-readable medium, comprising instructions for causing a computer system to:

a. provide for bi-direction information flow to and from an operating system running on the said computer system;

b. receive speech input spoken by a user converting said input into computer recognizable data under control of the said operating system;

c. recognize the said speech data by identifying the best match to known words or phases of a spoken language;

d. receive handwriting data from a user via a pen or stylus device under control of the said operating system;

e. recognize the said handwriting data from the, said user pen input and convert it into text, graphics or both;

f. display the said recognized speech text with any recognition errors, and displaying the said converted recognized handwriting data, at roughly the same time as the said speech text; and

g. alter by adding, changing or deleting said displayed text via said pen or stylus input device.

7. A computer program as recited in claim 6, in which the handwriting data from said pen input device is displayed as electronic ink on said display, wherein the displayed electronic ink roughly tracks the pen tip'"'"'s path during handwriting, and wherein the delay is measured almost immediately after the recognition process is completed and ending at the time of a command.

8. A computer program as recited in claim 6, in which the user has the option to display said recognized handwriting data at a setable a time delay, wherein the delay interval is measured beginning almost immediately after recognition process and is completed after a set delay time.

9. A computer program as recited in claim 6, in which the user has the option to display said recognized handwriting data at a time delay, wherein the delay is measured beginning almost immediately after recognition process is completed and extending to the time at which a command given by the user.

10. A computer program as recited in claim 6, further comprising instructions for causing a computer system to provide computer telephony functions, wherein voice and data information is communicated to and from telephone like communication networks, and wherein incoming speech information from a remote telephone is received, recognized and displayed in similar manner as the local speech input.

11. A computer program as recited in claim 10, in which the telephone like communication networks include the Internet network or similar wide area computer networks.

12. A computer program as recited in claim 6, further comprising instructions for causing a computer system to display at least one screen showing setup menus with speech recognition, handwriting recognition, and optionally telephony program information.

13. A computer program as recited in claim 6, further comprising instructions for causing a computer system to have at least one integrated voice output program for audio output of recorded or realtime speech information.