Integrated handwriting and speed recognition systems

• US 6,415,256 B1
• Filed: 11/27/2000
• Issued: 07/02/2002
• Est. Priority Date: 12/21/1998
• Status: Expired due to Term

• Current Assignee: NetAirus Technologies LLC
• Original Assignee: Richard J. Ditzik
• Inventors: Ditzik, Richard Joseph
• Primary Examiner(s): Dorvil, Richemond
• Assistant Examiner(s): Nolan, Daniel A.
• Application Number:
  US09/723,192
• Time in Patent Office: 582 Days
• Field of Search: 704/260, 704/270, 704/231, 704/235, 704/251, 704/275, 345/169, 708/100, 708/141, 708/131
• US Class Current: 704/231
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06V 10/987 with the intervention of an operator
• CPC Class Codes: G06V 30/1423 the instrument generating sequences of position coordinates corresponding to handwriting preprocessing or recognising digital ink G06V30/32

Abstract

A computer system with speech recognition system and handwriting recognition system are disclosed that work closely together to improve the total recognition accuracy of each alone. The handwriting recognition system may include a pen/stylus input device and associated program functions. The system or programs may be combined with computer telephony functions to provide intergrated applications having voice output programs, Internet access, e-mail/v-mail and personal information manager functions. The computer system can recognize speaker-dependent and speaker-independent speech, converting this information to computer recognizable text, which may be displayed onto a display device in near realtime. Speech recognition errors may be corrected via a pen input device, and the pen information may be recognized, converted to text and graphics. This data may then be displayed at near realtime or displayed later at a user specified time. Recognized handwritten pen information may be intergrated into the speech recognized text and stored in a data storage system.

Claims

1. A computer program, residing on one or more computer-readable mediums, comprising instructions for causing at least one computer system to:

a) control data and information flow to and from said computer system and at least one user interface;

b) receive speech input data spoken by a user via speech input means and convert said input data into computer recognizable data under control of said computer system;

c) recognize said speech data by identifying best matches to known words or phases of a spoken language and output recognized speech text or data;

d) receive handwriting data from a user via a pen input means under control of said computer system, convert this data to electronic ink form of data and, at the option of the user, select recognition of said handwriting data;

e) relate said recognized speech data with said recognized handwriting data, at the option of said user, so that enhanced understanding of said information is accomplished; and

f) format for display said recognized speech data, said recognized handwriting data, or said converted electronic ink data.

2. A computer program as recited in claim 1, further comprising additional one or more cycles of speech and/or pen input with associated recognition processing which is accomplished to improve computer program recognition functions and user understanding of said information.

3. A computer program as recited in claim 1, in which said computer program is interfaced to the Internet via external communication means, wherein said computer program provides user interfaces for Internet applications.

4. A computer program as recited in claim 1, in which said computer program acts to control user interfaces for application programs, wherein typical editing functions can be accomplished via said additional pen inputs or additional speech inputs.

5. A computer program as recited in claim 1, in which said additional speech and pen input functions are coordinated, wherein said computer program provides telephony or Internet functions.

6. A computer program as recited in claim 1, in which said computer program resides in personal digital assistants or cellular phones devices, wherein said computer program controls user interface functions based on said pen input and speech input functions.

7. A computer program as recited in claim 6, in which said personal digital assistant or cellular phone includes handwriting transcription functions.

8. A computer program as recited in claim 6, in which said personal digital assistant or cellular phone includes audio recording functions, wherein audio information is recorded but not recognized into computer readable text.

9. A computer program as recited in claim 6, in which said personal digital assistant or cellular phone is connected to the Internet or telephony network via wireless communication means.

10. A computer program as recited in claim 6, in which said personal digital assistant or cellular phone includes speech transcription functions.

11. A computer program as recited in claim 1, in which said additional speech and pen input functions are coordinated to compliment each other'"'"'s functions.

12. A computer program as recited in claim 1, in which said altering functions can be accomplished, wherein understanding of said user interface can be assisted by session a setup function.

13. A computer program as recited in claim 1, in which said computer program controls two or more displayable cursors that are independently controlled by said computer program and user inputs.

14. A computer program, residing on one or more computer-readable mediums, comprising instructions for causing at least one computer system to:

a) control certain program functions of computer system, including data input and data output of data;

b) receive speech input spoken by a user and convert said speech input into computer recognizable data under control of said computer system;

c) recognize said speech data by identifying best matches to known words or phases of a spoken language, wherein said received speech and recognized speech forms speech input means;

d) receive handwriting data from a user via a pen input device under control of said computer system and recognize said handwriting data by identifying best matches to known written words or symbols, wherein the meaning of said handwriting data is either consistent with or inconsistent with said speech input data; and

e) combine recognition results of said handwriting and said speech recognition data, in a manner to provide output text or data with improved recognition accuracy.

15. A computer program as recited in claim 14, further causing the computer system to format said output text or data for display on a user interface.

16. A computer program as recited in claim 14, further causing the computer system to

a) receive user input to select one or more portions of said output text or data; and

b) process additional cycles of speech input functions or pen input functions for additional information recognition on said user selected portion of said output text or data.

17. A computer program as recited in claim 14, in which said pen input recognition and speech input recognition occurs at roughly realtime.

18. A computer program as recited in claim 14, in which said computer program is interfaced to the Internet via external communication means, wherein said computer program provides user interfaces and processing for Internet applications.

19. A computer program as recited in claim 14, in which said computer program is adapted to reside in personal digital assistants or cellular phones devices, wherein said computer program controls user interface functions using said pen input and speech input functions.

20. A method of controlling a user interface for viewing and control of a speech and pen data processing system, said method comprising the steps of:

a) displaying text, characters images, and/or graphics on a display screen of a display device;

b) running an operating system supporting a graphic user interface for example Windows™

on said display screen of said display device;

c) accepting pen input data from a pen input system and showing said pen input data on said display screen;

d) accepting speech input from a speech input means, recognizing the speech input and showing recognized speech text on said display screen; and

e) displaying, at the option of the user, system setup data of said speech and pen data processing system on said display screen.

21. A method of controlling a user interface as recited in claim 20, in which said pen input data is, at the option of the user, either recognized before said pen input data is shown on said display screen.

22. An computer operating system with a graphic user interface controlling a digital computer system, the operating system comprising the steps:

a) controlling multiple cursor operations in a single application program

b) accepting input data from first input device under control of said controlling multiple cursor operation step, and generating a first cursor;

c) generating data for the display of said first cursor under control of said controlling multiple cursor operation step;

d) accepting input data from second input device and generating a second cursor, under control of said controlling multiple cursor operation step;

e) generating data for the display of said second cursor under control of said controlling multiple cursor operation step; and

f) said first and second cursors are both visible to the user and active in the graphical user interface screen simultaneously, for example during an input device operation, and adapted to being independently controlled by said first and second input devices simultaneously.