Method and system for data entry of handwritten symbols

• US 5,956,423 A
• Filed: 01/10/1994
• Issued: 09/21/1999
• Est. Priority Date: 06/17/1991
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Bishop, Bryon Dean, Frink, Lloyd
• Primary Examiner(s): Couso, Jose L.
• Application Number:
  US08/179,813
• Time in Patent Office: 2,080 Days
• Field of Search: 178/18-20, 345/179-183, 382/100, 382/187, 382/309
• US Class Current: 382/187
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06V 30/1423 the instrument generating sequences of position coordinates corresponding to handwriting preprocessing or recognising digital ink G06V30/32
• CPC Class Codes: G06V 30/32 Digital ink

Abstract

A method and system for data entry of handwritten text into a computer program that is not designed to accept handwritten text is provided. In preferred embodiments, the computer program is designed to operate in a windowing environment. A data entry program receives handwritten data, recognizes the data, and sends the recognized data to the computer program. The computer program processes the recognized data as if it had been entered from the keyboard. According to the present invention, the data entry program overlaps the window of the computer program with an invisible window. To the user, it looks as if the computer program is accepting handwritten data directly.

Claims

1. A system for inputting handwritten data into a target computer program, the target computer program designed to receive data from a computer keyboard and not designed to accept handwritten data, the target computer program having a window for displaying data on a display device, the window having a data entry portion, the handwritten data comprising symbols, each symbol having a corresponding symbol code of a coding scheme, the system comprising:

a computer for operating a plurality of computer programs;

an electronic tablet connected to the computer, wherein the electronic tablet inputs raw handwritten data, sends the raw handwritten data to the computer, displays ink corresponding to the raw handwritten data, receives data from the computer, and displays the data received from the computer;

a windowing environment computer program operating on the computer, the windowing environment computer program for controlling windows associated with the computer programs operating on the computer, wherein the windowing environment computer program controls the displaying of the window of the target computer program on the electronic tablet;

a hook computer program for detecting the start of writing on the electronic tablet wherein the writing is over a data entry portion of the window of the target computer program;

a recognition system that, in response to a request by a requesting computer program to return symbol codes, inputs the raw handwritten data from the electronic tablet, controls the inking of the electronic tablet, identifies the symbol codes corresponding to the symbols of the raw handwritten data and returns the identified symbol codes to the requesting computer program; and

a data entry computer program for displaying an invisible window when the hook computer program detects the start of writing, for requesting the recognition system to return symbol codes, for converting the symbol codes into virtual keyboard events without input from the computer keyboard, and for sending the virtual keyboard events to the target computer program, whereby the target computer program processes the virtual keyboard events as if the corresponding symbols were input from the computer keyboard without modification of the target computer program.

2. The system of claim 1 wherein the data entry computer program sends the returned symbol codes to the windowing environment computer program as virtual events and wherein the windowing environment computer program sends the returned symbol codes to the target computer program as keyboard events.

24. The system of claim 1 wherein the recognition system identifies symbol codes as codes of the American Standard Code for Information Interchange.

3. A method of sending data to a non-pen computer program designed to accept data from a keyboard, the method comprising the steps of:

inputting a handwritten gesture from a user of the computer program;

recognizing the handwritten gesture as a gesture symbol;

converting the gesture symbol to equivalent keyboard events; and

sending the equivalent keyboard events to the computer program for processing, wherein the computer program can process the equivalent keyboard events without the need for modifying the computer program.

4. A data processing method for inputting data into a non-pen computer program, the non-pen computer program designed to process data input as events, the method comprising the steps of:

inputting a handwritten gesture from a user of the non-pen computer program, the gesture having a function;

recognizing the gesture as a gesture symbol;

converting the gesture symbol to an equivalent event, the equivalent event representing the function of the gesture symbol; and

processing the equivalent event whereby the non-pen computer program effects the function of the handwritten gesture.

5. The method of claim 4 wherein the steps of inputting, recognizing, and converting are performed by a data entry computer program and including the step of sending the equivalent events from the data entry computer program to the computer program.

6. The method of claim 5 wherein the events include keyboard events.

7. The method of claim 5 wherein the events include mouse events.

8. The method of claim 5 wherein the events include messages.

9. The method of claim 4 wherein the computer program has an associated window and executes on a computer that is operatively connected to an electronic tablet, the electronic tablet having an associated pen, and wherein the step of inputting the handwritten gesture includes the step of detecting that the associated pen has contacted the electronic tablet within an area corresponding to the associated window whereby the detection indicates the start of the input of the handwritten gesture.

10. A method for inputting handwritten data into a non-pen computer program, the computer program designed to receive input from a computer keyboard, the handwritten data comprising symbols, each symbol having a corresponding symbol code of a coding scheme, the method comprising the steps of:

detecting an indication that handwritten data is to be input into the computer program;

creating an invisible window to accept the handwritten data;

inputting the handwritten data through the invisible window in response to detecting the indication;

identifying the symbol codes corresponding to symbols of the handwritten data; and

sending the identified symbol codes to the computer program as virtual keyboard events without input from the computer keyboard as symbols entered into the computer program through the computer keyboard, wherein the computer program processes the identified symbol codes without the need for modifying the computer program to process handwritten data.

11. The method of claim 10 wherein the computer program is designed to operate in a windowing environment.

12. The method of claim 10 wherein the computer program has an associated window and executes on a computer that is operatively connected to an electronic tablet, the electronic tablet having an associated pen, and wherein the step of detecting an indication includes detecting that the associated pen has contacted the electronic tablet within an area corresponding to the associated window.

13. The method of claim 10 including the steps of:

converting the identified symbol codes to equivalent mouse or keyboard events or messages.

14. The method of claim 10 wherein the steps of detecting, creating, inputting, and identifying are performed by a second computer program and including the step of sending the identified symbol codes from the second computer program to the computer program.

15. A method of inputting handwritten data into a non-pen computer program, the computer program executing on a computer with an electronic tablet and pen, the non-pen computer program having a first window with data entry areas, the first window being displayed on the electronic tablet, the handwritten data comprising symbols, each symbol having a corresponding symbol code of a coding scheme, the method comprising the steps of:

detecting when the pen contacts the electronic tablet within an area corresponding to a data entry area of the first window;

creating a second window to input handwritten data in response to the detection of pen contact;

inputting the handwritten data through the second window;

identifying the symbol codes corresponding to the symbols of the handwritten data; and

sending the identified symbol codes to the non-pen computer program.

16. A computer system for inputting handwritten data into a non-pen computer program comprising:

a computer with a computer memory that stores the non-pen computer program;

an electronic tablet connected to the computer, the electronic tablet for receiving handwritten data, the handwritten data comprising symbols, each symbol having a corresponding symbol code of a coding scheme; and

a handwritten data entry computer program stored in the computer memory for receiving handwritten data from the electronic tablet, for identifying the symbol codes corresponding to the symbols of the handwritten data, and for sending the identified symbol codes to the non-pen computer program as non-handwritten data, whereby the non-pen computer program processes the identified symbol codes as if the identified symbol codes were received by the computer system as non-handwritten data without the need for modifying the computer program.

17. The computer system of claim 16 including a display device connected to the computer for displaying output of the computer programs, wherein the target computer program displays output on a display device and the handwritten data entry computer program creates a window that logically overlaps the output of the target computer program, the window for receiving handwritten data to effect the appearance to a user of the computer system that the handwritten data is input directly into the target computer program.

18. A method in a computer system of inputting data through an alternate input device into a target computer program, the target computer program designed to input data from a primary input device and not designed to input data through the alternate input device, the method comprising the steps of:

executing a data entry computer program, wherein the data entry computer program receives data through the electronic tablet, converts the data to virtual input events of the primary input device, and sends the virtual input events to the target computer program; and

,

executing the target computer program, wherein the target computer program receives the virtual input events sent from the data entry computer program and processes the virtual input events as if the virtual input events were input directly to the target computer program by the primary input device without the need for modifying the target computer program to accommodate data from the alternate input device.

19. The method of claim 18 wherein the virtual input events correspond to keyboard events.

20. The method of claim 18 wherein the virtual input events correspond to mouse events.

21. The method of claim 18 wherein the computer system includes a display device for displaying output of the target computer program, wherein the target computer program displays output on the display device and the data entry computer program creates a window that logically overlaps the output of the target computer program, the window for receiving data input through the alternate input device which includes an electronic tablet to effect the appearance to a user of the computer system that the data input through the electronic tablet is input directly into the target computer program.

22. The method of claim 21 wherein the data input through the electronic tablet is handwritten data.

23. The method of claim 22 wherein the data input through the electronic tablet corresponds to mouse input.