Method and apparatus for acquiring and organizing ink information in pen-aware computer systems

• US 7,894,641 B2
• Filed: 07/17/2009
• Issued: 02/22/2011
• Est. Priority Date: 03/07/2000
• Status: Expired due to Term

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Yaeger, Larry S., Fabrick, Richard W. II, Pagallo, Giulia M.
• Primary Examiner(s): Le; Brian Q
• Application Number:
  US12/505,249
• Publication Number:
  US 20090279783A1
• Time in Patent Office: 585 Days
• Field of Search: 382/119, 382187-189, 382/315, 345/179, 345/173, 178/18.03
• US Class Current: 382/119
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06V 30/1423 the instrument generating sequences of position coordinates corresponding to handwriting preprocessing or recognising digital ink G06V30/32

Abstract

An ink manager running at a computer system receives ink information entered at a pen-based input/display device and accumulates the ink information into ink strokes. The ink manager communicates with a handwriting recognition engine and includes an ink phrase termination engine that is configured to detect the occurrence of one or more ink phrase termination events by examining the ink information. Upon the occurrence of an ink phrase termination event, the ink manager notifies the handwriting recognition engine and organizes the preceding ink strokes into an ink phrase data structure. The ink manager may also pass the ink phrase to an application executing on the computer system that is associated with the ink information, and it, in response, may return a reference pointer and a recognition context to the ink manager. The reference pointer and recognition context are then appended to the ink phrase data structure.

Claims

1. A method comprising:

receiving ink information generated from a user input;

organizing the ink information into corresponding ink strokes;

examining the ink information to determine whether an ink phrase termination event has occurred, and

in response to the occurrence of an ink phrase termination event, storing the ink strokes received prior to the termination event as one or more ink phrases in a designated ink phrase data structure; and

in response to receiving a reference context from a client application affiliated with the un-recognized ink strokes of the ink phrase, associating the reference context with the ink strokes, wherein the reference context is associated with the respective ink phrase by appending the reference context to the designated ink phrase data structure.

2. The method of claim 1, further comprising passing the un-recognized ink strokes of the respective ink phrase to the client application in response to the ink phrase termination event.

3. The method of claim 1, further comprising:

generating one or more recognition hypotheses for the ink strokes of the ink phrase data structure; and

passing the one or more recognition hypotheses to the client application together with the respective reference context.

4. The method of claim 1, where the ink information further includes out-of-proximity data that corresponds to a pen being lifted above a surface of an input device, the method further comprising:

examining the ink information to detect out-of-proximity data; and

identifying the occurrence of an ink phrase termination event in response to detecting out-of-proximity data.

5. The method of claim 1, where the client application defines a form for display on an input device, the form having one or more data entry fields for receiving handwritten information, the method further comprising:

receiving a set of bounding coordinates established by the client application for the one or more data entry fields;

comparing the ink information from the input device with the bounding coordinates of the one or more data entry fields; and

identifying the occurrence of an ink phrase termination event in response to detecting ink information moving outside of the bounding coordinates for at least one of the one or more data entry fields.

6. The method of claim 1, further comprising:

applying a word segmentation model to the ink strokes as they are organized; and

identifying the occurrence of an ink phrase termination event when the word segmentation model determines that a given ink stroke is part of a new word relative to an immediately prior ink stroke.

7. The method of claim 1, further comprising:

initiating a time-out mechanism upon receipt of each ink data point; and

identifying the occurrence of an ink phrase termination event when the time-out expires prior to receiving a next sequential ink data point.

8. The method of claim 7, where the ink information further includes out-of-proximity data that corresponds to a pen being lifted above a surface of an input device, the method further comprising:

examining the ink information to detect out-of-proximity data; and

identifying the occurrence of an ink phrase termination event in response to detecting out-of-proximity data.

9. A system comprising:

a pen driver coupled to a pen-based input tablet and configured to collect and organize received ink information into ink strokes;

an ink memory area organized into one or more ink phrase data structures; and

an ink manager coupled to the pen driver for receiving the ink strokes, the ink manager having an ink phrase termination engine configured to detect the occurrence of an ink phrase termination event from the ink information;

whereby the ink information entered at the pen-based input tablet is associated with a client application; and

the ink manager stores the ink strokes received prior to the ink phrase termination event in a selected ink phrase data structure and, in response to receiving from the client application a reference context affiliated with the un-recognized ink strokes of the ink phrase, associates the reference context with the ink strokes, where the ink manager associates the reference context with the un-recognized ink strokes by appending the reference context to the selected ink phrase data structure.

10. The system of claim 9, where the ink manager, in response to the occurrence of an ink phrase termination event, passes the un-recognized ink strokes of the respective ink phrase to the client application.

11. The system of claim 9, where the ink phrase termination engine is configured to initiate a time-out for each ink stroke and where the termination engine identifies the occurrence of an ink phrase termination event when the time-out expires before the next sequential ink stroke is detected.

12. The system of claim 11, where the pen-based input tablet has a surface and the ink information generated by the tablet includes out-of-proximity data corresponding to the pen being lifted above the surface of the tablet, and where the termination engine detects the occurrence of an ink phrase termination event upon detecting out-of-proximity data from the tablet.

13. The system of claim 9, further comprising:

one or more handwriting recognition engines for generating hypotheses based on the ink information entered at the pen-based tablet; and

a handwriting recognition manager coupled to both the ink manager and the one or more handwriting recognition engines, the handwriting recognition manager configured and arranged to coordinate operation of the one or more handwriting recognition engines, where;

the ink strokes received at the ink manager are passed to the handwriting recognition manager, and

the ink manager notifies the handwriting recognition manager of the occurrence of each ink phrase termination event and, in response, the handwriting recognition manager directs a selected handwriting recognition engine to generate one or more hypotheses for the ink strokes corresponding to the respective ink phrase.

14. The system of claim 13, where the handwriting recognition manager in cooperation with the selected handwriting recognition engine employs a word segmentation model to the ink strokes as they are received by the ink manager and, in response to determining that a given ink stroke represents a new word, issues an ink phrase termination signal to the ink manager.

15. The system of claim 13, where:

the client application is configured to define at least one data entry field for display on the tablet and to establish corresponding boundary coordinates for the at least one data entry field, and

the termination engine identifies the occurrence of an ink phrase termination event when an ink stroke or portion thereof is outside of the boundary coordinates for the at least one data entry field.

16. The system of claim 13, where the ink manager:

in response to receiving from the client application a reference context affiliated with the un-recognized ink strokes of the ink phrase, associates the reference context with the ink strokes, and

in response to a request by the client application, returns the affiliated reference context to the client application together with the one or more hypotheses.

17. The system of claim 13, where in response to receiving an indication that the client application has consumed the un-recognized ink strokes, the ink manager directs the handwriting recognition manager not to generate one or more hypotheses for the ink strokes.

18. The system of claim 13, where

in response to receiving the un-recognized ink strokes, the client application establishes a corresponding recognition context of the ink strokes, and

the handwriting recognition manager receives the recognition context and directs the selected handwriting recognition engine to utilize the recognition context in generating the one or more hypotheses.

19. The system of claim 18, where the one or more hypotheses generated are provided to the client application.

20. A method comprising:

converting a received user input into ink information;

converting the ink information into one or more strokes;

determining whether a phrase termination event has occurred such that when a phrase termination event occurs;

defining an unrecognized phrase from the one or more strokes received prior to the phrase termination event;

determining whether to recognize the phrase or send the unrecognized phrase to a distinct application;

when it is determined to send the unrecognized phrase to the distinct application, sending the unrecognized phrase to the application;

when it is determined to recognize the phrase, requesting recognition of the ink information of the phrase and sending recognition results to the application.

21. The method of claim 20, further comprising:

performing pre-recognition processing of the ink information as the ink information is being organized into the one or more strokes.

22. The method of claim 21, where the pre-recognition processing includes applying a word segmentation model to the ink strokes as they are received.

23. The method of claim 21, where the pre-recognition processing is used to identify a phrase termination event.

24. The method of claim 20, where determining whether a phrase termination event has occurred includes identifying a break in received user input.

25. The method of claim 20, where when it is determined to recognize the phrase, sending recognition results to the application includes generating one or more recognition hypotheses from the ink strokes of the ink phrase and passing the hypotheses to the application.