Method for correcting handwriting on a pen-based computer

• US 5,710,831 A
• Filed: 06/07/1995
• Issued: 01/20/1998
• Est. Priority Date: 07/30/1993
• Status: Expired due to Term

• Current Assignee: Apple Inc.
• Original Assignee: Apple Computer Incorporated (Apple Inc.)
• Inventors: Meier, John R., Capps, Stephen P., Tou, Frederich N., Beernink, Ernest H.
• Primary Examiner(s): Razavi, Michael T.
• Assistant Examiner(s): CHANG, JON CARLTON
• Application Number:
  US08/476,886
• Time in Patent Office: 958 Days
• Field of Search: 382/189, 382/187, 382/186, 382/310, 382/309, 382/311, 382/179, 395/179, 395/792, 395/793, 395/795
• US Class Current: 382/189
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06V 10/98 Detection or correction of errors, e.g. by rescanning the pattern or by human intervention; Evaluation of the quality of the acquired patterns
• CPC Class Codes: G06V 30/1423 the instrument generating sequences of position coordinates corresponding to handwriting preprocessing or recognising digital ink G06V30/32
• CPC Class Codes: G06V 30/373 using a special pattern or subpattern alphabet

Abstract

A method for correcting textual material recognized on a pen-based computer in accordance with the present invention includes the step of receiving an input on a screen of the computer system. The input is then recognized, and a bounding box defining the bounds of the input is defined. A determination of the intention and placement of the recognized input is then made according to the relationship between the bounding box of the recognized input and the bounding boxes of the already existing textual objects. When a substantial overlap is found of the recognized input with an existing textual object and the number of characters of the recognized input is less than a predetermined number, the recognized input is considered as an overwrite of the existing text. If the conditions are not met, the recognized input is considered as an insertion of a new textual object, i.e., a new word. A best potential character is determined from the recognized input if the overwrite conditions are met, and the most overlapped character of the overlapped textual object is replaced by the best potential character.

Claims

1. A method for correcting text objects displayed on a display screen of a pen based computer system, the method comprising the steps of:

a) receiving an initial input on the display screen corresponding to at least one text object;

b) recognizing the initial input as said at least one text object having at least one character, wherein the recognized initial input includes a text object bounding box;

c) receiving an additional input on the display screen such that an additional input bounding box is defined for said additional input that includes at least one character;

d) determining whether said additional input is intended for overwriting based on an amount the additional input overlaps at least a portion of said text object by determining if there is substantial overlap between said additional input bounding box and said text object bounding box;

e) re-recognizing the initial input in conjunction with said additional input when it has been determined that the additional input was intended for overwriting;

f) determining a best potential replacement character of said text object based upon a result of said re-recognizing step; and

g) correcting said text object using the best potential replacement character result and displaying the result.

2. A method for correcting textual material as recited in claim 1 wherein when it is determined that said additional input is intended for overwriting, it is determined to be a correction gesture, and when more than a predefined number of characters of said text object are overlapped by the additional input, the additional input is treated as a word object insertion gesture as opposed to a correction gesture.

3. A method for correcting textual material as recited in claim 2 wherein the predefined number of overlapped characters is two characters.

4. A method for correcting textual material as recited in claim 2 wherein when the input substantially overlaps a written character and the word correction gesture is recognized, the most overlapped written character is replaced by the additional input.

5. A method for correcting textual material as recited in claim 2 wherein the step of determining the best potential replacement character result includes the substeps of:

determining whether a most overlapped character is in the same textual position as the last corrected character;

clearing an attempted character string variable when the most overlapped character is in the same textual position as the last corrected character;

adding the most overlapped character to the attempted character string variable; and

creating a replacement list that prioritizes multiple potential interpretations in a re-recognition result based at least in part upon the characters in the attempted character string variable when the re-recognition result has a potential interpretation that is not in the attempted character string variable, wherein the best potential replacement character is selected based at least in part upon the prioritization in the replacement list.

6. A method for correcting textual material as recited in claim 5 wherein the replacement list is formulated by first listing the potential interpretations of the re-recognition result that were not in the attempted character list in the order they were prioritized by a recognizer and then listing the potential interpretations of the re-recognition result that were in the attempted character list in the order they were prioritized by the recognizer.

7. A method for correcting textual material as recited in claim 5 wherein the best potential replacement character is selected by sequentially inserting the characters in the replacement list into the selected word and comparing the resultant word to words in a dictionary means, wherein the first character associated with a resultant word that is found in the dictionary is selected as the best potential replacement character.

8. A method for correcting textual material displayed on a display screen of a pen based computer system, the method comprising the steps of:

a) receiving an input with a stylus on a dual function display screen of a pen-based computer system;

b) recognizing the input, the recognition step including the substep of defining a bounding box that bounds the input, wherein the input includes at least one character;

c) determining whether the input is a word correction gesture, wherein the word correction gesture determining step includes the substeps of,

determining whether the bounding box of the input substantially overlaps a displayed textual object having at least one character, wherein when them is not substantially overlap, the input is not considered to be a word correction gesture,

determining whether more than a predetermined number of characters of said displayed textual object are overlapped by the input, wherein when more than the predetermined number of characters are overlapped by a determined amount the input is not considered to be a word correction gesture,

determining whether the input is positioned entirely within a multi-space region, wherein when the input is positioned entirely within a multi-space, the input is not considered to be a word correction gesture,

d) determining an insertion position for the input and a selected word for correction when the input is determined to be a word correction gesture, wherein the insertion determining step includes the substeps of,

selecting the closest adjacent word to the midpoint of the input as the selected word for correction when the midpoint of the input is over a space character and setting the insert position of the input to an adjacent end of the selected word;

selecting the position of a most overlapped textual character as the insertion position when the input substantially overlaps textual characters and selecting the word associated with the most overlapped textual character as the selected word for correction;

e) re-recognizing the input in conjunction with characteristics of the selected word when the input is determined to be a word correction gesture and providing a plurality of potential new interpretations as a result of said re-recognizing;

f) determining a best potential replacement character result based upon a result of said re-recognizing step, by selecting said best potential replacement character as the highest probability of it being the correct replacement character based upon said plurality of potential new interpretations; and

g) correcting the selected word using the best potential replacement character result and displaying the result.

9. A method for correcting textual material as recited in claim 8 wherein the step of determining the best potential replacement character result includes the substeps of:

determining whether a most overlapped character element is in the same textual position as the last corrected character;

clearing an attempted character string variable when the most overlapped character element is in the same textual position as the last corrected character;

adding the most overlapped character dement to the attempted character string variable;

creating a replacement list that prioritizes multiple potential interpretations in a re-recognition result wherein the replacement list is formulated by first listing the potential interpretations of the re-recognition result that were not in the attempted character list in the order they were prioritized by the recognizer and then listing the potential interpretations of the re-recognition result that were in the attempted character list in the order they were prioritized by the recognizer; and

selecting the best potential replacement character by sequentially inserting the characters in the replacement list into the selected word and comparing the resultant word to words in a dictionary means, wherein the first character associated with a resultant word that is found in the dictionary is selected as the best potential replacement character.

10. In a computer system including a processing means, a display screen controlled by the processing means and a stylus for permitting a user to write on the display screen in order to input information into the processing means, an arrangement for correcting textual material displayed on a computer screen comprising:

means for determining the location of an input written onto the screen by the user;

a recognizer for recognizing the input as a text object including at least one character;

means for determining whether the input is a correction gesture intended to correct a word displayed on the display screen based on a determined amount of overlap of the input written onto the screen over the textual material that is displayed on the screen;

means for causing the recognizer to re-recognize the input in context when the input is determined to be part of a correction gesture, wherein said correction gesture also represents a correction character to replace at least one character of said text object;

means for selecting a particular re-recognition interpretation from a plurality of potential re-recognition interpretations based in part on whether particular interpretations have already been tried; and

means for replacing said at least one character of said text object with the selected re-recognition interpretation when a recognized input is determined to be part of a correction gesture.

11. A method of correcting textual material displayed on a display screen of a pen based computer system, the method comprising the steps of:

a) receiving an input on the display screen, wherein the input includes at least one character;

b) initially recognizing the input wherein the initial input recognition identifies a set of potential interpretations, each potential interpretation having a probability associated therewith, wherein the set includes at least one potential interpretation;

c) determining whether the recognized input is a word correction gesture based in part on g determination of whether the recognized input overlaps by a determined mount textual material that is displayed on the screen and if so, determining an insertion position for the recognized input and a selected word for correction;

d) recognizing the input a second time in conjunction with a context of the selected word for correction when the input is determined to be a word correction gesture;

e) determining a best potential replacement character result based at least in part upon a result of said step of recognizing the input in conjunction with a context of the selected word for correction; and

f) correcting the selected word using the best potential replacement character result and displaying the result, whereby corrections to textual material displayed on the display screen are made by overwriting the text to be corrected.

12. A method of correcting textual material as recited in claim 11 wherein when more than a predefined number of characters are overlapped by the input, the input is treated as a word insertion gesture as opposed to a correction gesture.

13. A method of correcting textual material as recited in claim 12 wherein the predefined number of overlapped characters is two characters.

14. A method of correcting textual material as recited in claim 11 wherein when the input substantially overlaps a written character and a word correction gesture is recognized, the most overlapped written character is replaced by the input.

15. A method of correcting textual material as recited in claim 14 wherein when a midpoint of the input is over a particular written character, that particular written character is considered the most overlapped character.

16. A method of correcting textual material as recited in claim 11 wherein when a midpoint of the input is over a tab or past the end of a line and the input is determined to be a word correction gesture, the insert position of the input is designated as an adjacent end of the closest adjacent written word.

17. A method of correcting textual material as recited in claim 11 wherein when a midpoint of the input is over a space character and the input is determined to be a word correction gesture, the insert position of the input is designated as an adjacent end of the closest adjacent written word.

18. A method as recited in claim 11 further comprising the steps of:

determining whether a bounding box of the input substantially overlaps a space between sequential words in a textual object; and

designating the insertion position as a closest end of an adjacent word when the bounding box of the input substantially overlaps a space between sequential words in the textual object.

19. A method of correcting textual material as recited in claim 11 further including the substeps of:

determining whether a most overlapped character element is in the same textual position as a last input correction character;

clearing a character string variable associated with the input when the most overlapped character element is in the same textual position as the last corrected character;

adding the most overlapped character element to the attempted character string variable; and

creating a replacement list that prioritizes multiple potential interpretations in a re-recognition result based at least in part upon the characters in the attempted character string variable when the re-recognition result has a potential interpretation that is not in the attempted character string variable, wherein the best potential replacement character is selected based at least in part upon the prioritization in the replacement list.

20. A method of correcting textual material as recited in claim 19 wherein the replacement list is formulated by first listing the potential interpretations of the result of the step of recognizing the input in conjunction with a context of the selected word for correction that were not in the attempted character list in the order they were prioritized by the recognizer and then listing the potential interpretations of the result of the step of recognizing the input in conjunction with a context of the selected word for correction that were in the attempted character list in the order they were prioritized by the recognizer.

21. A method of correcting textual material as recited in claim 19 wherein the step of recognizing the input in conjunction with a context of the selected word for correction is accomplished by sequentially inserting the characters in the replacement list into the selected word and comparing the resultant word to words in a dictionary, wherein the first character associated with a resultant word that is found in the dictionary is selected as the best potential replacement character.

22. A method as recited in claim 21 wherein when all potential interpretations found in the re-recognition result are also in the attempted character list, the best potential replacement character result is selected to be the top choice of the recognizer.

23. A method of correcting textual material as recited in claim 11 wherein when the input is positioned entirely within a tab, the input is treated as a word insertion as opposed to a correction.

24. A method of correcting textual material as recited in claim 11 wherein when the input is positioned entirely within a multi-space region that includes at least three consecutive spaces, the input is treated as a word insertion as opposed to a correction.

25. A method of correcting textual material as recited in claim 11 wherein:

the recognition step including the substep of defining a bounding box that bounds the input;

the word correction gesture determining step includes the substep of determining whether the bounding box of the input substantially overlaps a displayed textual object, wherein when there is no substantial overlap, the input is not considered to be a word correction gesture.

26. A method of correcting textual material displayed on a display screen of a pen based computer system, the method comprising the steps of:

a) receiving an input on the display screen;

b) recognizing the input, wherein the input recognition identifies a set of potential interpretations, each potential interpretation having a probability associated therewith, wherein the set includes at least one potential interpretation, the recognition step further including the substep of defining a bounding box that bounds the input;

c) determining whether the input is a word correction gesture, wherein the word correction gesture determining step includes the substeps of,

determining whether the bounding box of the input substantially overlaps a displayed textual object by a determined amount, wherein when there is no substantial overlap, the input is not considered to be a word correction gesture,

determining whether more than a predetermined number of characters are overlapped by the input, wherein when more than the predetermined number of characters are overlapped, the input is not considered to be a word correction gesture,

determining whether the input is positioned entirely within a multi-space region, wherein when the input is positioned entirely within a multi-space, the input is not considered to be a word correction gesture,

d) determining an insertion position for the input and a selected word for correction when the input is determined to be a word correction gesture, wherein the insertion determining step includes the substeps of,

selecting the closest adjacent word to the midpoint of the input as the selected word for correction when the midpoint of the input is over a space character and setting the insert position of the input to an adjacent end of the selected word;

selecting the position of a most overlapped textual character as the insertion position when the input substantially overlaps textual characters and selecting the word associated with the most overlapped textual character as the selected word for correction;

e) re-recognizing the input in conjunction with characteristics of the selected word when the input is determined to be a word correction gesture, the re-recognizing step being arranged to reassign probabilities to the potential interpretations when the probabilities calculated in the re-recognizing step for each potential interpretation differ from the probabilities associated with the potential interpretations of the recognizing step;

f) using the re-recognition results to create at least one resultant word based upon the potential interpretations and comparing the resultant word to words in a dictionary to further facilitate recognition of the input;

g) determining a best potential replacement character result based at least in part upon a result of said resultant word comparing step; and

h) correcting the selected word using the best potential replacement character result and displaying the result.

27. A method of correcting textual material as recited in claim 26 further including the substeps of:

determining whether a most overlapped character element is in the same textual position as the last corrected character;

clearing a character string variable associated with the attempted input when the most overlapped character element is in the same textual position as the last corrected character;

adding the most overlapped character element to the attempted character string variable; and

creating a replacement list that prioritizes multiple potential interpretations in a re-recognition result wherein the replacement list is formulated by first listing the potential interpretations of the recognition result that were not in the attempted character list in the order they were prioritized by the recognizer and then listing the potential interpretations of the re-recognition result that were in the attempted character list in the order they were prioritized by the recognizer; and

wherein the comparing step includes sequentially inserting the characters in the replacement list into the selected word and comparing the resultant word to words in a dictionary, and wherein the first character associated with a resultant word that is found in the dictionary is selected as the best potential replacement character.

28. In a computer system including a processing means, a display screen controlled by the processing means and a stylus for permitting a user to write on the display screen in order to input information into the processing means, an arrangement for correcting textual material displayed on a computer screen comprising:

means for determining the location of an input written onto the screen by the user, wherein the input includes at least one character;

a recognizer for recognizing the input;

means for determining whether the recognized input is a correction gesture intended to correct a word displayed on the display screen based in part on a determined amount of relative overlap of the input written onto the semen and the location where textual material is displayed on the screen;

means for causing the recognizer to re-recognize the input in context with the present textual material, when the input is determined to be part of a correction gesture;

means for selecting a particular re-recognition interpretation from a plurality of potential re-recognition interpretations based in pan on whether particular interpretations have already been tried; and

means for replacing an overwritten character with the selected re-recognition interpretation when a recognized input is determined to be part of a correction gesture.

29. A method of correcting existing textual material displayed on a display screen of a pen based computer system, the method comprising the steps of:

a) receiving a handwritten input on the display screen, wherein the input includes at least one character;

b) recognizing the input, wherein the input recognition identifies a set of potential interpretations, each potential interpretation having a probability associated therewith, wherein the set includes at least one potential interpretation;

c) determining whether the recognized handwritten input is a word correction gesture based in part on a determination of whether the input overlaps by a determined amount the existing textual material that is displayed on the screen and if so, determining an insertion position for the input and a selected word for correction;

d) re-recognizing the input taking into consideration characteristics of the selected word for correction in conjunction with the set of potential characters when the input is determined to be a word correction gesture to facilitate improved recognition of the input, the re-recognizing step being arranged to reassign probabilities for the potential interpretations when appropriate;

e) determining a best potential replacement character result from the set of potential interpretation based at least in part upon a result of said re-recognizing step; and

f) correcting the selected word using the best potential replacement character result and displaying the result, whereby corrections to existing textual material displayed on the display screen we made by overwriting the text to be corrected.

30. A method as recited in claim 29 wherein the characteristics of the selected word taken into consideration in the recognizing step includes a baseline of the selected word.

31. A method as recited in claim 30 wherein the characteristics of the selected word taken into consideration in the re-recognizing step further includes a height of the selected word.

32. A method of correcting textual material displayed on a display screen of a pen based computer system, the method comprising the steps of:

a) receiving a handwritten input on the display screen, wherein the input includes at least one character;

b) recognizing the input, wherein the input recognition identifies a set of potential interpretations, each potential interpretation having a probability associated therewith, wherein the set includes at least one potential interpretation;

c) determining whether the recognized input is a word correction gesture based in part on a determination of whether the input overlaps by a determined mount textual material that is displayed on the screen and if so, determining an insertion position for the input and a selected word for correction;

d) using the potential interpretations to create a set of at least one resultant word and comparing the resultant word set to words in a dictionary when the input is determined to be a word correction gesture to facilitate improved recognition of the input;

e) determining a best potential replacement character result based upon a result of the resultant word comparing step; and

f) correcting the selected word using the best potential replacement character result and displaying the result, whereby corrections to textual material displayed on the display screen are made by overwriting the text to be corrected.

33. A method of correcting textual material as recited in claim 32 wherein:

a replacement list is formulated by first listing the potential interpretations that are not previously attempted interpretations of the character being replaced in the order they are prioritized by a recognizer and then listing the potential interpretations that were previously attempted interpretations of the character being replaced in the order they were prioritized by the recognizer; and

the replacement list is used to prioritize the potential interpretations used to create the resultant word set.

34. A method of correcting textual material as recited in claim 33 wherein the comparing step is accomplished by sequentially inserting the characters in the replacement list into the selected word and comparing the resultant word to words in a dictionary, wherein the first character associated with a resultant word that is found in the dictionary is selected as the best potential replacement character.

35. A method as recited in claim 33 wherein when all potential interpretations were previously attempted interpretations, the best potential replacement character result is selected to be a top choice of the recognizer.

36. A method of correcting textual material as recited in claim 32 wherein the comparing step is accomplished by sequentially inserting potential interpretations into the selected word and comparing the resultant word to words in a dictionary, wherein the first character associated with a resultant word that is found in the dictionary is selected as the best potential replacement character.

37. A method for correcting textual material as recited in claim 32 wherein the selected word context recognizing step re-recognizes the input in conjunction with characteristics of the selected word for correction when the input is determined to be a word correction gesture, the re-recognizing step being arranged to reassign probabilities to the potential interpretations when the probabilities calculated in the re-recognizing step for each potential interpretation differ from the probabilities associated with the potential interpretations of the recognizing step.

38. A method for correcting textual material as recited in claim 32 wherein the selected word context recognizing step uses the input recognition results to create at least one resultant word based upon the potential interpretations and compares the resultant word to words in a dictionary to further facilitate recognition of the input.