Pen-based handwritten character recognition and storage system

• US 20010038711A1
• Filed: 01/05/2001
• Published: 11/08/2001
• Est. Priority Date: 01/06/2000
• Status: Active Grant

• Current Assignee: Zen Optical Technology, LLC
• Original Assignee: Zen Optical Technology, LLC
• Inventors: Williams, David R., Richter, Kathie S.
• Granted Patent: US 6,968,083 B2
• Time in Patent Office: Days
• US Class Current: 382/181
• CPC Class Codes: G06V 10/12 Details of acquisition arrangements; Constructional details thereof
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 30/1423 the instrument generating sequences of position coordinates corresponding to handwriting preprocessing or recognising digital ink G06V30/32

Abstract

The present invention is directed to an improved system and method in which a pen-sized and shaped device detects, recognizes and stores handwriting as it is written by the device. The invention employs both an active feedback network and a character recognition methodology of partitioning detected input into character components. The active feedback network continually monitors device output to determine the sufficiency of the data input. If the data input is insufficient, the device modifies its detection methodology to obtain data the device readily recognizes. Data recognition is performed in multiple asynchronous processes. Elements of individual characters are sampled by the detector. Character elements are processed and recognized on this elemental level. Recognized character elements are stored for subsequent assembly and recognition on a character level. Thus, preferably two recognition sub-processes take place, one on a character element level and another on a character level.

Claims

1. A device for writing and processing handwriting comprising:

a marking element for making strokes comprising a character on a surface;

a detector for detecting each stroke on the surface; and

a processor coupled to the detector, characterizing each detected stroke as one in a set of reference strokes.

2. The device according to

claim 1

, wherein the photo detector detects the strokes at a periodic rate which is adjusted based on input from an active feedback mechanism.

3. The device of

claim 1

, wherein the processor identifies the character by combining the recognized strokes of the character and comparing the combined recognized strokes with a reference set of combined recognized strokes.

4. The device of

claim 3

, wherein the processor is comprised of a first sub-processor for characterizing each detected stroke as one in a set of reference strokes, and a second sub-processor for identifying the character, the first and second sub-processors functioning asynchronously.

5. The device of

claim 1

, wherein the processor characterizes each detected stroke as one in a set of reference strokes by representing each detected stroke as a polynomial representation, comparing the polynomial representation of each detected stroke with polynomial representations of the reference strokes, and selecting for each detected stroke a reference stroke whose polynomial representation is sufficiently similar to the polynomial representation of the detected stroke.

6. The device of

claim 1

, wherein the processor characterizes each detected stroke as one in a set of reference strokes by representing each detected stroke as a vector representation, comparing the vector representation of each detected stroke with vector representations of the reference strokes, and selecting for each detected stroke a reference stroke whose vector representation is sufficiently similar to the vector representation of the detected stroke.

7. The device of

claim 1

further comprising a character output mechanism for outputting a signal representing the character.

8. The device of

claim 1

, wherein the detector detects the strokes in the temporal order that the strokes are made.

12. The device according to

claim 1

, wherein the detector comprises:

a photo emitter mounted on a first side of the device, wherein said photo emitter emits light towards the marking element to illuminate the strokes; and

a photo detector mounted on a second side of the device, wherein said photo detector (a) has a field of view that includes the marking element, (b) detects the strokes using the light reflected off the surface, (c) converts the detected strokes into electronic signals, and (d) sends the electronic signals to the processor.

13. The device according to

claim 12

, wherein the first side is opposite the second side.

14. The device according to

claim 12

, wherein the photo emitter and the photo detector are mounted adjacent to the marking element.

15. The device according to

claim 12

, wherein the photo detector detects the strokes at a periodic rate which is adjusted based on input from an active feedback mechanism.

16. The device according to

claim 12

, wherein the light and the field of view are approximately centered upon the marking element.

17. The device according to

claim 12

, wherein the photo emitter emits constant light and the photo detector detects diffusely reflected light.

18. The device according to

claim 12

, wherein the photo emitter emits pulsed light and the photo detector detects spectrally reflected light.

19. The device according to

claim 1

, wherein the detector comprises:

a photo emitter mounted on a first side of the device, wherein said photo emitter emits light towards the marking element to illuminate the strokes;

a first photo detector mounted on a second side of the device, wherein said first photo detector (a) has a first field of view that includes the marking element, (b) detects the strokes using the light reflected off the surface, (c) converts the detected strokes into electronic signals, and (d) sends the electronic signals to the processor; and

a second photo detector mounted on a third side of the device, wherein said second photo detector (a) has a second field of view that includes the marking element, (b) detects the strokes using the light reflected off the surface, (c) converts the detected strokes into electronic signals, and (d) sends the electronic signals to the processor.

20. The device according to

claim 19

, wherein the second side is opposite the third side.

21. The device according to

claim 19

, wherein the photo emitter, the first photo detector, and the second photo detector are mounted adjacent to the marking element.

22. The device according to

claim 19

, wherein the first photo detector and the second photo detector detect the strokes at a periodic rate which is adjusted based on input from an active feedback mechanism.

23. The device according to

claim 19

, wherein the light, the first field of view, and the second field of view are approximately centered upon the marking element.

24. The device according to

claim 19

, wherein the first field of view overlaps the second field of view.

25. The device according to

claim 19

, wherein the photo emitter emits constant light and the first photo detector and the second photo detector detect diffusely reflected light.

26. The device according to

claim 19

, wherein the photo emitter emits pulsed light and the first photo detector and the second photo detector detect spectrally reflected light.

27. The device according to

claim 1

, wherein the detector comprises:

a multi-segment photo emitter mounted on the device, wherein said multi-segment photo emitter emits light towards the marking element to illuminate the strokes;

a first photo detector mounted on a first side of the device, wherein said first photo detector (a) has a first field of view that includes the marking element, (b) detects the strokes using the light reflected off the surface, (c) converts the detected strokes into electronic signals, and (d) sends the electronic signals to the processor; and

a second photo detector mounted on a second side of the device, wherein said second photo detector (a) has a second field of view that includes the marking element, (b) detects the strokes using the light reflected off the surface, (c) converts the detected strokes into electronic signals, and (d) sends the electronic signals to the processor.

28. The device according to

claim 27

, wherein the multi-segment photo emitter is in the shape of a ring, with the center of the ring perpendicular to a z-axis that passes through a center of the marking element and is parallel a major axis of the device.

29. The device according to

claim 27

, wherein the first side is opposite the second side.

30. The device according to

claim 27

, wherein the multi-segment photo emitter, the first photo detector, and the second photo detector are mounted adjacent to the marking element.

31. The device according to

claim 27

, wherein the first photo detector and the second photo detector detect the strokes at a periodic rate which is adjusted based on input from an active feedback mechanism.

32. The device according to

claim 27

, wherein the light, the first field of view, and the second field of view are approximately centered upon the marking element.

33. The device according to

claim 27

, wherein the first field of view overlaps the second field of view.

34. The device according to

claim 27

, wherein the multi-segment photo emitter emits constant light and the first photo detector and the second photo detector detect diffusely reflected light.

35. The device according to

claim 27

, wherein the multi-segment photo emitter emits pulsed light and the first photo detector and the second photo detector detect spectrally reflected light.

36. The device according to

claim 1

, wherein the detector comprises:

an array that includes at least one photo emitter and at least one photo detector;

wherein each at least one photo emitter emits light towards the marking element to illuminate the strokes; and

wherein each at least one photo detector (a) has a field of view that includes the marking element, (b) detects the strokes using the light reflected off the surface, (c) converts the detected strokes into electronic signals, and (d) sends the electronic signals to the processor.

37. The device according to

claim 36

, wherein the array is in the shape of a ring, with the center of the ring perpendicular to a z-axis that passes through a center of the marking element and is parallel a major axis of the device.

38. The device according to

claim 36

, wherein the array contains eight elements, each element being either a photo emitter or a photo detector.

39. The device according to

claim 38

, wherein each element is equally spaced within the array.

40. The device according to

claim 36

, wherein the array is mounted adjacent to the marking element.

41. The device according to

claim 36

, wherein each at least one photo detector detects the strokes at a periodic rate which is adjusted based on input from an active feedback mechanism.

42. The device according to

claim 36

, wherein the light and the field of view are approximately centered upon the marking element.

43. The device according to

claim 36

, wherein each field of view overlaps at least one other field of view.

44. The device according to

claim 36

, wherein each at least one photo emitter emits constant light and each at least one photo detector detects diffusely reflected light.

45. The device according to

claim 36

, wherein each at least one photo emitter emits pulsed light and each at least one photo detector detects spectrally reflected light.

46. The device according to

claim 1

, wherein marking element comprises a ball.

47. The device according to

claim 46

, wherein the detector comprises:

a first microfeeler in contact with a first side of the ball, said first microfeeler changing impedance as the ball rotates;

a first friction roller in contact with the first microfeeler, wherein said first friction roller detects impedance changes in the first microfeeler as electronic signals and sends the electronic signals to the processor; and

a second microfeeler in contact with a second side of the ball, said second microfeeler changing impedance as the ball rotates;

a second friction roller in contact with the second microfeeler, wherein said second microfeeler detects impedance changes in the second microfeeler as electronic signals and sends the electronic signal to the processor.

48. The device according to

claim 47

, wherein the first side, the second side, and a z-axis are mutually orthogonal, wherein the z-axis passes through a center of the ball and is parallel to a major axis of the device.

49. The device according to

claim 47

, wherein the impedance changes are detected at a periodic rate which is adjusted based on input from an active feedback mechanism.

50. The device according to

claim 46

, wherein the ball comprises a plurality of discrete magnetic domains asymmetrically distributed over the surface of the ball, said plurality of discrete magnetic domains being uniform in size.

51. The device according to

claim 50

, wherein the detector comprises:

a first induction coil having evenly spaced symmetrical windings, wherein said first induction coil (a) is positioned adjacent to a first side of the ball, (b) senses ball movement using the plurality of discrete magnetic domains, (c) produces electronic signals based on the ball movement, and (d) sends the electronic signals to the processor; and

a second induction coil having evenly spaced symmetrical windings, wherein said second induction coil (a) is positioned adjacent to a second side of the ball, (b) senses ball movement using the plurality of discrete magnetic domains, (c) produces electronic signals based on the ball movement, and (d) sends the electronic signals to the processor.

52. The device according to

claim 51

, wherein the first side, the second side, and a z-axis are mutually orthogonal, wherein the z-axis passes through a center of the ball and is parallel to major axis of the device.

53. The device according to

claim 51

, wherein the impedance changes are detected at a periodic rate which is adjusted based on input from an active feedback mechanism.

54. The device according to

claim 46

, wherein the ball includes a plurality of discrete magnetic domains distributed in a characterized non-uniform pattern over the surface of the ball, said plurality of discrete magnetic domains being non-uniform in size.

55. The device according to

claim 54

, wherein the detector comprises:

a first induction coil having asymmetrical windings with linearly increasing spacing, wherein said first induction coil (a) is positioned adjacent to a first side of the ball, (b) senses ball movement using the plurality of discrete magnetic domains, (c) produces electronic signals based on the ball movement, and (d) sends the electronic signals to the processor; and

a second induction coil having asymmetrical windings with linearly increasing spacing, wherein said second induction coil (a) is positioned adjacent to a second side of the ball, (b) senses ball movement using the plurality of discrete magnetic domains, (c) produces electronic signals based on the ball movement, and (d) sends the electronic signals to the processor.

56. The device according to

claim 55

, wherein the first side, the second side, and a z-axis are mutually orthogonal, wherein the z-axis passes through a center of the ball and is parallel to major axis of the device.

57. The device according to

claim 55

, wherein the impedance changes are detected at a periodic rate which is adjusted based on input from an active feedback mechanism.

58. The device according to

claim 46

, wherein the ball includes an outer shell, said outer shell comprising a single magnetic domain.

59. The device according to

claim 58

, wherein the detector comprises:

a first induction coil having evenly spaced symmetrical windings, wherein said first induction coil (a) is positioned adjacent to a first side of the ball, (b) senses ball movement using the plurality of discrete magnetic domains, (c) produces electronic signals based on the ball movement, and (d) sends the electronic signals to the processor; and

a second induction coil having evenly spaced symmetrical windings that are uniform in size, wherein said second induction coil (a) is positioned adjacent to a second side of the ball, (b) senses ball movement using the plurality of discrete magnetic domains, (c) produces electronic signals based on the ball movement, and (d) sends the electronic signals to the processor.

60. The device according to

claim 59

, wherein the first side, the second side, and a z-axis are mutually orthogonal, wherein the z-axis passes through a center of the ball and is parallel to major axis of the device.

61. The device according to

claim 59

, wherein the impedance changes are detected at a periodic rate which is adjusted based on input from an active feedback mechanism.

9. A device for writing and processing handwriting comprising:

a marking element for providing on a surface a representation of a character;

a detector operating according to a detection parameter having a value, the detector detecting the character on the surface;

a processor for comparing the character to a set of reference characters and determining therefrom whether the character is represented by any one of the reference characters; and

an active feedback mechanism for modifying the value of the detection parameter if the character is represented by none of the reference characters.

10. The device of

claim 9

, wherein the detection parameter is a sampling rate of the detector.

11. The device of

claim 9

further comprising a character output mechanism for outputting a signal representing the character.

62. A method of detecting handwriting comprising the steps of:

(a) illuminating strokes comprising a character marked on a surface;

(b) detecting light reflected off the surface;

(c) converting the detected light into an electronic signal;

(d) interpreting the electronic signal to identify the strokes; and

(e) identifying the strokes as the character.

63. The method according to

claim 62

, wherein step (b) is performed by detecting the light at a periodic rate.

64. The method according to

claim 63

, with the additional step of adjusting the periodic rate if the set of strokes cannot be identified as the character in step (e), wherein an active feedback mechanism is used to determine the amount by which the rate is adjusted.

65. The method according to

claim 62

, wherein step (d) is performed by:

characterizing each detected stroke as one in a set of reference strokes by comparing each detected stroke with the reference strokes; and

selecting for each detected stroke a reference stroke that is sufficiently similar to the detected stroke.

66. In a pen utilizing a ball for marking strokes comprising a character on a surface, a method of detecting handwriting comprising the steps of:

(a) detecting the strokes by directly detecting rotations of the ball;

(b) converting the detected rotations into electronic signals;

(c) interpreting the electronic signals to identify the strokes; and

(d) identifying the strokes as the character.

67. The method according to

claim 66

, wherein step (d) is performed by:

characterizing each detected stroke as one in a set of reference strokes by comparing each detected stroke with the reference strokes; and

selecting for each detected stroke a reference stroke that is sufficiently similar to the detected stroke.

68. The method according to

claim 66

, wherein step (a) is performed by:

generating a first change in impedance as the ball rotates along a first axis;

generating a second change in impedance as the ball rotates along a second axis;

measuring the first change in impedance; and

measuring the second change in impedance.

69. The method according to

claim 68

, wherein the first axis, the second axis, and a z-axis are mutually orthogonal, wherein the z-axis passes through a center of the ball and is parallel to a major axis of the pen.

70. The method according to

claim 66

, wherein the ball includes at least one magnetic domain.

71. The method according to

claim 70

, wherein step (a) comprises the steps of:

generating a first current along a first axis using the at least one magnetic domain;

generating a second current along a second axis using the at least one magnetic domain;

measuring the first current; and

measuring the second current.

72. The method according to

claim 71

, wherein the first axis, the second axis, and a z-axis are mutually orthogonal, wherein the z-axis passes through a center of the ball and is parallel to a major axis of the pen.