Digital pen using visible image and autocorrelation of substrate pattern

• US 6,650,320 B1
• Filed: 08/16/2000
• Issued: 11/18/2003
• Est. Priority Date: 08/16/2000
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: International Business Machines Corporation
• Inventors: Zimmerman, Thomas Guthrie
• Primary Examiner(s): Hjerpe, Richard
• Assistant Examiner(s): Eisen, Alexander
• Application Number:
  US09/640,133
• Time in Patent Office: 1,189 Days
• Field of Search: 345/179, 345/183, 345/166, 178/18.01, 178/18.03, 178/18.05, 178/19.01, 178/19.04, 178/19.05
• US Class Current: 345/179
• CPC Class Codes: G06F 3/0317 in co-operation with a patterned surface, e.g. absolute position or relative movement detection for an optical mouse or pen positioned with respect to a coded surface
• CPC Class Codes: G06F 3/0321 by optically sensing the absolute position with respect to a regularly patterned surface forming a passive digitiser, e.g. pen optically detecting position indicative tags printed on a paper sheet constructional details of pen-shaped pointing devices G06F3/03545, G06F3/03542, G06F3/037
• CPC Class Codes: G06F 3/03545 Pens or stylus

Abstract

A digital pen that has an ink writing tip includes a light source on a pen body that directs light toward paper across which the writing tip is stroked. A CMOS camera or CCD is also mounted on the pen body for detecting reflections of the light. A processor in the pen body determines relative pen motion based on the reflections. A contact sensor such as an FSR on the pen body senses when the tip is pressed against the paper, with positions being recorded on a non-volatile memory in the pen body only when the contact sensor indicates that the pen is against the paper. Periodically, key frames are stored in memory but not for every cycle. The memory can be later engaged with a handwriting recognition device, such as a PC, to correlate the key frames and positions to alpha-numeric characters. Ordinary paper or quad-ruled paper can be used, and, if desired, special bar-coded paper can also be used, so that the PC can determine absolute pen position.

Claims

1. A digital pen, comprising:

an elongated body defining a writing tip;

a light source oriented on the body to direct light onto a substrate against which the writing tip can be positioned to deposit a substance thereon;

a contact sensor generating a contact signal when the writing tip abuts a surface;

a detector receiving reflections of light from the substrate; and

a pen processor receiving signals from the detector and in response thereto outputting delta signals representative of differences between successive positions of the pen by executing a bitwise XOR of at least a subset of a current frame of pixels against a previous frame of pixels at plural potential positions, a delta position being based on the potential position having a highest XOR sum, wherein when the contact signal indicates the writing tip is positioned against the substrate and a confidence threshold is reached, a delta signal is generated every cycle and a key frame is imaged and stored only once every “

N”

cycles, wherein “

N”

is an integer greater than one, wherein the pen processor has a local memory and wherein when a confidence threshold is reached as defined by a comparison between the current frame and last frame, a delta signal is stored in a memory supported by the body when the contact signal indicates the body is positioned against the substrate, and otherwise the delta signal is stored in the local memory of the pen processor.

2. The device of claim 1, wherein at least some of the light detected by the detector is from the substance.

3. The device of claim 2, wherein at least some of the light detected by the detector is from an optical code on the substrate.

4. The device of claim 1, wherein the memory supported by the body stores signals from the pen processor, whereby the signals can be retrieved from the memory supported by the pen body and processed to digitally represent writing input.

5. The device of claim 4, further comprising a contact sensor supported by the body, the contact sensor outputting a contact signal representative of whether the body is positioned against the substrate, wherein the pen processor selectively sends delta signals to the memory supported by the body based at least in part on the contact signal.

6. The device of claim 3, wherein the optical codes detectable by the detector represent at least positions, the detector outputting at least one code signal in response to detecting an optical code for determining an absolute position of the body.

7. The device of claim 5, wherein in addition to generating delta signals, the pen processor periodically stores key frames, each key frame being an image of the substrate.

8. The device of claim 1, wherein when the contact signal indicates the body is positioned against the substrate and the confidence threshold is not related for a low confidence cycle, a key frame is imaged and stored in memory supported by the body for the low confidence cycle.

9. The device of claim 1, further comprising a host processor receiving signals from the pen processor and accessing a library of optical patterns to determine a type of optical pattern on the substrate.

10. A digital input device, comprising:

an elongated body defining a distal tip configured for depositing a substance onto a substrate;

at least one light source oriented on the body to direct light toward the vicinity of the distal tip;

at least one detector oriented on the body to receive light from a substrate against which the distal from time to time can be positioned;

at least one contact sensor on the body; and

at least one pen processor on the body, the processor communicating with the detector and the contact sensor to determine, each cycle period, whether to generate delta signals and to store in memory onboard the body, each update period greater than a cycle period, a key frame image of the substrate, wherein the pen processor has a local memory and wherein when a confidence threshold is reached as defined by a comparison between a current frame and a last frame, a delta signal is stored in the memory onboard the body when the contact signal indicates the writing tip is positioned against the substrate, and otherwise the delta signal is stored in the local memory of the pen processor.

11. The device of claim 10, wherein delta signals are generated by the pen processor, the delta signals representing differences between successive positions of the distal tip, the pen processor generating a delta signal by executing a bitwise XOR of at least a subset of a current frame of pixels against a previous frame of pixels at plural potential positions, a delta position being based on the potential position having a highest XOR sum.

12. The device of claim 11, wherein at least some of the light received by the detector is from the substance.

13. The device of claim 12, wherein at least some of the light detected by the detector is from an optical code on the substrate.

14. The device of claim 11, wherein the memory stores delta signals from the pen processor, whereby the delta signals can be retrieved from the memory and processed by a handwriting recognition module to render signals representing alpha-numeric characters.

15. The device of claim 14, further comprising a contact sensor supported by the body, the contact sensor outputting a contact signal representative of whether the writing tip is positioned against the substrate, wherein the pen processor selectively sends delta signals to the memory based at least in part on the contact signal.

16. The device of claim 13, wherein the optical codes detectable by the detector represent at least positions, the detector outputting at least one code signal in response to detecting an optical code for determining an absolute position of the writing tip.

17. The device of claim 10, wherein when the contact signal indicates the writing tip is positioned against the substrate and the confidence threshold is reached, a delta signal is generated every cycle and a key frame is imaged and stored upon the detection of pen up and pen down events.

18. The device of claim 17, wherein when the contact signal indicates the writing tip is positioned against the substrate and the confidence threshold is not reached for a low confidence cycle, a key frame is imaged and stored in the memory onboard the body for the low confidence cycle.

19. The device of claim 10, further comprising a host processor receiving signals from the pen processor and accessing a library of optical patterns to determine a type of optical pattern on the substrate.

20. A computer-implemented method for digitizing strokes of a writing tip across a substrate, comprising:

detecting light from a substrate each cycle period;

comparing a frame of light received from the substrate in a second cycle to a frame of received light in a first cycle immediately preceding the second cycle to generate a delta signal;

occasionally but not every cycle generating at least one key frame;

storing the key frames and selectively storing delta signals in memory in a pen body associated with the writing tip;

transferring the key frames and delta signals to a host processor; and

using the host processor to correlate the delta signals and key frames to a digital representation of writing input, wherein when a confidence threshold is reached as defined by a comparison between the frame of light received in the first cycle and the frame of light received in the second cycle, storing a delta signal in the memory in the pen body when the contact signal indicates the writing tip is positioned against the substrate, and otherwise storing the delta signal in a local memory.

21. The method of claim 20, further comprising, at the host processor, accessing a library of bar codes to determine, using at least one key frame, whether a bar code is on the substrate.

22. The method of claim 21, wherein the method includes, if a bar code is on the substrate, using bar codes in key frames to determine absolute writing tip position.

23. The method of claim 20, wherein delta signals are generated by executing a bitwise XOR of at least a subset of a current frame of pixels against a previous frame of pixels at plural potential positions, a delta position being based on the potential position having a highest XOR sum.

24. The method of claim 23, further comprising detecting light from a substance deposited from the writing tip onto the substrate.

25. The method of claim 24, further comprising outputting a contact signal representative of whether the writing tip is positioned against the substrate, and selectively sending delta signals to the memory in the pen body based at least in part on the contact signal.

26. The method of claim 20, further comprising, when the contact signal indicates the writing tip is positioned against the substrate and the confidence threshold is reached, generating a delta signal every cycle and imaging and storing a key frame only once every “

N”

cycles, wherein “

N”

is an integer greater than one.

27. The method of claim 26, further comprising, when the contact signal indicates the writing tip is positioned against the substrate and the confidence threshold is not reached for a low confidence cycle, imaging and storing a key frame for the low confidence cycle.

28. The method of claim 20, wherein when the contact signal indicates the writing tip is positioned against the substrate and the confidence threshold is reached, a delta signal is generated every cycle and a key frame is imaged and stored at least upon detection of pen up and pen down events.

29. The method of claim 28, wherein optical codes are detected to represent at least positions, the method including outputting at least one code signal in response to detecting an optical code for determining an absolute position of the writing tip.

30. The method of claim 20, wherein the digital representation of writing input is selected from among the group including:

bitmap, vector, and alpha-numeric character representation.