Apparatus and method for tracking handwriting from visual input

• US 6,044,165 A
• Filed: 06/15/1995
• Issued: 03/28/2000
• Est. Priority Date: 06/15/1995
• Status: Expired due to Term

• Current Assignee: California Institute of Technology
• Original Assignee: California Institute of Technology
• Inventors: Perona, Pietro, Munich, Mario, Soatto, Steffano
• Primary Examiner(s): Bella, Matthew C.
• Application Number:
  US08/490,678
• Time in Patent Office: 1,748 Days
• Field of Search: 382/313, 382/314, 382/315, 382/186, 382/187, 382/188, 382/189, 382/100, 382/103, 382/119, 345/179, 345/156
• US Class Current: 382/103
• CPC Class Codes: G06V 30/1423 the instrument generating sequences of position coordinates corresponding to handwriting preprocessing or recognising digital ink G06V30/32
• CPC Class Codes: G06V 30/228 of three-dimensional handwriting, e.g. writing in the air

Abstract

A system for detecting movement of a writing implement relative to a writing surface to determine the path of the writing implement. The writing implement tip is determined within the image and used to form a kernel. The determination is made either manually, by looking for a predetermined pen tip shape, or by looking for a position of maximum motion in the image. That kernel is tracked from frame to frame to define the path of the writing implement. The tracking is accomplished by correlating the kernel to the image: either to the whole image, to a portion of the image near the last position of the kernel, or to a portion of the image predicted by a prediction filter.

Claims

1. A system for monitoring movement carried out by a writing implement on a writing surface using a visual input, comprising:

a camera of a type which monitors said writing implement relative to said writing surface, and produces an image signal indicative thereof;

a tracking element, which receives said image signal, recognizes a portion of the image which represents a portion of the writing implement in said image signal by comparing portions of said image signal to a writing implement image and processes said image signal to convert said image signal to a value indicative of position and/or movement of the writing implement relative to the writing surface; and

an image processor, operating to convert said value indicative of position and/or movement into coordinates indicating a path of movement of said writing implement relative to said writing surface.

2. The system as in claim 1 wherein said camera monitors the visual input of a user writing on a piece of paper.

3. A system as in claim 1 wherein said camera monitors a visual input of a user writing on a writable board.

4. A system as in claim 1 wherein said image processor includes a handwriting recognition unit, and further comprising an output unit, coupled to said image processor, which displays handwriting-recognized information.

5. A system as in claim 1 wherein said tracking element processes said image signals by:

1) determining the portion that represents a tip of the writing implement in said image signal; and

2) following motion of said portion to determine a path defined thereby.

6. A system as in claim 5 wherein said tracking element determines a motion of a centroid of said tip, and wherein said tracking element also corrects for an offset between a centroid of the tip and an actual position of the tip.

7. A system as in claim 5 wherein said processing element detects the tip by accepting a manual input indicating the position of the tip.

8. A system as in claim 5 further comprising a template memory storing at least one image template indicating an image portion of a specific writing implement pen tip, and wherein said tracking element correlates said template with said image signal to determine a position of a tip matching said template within said image signal.

9. A system as in claim 8 wherein said template memory stores a plurality of said image templates, and said tracking element searches said image signal to determine any of said plurality of said image templates in said image signal.

10. A system as in claim 5 wherein said image element detects said tip by detecting an area in the image signal which has maximum motion.

11. A system as in claim 5 wherein said tracking element follows motion by correlating a kernel image of the tip to positions of matching features in said image signal.

12. A system as in claim 11 wherein said tracking element carries out said correlating by determining a kernel indicative of a shape of said tip;

multiplying said kernel by a plurality of portions of the image signal; and

taking a maximum of said multiplications by said portions as being a correct position.

13. A system as in claim 11 wherein said correlating monitors positions in the image signal which are within a predetermined distance of a last position in the image signal.

14. A system as in claim 11 wherein said tracking element also carries out a prediction operation to predict a most likely position of the tip, said correlating carried out in said most likely position of the tip.

15. A system as in claim 11, further comprising a pyramid filter which filters said image signal to produce a reduced-resolution image signal;

wherein said correlating is carried out between said reduced-resolution image signal and said kernel.

16. A system as in claim 1, further comprising a memory for storing frames of information from said camera, said frames obtained at spaced time increments.

17. A system as in claim 16, wherein said tracking element determines a first condition of said writing implement relative to said writing surface, and does not store information in said memory while said first condition is determined to exist.

18. A system as in claim 17, wherein said first condition is said writing implement being spaced from said writing surface.

19. A writing processing system, comprising:

a first device which obtains video images of a part of a writing implement relative to a writing surface;

a processor, receiving said video images from said first device, and processing said video images to determine a position of the writing implement therein over multiple frames defined by said video images, to produce a tracking signal indicative of position and/or movement of the writing implement over said multiple frames, said processor recognizing different kinds of writing implements by storing a plurality of different template images, said different template images including different shapes of writing implement tips therein, determining a template image indicating a tip of a writing implement within a frame of the video image, and tracking a position of said template image within said frame, and said tracking signal indicating position and/or movement of said any of said different kinds of writing implements including a writing implement which is dark relative to its background; and

an output unit, which produces an output signal indicative of said tracking signal.

20. The system as in claim 19 wherein said first device monitors the visual input of a user writing on a piece of paper.

21. A system as in claim 19 wherein said first device monitors a visual input of a user writing on a writable board.

22. A system as in claim 19 wherein said output unit includes a handwriting recognizer, which recognizes handwriting based on said tracking signal and which displays recognized information.

23. A system as in claim 19 wherein said processor processes said video image by:

1) determining an image of a tip of the writing implement in said video images; and

2) following motion of said tip to determine a path defined thereby.

24. A system as in claim 23 wherein said processor follows a motion of a centroid near the tip, and wherein said processing element also corrects for an offset between a centroid of the tip and an actual position of the tip.

25. A system as in claim 23 wherein said processor detects the tip by accepting a manual input indicating the position of the tip.

26. A system as in claim 23 further comprising a template memory storing at least one image template indicating a image portion of a specific writing implement tip, and wherein said processing element correlates said template with said video image to determine a position of a tip matching said template within said video images.

27. A system as in claim 26 wherein said template memory stores a plurality of said templates, and said processing element searches said video images to determine any of said plurality of said templates in said video images.

28. A system as in claim 23 wherein said processing element detects said tip by detecting an area in the video images which has maximum motion.

29. A system as in claim 23 wherein said processing element follows motion by correlating a kernel image of the tip to positions of matching features in the video image.

30. A system as in claim 29 wherein said processing element carries out said correlation by determining a kernel indicative of a shape of said pen tip,

multiplying said kernel by a plurality of portions in the video image;

and

taking a maximum of said multiplications by said portion as being a correct position.

31. A system as in claim 29 wherein said correlating monitors positions in the video image which are within a predetermined distance of a last position in the video image.

32. A system as in claim 29 wherein said processor also carries out a prediction operation to predict a most likely position of the tip, said correlation carried out in said most likely position of the tip.

33. A system as in claim 19 wherein said processor operates by determining an area of the video image which has maximum motion, determining a centroid of said area of the image which has the maximum motion, and determining a distance between a position of a tip of the writing implement and the centroid.

34. A system as in claim 33 wherein said maximum motion is obtained by using a low pass filter to blur the image and find the maximum motion position.

35. A system as in claim 26 wherein said processing element finds an area of maximum motion of the image if no portion of the image matches a template.

36. A system as in claim 19 wherein said processing element determines whether the writing implement is touching the writing surface, and said output indicative of movement of the writing implement is produced only when said writing implement is touching the writing surface.

37. A system as in claim 36 further comprising a second camera, located in a different position spaced from the image, to triangulate x, y and z positions;

said x and y positions used for outputting positions of movement, and said z position used for determining whether the writing implement is touching the surface.

38. A system as in claim 32 wherein said prediction is carried out using a Kalman filter.

39. A system as in claim 19, wherein said processing element determines a first condition of said writing implement relative to said writing surface, and does not store information in said memory while said first condition is determined.

40. A method as in claim 39, wherein said desired point in said video image is a position of a tip of the writing implement, and said center point is a centroid of the kernel.

41. A method as in claim 40, wherein said tracing comprises obtaining a video image of a writing implement relative to a writing surface at a first time;

determining a portion of said image representing a kernel to be tracked in position;

obtaining a position of said kernel in said video image at said first time;

obtaining a video image of the writing implement relative to the writing surface at a second time;

obtaining a position of said kernel in said video image at said second time;

determining a distance between a desired point in said video image to be tracked, and a center point of said kernel; and

correcting said positions using said distance, and determining a path between said first and second positions.

42. A method of monitoring the movement of a writing implement on a writing surface, comprising:

obtaining a video image of a writing implement relative to a writing surface at a first time;

determining a portion of said image representing a kernel to be tracked in position;

obtaining a position of said kernel in said video image at said first time;

obtaining a video image of the writing implement relative to the writing surface at a second time;

obtaining a position of said kernel in said video image at said second time;

determining a distance between a desired point in said video image to be tracked, and a center point of said kernel;

correcting said positions using said distance, and determining a path between said first and second positions;

monitoring whether the writing implement is in contact with the writing surface, and wherein said determining comprises determining that said information is undesired when the writing implement is out of contact with the writing surface.