Capturing hand motion

• US 20030095708A1
• Filed: 11/21/2001
• Published: 05/22/2003
• Est. Priority Date: 11/21/2001
• Status: Active Grant

• Current Assignee: Candledragon, Inc.
• Original Assignee: Candledragon, Inc.
• Inventors: Pittel, Arkady
• Granted Patent: US 7,257,255 B2
• Time in Patent Office: Days
• US Class Current: 382/187
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/0418 for error correction or compensation, e.g. based on parallax, calibration or alignment
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/0428 by sensing at the edges of the touch surface the interruption of optical paths, e.g. an illumination plane, parallel to the touch surface which may be virtual sensing beam interruptions in a planar beam grid of an optical touch-screen G06F3/0421
• CPC Class Codes: G06V 30/1423 the instrument generating sequences of position coordinates corresponding to handwriting preprocessing or recognising digital ink G06V30/32
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A portable electronic device, a digital camera associated with the portable electronic device, and software configured to run on the portable electronic device and to derive handwriting and control information from hand motion of a writing instrument in the vicinity of the digital camera.

Claims

1. Apparatus comprising

a portable electronic device,

a digital camera associated with the portable electronic device, and

software configured to run on the portable electronic device and to derive handwriting and control information from hand motion of a writing instrument in the vicinity of the digital camera.

2. The apparatus of claim 1 in which the portable electronic device comprises a mobile telephone or a personal digital assistant.

3. The apparatus of claim 1 in which the digital camera is attached to the portable electronic device.

4. The apparatus of claim 1 in which the digital camera is incorporated in the portable electronic device.

5. The apparatus of claim 1 also including an infrared filter arranged to filter light being received from the writing instrument.

6. The apparatus of claim 1 also including a lens arranged to alter the focal length and/or depth of field of the digital camera.

7. The apparatus of claim 1 also including a mechanism configured to enable the digital camera to be attached to a writing surface.

8. The apparatus of claim 7 in which the mechanism comprises a suction device configured for attachment to a white board.

9. The apparatus of claim 7 in which the mechanism comprises a clip configured to grasp paper.

10. The apparatus of claim 7 in which the portable electronic device includes a writing surface.

11. The apparatus of claim 10 in which the writing surface is on a protective cover.

12. The apparatus of claim 1 in which the software is configured to define a mapping between a sensor surface in the digital camera and a space in which the hand motion is occurring.

13. The apparatus of claim 1 in which the software is configured to define the mapping in response to calibration steps that include a user marking three locations in the space in which the hand motion is occurring.

14. The apparatus of claim 1 in which the software is configured to derive the location and trajectory of the hand motion.

15. The apparatus of claim 1 in which the software is configured to generate the handwriting and control information based on processing cycles each associated with one location of the writing instrument.

16. The apparatus of claim 1 in which the software is configured to discriminate light received from the writing instrument by locking onto a carrier frequency at which light from writing instrument is modulated.

17. The apparatus of claim 1 in which the software is configured to determine a tilt of the writing instrument relative to a direction normal to a writing surface.

18. The apparatus of claim 1 in which the portable electronic device includes a display and the trajectory of the hand motion is shown on the display in real-time.

19. The apparatus of claim 18 in which the display is not touch-sensitive.

20. The apparatus of claim 1 in which the portable electronic device includes a digital signal processing chip and a general purpose microprocessor and the software is run in part on the chip and in part on the microprocessor.

21. The apparatus of claim 1 in which the portable electronic device includes a wireless communication facility and the software is configured to communicate the handwriting and control information to a remote location.

22. The apparatus of claim 1 in which the digital camera is configured to receive light that has been reflected from the writing instrument.

23. The apparatus of claim 1 in which the digital camera comprises a still camera.

24. The apparatus of claim 1 in which the digital camera comprises a video-capable camera.

25. The apparatus of claim 1 also including an infra-red beacon configured to be directed at the writing instrument.

26. The apparatus of claim 1 in which the software is configured to apply pattern recognition to signals from the digital camera.

27. A method comprising

exposing a sensor of a digital camera to a writing instrument that is being subjected to hand motion, and

in a device associated with the digital camera, processing the data to infer handwriting and/or control information based on the hand motion.

28. The method of claim 27 in which the exposing of the sensor includes receiving light that originates from light sources on the writing instrument.

29. The method of claim 27 in which the exposing of the sensor includes exposing the sensor to a trace or other marking left by the writing instrument.

30. The method of claim 29 in which the trace or other marking includes ink selected to increase a signal-to-noise ratio of light received by the sensor.

31. The method of claim 27 in which the exposing of the sensor includes processing of an image of a tip of a writing instrument.

32. The method of claim 31 in which the tip of the writing instrument is characterized by being of high contrast with the environment in which it is used.

33. The method of claim 27 in which the sensor is exposed to hand motion occurring with respect to any arbitrary surface at any arbitrary angle.

34. The method of claim 27 also including

calibrating the digital camera in a manner to permit correctly inferring linear hand motions.

35. The method of claim 27 also including calibrating the processing of data using information derived when the writing instrument is touched to at least two points on a writing surface.

36. The method of claim 27 also including automatically switching the digital camera from one format to another by coupling the camera, or a device to which it is connected, to another mechanism.

37. The method of claim 27 also including adjusting a tilt and/or swivel position of the camera for better coverage of a writing surface.

38. The method of claim 27 in which light sources on the writing instrument are operated to enabling filtering of noise and interference.

39. The method of claim 27 in which light sources on the writing instrument are sequenced to encode functionality that includes at least one of erasing or biometrics of handwriting.

40. The method of claim 27 in which there are enough light sources associated with the writing instrument to prevent loss of tracking when one or more of the light sources are blocked.

41. A method comprising

using a two-dimensional sensor in a digital camera to permit subpixel calculations of a position of a writing instrument that is in the field of view of the camera,

averaging lines, and

applying an algorithm to derive a final position from the averaged lines.