Capturing hand motion

• US 7,257,255 B2
• Filed: 11/21/2001
• Issued: 08/14/2007
• Est. Priority Date: 11/21/2001
• Status: Expired due to Fees

• Current Assignee: Candledragon, Inc.
• Original Assignee: Candledragon, Inc.
• Inventors: Pittel, Arkady
• Primary Examiner(s): MARIAM, DANIEL G
• Application Number:
  US09/991,539
• Publication Number:
  US 20030095708A1
• Time in Patent Office: 2,092 Days
• Field of Search: 382186-189, 382/313, 178/19.01, 178/19.05, 345/179
• US Class Current: 382/187
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/0418 for error correction or compensation, e.g. based on parallax, calibration or alignment
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/0428 by sensing at the edges of the touch surface the interruption of optical paths, e.g. an illumination plane, parallel to the touch surface which may be virtual sensing beam interruptions in a planar beam grid of an optical touch-screen G06F3/0421
• CPC Class Codes: G06V 30/1423 the instrument generating sequences of position coordinates corresponding to handwriting preprocessing or recognising digital ink G06V30/32
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A portable electronic device, a digital camera associated with the portable electronic device, and software configured to run on the portable electronic device and to derive handwriting and control information from hand motion of a writing instrument in the vicinity of the digital camera.

Claims

1. Apparatus comprising

a portable electronic device comprising

a digital camera, and

a processor configured to derive handwriting or control information based on hand motion of a writing instrument from infared light received by the digital camera directly from the writing instrument when the writing instrument is used on a writing surface, the writing instrument and writing surface being separate from the apparatus.

2. The apparatus of claim 1 in which the portable electronic device comprises a mobile telephone or a personal digital assistant.

3. The apparatus of claim 1 also including an infrared filter arranged to filter light being received from the writing instrument.

4. The apparatus of claim 1 also including a lens arranged to alter the focal length and/or depth of field of the digital camera.

5. The apparatus of claim 1 also including a mechanism configured to enable the portable electronic device to be attached to a writing surface.

6. The apparatus of claim 5 in which the mechanism comprises a suction device configured for attachment to a white board.

7. The apparatus of claim 5 in which the mechanism comprises a clip configured to grasp paper.

8. The apparatus of claim 1 also comprising a writing surface.

9. The apparatus of claim 8 in which the writing surface is on a protective cover.

10. The apparatus of claim 1 in which the processor is configured to define a mapping between a sensor surface in the digital camera and a space in which the writing instrument is located.

11. The apparatus of claim 1 in which the processor is configured to define the mapping in response to calibration steps that include a user marking three locations in the space in which the writing instrument is located.

12. The apparatus of claim 1 in which the processor is configured to derive the location and trajectory of the writing instrument.

13. The apparatus of claim 1 in which the processor is configured to generate the handwriting and control information based on processing cycles each associated with one location of the writing instrument.

14. The apparatus of claim 1 in which the processor is configured to discriminate light received from the writing instrument from other light by locking onto a carrier frequency at which light from writing instrument is modulated.

15. The apparatus of claim 1 in which the processor is configured to determine a tilt of the writing instrument relative to a direction normal to a writing surface.

16. The apparatus of claim 1 in which the portable electronic device also comprises a display, and in which the processor is configured to cause the display to show the trajectory of the writing instrument in real-time.

17. The apparatus of claim 16 in which the display is not touch-sensitive.

18. The apparatus of claim 1 in which the processor comprises a digital signal processing chip and a general purpose microprocessor and software is run in part on the chip and in part on the microprocessor.

19. The apparatus of claim 1 in which the portable electronic device also comprises a wireless communication facility and in which the processor is configured to communicate the handwriting or control information to a remote location.

20. The apparatus of claim 1 in which the digital camera is configured to receive light that has been reflected from the writing instrument.

21. The apparatus of claim 1 in which the digital camera comprises a still camera.

22. The apparatus of claim 1 in which the digital camera comprises a video camera.

23. The apparatus of claim 1 also including an infra-red beacon configured to be directed at the writing instrument.

24. The apparatus of claim 1 in which the processor is configured to apply pattern recognition to signals from the digital camera.

25. A method comprising

in a portable electronic device comprising a digital camera and a processor,

receiving images at the digital camera comprising infared light received directly from a writing instrument being used on a writing surface, the writing instrument and writing surface being separate from the device, and

deriving handwriting and control information based on hand motion of the writing instrument, from the images.

26. Apparatus comprising

a portable electronic device comprising

a digital camera, and

a processor configured to

derive handwriting or control information based on hand motion of a writing instrument from light received by the digital camera directly from the writing instrument when the writing instrument is used on a writing surface, the writing instrument and the writing surface being separate from the apparatus, and

define a mapping between a sensor surface in the digital camera and a in response to calibration steps that include a user marking three locations on the writing surface.