System and method for capturing hand annotations

• US 9,092,129 B2
• Filed: 03/15/2011
• Issued: 07/28/2015
• Est. Priority Date: 03/17/2010
• Status: Active Grant

• Current Assignee: Logitech Europe S A (Logitech International SA)
• Original Assignee: Logitech Europe S A (Logitech International SA)
• Inventors: Abdo, Samer, Piot, Julien, Dizac, Greg, Milhe, Richard, Bourelly, Guillaume, Meisser, Matthieu, Pedrotta, Fabiano, Tillitt, Candice Patricia, Loeb, Alice Estelle, Brisebras, Thomas Gatan
• Primary Examiner(s): Lao, Lun-Yi
• Assistant Examiner(s): KHAN, IBRAHIM A
• Application Number:
  US13/047,962
• Publication Number:
  US 20110248941A1
• Time in Patent Office: 1,596 Days
• Field of Search: 345173-184, 345/104, 178 1801- 1806, 349/12
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/04101 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup
• CPC Class Codes: G06F 2203/04104 Multi-touch detection in digitiser, i.e. details about the simultaneous detection of a plurality of touching locations, e.g. multiple fingers or pen and finger
• CPC Class Codes: G06F 3/044 by capacitive means
• CPC Class Codes: G06F 3/045 using resistive elements, e.g. a single continuous surface or two parallel surfaces put in contact
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

A capture device for remote, virtual on screen data input by hand annotation comprises at least three functional layers including a bottom rigid layer, a middle pressure sensor layer and a top flexible layer. The bottom rigid layer has a surface that provides a mechanical support for writing. The middle pressure sensor layer is adapted to measuring a pressure array or map on the capture active area and to send data representing the measured pressure to a personal computer. The top flexible touch-sensitive passive LCD display layer includes an LCD surface by which whatever is written down on the LCD is impressed graphically due to its liquid crystal physical properties wherein applied pressure changes the crystal particles orientation and light properties, such that when a stylus presses against a writing surface thereof, it leaves a visible trace allowing the user to produce a drawing though no real ink has flown.

Claims

1. A capture device for capturing and processing hand annotation data that comprises at least three functional layers including:

(a) a bottom rigid layer having a surface that provides a mechanical support for writing;

(b) a top flexible touch sensitive passive LCD display layer including particles which change light properties in response to pressure, to produce an apparent ink display;

(c) a middle pressure sensor layer for measuring touch pressure electrically independently from the top flexible touch sensitive passive LCD

(d) a processing circuit directly coupled to the middle pressure sensor and configured order to create a pressure array or map of a capture active area indicating location and amount of pressure and to send and process the pressure data as inputs to a personal computer separate from the capture device; and

(e) an LCD control configured to erase the apparent ink display of the passive LCD display layer in response to an activate signal.

2. The capture device of claim 1, wherein the flexible touch-sensitive passive LCD display is a reflective bistable cholesteric liquid crystal laminated between two conductive-polymer coated polyethyleneterephthalate substrates.

3. The capture device of claim 1, wherein the passive LCD display layer is provided in a variety of colors.

4. The capture device of claim 1, wherein the pressure sensor layer comprises a plurality of voltage sources and multiple ammeters, defining a plurality of electrical conductive paths, whose conductivity is modulated by the applied pressure on that path, wherein at least one ammeter measures current emitted from more than one of said voltage sources.

5. The capture device of claim 4, wherein the pressure sensor layer is a multilayer assembly, the multilayer assembly comprising:

a. a first flexible layer in which parallel conductive paths are embedded;

b. a second conductive separation layer; and

c. a third flexible layer in which parallel conductive paths are embedded substantially perpendicular to the conductive paths of the first layer, wherein compression of the multilayer assembly changes the conductivity across the layers.

6. The capture device of claim 1, wherein the data captured is processed in association with a time stamps, measured pressure, touchpoint identifiers, proximity and type.

7. The device of claim 6, wherein the data is processed into a pressure map.

8. The capture device of claim 1, further comprising a control board which classifies the measured pressure map into various pressure points together with associated position and force.

9. The capture device of claim 1 wherein the device is coupled to the personal computer by a connection device selected from a group of connection devices consisting of USB, Bluetooth, other 2.4 GHz RF link, SPI or I2C interface, thereby permitting bi-directional communication of the device with the PC.

10. A capture device of claim 1 for enabling virtual input on a remote display, the capture device further comprising:

at least one proximity sensor adapted to dynamically recognize the movement of at least one target in the proximity of the capture device;

a data connection device adapted to transmit signals from the proximity sensor to a processor communicatively coupled to the remote display, and

encoded instructions for, when a target is detected, overlaying a real-time, virtual representation of the target on the remote display in an orientation which represents the real world orientation of the target to the proximity sensor.

11. A system including a peripheral capture device for enabling virtual input on a remote display, the peripheral device including:

(a) a capture device of claim 1, further adapted to dynamically classify and recognize the movement of at least one target in the proximity of the capture device; and

(b) a data connection device adapted to transmit signals from the proximity sensor to a processor communicatively coupled to the remote display and to cooperate therewith so as to, when detected, overlay a real-time, virtual representation of the target over a representation of the input fields.

12. The system of claim 11, wherein the capture device is adapted to classify and recognize inputs made via a suitable protocol.

13. The system of claim 12, wherein the inputs classified and recognized are selected from a group of inputs consisting of keying. inking. gesturing, and pointing using a finger.

14. The system of claim 12, wherein the inputs classified and recognized are selected from a group of inputs consisting of keying, inking, gesturing, and pointing using a stylus.

15. The system of claim 12, wherein the inputs classified and recognized are used to turn on or turn off functionality to suit the task at hand.

16. The system of claim 12, wherein the suitable protocol is a classification process of sensed data.

17. The system of claim 16, wherein the sensed data includes any one of a group of sensed data consisting of proximity, distance, landing speed, touch, contact area, pressure segmentation, pressure distribution, heat, shape, footprint, pattern, capacitance, measured wavelength, biometric data, flux, induction, sound, and conductivity.

18. The system of claim 12 wherein the protocol used is combined with inputs to other input devices to define more complex meanings.

19. A system including a peripheral capture device for enabling virtual input on a remote display, the peripheral device including:

a capture device of claim 1, further adapted to dynamically classify and recognize a target parameter generated by at least one target in the proximity of the capture device; and

a data connection device adapted to transmit signals from the proximity sensor to a processor communicatively coupled to the remote display and to cooperate therewith so as to;

(i) construct a representation of input fields on the display, and

(ii) when detected, overlay a real-time, virtual representation of the target over the representation of the input fields.

20. The system of claim 19, wherein the target is one of a group of targets consisting of a user'"'"'s hand or hands, finger or fingers, arm or arms, a stylus or styluses, and a pointer or pointers.

21. The system of claim 19, wherein the target parameter is a parameter of sufficient proximity, at which proximity a touch signal indicating touch is sent to the processor, thereby allowing traditional keypad use with the benefits of touch pad use.

22. The system of claim 19, wherein the proximity sensor is selected from a group of proximity sensors consisting of capacitive, infrared, electromagnetic, read switch, hall effect, resistive variation, conductive variation, echo, radio waves, heat detection, eddy currents, optical pattern recognition technologies and micro air flux change.

23. The system of claim 19, further comprising a multi-touch input surface.

24. The system of claim 23, wherein the multi-touch input surface is integrated onto a housing which is separable from a principle input surface permitting keying.

25. The system of claim 19, where the representation of the input fields for display in a window of a display is a representation of a virtual keyboard.

26. The system of claim 19, wherein the representation of input fields for display in a window of the display is transparent, permitting viewing of screen content visually underneath the representation of the input fields.

27. The system of claim 19 wherein the processor includes instructions in an instruction set for automatic system activation when the proximity sensor detects a target in appropriate proximity to the capture device.

28. The system of claim 27, wherein, upon automatic system activation, a representation of the target is displayed on the display.

29. The system of claim 27, wherein, upon automatic system activation, a representation of the input fields is displayed on the display.

30. The system of claim 19, where the representation of the target is presented using a depth cue selected from a group of depth cues consisting of:

variation of target size;

variation of target color and/or transparency;

variation of target shadow relative position;

variation of target shadow color and/or transparency;

variation of target shadow blur;

displaying arrows encoding the distance between the target and the input device surface; and

by a sound cue or a variation in sound emitted by an associated sound system as the target approaches or retreats from the input device surface.

31. The system of claim 19, wherein the virtual representation of the target is a simplified representation in which only an input end of the target is displayed oriented accurately with respect to the representation of the input fields.

32. The system of claim 19, wherein the end of the target opposite to the input end is presented in a simplified manner.

33. The system of claim 19, wherein the capture device is adapted to classify and recognize inputs made via a suitable protocol.

34. The system of claim 33, wherein the inputs classified and recognized are selected from a group of inputs consisting of keying, inking, gesturing, pointing using a finger.

35. The system of claim 33, wherein the inputs classified and recognized are selected from a group of inputs consisting of keying, inking, gesturing, pointing using a stylus.

36. The system of claim 33, wherein the inputs classified and recognized are used to turn on or turn off functionality to suit the task at hand.

37. The system of claim 33, wherein the suitable protocol is a classification process of sensed data.

38. The system of claim 37, wherein the sensed data includes any one of a group of sensed data consisting of proximity, distance, landing speed, touch, contact area, pressure segmentation, pressure distribution, heat, shape, footprint, pattern, capacitance, measured wavelength, biometric data, flux, induction, sound, and conductivity.

39. The system of claim 33 wherein the protocol used is combined with inputs to other input devices to define more complex meanings.

40. A system is provided for reproducing and displaying on a display the input relationship of a target, thereby allowing coordination of interactions of a user to be made by reference to the displayed virtual representations, the system including:

a. a capture device according to claim 1; and

b. an instruction set executable by the processor wherein, when input and/or proximity data are received from the input device by the processor, the processor constructs a representation of input fields for display in a window of the display, wherein further, the processor constructs and overlays a real-time, virtual representation of a target detected by the input device over such constructed representation.

41. The capture device of claim 1 wherein the top flexible touch-sensitive passive LCD display layer has an LCD surface wherein applied pressure changes the crystal particles orientation and light properties, such that when a stylus presses against a writing surface thereof, a visible trace is left which allows the user to produce a drawing though no real ink has flown.