SYSTEM AND METHOD FOR CAPTURING HAND ANNOTATIONS

• US 20110248941A1
• Filed: 03/15/2011
• Published: 10/13/2011
• Est. Priority Date: 03/17/2010
• Status: Active Grant

• Current Assignee: Logitech Europe S A (Logitech International SA)
• Original Assignee: Logitech Europe S A (Logitech International SA)
• Inventors: Pedrotta, Fabiano, Dizac, Greg, Tillitt, Candice Patricia, Milhe, Richard, Meisser, Matthieu, Piot, Julien, Abdo, Samer, Brisebras, Thomas Gaetan, Loeb, Alice Estelle, Bourelly, Guillaume
• Granted Patent: US 9,092,129 B2
• Time in Patent Office: Days
• US Class Current: 345/173
• CPC Class Codes: G06F 2203/04101 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup
• CPC Class Codes: G06F 2203/04104 Multi-touch detection in digitiser, i.e. details about the simultaneous detection of a plurality of touching locations, e.g. multiple fingers or pen and finger
• CPC Class Codes: G06F 3/044 by capacitive means
• CPC Class Codes: G06F 3/045 using resistive elements, e.g. a single continuous surface or two parallel surfaces put in contact
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

A capture device for remote, virtual on screen data input by hand annotation comprises at least three functional layers including a bottom rigid layer, a middle pressure sensor layer and a top flexible layer. The bottom rigid layer has a surface that provides a mechanical support for writing. The middle pressure sensor layer is adapted to measuring a pressure array or map on the capture active area and to send data representing the measured pressure to a personal computer. The top flexible touch-sensitive passive LCD display layer includes an LCD surface by which whatever is written down on the LCD is impressed graphically due to its liquid crystal physical properties wherein applied pressure changes the crystal particles orientation and light properties, such that when a stylus presses against a writing surface thereof, it leaves a visible trace allowing the user to produce a drawing though no real ink has flown.

Claims

1. A capture device for capturing and processing hand annotation data comprises at least three functional layers including:

(a) a bottom rigid layer having a surface that provides a mechanical support for writing;

(b) a middle pressure sensor layer adapted to measuring a pressure array or map on the capture active area and to send data representing the measured pressure to a personal computer; and

(c) a top flexible touch-sensitive passive LCD display layer having an LCD surface wherein applied pressure changes the crystal particles orientation and light properties, such that when a stylus presses against a writing surface thereof, a visible trace is left which allows the user to produce a drawing though no real ink has flown.

2. The capture device of claim 1, wherein the flexible touch-sensitive passive LCD display is a reflective bistable cholesteric liquid crystal laminated between two conductive-polymer coated polyethyleneterephthalate substrates.

3. The capture device of claim 1, wherein the passive LCD display layer is provided in a variety of colors.

4. The capture device of claim 1, wherein the pressure sensor layer comprises a plurality of voltage sources and multiple ammeters, defining a plurality of electrical conductive paths, whose conductivity is modulated by the applied pressure on that path, wherein at least one ammeter measures current emitted from more than one of said voltage sources.

5. The capture device of claim 4, wherein the pressure sensor layer is a multilayer assembly comprising:

a. a first flexible layer in which parallel conductive paths are embedded;

b. a second conductive separation layer; and

c. a third flexible layer in which parallel conductive paths are embedded substantially perpendicular to the conductive paths of the first layer, wherein compression of the laminate changes the conductivity across the layers.

6. The capture device of claim 1, wherein the data captured is processed in association with a time stamps, measured pressure, touchpoint identifiers, proximity and type.

7. The device of claim 6, wherein the data is processed into a pressure map.

8. The capture device of claim 1, further comprising a control board which classifies the measured pressure map into various pressure points together with associated position and force.

9. The capture device of claim 1 wherein the device is connected to the personal computer by a connection device selected from a group of connection devices consisting of USB, Bluetooth, other 2.4 GHz RF link, SP1 or I2C interface, thereby permitting bi-directional communication of the device with the PC.

10. A capture device of claim 1 for enabling virtual input on a remote display, the capture device further comprising:

at least one proximity sensor adapted to dynamically recognize the movement of at least one target in the proximity of the capture device;

a data connection device adapted to transmit signals from the proximity sensor to a processor communicatively coupled to the remote display, and

encoded instructions for, when a target is detected, overlaying a real-time, virtual representation of the target on the remote display in an orientation which represents the real world orientation of the target to the proximity sensor.

11. A system including a peripheral capture device for enabling virtual input on a remote display, the peripheral device including:

(a) a capture device of claim 1, further adapted to dynamically classify and recognize the movement of at least one target in the proximity of the capture device; and

(b) a data connection device adapted to transmit signals from the proximity sensor to a processor communicatively coupled to the remote display and to cooperate therewith so as to, when detected, overlay a real-time, virtual representation of the target over a representation of the input fields.

12. The system of claim 11, wherein the capture device is adapted to classify and recognize inputs made via a suitable protocol.

13. The system of claim 12, wherein the inputs classified and recognized are selected from a group of inputs consisting of keying, inking, gesturing, and pointing using a finger.

14. The system of claim 12, wherein the inputs classified and recognized are selected from a group of inputs consisting of keying, inking, gesturing, and pointing using a stylus.

15. The system of claim 12, wherein the inputs classified and recognized are used to turn on or turn off functionality to suit the task at hand.

16. The system of claim 12, wherein the suitable protocol is a classification process of sensed data.

17. The system of claim 16, wherein the sensed data includes any one of a group of sensed data consisting of proximity, distance, landing speed, touch, contact area, pressure segmentation, pressure distribution, heat, shape, footprint, pattern, capacitance, measured wavelength, biometric data, flux, induction, sound, and conductivity.

18. The system of claim 12 wherein the protocol used is combined with inputs to other input devices to define more complex meanings.

19. A system including a peripheral capture device for enabling virtual input on a remote display, the peripheral device including:

a capture device of claim 1, further adapted to dynamically classify and recognize a target parameter generated by at least one target in the proximity of the capture device; and

a data connection device adapted to transmit signals from the proximity sensor to a processor communicatively coupled to the remote display and to cooperate therewith so as to;

(i) construct a representation of input fields on the display, and

(ii) when detected, overlay a real-time, virtual representation of the target over the representation of the input fields.

20. The system of claim 19, wherein the target is one of a group of targets consisting of a user'"'"'s hand or hands, finger or fingers, arm or arms, a stylus or styluses, and a pointer or pointers.

21. The system of claim 19, wherein the target parameter is a parameter of sufficient proximity, at which proximity a touch signal indicating touch is sent to the processor, thereby allowing traditional keypad use with the benefits of touch pad use.

22. The system of claim 19, wherein the proximity sensor is selected from a group of proximity sensors consisting of capacitive, infrared, electromagnetic, read switch, hall effect, resistive variation, conductive variation, echo, radio waves, heat detection, eddy currents, optical pattern recognition technologies and micro air flux change.

23. The system of claim 19, further comprising a multi-touch input surface.

24. The system of claim 23, wherein the multi-touch input surface is integrated onto a housing which is separable from a principle input surface permitting keying.

25. The system of claim 19, where the representation of the input fields for display in a window of a display is a representation of a virtual keyboard.

26. The system of claim 19, wherein the representation of input fields for display in a window of the display is transparent, permitting viewing of screen content visually underneath the representation of the input fields.

27. The system of claim 19 wherein the processor includes instructions in an instruction set for automatic system activation when the proximity sensor detects a target in appropriate proximity to the capture device.

28. The system of claim 27, wherein, upon automatic system activation, a representation of the target is displayed on the display.

29. The system of claim 27, wherein, upon automatic system activation, a representation of the input fields is displayed on the display.

30. The system of claim 19, where the representation of the target is presented using a depth cue selected from a group of depth cues consisting of:

variation of target size;

variation of target color and/or transparency;

variation of target shadow relative position;

variation of target shadow color and/or transparency;

variation of target shadow blur;

displaying arrows encoding the distance between the target and the input device surface; and

by a sound cue or a variation in sound emitted by an associated sound system as the target approaches or retreats from the input device surface.

31. The system of claim 19, wherein the virtual representation of the target is a simplified representation in which only an input end of the target is displayed oriented accurately with respect to the representation of the input fields.

32. The system of claim 19, wherein the end of the target opposite to the input end is presented in a simplified manner.

33. The system of claim 19, wherein the capture device is adapted to classify and recognize inputs made via a suitable protocol.

34. The system of claim 33, wherein the inputs classified and recognized are selected from a group of inputs consisting of keying, inking, gesturing, pointing using a finger.

35. The system of claim 33, wherein the inputs classified and recognized are selected from a group of inputs consisting of keying, inking, gesturing, pointing using a stylus.

36. The system of claim 33, wherein the inputs classified and recognized are used to turn on or turn off functionality to suit the task at hand.

37. The system of claim 33, wherein the suitable protocol is a classification process of sensed data.

38. The system of claim 37, wherein the sensed data includes any one of a group of sensed data consisting of proximity, distance, landing speed, touch, contact area, pressure segmentation, pressure distribution, heat, shape, footprint, pattern, capacitance, measured wavelength, biometric data, flux, induction, sound, and conductivity.

39. The system of claim 33 wherein the protocol used is combined with inputs to other input devices to define more complex meanings.

40. A system is provided for reproducing and displaying on a display the input relationship of a target, thereby allowing coordination of interactions of a user to be made by reference to the displayed virtual representations, the system including:

a. a capture device according to claim 1; and

b. an instruction set executable by the processor wherein, when input and/or proximity data are received from the input device by the processor, the processor constructs a representation of input fields for display in a window of the display, wherein further, the processor constructs and overlays a real-time, virtual representation of a target detected by the input device over such constructed representation.

42. A method is provided for providing touch screen-like input functionality to a display remotely from the display in which inputs are made to a remote capture device, the method using the capture device of claim 1 and including the steps of:

a. reading proximity signals from each proximity sensing electrode;

b. checking if proximity signals are above a feature detection threshold and, if so classifying them as high proximity signals;

c. classifying high proximity signals into clusters based on corresponding sensing electrode locations which indicate a single feature detection;

d. identifying the local highest proximity signal, for each cluster; and

e. calculating the XYZ position of each feature by processing each local highest proximity signal with adjacent proximity electrode signals using triangulation methods; and

f. displaying each feature on the virtual keyboard at correct X-Y location and using depth cues corresponding to Z position.

41. A method is provided for providing touch screen-like input functionality to a display remotely from the display, the method including the steps of:

a. detecting proximity of one or more targets to a remote input device;

b. processing on a processor the 3D location of the one or more targets using the proximity data;

c. displaying a virtual representation of an input area on the display connected to the processor;

d. calculating relative position and transmitting such relative position information to the processor; and

e. displaying a virtual representation of the one or more targets dynamically, in real time, oriented with respect to the virtual touch screen input device as such one or more targets are detected in relation to the physical input device.

43. A method of operating a multilayer touch sensitive interface of at least three layers comprising

a. a first flexible layer in which parallel conductive paths are embedded;

b. a second conductive separation layer; and

c. a third flexible layer in which parallel conductive paths are embedded substantially perpendicular to the conductive paths of the first layer, wherein compression of the laminate changes the conductivity across the layers, the method including the steps of;

connecting voltage sources to the first layer conductive paths, driving said voltage sources simultaneously, where each source is modulated with a different modulation function, said modulation function taken from a set of orthogonal functions;

connecting ammeters to the third layer conductive paths, measuring current emitted from at least one voltage source;

cross-correlating measured current with each of the modulation functions contributing to current, to thereby determine the current flowing into at least one ammeter from each of said voltage source.