Method and apparatus for integrating manual input
First Claim
1. A sensing device that is sensitive to changes in self-capacitance brought about by changes in proximity of a touch device to the sensing device, the sensing device comprising:
- two electrical switching means connected together in series having a common node, an input node, and an output node;
a dielectric-covered sensing electrode connected to the common node between the two switching means;
a power supply providing an approximately constant voltage connected to the input node of the series-connected switching means;
an integrating capacitor to accumulate charge transferred during multiple consecutive switching of the series connected switching means;
another switching means connected in parallel across the integrating capacitor to deplete its residual charge; and
a voltage-to-voltage translation device connected to the output node of the series-connected switching means which produces a voltage representative of the proximity of the touch device to the sensing device.
9 Assignments
0 Petitions
Accused Products
Abstract
Apparatus and methods are disclosed for simultaneously tracking multiple finger and palm contacts as hands approach, touch, and slide across a proximity-sensing, compliant, and flexible multi-touch surface. The surface consists of compressible cushion, dielectric, electrode, and circuitry layers. A simple proximity transduction circuit is placed under each electrode to maximize signal-to-noise ratio and to reduce wiring complexity. Such distributed transduction circuitry is economical for large surfaces when implemented with thin-film transistor techniques. Scanning and signal offset removal on an electrode array produces low-noise proximity images. Segmentation processing of each proximity image constructs a group of electrodes corresponding to each distinguishable contact and extracts shape, position and surface proximity features for each group. Groups in successive images which correspond to the same hand contact are linked by a persistent path tracker which also detects individual contact touchdown and liftoff. Combinatorial optimization modules associate each contact'"'"'s path with a particular fingertip, thumb, or palm of either hand on the basis of biomechanical constraints and contact features. Classification of intuitive hand configurations and motions enables unprecedented integration of typing, resting, pointing, scrolling, 3D manipulation, and handwriting into a versatile, ergonomic computer input device.
-
Citations
118 Claims
-
1. A sensing device that is sensitive to changes in self-capacitance brought about by changes in proximity of a touch device to the sensing device, the sensing device comprising:
-
two electrical switching means connected together in series having a common node, an input node, and an output node;
a dielectric-covered sensing electrode connected to the common node between the two switching means;
a power supply providing an approximately constant voltage connected to the input node of the series-connected switching means;
an integrating capacitor to accumulate charge transferred during multiple consecutive switching of the series connected switching means;
another switching means connected in parallel across the integrating capacitor to deplete its residual charge; and
a voltage-to-voltage translation device connected to the output node of the series-connected switching means which produces a voltage representative of the proximity of the touch device to the sensing device. - View Dependent Claims (2, 3, 4, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29)
one of a rigid or flexible surface;
a two-dimensional array of the sensing devices of claim 1 arranged on the surface with their output nodes connected together and sharing the same integrating capacitor, charge depletion switch, and voltage-to-voltage translation device;
control circuitry for sequentially enabling each of the sensor devices;
voltage measurement circuitry to convert sensor data to a digital code; and
circuitry for communicating the digital code to another electronic device.
-
-
8. A plurality of the multi-touch surface apparatuses of claim 7 arranged in the shape of one of a cube, a sphere, or any other three dimensional shape.
-
9. The multi-touch surface apparatus of claim 7, wherein the surface comprises a micro-dimensional surface.
-
10. The multi-touch surface apparatus of claim 7 being one of fabricated on or integrated with a display device.
-
11. The multi-touch surface apparatus of claim 10, wherein the display device comprises one of a liquid crystal display (LCD) or a light-emitting polymer display (LPD).
-
12. A multi-layer cover apparatus for the multi-touch surface apparatus of claim 7, comprising:
-
a compliant dielectric layer;
a deformable conductive layer formed on the dielectric layer, the conductive layer being electrically coupled to the voltage or current measurement device; and
a touch layer formed on the conductive layer.
-
-
13. The multi-layer concr apparatus of claim 12, wherein the touch layer has a symbol set printed thereon that can be removed and replaced with an alternative symbol set.
-
14. The multi-touch surface apparatus of claim 7, wherein the apparatus is ergonomically arched.
-
15. The multi-touch surface apparatus of claim 7, wherein the apparatus includes hand configuration visual indicators.
-
16. The multi-touch surface apparatus of claim 7, wherein the apparatus includes hand configuration audio indicators.
-
17. A multi-touch surface apparatus for detecting a spatial arrangement of multiple touch devices on or near the surface of the multi-touch apparatus comprising:
-
one of a rigid or flexible surface;
a two-dimensional array of the sensing devices of claim 2 arranged on the surface with their output nodes connected together and sharing the same current-to-voltage translation device;
control circuitry for sequentially enabling each of the sensor devices;
voltage measurement circuitry to convert sensor data to a digital code; and
circuitry for communicating the digital code to another electronic device.
-
-
18. A multi-touch surface apparatus for detecting a spatial arrangement of multiple touch devices on or near the surface of the multi-touch apparatus comprising:
-
one of a rigid or flexible surface;
a plurality of two-dimensional arrays of the sensing devices of claim 1 arranged on the surface in groups wherein the sensing devices within one group have their output nodes connected to corresponding sensing devices within other groups and share the same integrating capacitor, charge depletion switch, and voltage-to-voltage translation circuitry;
control circuitry for enabling a single sensor device from each two-dimensional array;
means for selecting the sensor voltage data from each two-dimensional array;
voltage measurement circuitry to convert sensor voltage data to a digital code; and
circuitry for communicating the digital code to another electronic device.
-
-
19. The multi-touch surface apparatus of claim 18, wherein the sensor voltage data selecting means comprises one of a multiplexing circuitry and a plurality of voltage measurement circuits.
-
20. A multi-touch surface apparatus for detecting a spatial arrangement of multiple touch devices on or near the surface of the multi-touch apparatus comprising:
-
one of a rigid or flexible surface;
a plurality of the sensing devices of claim 1 arranged on the surface;
control circuitry for sequentially enabling each of the sensor devices;
voltage measurement circuitry to convert sensor data to a digital code; and
circuitry for communicating the digital code to another electronic device.
-
-
21. A multi-touch surface apparatus for sensing diverse configurations and activities of touch devices and generating integrated manual input to one of an electronic or electro-mechanical device, the apparatus comprising:
-
an array of the proximity sensing devices of claim 1;
a dielectric cover having symbols printed thereon that represent action-to-be-taken when engaged by the touch devices;
scanning means for forming digital proximity images from the array of sensing devices;
calibrating means for removing background offsets from the proximity images;
recognition means for interpreting the configurations and activities of the touch devices that make up the proximity images;
processing means for generating input signals in response to particular touch device configurations and motions; and
communication means for sending the input signals to the electronic or electro-mechanical device.
-
-
22. The multi-touch surface apparatus of claim 21, wherein the symbols printed on the dielectric cover can be removed and replaced with an alternative symbol set.
-
23. The multi-touch surface apparatus of claim 21, wherein the dielectric cover has conductive fibers therein, the conductive fibers being oriented normal to the array of sensing devices for conducting the capacitive ettect of the touch devices on the array of sensing devices.
-
24. A multi-layer cover apparatus for the multi-touch surface apparatus of claim 21, the multi-layer cover apparatus comprising:
-
a compliant dielectric layer;
a deformable conductive layer formed on the dielectric layer, the conductive layer being electrically coupled to the voltage or current measurement device; and
a touch layer formed on the conductive layer.
-
-
25. The multi-touch surface apparatus of claim 21, wherein the apparatus is ergonomically arched.
-
26. The multi-touch surface apparatus of claim 21, wherein the apparatus includes hand configuration visual indicators.
-
27. The multi-touch surface apparatus of claim 21, wherein the apparatus includes hand configuration audio indicators.
-
28. The multi-touch surface apparatus of claim 21, being one of fabricated on or integrated with a display device.
-
29. The multi-touch surface apparatus of claim 28, wherein the display device comprises one of a liquid crystal display (LCD) or a light-emitting polymer display (LPD).
-
5. A sensing device that is sensitive to changes in self-capacitance brought about by changes in proximity of a touch device to the sensing device, the sensing device comprising:
-
two electrical switching means connected together in series having a common node, an input node, and an output node;
a dielectric-covered sensing electrode connected to the common node between the two switching means;
a power supply providing an approximately constant voltage connected to the input node of the series-connected switching means; and
an integrating current-to-voltage translation device connected to the output node of the series connected switching means, the current-to-voltage translation device producing a voltage representative of the proximity of the touch device to the sensing device.
-
-
6. A sensing device that is sensitive to changes in self-capacitance brought about by changes in proximity of a touch device to the sensing device, the sensing device comprising:
-
two electrical switching means connected together in series having a common node an input node, and an output node, a dielectric-covered sensing electrode connected to the common node between the two switching means; and
a power supply providing an approximately constant voltage connected to the input node of the series-connected switching means.
-
-
30. A multi-touch surface apparatus for sensing diverse configurations and activities of fingers and palms of one or more hands near the surface and generating integrated manual input to one of an electronic or electro-mechanical device, the apparatus comprising:
-
an array of proximity sensing means embedded in the surface;
scanning means for forming digital proximity images from the proximities measured by the sensing means;
image segmentation means for collecting into groups those proximity image pixels intensified by contact of the same distinguishable part of a hand;
contact tracking means for parameterizing hand contact features and trajectories as the contacts move across successive proximity images;
contact identification means for determining which hand and which part of the hand is causing each surface contact;
synchronization detection means for identifying subsets of identified contacts which touchdown or liftoff the surface at approximately the same time, and for generating command signals in response to synchronous taps of multiple fingers on the surface;
typing recognition means for generating intended key symbols from asynchronous finger taps;
motion component extraction means for compressing multiple degrees of freedom of multiple fingers into degrees of freedom common in two and three dimensional graphical manipulation;
chord motion recognition means for generating one of command and cursor manipulation signals in response to motion in one or more extracted degrees of freedom by a selected combination of fingers;
pen grip detection means for recognizing contact arrangements which resemble the configuration of the hand when gripping a pen, generating inking signals from motions of the inner fingers, and generating cursor manipulation signals from motions of the palms while the inner fingers are lifted; and
communication means for sending the sensed configurations and activities of finger and palms to one of the electronic or electro-mechanical device.
-
-
31. A multi-touch surface apparatus for sensing diverse configurations and activities of fingers and palms of one or more hands near the surface and generating integrated manual input to one of an electronic or electro-mechanical device, the apparatus comprising:
-
an array of proximity sensing means embedded in the surface;
scanning means for forming digital proximity images from proximities measured by the sensing means;
contact segmentation means for collecting proximity image pixels caused by the same hand part into groups;
contact tracking means for parameterizing hand contact features and trajectories as the contacts move across successive proximity images;
contact identification means for determining which hand and which part of the hand is causing each surface contact;
synchronization detection means for identifying subsets of identified contacts which touchdown or liftoff the surface at approximately the same time;
typing recognition means for generating intended key symbols from asynchronous finger taps;
motion component extraction means for compressing the dozens of degrees of freedom in motions of multiple fingers into the degrees of freedom common in two and three dimensional graphical manipulation;
chord motion recognition means for generating, command or cursor manipulation signals in response to motion in one or more extracted degrees of freedom by a selected combination of fingers; and
communication means for sending said generated input signals to the electronic or electro-mechanical device.
-
-
32. A multi-touch surface apparatus for sensing diverse configurations and activities of fingers and palms of one or more hands near the surface and generating integrated manual input to one of an electronic or electro-mechanical device, the apparatus comprising:
-
an array of proximity sensing means embedded in the surface;
scanning means for forming digital proximity images from proximities measured by the sensing means;
contact segmentation means for collecting proximity image pixels caused by the same hand part into groups;
contact tracking means for parameterizing hand contact features and trajectories as the contacts move across successive proximity images;
contact identification means for determining which hand and which part of the hand is causing each surface contact;
pen grip detection means for recognizing contact arrangements which resemble the configuration of the hand when gripping a pen, generating inking signals from motions of the inner fingers, and generating cursor manipulation signals from motions of the palms while the inner fingers are lifted; and
communication means for sending said generated input signals to the electronic or electro-mechanical device.
-
-
33. A method for tracking and identifying hand contacts in a sequence of proximity images in order to support interpretation of hand configurations and activities related to typing, multiple degree-of-freedom manipulation via chords, and handwriting, the method comprising the steps of:
-
segmenting each proximity image into groups of electrodes which indicate significant proximity, each group representing proximity of a distinguishable hand part or other touch device;
extracting total proximity, position, shape, size, and orientation parameters from each group of electrodes;
tracking, group paths through successive proximity images including detection of path endpoints at contact touchdown and liftoff;
computing velocity and filtered position vectors along each path;
assigning a hand and finger identity to each contact path by incorporating relative path positions and velocities, individual contact features, and previous estimates of hand and finger positions; and
maintaining estimates of hand and finger positions from trajectories of paths currently assigned to the fingers, wherein the estimates provide high level feedback to bias segmentations and identifications in future images.
-
-
34. A method for filtering and segmenting hand contacts in a sequence of proximity images in order to support interpretation of various contact sizes, shapes, orientations, and spacings, the method comprising the steps of:
-
creating a smoothed copy of the most recent proximity image;
searching for pixels with locally maximum proximity in the smoothed proximity image;
searching outward from each local maximum pixel for contact boundary pixels using boundary tests of pixel and neighboring pixel proximities which depend on properties of hand contacts expected in a segmentation region of the pixel;
forming groups from those pixels surrounding each local maximum pixel up to and including the boundary pixels;
combining groups of pixels which partially overlap;
extracting group positions and features by fitting an ellipse to each group of pixels; and
updating positions of the segmentation regions of the pixels in response to further analysis of the position and features extracted from each group of pixels. - View Dependent Claims (35, 36, 37)
-
-
38. A method for associating into paths those surface contacts from successive proximity images caused by the same hand part and detecting liftoff from and touchdown onto the surface by each hand part, the method comprising the steps of:
-
predicting the current positions of hand parts from their velocity along existing paths;
finding for each of a group of pixels in current proximity image the existing path with a closest predicted path position;
finding for each existing path the pixel group whose centroid is closest to the predicted path position and whose centroid is within a path-dependent tracking radius;
pairing each pixel group with its closest path if the pixel group is also the closest pixel group to the path;
starting new paths for remaining unpaired pixel groups;
deactivating paths which have no pairable pixel groups within the path-dependent tracking radius; and
updating path parameters from the measured parameters of the pixel group paired with each path.
-
-
39. A method of computing hand and finger position offsets from the measured positions of individual hand contacts on a multi-touch surface for the purpose of biasing future hand contact identifications or morphing the key layout in an integrated manual input device, the method comprising the steps of:
-
establishing fingertip, thumb, or palm identities for each contact;
establishing an offset weighting for each contact;
computing a hand position offset, wherein the offset is a weighted average of the difference between a measured position of each contact and a predetermined default position of the hand part which corresponds to an established identity of the contact; and
computing a finger position offset by subtracting a predetermined default position of an associated hand part of the contact and the hand position offset from a measured position of the contact. - View Dependent Claims (40, 41)
computing the confidence in current contact identifications from the amount of information available for the identifications;
computing a weighted average of the individual hand contact velocities;
predicting a current hand and finger offset from the previous offset estimates and the weighted average velocity;
computing current hand and finger offsets from current contact identities and measured contact locations;
setting the currently measured hand and finger offsets to zero if the hand has no contacts in the current image; and
updating the hand and finger offset estimates to the weighted average of the predicted offsets and currently measured offsets, wherein the relative weighting given to the measured offsets increases in proportion to the confidence level in the current identifications.
-
-
41. The method of claim 40, wherein a hand sliding to the opposite side of the surface eliminates the estimated position of a lifted hand by imposing a minimum separation between the estimated hand positions, and permitting the hand with a highest identification confidence override the estimated position of the other hand.
-
42. A method for establishing identities of hand contacts on a multi-touch surface using relative contact positions and features, the method comprising the steps of:
-
defining a template of hand part attractor points on the surface, the attractor points for each hand roughly forming a ring;
computing a matrix of distances from each surface contact to each attractor point;
weighting the distances between each surface contact and each attractor point according to how closely measured contact features such as proximity to a surface, shape, size, eccentricity, orientation, distance to nearest neighbor contact, and velocity match features typical of the hand part the attractor point represents;
finding a one-to-one mapping of the surface contacts to the attractor points that minimizes a sum of distances between each surface contact and its corresponding attractor point; and
recognizing particular hand configurations from the number and features of surface contacts assigned to particular subsets of the attractor points. - View Dependent Claims (43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59)
finding an innermost finger contact by searching for a contact assigned to a filled attractor point corresponding to an innermost finger;
computing a thumb factor as a function of the innermost finger contact relative to other finger contacts;
shifting the innermost finger contact to a thumb attractor point if the innermost finger contact is not already assigned to the thumb attractor point and the thumb factor is above a predetermined thumb threshold; and
shifting the innermost finger contact to a fingertip attractor point if the innermost finger contact is currently assigned to the thumb attractor point and the thumb factor is below the predetermined thumb threshold.
-
-
57. The method of claim 56, wherein the thumb actor is high if the orientation and size of the innermost finger contact are greater than those of other finger contacts.
-
58. The method of claim 56, wherein the thumb factor is high if the separation, angle, and velocity of the innermost finger contact relative to other finger contacts are in ranges unique to opposable thumb presence or motion.
-
59. The method of claim 56, wherein the verification step is performed when one of a fingertip or thumb attractor point is left unfilled by the attractor points minimization step.
-
60. A method for ordering surface contacts and establishing finger, thumb, and palm identities, the method comprising the steps of:
-
finding a shortest path connecting all of the contacts assumed to be from a given hand;
passing through each contact once to form an ordered loop;
finding an innermost contact in the ordered loop;
determining whether the innermost contact is a thumb, fingertip, or palm contact from contact and inter-contact features of the innermost contact; and
assigning thumb, fingertip, or palm identities to non-innermost contacts based upon the features of the contacts, assignment of the innermost contacts, vertical position relative to assigned contacts, and the loop ordering.
-
-
61. An apparatus for distinguishing palm heel contacts from other types of hand contacts in a system for recognizing hand activity on a multi-touch surface and generating input signals to a competing device therefrom, the apparatus comprising:
-
means for finding the nearest neighbor contact of a given contact in a plane of the surface; and
means for suppressing identification of the given contact as a palm heel contact it a neighbor contact exists and is closer to the given contact than the anatomical separation between inner and outer portions of a palm heel. - View Dependent Claims (62)
-
-
63. An apparatus for distinguishing palm heel contacts from other types of hand contacts in a system for recognizing hand activity on a multi-touch surface and generating input signals to a competing device therefrom, the apparatus comprising:
-
means for measuring the total proximity, orientation, and eccentricity of all contacts;
means for encouraging identification of a given contact as a palm heel contact if its ratio of total proximity to eccentricity is larger than for a typical fingertip contact; and
means for encouraging identification of a given contact as a palm heel contact as its orientation approaches the expected slant of a palm heel.
-
-
64. An apparatus for distinguishing thumb contacts from other types of hand contacts in a system for recognizing hand activity on a multi-touch surface and generating input signals to a competing device therefrom, the apparatus comprising:
-
means for measuring the size and orientation of all contacts;
means for encouraging identification of a given contact as a thumb contact if its size is larger than a typical fingertip contact;
means for discouraging identification of a given contact as a thumb contact if its size is larger than a typical thumb contact; and
means for encouraging identification of a given contact as a thumb contact as its orientation approaches the expected slant of the thumb.
-
-
65. A method for determining which hand causes each surface contact detected on a multi-touch surface so that input signals generated by hand activity on the surface can depend on the identity of the hand performing the activity and so that multiple hands can perform independent activities on the surface simultaneously, the method comprising the steps of:
-
defining a template of hand part attractor points on the surface, the attractor points for each hand approximately forming a ring;
generating partitions which divide the set of all surface contacts into left hand clusters and right hand clusters;
assigning finger and palm identities to the contacts within each cluster;
computing for each partition an assignment fitness measure which represents the biomechanical consistency of the fit of contact clusters to their assigned attractor rings;
choosing the partition which has the best assignment fitness measure as the partition containing the true contact identities; and
recognizing each hand'"'"'s configuration from the combination of and features of surface contacts assigned within each attractor ring of the best partition. - View Dependent Claims (66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78)
constructing approximately vertical contours between each horizontally adjacent contact; and
constructing a partition from each contour by tentatively assigning contacts which are positioned to the left of a contour to the left hand cluster and contacts to the right of a contour to the right hand cluster.
-
-
70. The method of claim 69, wherein the hand assignments of previously identified contacts can be locked so to not depend on which side of the dividing contour the contacts lie, while assignments of new contacts still depend on which side of the contour they lie.
-
71. The method of claim 65, wherein each attractor ring is translated, scaled and/or rotated to match previous position estimates for the hand corresponding to the attractor ring.
-
72. The method of claim 65, wherein the attractor points within each ring are individually offset by previously estimated finger offsets.
-
73. The method of claim 65, wherein the assignment fitness measure is a total cost computed as a weighted sum of distances from each contact to its assigned attractor point in the attractor ring of its assigned hand cluster, and wherein the best partition is the one with the lowest total cost.
-
74. The method of claim 73, wherein the distances between each surface contact and each attractor point are weighted according to how closely measured contact features, such as proximity to the surface, shape, size, eccentricity, orientation, distance to nearest neighbor contact, and velocity, match features typical of the hand part the attractor point represents.
-
75. The method of claim 73, wherein a separation between the innermost finger and the next innermost finger is compared with a separation between the outermost finger and next outermost finger to obtain a handedness weighting which increases the total cost for a hand as the outermost separation becomes larger than is biomechanically consistent.
-
76. The method of claim 73, wherein a hand portion of the total cost is decreased by a cluster velocity weighting function when the average velocity of the contact cluster of the hand indicates the hand is returning to its associated side of the surface.
-
77. The method of claim 73, wherein a hand portion of the total cost is increased by a palm cohesion weighting function when the contacts assigned to the palms of a hand are scattered over an area larger than the anatomical size of an outstretched palm.
-
78. The method of claim 73, wherein the total cost of a partition is increased by a interhand separation weighting when the measured separation between contacts tentatively assigned to opposite hand clusters indicates that the hands may be overlapping or close to touching.
-
79. A method for integrally extracting multiple degrees of freedom of hand motion from sliding motions of two or more fingers of a hand across a multi-touch surface, one of the fingers preferably being the opposable thumb, the method comprising the steps of:
-
tracking, across successive scans of the proximity sensor array the trajectories of individual hand parts on the surface;
finding an innermost and an outermost finger contact from contacts identified as fingers on the given hand;
computing a scaling velocity component from a change in a distance between the innermost and outermost finger contacts;
computing a rotational velocity component from a change in a vector angle between the innermost and outermost finger contacts;
computing a translation weighting for each contacting finger;
computing translational velocity components in two dimensions from a translation weighted average of the finger velocities tangential to surface;
suppressively filtering components whose speeds are consistently lower than the fastest components;
transmitting the filtered velocity components as control signals to an electronic or electro-mechanical device. - View Dependent Claims (80, 81, 82, 83, 84, 85)
downscaling each velocity component in proportion to a function of its average speed compared to the other average component speeds;
dead-zone filtering each downscaled velocity component wherein the width of the dead-zone depends on the distribution of the current component speeds.
-
-
85. The method of claim 79, wherein the orientation of an ellipse fitted to the thumb contact after each successive sensor array scan is transmitted as an additional degree of freedom control signal.
-
86. A method for integrally extracting roll and tilt degrees of freedom of hand motion from pressure changes of three or more non-collinear hand contacts comprising any of thumbs, fingertips or palms, the method comprising the steps of:
-
tracking across successive proximity images the trajectories of individual hand parts on the surface;
measuring proximities from each hand contact in a calibration proximity image once all available hand contacts have been stabilized;
computing an average hand contact position from a post-calibration proximity image, wherein all hand contacts are weighted equally;
computing a weighted average hand contact position from a post-calibration proximity image, wherein each hand contact is weighted according to the ratio of its current proximity to its calibrated proximity;
computing for each post-calibration proximity image the difference vector between the weighted average hand contact position and the average hand contact position;
dead-zone filtering the difference vector to remove variations in proximity due to unintentional posture shifts; and
transmitting the filtered difference vector from each post-calibration proximity image as roll and tilt control signals to an electronic or electro-mechanical device. - View Dependent Claims (87, 88, 89, 90)
-
-
91. A manual input integration method for supporting diverse hand input activities such as resting the hands, typing, multiple degree-of-freedom manipulation, command gesturing and handwriting on a multi-touch surface, the method enabling users to instantaneously switch between the input activities by placing their hands in different configurations comprising distinguishable combinations of relative hand contact timing, proximity, shape, size, position, motion and/or identity across a succession of surface proximity images, the method comprising the steps of:
-
tracking each touching hand part across successive proximity images;
measuring the times when each hand part touches down and lifts off the surface;
detecting when hand parts touch down or lift off simultaneously;
producing discrete key symbols when the user asynchronously taps, holds, or slides a finger on key regions defined on the surface;
producing discrete mouse button click commands, key commands, or no signals when the user synchronously taps two or more fingers from the same hand on the surface;
producing gesture commands or multiple degree-of-freedom manipulation signals when the user slides two or more fingers across the surface; and
sending the produced symbols, commands and manipulation signals as input to an electronic or an electro-mechanical device. - View Dependent Claims (92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104)
establishing the finger or palm identity of each surface contact;
measuring the relative positions and proximities of the identified contacts to determine whether the inner fingers are pinched while the outer fingers curl under the palm exposing their knuckles to rest on the surface;
entering a handwriting mode for the hand if the above unique finger arrangement is detected;
producing inking signals from the motions of the inner fingers on the surface while in handwriting mode;
producing stylus lift signals each time the inner fingers lift off the surface while in handwriting mode;
sending the inking signals to an electronic device for capture, display, or recognition; and
leaving the handwriting mode after the hand has lifted and remained off the surface for a substantial time or if a non-pinched finger configuration is measured.
-
-
96. The method of claim 95, wherein while in handwriting mode but the inner fingers are lifted, sliding and tapping motions of the palm heels produce cursor manipulation and clicking signals which are sent to the electronic device.
-
97. The method of claim 95, wherein a stylus held between the pinched fingers touches the surface instead of the pinched fingers themselves to indicate pinch configuration, and wherein inking signals are measured from motion of the stylus.
-
98. The method of claim 91, wherein the layout of key regions defined on the surface is morphed to fit the user'"'"'s hand size and current position, the method comprising the following steps:
-
defining a default key layout whose home row key regions lie roughly at predetermined default positions of the fingertips;
identifying what hand part each surface contact comes from;
detecting a layout homing gesture when all five fingers of a hand are placed on the surface in a partially closed posture;
measuring during the layout homing gesture the position offsets of the homing hand and fingers with respect to the predetermined default finger positions;
translating the key regions normally typed by the hand by the measured hand and finger offsets such that each home row key region lies at approximately the measured position of its corresponding finger; and
updating the displayed positions of the key region symbols on a visual display embedded in the surface.
-
-
99. The method of claim 91, wherein the layout of key regions defined on the surface is morphed to fit the user'"'"'s hand size and current position, the method comprising the following steps:
-
identifying what hand part each surface contact comes from;
detecting a layout homing gesture when all five fingers of a hand are placed on the surface in a partially closed posture;
measuring during the layout homing gesture the position of each finger on the surface;
translating each home row key region and its neighboring keys by an amount such that the new position of the home row key region is approximately the same as the measured position of its corresponding finger; and
updating the displayed positions of the key region symbols on a visual display embedded in the surface.
-
-
100. The method of claim 91, wherein typing while the fingers mostly rest on the surface is made easier by not requiring finger liftoff quickly following each press of a key region, the method comprising the following steps:
-
measuring the relative impulsiveness or forcefulness of finger touchdowns;
producing key symbols from liftoff and impulsive or forceful touchdown of a finger while most fingers on the same hand are resting on the surface even if the finger continues to rest on the surface without quickly lifting back off the surface; and
not producing key symbols when finger touchdowns are gentle or synchronous with other fingers.
-
-
101. The method of claim 91, wherein typematic or automatic key repetition when a finger is held on a key is emulated despite the fact that fingers which stay on the surface for extended periods are normally ignored to support hand resting, the method comprising the steps of:
-
issuing a first keypress signal after a holding finger has touched down and remained on a desired key region for at least a hold setup time interval and all other fingers on same hand leave the surface within a release setup time after holding finger touched down;
periodically issuing additional keypress signals every repeat time interval subsequent to the second keypress signal as long as the holding finger continues touching the desired key region; and
ceasing repetitive issuance of the additional keypress signals when the holding finger lifts off the surface.
-
-
102. The method of claim 101, wherein touchdown, resting or liftoff of hand contacts identified as palms on either hand does not affect the typematic state.
-
103. The method of claim 101, wherein the cycle of keypress signal generation continues irrespective of whether other fingers touch down and rest on the surface subsequent to issuing the first keypress signal.
-
104. The method of claim 101, wherein the repeat time interval is continuously adjusted to be inversely proportional to current measurements of holding finger proximity or pressure.
-
105. A method for choosing what kinds of input signals will be generated and sent to an electronic or electro-mechanical device in response to tapping or sliding of fingers on a multi-touch surface, the method comprising the following steps:
-
identifying each contact on the surface as either a thumb, fingertip or palm;
measuring the times when each hand part touches down and lifts off the surface;
forming a set of those fingers which touch down from the all finger floating state before any one of the fingers lifts back off the surface;
choosing the kinds of input signals to be generated by further distinctive motion of the fingers from the combination of finger identities in the set;
generating input signals of this kind when further distinctive motions of the fingers occur;
forming a subset any two or more fingers which touch down synchronously after at least one finger has lifted back off the surface;
choosing a new kinds of input signals to be generated by further distinctive motion of the fingers from the combination of finger identities in the subset;
generating input signals of this new kind when further distinctive motions of the fingers occur; and
continuing to form new subsets, choose and generate new kinds of input signals in response to liftoff and synchronous touchdowns until all fingers lift off the surface. - View Dependent Claims (106, 107, 108, 109, 110, 111, 112, 113, 114)
-
-
115. A method for continuing generation of cursor movement or scrolling signals from a tangential motion of a touch device over a touch-sensitive input device surface after touch device liftoff from the surface if the touch device operator indicates that cursor movement continuation is desired by accelerating or failing to decelerate the tangential motion of the touch device before the touch device is lifted, the method comprising the following steps:
-
measuring, storing and transmitting to a computing device two or more representative tangential velocities during touch device manipulation;
computing and storing a liftoff velocity from touch device positions immediately prior to the touch device liftoff;
comparing the liftoff velocity with the representative tangential velocities, and entering a mode for continuously moving the cursor if a tangential liftoff direction approximately equals the representative tangential directions and a tangential liftoff speed is greater than a predetermined fractional multiple of representative tangential speeds;
continuously transmitting cursor movement signals after liftoff to a computing device such that the cursor movement velocity corresponds to one of the representative tangential velocities; and
ceasing transmission of the cursor movement signals when the touch device engages the surface again, if comparing means detects significant deceleration before liftoff, or if the computing device replies that the cursor can move no farther or a window can scroll no farther. - View Dependent Claims (116, 117)
-
-
118. A method for mapping gestures performed on a multi-touch surface by a right hand to simulate mouse manipulations, the method comprising the steps of:
-
reserving asynchronous single-finger motions for typing and five-finger motions for hand resting;
generating mouse pointer motion signals in response to translational slides of two fingertips;
generating a single mouse click signal in response to a synchronized tap of two fingertips;
generating mouse drag signals in response to translational slides of three fingertips;
generating a double mouse click signal in response to a synchronized tap of three fingertips;
generating window scrolling signals in response to translational slides of four fingertips;
generating a cut to clipboard signal in response to a pinching motion between the thumb and a fingertip;
generating a copy to clipboard signal in response to a synchronized tap of the thumb and a fingertip; and
generating a paste from clipboard signal in response to a movement of the thumb and a fingertip away from each other.
-
Specification