Method and apparatus for integrating manual input

• US 6,323,846 B1
• Filed: 01/25/1999
• Issued: 11/27/2001
• Est. Priority Date: 01/26/1998
• Status: Expired due to Term

• Current Assignee: Apple Inc.
• Original Assignee: University of Delaware
• Inventors: Westerman, Wayne, Elias, John G.
• Primary Examiner(s): Hjerpe, Richard
• Assistant Examiner(s): Laneau, Ronald
• Application Number:
  US09/236,513
• Time in Patent Office: 1,037 Days
• Field of Search: 345/173, 345/174, 345/178, 345/184, 345/156, 345/158
• US Class Current: 345/173
• CPC Class Codes: G06F 18/24137 Distances to cluster centroïds
• CPC Class Codes: G06F 2203/04101 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup
• CPC Class Codes: G06F 2203/04104 Multi-touch detection in digitiser, i.e. details about the simultaneous detection of a plurality of touching locations, e.g. multiple fingers or pen and finger
• CPC Class Codes: G06F 2203/04106 Multi-sensing digitiser, i.e. digitiser using at least two different sensing technologies simultaneously or alternatively, e.g. for detecting pen and finger, for saving power or for improving position detection
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/0235 using chord techniques G06F3/0234 takes precedence
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/0412 Digitisers structurally integrated in a display
• CPC Class Codes: G06F 3/04166 Details of scanning methods, e.g. sampling time, grouping of sub areas or time sharing with display driving Synchronisation with the driving of the display or the backlighting unit to avoid interferences generated internally G06F3/04184
• CPC Class Codes: G06F 3/04186 Touch location disambiguation
• CPC Class Codes: G06F 3/0446 using a grid-like structure of electrodes in at least two directions, e.g. using row and column electrodes
• CPC Class Codes: G06F 3/0447 Position sensing using the local deformation of sensor cells
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06V 30/19173 Classification techniques
• CPC Class Codes: G06V 30/1983 Syntactic or structural pattern recognition, e.g. symbolic string recognition
• CPC Class Codes: G06V 40/107 : Static hand or arm
• CPC Class Codes: H03K 17/9622 : using a plurality of detectors, e.g. keyboard
• CPC Class Codes: H03K 2217/960725 : Charge-transfer

Abstract

Apparatus and methods are disclosed for simultaneously tracking multiple finger and palm contacts as hands approach, touch, and slide across a proximity-sensing, compliant, and flexible multi-touch surface. The surface consists of compressible cushion, dielectric, electrode, and circuitry layers. A simple proximity transduction circuit is placed under each electrode to maximize signal-to-noise ratio and to reduce wiring complexity. Such distributed transduction circuitry is economical for large surfaces when implemented with thin-film transistor techniques. Scanning and signal offset removal on an electrode array produces low-noise proximity images. Segmentation processing of each proximity image constructs a group of electrodes corresponding to each distinguishable contact and extracts shape, position and surface proximity features for each group. Groups in successive images which correspond to the same hand contact are linked by a persistent path tracker which also detects individual contact touchdown and liftoff. Combinatorial optimization modules associate each contact'"'"'s path with a particular fingertip, thumb, or palm of either hand on the basis of biomechanical constraints and contact features. Classification of intuitive hand configurations and motions enables unprecedented integration of typing, resting, pointing, scrolling, 3D manipulation, and handwriting into a versatile, ergonomic computer input device.

Claims

1. A sensing device that is sensitive to changes in self-capacitance brought about by changes in proximity of a touch device to the sensing device, the sensing device comprising:

two electrical switching means connected together in series having a common node, an input node, and an output node;

a dielectric-covered sensing electrode connected to the common node between the two switching means;

a power supply providing an approximately constant voltage connected to the input node of the series-connected switching means;

an integrating capacitor to accumulate charge transferred during multiple consecutive switching of the series connected switching means;

another switching means connected in parallel across the integrating capacitor to deplete its residual charge; and

a voltage-to-voltage translation device connected to the output node of the series-connected switching means which produces a voltage representative of the proximity of the touch device to the sensing device.

2. The sensing device of claim 1, wherein the electrical switching means comprise semiconductor transistors that are switched on and off by dedicated control circuitry.

3. By The sensing device of claim 1, wherein the electrical switching means comprise polymer transistors that are switched on and off by dedicated control circuitry.

4. The sensing device of claim 1, wherein the electrical switching means comprise thin film transistors that are switched on and off by dedicated control circuitry.

7. A multi-touch surface apparatus for detecting a spatial arrangement of multiple touch devices on or near the surface of the multi-touch apparatus comprising:

one of a rigid or flexible surface;

a two-dimensional array of the sensing devices of claim 1 arranged on the surface with their output nodes connected together and sharing the same integrating capacitor, charge depletion switch, and voltage-to-voltage translation device;

control circuitry for sequentially enabling each of the sensor devices;

voltage measurement circuitry to convert sensor data to a digital code; and

circuitry for communicating the digital code to another electronic device.

8. A plurality of the multi-touch surface apparatuses of claim 7 arranged in the shape of one of a cube, a sphere, or any other three dimensional shape.

9. The multi-touch surface apparatus of claim 7, wherein the surface comprises a micro-dimensional surface.

10. The multi-touch surface apparatus of claim 7 being one of fabricated on or integrated with a display device.

11. The multi-touch surface apparatus of claim 10, wherein the display device comprises one of a liquid crystal display (LCD) or a light-emitting polymer display (LPD).

12. A multi-layer cover apparatus for the multi-touch surface apparatus of claim 7, comprising:

a compliant dielectric layer;

a deformable conductive layer formed on the dielectric layer, the conductive layer being electrically coupled to the voltage or current measurement device; and

a touch layer formed on the conductive layer.

13. The multi-layer concr apparatus of claim 12, wherein the touch layer has a symbol set printed thereon that can be removed and replaced with an alternative symbol set.

14. The multi-touch surface apparatus of claim 7, wherein the apparatus is ergonomically arched.

15. The multi-touch surface apparatus of claim 7, wherein the apparatus includes hand configuration visual indicators.

16. The multi-touch surface apparatus of claim 7, wherein the apparatus includes hand configuration audio indicators.

17. A multi-touch surface apparatus for detecting a spatial arrangement of multiple touch devices on or near the surface of the multi-touch apparatus comprising:

one of a rigid or flexible surface;

a two-dimensional array of the sensing devices of claim 2 arranged on the surface with their output nodes connected together and sharing the same current-to-voltage translation device;

control circuitry for sequentially enabling each of the sensor devices;

voltage measurement circuitry to convert sensor data to a digital code; and

circuitry for communicating the digital code to another electronic device.

18. A multi-touch surface apparatus for detecting a spatial arrangement of multiple touch devices on or near the surface of the multi-touch apparatus comprising:

one of a rigid or flexible surface;

a plurality of two-dimensional arrays of the sensing devices of claim 1 arranged on the surface in groups wherein the sensing devices within one group have their output nodes connected to corresponding sensing devices within other groups and share the same integrating capacitor, charge depletion switch, and voltage-to-voltage translation circuitry;

control circuitry for enabling a single sensor device from each two-dimensional array;

means for selecting the sensor voltage data from each two-dimensional array;

voltage measurement circuitry to convert sensor voltage data to a digital code; and

circuitry for communicating the digital code to another electronic device.

19. The multi-touch surface apparatus of claim 18, wherein the sensor voltage data selecting means comprises one of a multiplexing circuitry and a plurality of voltage measurement circuits.

20. A multi-touch surface apparatus for detecting a spatial arrangement of multiple touch devices on or near the surface of the multi-touch apparatus comprising:

one of a rigid or flexible surface;

a plurality of the sensing devices of claim 1 arranged on the surface;

control circuitry for sequentially enabling each of the sensor devices;

voltage measurement circuitry to convert sensor data to a digital code; and

circuitry for communicating the digital code to another electronic device.

21. A multi-touch surface apparatus for sensing diverse configurations and activities of touch devices and generating integrated manual input to one of an electronic or electro-mechanical device, the apparatus comprising:

an array of the proximity sensing devices of claim 1;

a dielectric cover having symbols printed thereon that represent action-to-be-taken when engaged by the touch devices;

scanning means for forming digital proximity images from the array of sensing devices;

calibrating means for removing background offsets from the proximity images;

recognition means for interpreting the configurations and activities of the touch devices that make up the proximity images;

processing means for generating input signals in response to particular touch device configurations and motions; and

communication means for sending the input signals to the electronic or electro-mechanical device.

22. The multi-touch surface apparatus of claim 21, wherein the symbols printed on the dielectric cover can be removed and replaced with an alternative symbol set.

23. The multi-touch surface apparatus of claim 21, wherein the dielectric cover has conductive fibers therein, the conductive fibers being oriented normal to the array of sensing devices for conducting the capacitive ettect of the touch devices on the array of sensing devices.

24. A multi-layer cover apparatus for the multi-touch surface apparatus of claim 21, the multi-layer cover apparatus comprising:

a compliant dielectric layer;

a deformable conductive layer formed on the dielectric layer, the conductive layer being electrically coupled to the voltage or current measurement device; and

a touch layer formed on the conductive layer.

25. The multi-touch surface apparatus of claim 21, wherein the apparatus is ergonomically arched.

26. The multi-touch surface apparatus of claim 21, wherein the apparatus includes hand configuration visual indicators.

27. The multi-touch surface apparatus of claim 21, wherein the apparatus includes hand configuration audio indicators.

28. The multi-touch surface apparatus of claim 21, being one of fabricated on or integrated with a display device.

29. The multi-touch surface apparatus of claim 28, wherein the display device comprises one of a liquid crystal display (LCD) or a light-emitting polymer display (LPD).

5. A sensing device that is sensitive to changes in self-capacitance brought about by changes in proximity of a touch device to the sensing device, the sensing device comprising:

two electrical switching means connected together in series having a common node, an input node, and an output node;

a dielectric-covered sensing electrode connected to the common node between the two switching means;

a power supply providing an approximately constant voltage connected to the input node of the series-connected switching means; and

an integrating current-to-voltage translation device connected to the output node of the series connected switching means, the current-to-voltage translation device producing a voltage representative of the proximity of the touch device to the sensing device.

6. A sensing device that is sensitive to changes in self-capacitance brought about by changes in proximity of a touch device to the sensing device, the sensing device comprising:

two electrical switching means connected together in series having a common node an input node, and an output node,

a dielectric-covered sensing electrode connected to the common node between the two switching means; and

a power supply providing an approximately constant voltage connected to the input node of the series-connected switching means.

30. A multi-touch surface apparatus for sensing diverse configurations and activities of fingers and palms of one or more hands near the surface and generating integrated manual input to one of an electronic or electro-mechanical device, the apparatus comprising:

an array of proximity sensing means embedded in the surface;

scanning means for forming digital proximity images from the proximities measured by the sensing means;

image segmentation means for collecting into groups those proximity image pixels intensified by contact of the same distinguishable part of a hand;

contact tracking means for parameterizing hand contact features and trajectories as the contacts move across successive proximity images;

contact identification means for determining which hand and which part of the hand is causing each surface contact;

synchronization detection means for identifying subsets of identified contacts which touchdown or liftoff the surface at approximately the same time, and for generating command signals in response to synchronous taps of multiple fingers on the surface;

typing recognition means for generating intended key symbols from asynchronous finger taps;

motion component extraction means for compressing multiple degrees of freedom of multiple fingers into degrees of freedom common in two and three dimensional graphical manipulation;

chord motion recognition means for generating one of command and cursor manipulation signals in response to motion in one or more extracted degrees of freedom by a selected combination of fingers;

pen grip detection means for recognizing contact arrangements which resemble the configuration of the hand when gripping a pen, generating inking signals from motions of the inner fingers, and generating cursor manipulation signals from motions of the palms while the inner fingers are lifted; and

communication means for sending the sensed configurations and activities of finger and palms to one of the electronic or electro-mechanical device.

31. A multi-touch surface apparatus for sensing diverse configurations and activities of fingers and palms of one or more hands near the surface and generating integrated manual input to one of an electronic or electro-mechanical device, the apparatus comprising:

an array of proximity sensing means embedded in the surface;

scanning means for forming digital proximity images from proximities measured by the sensing means;

contact segmentation means for collecting proximity image pixels caused by the same hand part into groups;

contact tracking means for parameterizing hand contact features and trajectories as the contacts move across successive proximity images;

contact identification means for determining which hand and which part of the hand is causing each surface contact;

synchronization detection means for identifying subsets of identified contacts which touchdown or liftoff the surface at approximately the same time;

typing recognition means for generating intended key symbols from asynchronous finger taps;

motion component extraction means for compressing the dozens of degrees of freedom in motions of multiple fingers into the degrees of freedom common in two and three dimensional graphical manipulation;

chord motion recognition means for generating, command or cursor manipulation signals in response to motion in one or more extracted degrees of freedom by a selected combination of fingers; and

communication means for sending said generated input signals to the electronic or electro-mechanical device.

32. A multi-touch surface apparatus for sensing diverse configurations and activities of fingers and palms of one or more hands near the surface and generating integrated manual input to one of an electronic or electro-mechanical device, the apparatus comprising:

an array of proximity sensing means embedded in the surface;

scanning means for forming digital proximity images from proximities measured by the sensing means;

contact segmentation means for collecting proximity image pixels caused by the same hand part into groups;

contact tracking means for parameterizing hand contact features and trajectories as the contacts move across successive proximity images;

contact identification means for determining which hand and which part of the hand is causing each surface contact;

pen grip detection means for recognizing contact arrangements which resemble the configuration of the hand when gripping a pen, generating inking signals from motions of the inner fingers, and generating cursor manipulation signals from motions of the palms while the inner fingers are lifted; and

communication means for sending said generated input signals to the electronic or electro-mechanical device.

33. A method for tracking and identifying hand contacts in a sequence of proximity images in order to support interpretation of hand configurations and activities related to typing, multiple degree-of-freedom manipulation via chords, and handwriting, the method comprising the steps of:

segmenting each proximity image into groups of electrodes which indicate significant proximity, each group representing proximity of a distinguishable hand part or other touch device;

extracting total proximity, position, shape, size, and orientation parameters from each group of electrodes;

tracking, group paths through successive proximity images including detection of path endpoints at contact touchdown and liftoff;

computing velocity and filtered position vectors along each path;

assigning a hand and finger identity to each contact path by incorporating relative path positions and velocities, individual contact features, and previous estimates of hand and finger positions; and

maintaining estimates of hand and finger positions from trajectories of paths currently assigned to the fingers, wherein the estimates provide high level feedback to bias segmentations and identifications in future images.

34. A method for filtering and segmenting hand contacts in a sequence of proximity images in order to support interpretation of various contact sizes, shapes, orientations, and spacings, the method comprising the steps of:

creating a smoothed copy of the most recent proximity image;

searching for pixels with locally maximum proximity in the smoothed proximity image;

searching outward from each local maximum pixel for contact boundary pixels using boundary tests of pixel and neighboring pixel proximities which depend on properties of hand contacts expected in a segmentation region of the pixel;

forming groups from those pixels surrounding each local maximum pixel up to and including the boundary pixels;

combining groups of pixels which partially overlap;

extracting group positions and features by fitting an ellipse to each group of pixels; and

updating positions of the segmentation regions of the pixels in response to further analysis of the position and features extracted from each group of pixels.

35. The method of claim 34, wherein sloppy segmentation regions of the pixels include rectangular areas under expected palm locations where only proximity valleys between the palm heels or pixels near a background signal level act as boundaries, and wherein a remaining image portion is a strict segmentation region that establishes group boundaries at a directional proximity minima encountered in a direction of search outward from a given local maximum pixel.

36. The method of claim 34, wherein segmentation region rules of a hand touching the surface override segmentation region rules of a hand not contacting the surface.

37. The method of claim 34, wherein verified properties of each group of pixels in previous images are fed back as estimated hand offsets to adjust alignment of the segmentation regions for the current image.

38. A method for associating into paths those surface contacts from successive proximity images caused by the same hand part and detecting liftoff from and touchdown onto the surface by each hand part, the method comprising the steps of:

predicting the current positions of hand parts from their velocity along existing paths;

finding for each of a group of pixels in current proximity image the existing path with a closest predicted path position;

finding for each existing path the pixel group whose centroid is closest to the predicted path position and whose centroid is within a path-dependent tracking radius;

pairing each pixel group with its closest path if the pixel group is also the closest pixel group to the path;

starting new paths for remaining unpaired pixel groups;

deactivating paths which have no pairable pixel groups within the path-dependent tracking radius; and

updating path parameters from the measured parameters of the pixel group paired with each path.

39. A method of computing hand and finger position offsets from the measured positions of individual hand contacts on a multi-touch surface for the purpose of biasing future hand contact identifications or morphing the key layout in an integrated manual input device, the method comprising the steps of:

establishing fingertip, thumb, or palm identities for each contact;

establishing an offset weighting for each contact;

computing a hand position offset, wherein the offset is a weighted average of the difference between a measured position of each contact and a predetermined default position of the hand part which corresponds to an established identity of the contact; and

computing a finger position offset by subtracting a predetermined default position of an associated hand part of the contact and the hand position offset from a measured position of the contact.

40. The method of claim 39, wherein conservative estimates of hand and finger position offsets are maintained across a succession of proximity images even when hand contact identifications become unreliable, the method comprising the steps of:

computing the confidence in current contact identifications from the amount of information available for the identifications;

computing a weighted average of the individual hand contact velocities;

predicting a current hand and finger offset from the previous offset estimates and the weighted average velocity;

computing current hand and finger offsets from current contact identities and measured contact locations;

setting the currently measured hand and finger offsets to zero if the hand has no contacts in the current image; and

updating the hand and finger offset estimates to the weighted average of the predicted offsets and currently measured offsets, wherein the relative weighting given to the measured offsets increases in proportion to the confidence level in the current identifications.

41. The method of claim 40, wherein a hand sliding to the opposite side of the surface eliminates the estimated position of a lifted hand by imposing a minimum separation between the estimated hand positions, and permitting the hand with a highest identification confidence override the estimated position of the other hand.

42. A method for establishing identities of hand contacts on a multi-touch surface using relative contact positions and features, the method comprising the steps of:

defining a template of hand part attractor points on the surface, the attractor points for each hand roughly forming a ring;

computing a matrix of distances from each surface contact to each attractor point;

weighting the distances between each surface contact and each attractor point according to how closely measured contact features such as proximity to a surface, shape, size, eccentricity, orientation, distance to nearest neighbor contact, and velocity match features typical of the hand part the attractor point represents;

finding a one-to-one mapping of the surface contacts to the attractor points that minimizes a sum of distances between each surface contact and its corresponding attractor point; and

recognizing particular hand configurations from the number and features of surface contacts assigned to particular subsets of the attractor points.

43. The method of claim 42, wherein an attractor point which is unassigned to a real surface contact because there are less surface contacts than attractor points contributes nothing to the sum of distances.

44. The method of claim 42, wherein the distance metric used for computing the distance from a surface contact to an attractor point is the squared Euclidean distance.

45. The method of claim 42, wherein the attractor point positions correspond to the positions of hand part surface contacts measured when each hand is in a neutral posture.

46. The method of claim 42, wherein forepalm contacts of the hands are recognized by adding attractor points near the center of the attractor point ring of each hand, and weighting the distances to the forepalm attractor points so that contacts are assigned to the forepalm attractor points only if the hand produces enough contacts to nearly fill the attractor ring.

47. The method of claim 42, wherein the portion of an attractor template corresponding to parts of a particular hand is translated to remain centered on the last estimated position of that hand.

48. The method of claim 42, wherein a portion of the attractor points template corresponding to parts of a particular hand is rotated and/or scaled to match a previous estimated orientation and size of that particular hand.

49. The method of claim 42, wherein all of the surface contacts to be assigned are assumed to have come from the same hand and all of the attractor points represent parts of one hand.

50. The method of claim 42, wherein an additional hand identification means restricts assignment of the surface contacts to those attractor points which the hand identification means assigned to a particular hand.

51. The method of claim 42, wherein the measured distance from a contact to a thumb attractor point on a given hand is weighted so as to encourage assignment to the thumb attractor when the total contact proximity is greater than that of a typical fingertip but less than that of a typical palm heel.

52. The method of claim 42, wherein the measured distance from a contact to a thumb or inner palm heel attractor point on a given hand is weighted so as to encourage assignment to that attractor when the contact orientation approaches the expected slant of a thumb or inner palm heel on the given hand.

53. The method of claim 42, wherein the measured distance from a contact to a palm heel attractor point on a given hand is weighted so as to encourage assignment to the palm heel attractor when the measured contact width or ratio of total proximity to eccentricity exceeds that of a typical finger.

54. The method of claim 42, wherein additional contact and inter-contact features are incorporated during a verification step, the verification step comprising shifting assignments found in the attractor points minimization step to make the assignments more consistent with additional feature tests used in the verification step.

55. The method of claim 54, wherein the verification step checks horizontal position coordinates of contacts assigned to attractor points corresponding to fingertips to ensure they are in increasing order for right hand fingertips and in decreasing order for left hand fingertips.

56. The method of claim 54, wherein the verification step includes a thumb verification step comprising the following sub-steps:

finding an innermost finger contact by searching for a contact assigned to a filled attractor point corresponding to an innermost finger;

computing a thumb factor as a function of the innermost finger contact relative to other finger contacts;

shifting the innermost finger contact to a thumb attractor point if the innermost finger contact is not already assigned to the thumb attractor point and the thumb factor is above a predetermined thumb threshold; and

shifting the innermost finger contact to a fingertip attractor point if the innermost finger contact is currently assigned to the thumb attractor point and the thumb factor is below the predetermined thumb threshold.

57. The method of claim 56, wherein the thumb actor is high if the orientation and size of the innermost finger contact are greater than those of other finger contacts.

58. The method of claim 56, wherein the thumb factor is high if the separation, angle, and velocity of the innermost finger contact relative to other finger contacts are in ranges unique to opposable thumb presence or motion.

59. The method of claim 56, wherein the verification step is performed when one of a fingertip or thumb attractor point is left unfilled by the attractor points minimization step.

60. A method for ordering surface contacts and establishing finger, thumb, and palm identities, the method comprising the steps of:

finding a shortest path connecting all of the contacts assumed to be from a given hand;

passing through each contact once to form an ordered loop;

finding an innermost contact in the ordered loop;

determining whether the innermost contact is a thumb, fingertip, or palm contact from contact and inter-contact features of the innermost contact; and

assigning thumb, fingertip, or palm identities to non-innermost contacts based upon the features of the contacts, assignment of the innermost contacts, vertical position relative to assigned contacts, and the loop ordering.

61. An apparatus for distinguishing palm heel contacts from other types of hand contacts in a system for recognizing hand activity on a multi-touch surface and generating input signals to a competing device therefrom, the apparatus comprising:

means for finding the nearest neighbor contact of a given contact in a plane of the surface; and

means for suppressing identification of the given contact as a palm heel contact it a neighbor contact exists and is closer to the given contact than the anatomical separation between inner and outer portions of a palm heel.

62. The apparatus of claim 61, wherein the finding means ignores contacts identified as forepalm hand contacts.

63. An apparatus for distinguishing palm heel contacts from other types of hand contacts in a system for recognizing hand activity on a multi-touch surface and generating input signals to a competing device therefrom, the apparatus comprising:

means for measuring the total proximity, orientation, and eccentricity of all contacts;

means for encouraging identification of a given contact as a palm heel contact if its ratio of total proximity to eccentricity is larger than for a typical fingertip contact; and

means for encouraging identification of a given contact as a palm heel contact as its orientation approaches the expected slant of a palm heel.

64. An apparatus for distinguishing thumb contacts from other types of hand contacts in a system for recognizing hand activity on a multi-touch surface and generating input signals to a competing device therefrom, the apparatus comprising:

means for measuring the size and orientation of all contacts;

means for encouraging identification of a given contact as a thumb contact if its size is larger than a typical fingertip contact;

means for discouraging identification of a given contact as a thumb contact if its size is larger than a typical thumb contact; and

means for encouraging identification of a given contact as a thumb contact as its orientation approaches the expected slant of the thumb.

65. A method for determining which hand causes each surface contact detected on a multi-touch surface so that input signals generated by hand activity on the surface can depend on the identity of the hand performing the activity and so that multiple hands can perform independent activities on the surface simultaneously, the method comprising the steps of:

defining a template of hand part attractor points on the surface, the attractor points for each hand approximately forming a ring;

generating partitions which divide the set of all surface contacts into left hand clusters and right hand clusters;

assigning finger and palm identities to the contacts within each cluster;

computing for each partition an assignment fitness measure which represents the biomechanical consistency of the fit of contact clusters to their assigned attractor rings;

choosing the partition which has the best assignment fitness measure as the partition containing the true contact identities; and

recognizing each hand'"'"'s configuration from the combination of and features of surface contacts assigned within each attractor ring of the best partition.

66. The method of claim 65, wherein the hand assignments are re-computed only for proximity images in which new hand contacts are stabilizing.

67. The method of claim 65, wherein a reactivated path for a temporarily-removed contact regains its previous identity.

68. The method of claim 65, wherein each attractor point is placed at an expected position of a corresponding hand part.

69. The method of claim 65, wherein the partition generating step comprises the following sub-steps:

constructing approximately vertical contours between each horizontally adjacent contact; and

constructing a partition from each contour by tentatively assigning contacts which are positioned to the left of a contour to the left hand cluster and contacts to the right of a contour to the right hand cluster.

70. The method of claim 69, wherein the hand assignments of previously identified contacts can be locked so to not depend on which side of the dividing contour the contacts lie, while assignments of new contacts still depend on which side of the contour they lie.

71. The method of claim 65, wherein each attractor ring is translated, scaled and/or rotated to match previous position estimates for the hand corresponding to the attractor ring.

72. The method of claim 65, wherein the attractor points within each ring are individually offset by previously estimated finger offsets.

73. The method of claim 65, wherein the assignment fitness measure is a total cost computed as a weighted sum of distances from each contact to its assigned attractor point in the attractor ring of its assigned hand cluster, and wherein the best partition is the one with the lowest total cost.

74. The method of claim 73, wherein the distances between each surface contact and each attractor point are weighted according to how closely measured contact features, such as proximity to the surface, shape, size, eccentricity, orientation, distance to nearest neighbor contact, and velocity, match features typical of the hand part the attractor point represents.

75. The method of claim 73, wherein a separation between the innermost finger and the next innermost finger is compared with a separation between the outermost finger and next outermost finger to obtain a handedness weighting which increases the total cost for a hand as the outermost separation becomes larger than is biomechanically consistent.

76. The method of claim 73, wherein a hand portion of the total cost is decreased by a cluster velocity weighting function when the average velocity of the contact cluster of the hand indicates the hand is returning to its associated side of the surface.

77. The method of claim 73, wherein a hand portion of the total cost is increased by a palm cohesion weighting function when the contacts assigned to the palms of a hand are scattered over an area larger than the anatomical size of an outstretched palm.

78. The method of claim 73, wherein the total cost of a partition is increased by a interhand separation weighting when the measured separation between contacts tentatively assigned to opposite hand clusters indicates that the hands may be overlapping or close to touching.

79. A method for integrally extracting multiple degrees of freedom of hand motion from sliding motions of two or more fingers of a hand across a multi-touch surface, one of the fingers preferably being the opposable thumb, the method comprising the steps of:

tracking, across successive scans of the proximity sensor array the trajectories of individual hand parts on the surface;

finding an innermost and an outermost finger contact from contacts identified as fingers on the given hand;

computing a scaling velocity component from a change in a distance between the innermost and outermost finger contacts;

computing a rotational velocity component from a change in a vector angle between the innermost and outermost finger contacts;

computing a translation weighting for each contacting finger;

computing translational velocity components in two dimensions from a translation weighted average of the finger velocities tangential to surface;

suppressively filtering components whose speeds are consistently lower than the fastest components;

transmitting the filtered velocity components as control signals to an electronic or electro-mechanical device.

80. The method of claim 79, wherein the scaling velocity computed from a change in distance between the innermost and outermost finger contacts is supplemented with a measure of scaling velocity selective for symmetric scaling about a fixed point between the thumb and other fingers.

81. The method of claim 79, wherein the rotational velocity computed from a change in vector angle between the innermost and outermost finger contacts is supplemented with a measure of rotational velocity selective for symmetric rotational about a fixed point between the thumb and other fingers.

82. The method of claim 79, wherein the translation weightings of the innermost and outermost fingers are constant but the translation weightings of central fingers are inversely related to polar component speeds so as to prevent vertical translation bias while performing hand scaling and rotation but otherwise include all available fingers in the translation average.

83. The method of claim 79, wherein the translational weightings are related to the ratio of each finger'"'"'s speed to the speed of the fastest finger so that if the user chooses to move fewer fingers than are on the surface the gain between individual finger motion and cursor motion does not decrease.

84. The method of claim 79, wherein the suppressive filtering step comprises the following two sub steps:

downscaling each velocity component in proportion to a function of its average speed compared to the other average component speeds;

dead-zone filtering each downscaled velocity component wherein the width of the dead-zone depends on the distribution of the current component speeds.

85. The method of claim 79, wherein the orientation of an ellipse fitted to the thumb contact after each successive sensor array scan is transmitted as an additional degree of freedom control signal.

86. A method for integrally extracting roll and tilt degrees of freedom of hand motion from pressure changes of three or more non-collinear hand contacts comprising any of thumbs, fingertips or palms, the method comprising the steps of:

tracking across successive proximity images the trajectories of individual hand parts on the surface;

measuring proximities from each hand contact in a calibration proximity image once all available hand contacts have been stabilized;

computing an average hand contact position from a post-calibration proximity image, wherein all hand contacts are weighted equally;

computing a weighted average hand contact position from a post-calibration proximity image, wherein each hand contact is weighted according to the ratio of its current proximity to its calibrated proximity;

computing for each post-calibration proximity image the difference vector between the weighted average hand contact position and the average hand contact position;

dead-zone filtering the difference vector to remove variations in proximity due to unintentional posture shifts; and

transmitting the filtered difference vector from each post-calibration proximity image as roll and tilt control signals to an electronic or electro-mechanical device.

87. The method of claim 86, wherein reference proximities slowly adapt to decreases in individual contact proximity.

88. The method of claim 86, wherein the ratio of current hand contact proximity to its calibrated hand contact proximity is clipped to be greater than or equal to one.

89. The method of claim 86, wherein an additional total hand proximity component is computed from the average of all current hand contact proximity to calibrated contact proximity ratios, and the total hand proximity component is transmitted to computing device.

90. The method of claim 86, wherein the computation and transmission of hand roll and tilt rotational axes are initialized by resting all five fingers on the surface, tapping palms on the surface, and then resting palms on the surface.

91. A manual input integration method for supporting diverse hand input activities such as resting the hands, typing, multiple degree-of-freedom manipulation, command gesturing and handwriting on a multi-touch surface, the method enabling users to instantaneously switch between the input activities by placing their hands in different configurations comprising distinguishable combinations of relative hand contact timing, proximity, shape, size, position, motion and/or identity across a succession of surface proximity images, the method comprising the steps of:

tracking each touching hand part across successive proximity images;

measuring the times when each hand part touches down and lifts off the surface;

detecting when hand parts touch down or lift off simultaneously;

producing discrete key symbols when the user asynchronously taps, holds, or slides a finger on key regions defined on the surface;

producing discrete mouse button click commands, key commands, or no signals when the user synchronously taps two or more fingers from the same hand on the surface;

producing gesture commands or multiple degree-of-freedom manipulation signals when the user slides two or more fingers across the surface; and

sending the produced symbols, commands and manipulation signals as input to an electronic or an electro-mechanical device.

92. The method of claim 91, wherein production of discrete key symbols or mouse button click commands from single finger taps or finger chord taps is accompanied by transmission of activation signals to a light or sound feedback generating device.

93. The method of claim 91, wherein accidental synchronous finger taps during typing are prevented from disrupting the typing session by not producing input signals to a computing device when the accidental chord tap is the first detected chord tap since the last detected asynchronous key tap, the chord tap occurs within a typing timeout interval subsequent to the last detected asynchronous key tap, and no finger slides have been detected between the last detected asynchronous key tap and said accidental chord tap.

94. The method of claim 91, wherein hand resting is tolerated by suppressing a generation of output commands when all or nearly all of the fingers simultaneously engage the multi-touch surface and remain substantially stationary.

95. The method of claim 91, wherein user handwriting activity is distinguished from other input activities by its unique pen grip hand configuration with the following additional steps:

establishing the finger or palm identity of each surface contact;

measuring the relative positions and proximities of the identified contacts to determine whether the inner fingers are pinched while the outer fingers curl under the palm exposing their knuckles to rest on the surface;

entering a handwriting mode for the hand if the above unique finger arrangement is detected;

producing inking signals from the motions of the inner fingers on the surface while in handwriting mode;

producing stylus lift signals each time the inner fingers lift off the surface while in handwriting mode;

sending the inking signals to an electronic device for capture, display, or recognition; and

leaving the handwriting mode after the hand has lifted and remained off the surface for a substantial time or if a non-pinched finger configuration is measured.

96. The method of claim 95, wherein while in handwriting mode but the inner fingers are lifted, sliding and tapping motions of the palm heels produce cursor manipulation and clicking signals which are sent to the electronic device.

97. The method of claim 95, wherein a stylus held between the pinched fingers touches the surface instead of the pinched fingers themselves to indicate pinch configuration, and wherein inking signals are measured from motion of the stylus.

98. The method of claim 91, wherein the layout of key regions defined on the surface is morphed to fit the user'"'"'s hand size and current position, the method comprising the following steps:

defining a default key layout whose home row key regions lie roughly at predetermined default positions of the fingertips;

identifying what hand part each surface contact comes from;

detecting a layout homing gesture when all five fingers of a hand are placed on the surface in a partially closed posture;

measuring during the layout homing gesture the position offsets of the homing hand and fingers with respect to the predetermined default finger positions;

translating the key regions normally typed by the hand by the measured hand and finger offsets such that each home row key region lies at approximately the measured position of its corresponding finger; and

updating the displayed positions of the key region symbols on a visual display embedded in the surface.

99. The method of claim 91, wherein the layout of key regions defined on the surface is morphed to fit the user'"'"'s hand size and current position, the method comprising the following steps:

identifying what hand part each surface contact comes from;

detecting a layout homing gesture when all five fingers of a hand are placed on the surface in a partially closed posture;

measuring during the layout homing gesture the position of each finger on the surface;

translating each home row key region and its neighboring keys by an amount such that the new position of the home row key region is approximately the same as the measured position of its corresponding finger; and

updating the displayed positions of the key region symbols on a visual display embedded in the surface.

100. The method of claim 91, wherein typing while the fingers mostly rest on the surface is made easier by not requiring finger liftoff quickly following each press of a key region, the method comprising the following steps:

measuring the relative impulsiveness or forcefulness of finger touchdowns;

producing key symbols from liftoff and impulsive or forceful touchdown of a finger while most fingers on the same hand are resting on the surface even if the finger continues to rest on the surface without quickly lifting back off the surface; and

not producing key symbols when finger touchdowns are gentle or synchronous with other fingers.

101. The method of claim 91, wherein typematic or automatic key repetition when a finger is held on a key is emulated despite the fact that fingers which stay on the surface for extended periods are normally ignored to support hand resting, the method comprising the steps of:

issuing a first keypress signal after a holding finger has touched down and remained on a desired key region for at least a hold setup time interval and all other fingers on same hand leave the surface within a release setup time after holding finger touched down;

periodically issuing additional keypress signals every repeat time interval subsequent to the second keypress signal as long as the holding finger continues touching the desired key region; and

ceasing repetitive issuance of the additional keypress signals when the holding finger lifts off the surface.

102. The method of claim 101, wherein touchdown, resting or liftoff of hand contacts identified as palms on either hand does not affect the typematic state.

103. The method of claim 101, wherein the cycle of keypress signal generation continues irrespective of whether other fingers touch down and rest on the surface subsequent to issuing the first keypress signal.

104. The method of claim 101, wherein the repeat time interval is continuously adjusted to be inversely proportional to current measurements of holding finger proximity or pressure.

105. A method for choosing what kinds of input signals will be generated and sent to an electronic or electro-mechanical device in response to tapping or sliding of fingers on a multi-touch surface, the method comprising the following steps:

identifying each contact on the surface as either a thumb, fingertip or palm;

measuring the times when each hand part touches down and lifts off the surface;

forming a set of those fingers which touch down from the all finger floating state before any one of the fingers lifts back off the surface;

choosing the kinds of input signals to be generated by further distinctive motion of the fingers from the combination of finger identities in the set;

generating input signals of this kind when further distinctive motions of the fingers occur;

forming a subset any two or more fingers which touch down synchronously after at least one finger has lifted back off the surface;

choosing a new kinds of input signals to be generated by further distinctive motion of the fingers from the combination of finger identities in the subset;

generating input signals of this new kind when further distinctive motions of the fingers occur; and

continuing to form new subsets, choose and generate new kinds of input signals in response to liftoff and synchronous touchdowns until all fingers lift off the surface.

106. The method of claim 105, wherein all sets or subsets which contain the same number of fingertips choose the same kinds of input signals, such that sets or subsets are uniquely distinguished by the number of fingertips they contain and whether they contain the thumb.

107. The method of claim 105, wherein all sets or subsets which contain the same combination of thumb, index, middle, ring, and pinky fingers choose the same kinds of input signals.

108. The method of claim 105, wherein the method is applied to contacts identified as left hand parts independently from contacts identified as right hand parts.

109. The method of claim 105, wherein no input signals are generated after a set or subset is formed without further distinctive finger motions, to support resting of fingers on the surface.

110. The method of claim 105, wherein one of the distinctive finger motions is synchronized liftoff of a finger set or subset quickly following synchronized touchdown, and wherein each such motion generates a tap signal of the selected kind.

111. The method of claim 105, wherein one of the distinctive finger motions is sliding of the finger set or subset across the surface, and wherein such motion continuously generates slide signals of the selected kind which include measurements of the sliding motion.

112. The method of claim 105, wherein asynchronous touchdown quickly followed by liftoff of a finger forms a new subset of one finger and generates a tap event dependent on the location on the surface of the touchdown.

113. The method of claim 105, wherein generation of input signals is accompanied by generation of activation signals to a light or sound generating feedback device, and wherein the activation signals depend upon the kinds of input signals currently selected.

114. The method of claim 105, wherein a new subset of fingers is formed upon simultaneous finger release from the all fingers resting state, wherein this new subset consists of those fingers which remain on the surface, and wherein this new subset chooses a new kinds of input signals which can be generated in response to further distinctive finger motions.

115. A method for continuing generation of cursor movement or scrolling signals from a tangential motion of a touch device over a touch-sensitive input device surface after touch device liftoff from the surface if the touch device operator indicates that cursor movement continuation is desired by accelerating or failing to decelerate the tangential motion of the touch device before the touch device is lifted, the method comprising the following steps:

measuring, storing and transmitting to a computing device two or more representative tangential velocities during touch device manipulation;

computing and storing a liftoff velocity from touch device positions immediately prior to the touch device liftoff;

comparing the liftoff velocity with the representative tangential velocities, and entering a mode for continuously moving the cursor if a tangential liftoff direction approximately equals the representative tangential directions and a tangential liftoff speed is greater than a predetermined fractional multiple of representative tangential speeds;

continuously transmitting cursor movement signals after liftoff to a computing device such that the cursor movement velocity corresponds to one of the representative tangential velocities; and

ceasing transmission of the cursor movement signals when the touch device engages the surface again, if comparing means detects significant deceleration before liftoff, or if the computing device replies that the cursor can move no farther or a window can scroll no farther.

116. The method of claim 115, wherein one of the representative tangential velocities is a weighted average of several instantaneous velocities.

117. The method of claim 115, wherein the touch surface is a multi-touch surface, the touch devices are fingers, the cursor movement velocity is the hand translation, rotation, or scaling velocity extracted from the touching fingers, and the mode for continuously moving the cursor is entered when the velocity of the dominant hand motion component passes the deceleration test as the last fingers are lifted.

118. A method for mapping gestures performed on a multi-touch surface by a right hand to simulate mouse manipulations, the method comprising the steps of:

reserving asynchronous single-finger motions for typing and five-finger motions for hand resting;

generating mouse pointer motion signals in response to translational slides of two fingertips;

generating a single mouse click signal in response to a synchronized tap of two fingertips;

generating mouse drag signals in response to translational slides of three fingertips;

generating a double mouse click signal in response to a synchronized tap of three fingertips;

generating window scrolling signals in response to translational slides of four fingertips;

generating a cut to clipboard signal in response to a pinching motion between the thumb and a fingertip;

generating a copy to clipboard signal in response to a synchronized tap of the thumb and a fingertip; and

generating a paste from clipboard signal in response to a movement of the thumb and a fingertip away from each other.