Method and apparatus for integrating manual input

• US 6,888,536 B2
• Filed: 07/31/2001
• Issued: 05/03/2005
• Est. Priority Date: 01/26/1998
• Status: Expired due to Term

• Current Assignee: Apple Inc.
• Original Assignee: University of Delaware
• Inventors: Westerman, Wayne, Elias, John G.
• Primary Examiner(s): Wu, Xiao
• Assistant Examiner(s): LESPERANCE, JEAN E
• Application Number:
  US09/919,266
• Publication Number:
  US 20020015024A1
• Time in Patent Office: 1,372 Days
• Field of Search: 345/173, 345/174, 345/178, 345/184, 345/156, 345/158, 345/863
• US Class Current: 345/173
• CPC Class Codes: G06F 18/24137 Distances to cluster centroïds
• CPC Class Codes: G06F 2203/04101 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup
• CPC Class Codes: G06F 2203/04104 Multi-touch detection in digitiser, i.e. details about the simultaneous detection of a plurality of touching locations, e.g. multiple fingers or pen and finger
• CPC Class Codes: G06F 2203/04106 Multi-sensing digitiser, i.e. digitiser using at least two different sensing technologies simultaneously or alternatively, e.g. for detecting pen and finger, for saving power or for improving position detection
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/0235 using chord techniques G06F3/0234 takes precedence
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/0412 Digitisers structurally integrated in a display
• CPC Class Codes: G06F 3/04166 Details of scanning methods, e.g. sampling time, grouping of sub areas or time sharing with display driving Synchronisation with the driving of the display or the backlighting unit to avoid interferences generated internally G06F3/04184
• CPC Class Codes: G06F 3/04186 Touch location disambiguation
• CPC Class Codes: G06F 3/0446 using a grid-like structure of electrodes in at least two directions, e.g. using row and column electrodes
• CPC Class Codes: G06F 3/0447 Position sensing using the local deformation of sensor cells
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06V 30/19173 Classification techniques
• CPC Class Codes: G06V 30/1983 Syntactic or structural pattern recognition, e.g. symbolic string recognition
• CPC Class Codes: G06V 40/107 : Static hand or arm
• CPC Class Codes: H03K 17/9622 : using a plurality of detectors, e.g. keyboard
• CPC Class Codes: H03K 2217/960725 : Charge-transfer

Abstract

Apparatus and methods are disclosed for simultaneously tracking multiple finger and palm contacts as hands approach, touch, and slide across a proximity-sensing. compliant, and flexible multi-touch surface. The surface consists of compressible cushion, dielectric, electrode, and circuitry layers. A simple proximity transduction circuit is placed under each electrode to maximize signal-to-noise ratio and to reduce wiring complexity. Such distributed transduction circuitry is economical for large surfaces when implemented with thin-film transistor techniques. Scanning and signal offset removal on an electrode array produces low-noise proximity images. Segmentation processing of each proximity image constructs a group of electrodes corresponding to each distinguishable contact and extracts shape, position and surface proximity features for each group. Groups in successive images which correspond to the same hand contact are linked by a persistent path tracker which also detects individual contact touchdown and liftoff. Combinatorial optimization modules associate each contact'"'"'s path with a particular fingertip, thumb, or palm of either hand on the basis of biomechanical constraints and contact features. Classification of intuitive hand configurations and motions enables unprecedented integration of typing, resting, pointing, scrolling, 3D manipulation, and handwriting into a versatile, ergonomic computer input device.

Claims

1. A multi-touch surface apparatus for detecting a spatial arrangement of multiple touch devices on or near the surface of the multi-touch apparatus, comprising:

one of a rigid or flexible surface, the surface having, for each hand, a shallow depression running between the default index and pinky fingertip locations to provide tactile indication of fingertip home positions;

a plurality of the sensing devices arranged on the surface, each sensing comprising;

two electrical switching means connected together in series having a common node, an input node, and an output node,

a dielectric-covered sensing electrode connected to the common node between the two switching means,

a power supply providing an approximately constant voltage connected to the input node of the series-connected switching means,

an integrating capacitor to accumulate charge transferred during multiple consecutive switchings of the series connected switching means,

another switching means connected in parallel across the integrating capacitor to deplete its residual charge, and

a voltage-to-voltage translation device connected to the output node of the series-connected switching means which produces a voltage representative of the proximity of the touch device to the sensing device;

control circuitry for sequentially enabling each of the sensor devices;

voltage measurement circuitry to convert sensor data to a digital code; and

circuitry for communicating the digital code to another electronic device.

2. The multi-touch surface apparatus of claim 1, wherein the surface includes for each hand a shallow depression in the shape of an oblique oval at the default thumb position to provide tactile indication of thumb home position.

3. A multi-touch surface apparatus for detecting a spatial arrangement of multiple touch devices on or near the surface of the multi-touch apparatus, comprising:

one of a rigid or flexible surface, the surface having key region positions indicated by a raised dot near the center of selected regions, the raised portion of the dot being made from a conductive material to prevent weakening of a finger proximity signal as a finger is pushed up by the dot;

a plurality of the sensing devices arranged on the surface, each sensing comprising;

two electrical switching means connected together in series having a common node, an input node, and an output node,

a dielectric-covered sensing electrode connected to the common node between the two switching means,

a power supply providing an approximately constant voltage connected to the input node of the series-connected switching means,

an integrating capacitor to accumulate charge transferred during multiple consecutive switchings of the series connected switching means,

another switching means connected in parallel across the integrating capacitor to deplete its residual charge, and

a voltage-to-voltage translation device connected to the output node of the series-connected switching means which produces a voltage representative of the proximity of the touch device to the sensing device;

control circuitry for sequentially enabling each of the sensor devices;

voltage measurement circuitry to convert sensor data to a digital code; and

circuitry for communicating the digital code to another electronic device.

4. A method for mapping gestures performed on a multi-touch surface by a left hand to simulate text editing commands via arrow keys, the method comprising the steps of:

reserving asynchronous single-finger motions for typing and five-finger motions for hand resting;

generating arrow key signals in response to translational slides of two fingertips; and

generating shift-arrow key signals in response to translational slides of three fingertips.

5. A method for mapping surface contact gestures performed by a hand on a multi-touch surface to clipboard commands, comprising:

estimating fingertip or thumb identities for each surface contact;

preparing to generate clipboard commands when the surface contacts are identified as a thumb and one fingertip;

generating a cut to clipboard signal in response to a movement of the thumb and fingertip contacts toward each other;

generating a copy to clipboard signal in response to a synchronized tap of the thumb and fingertip contacts; and

generating a paste from clipboard signal in response to a movement of the thumb and fingertip contacts away from each other.

6. A method for mapping surface contact gestures performed by a hand on a multi-touch surface to cursor and object manipulation operations, comprising:

tracking and counting the surface contacts;

extracting hand translation and scaling motions from surface contact paths;

generating cursor motion signals in response to translational slides of two fingertips;

generating cursor selection and dragging signals in response to translational slides of three fingertips;

generating scrolling signals in response to translational slides of four fingertips; and

generating zooming signals in response to hand scaling slides of five fingertips.

7. A method for mapping surface contact gestures as recited in claim 6, wherein the surface is a two-handed, multi-touch surface;

gesture performance by one hand on a portion of the surface causes the cursor selection, scrolling, and zooming signals to be of the mouse-emulation type for manipulating a mouse cursor; and

gesture performance by the other hand on another portion of the surface causes the cursor selection, scrolling, and zooming signals to be of the arrow and editing key type for manipulating a text cursor.

8. A method for mapping surface contact gestures as recited in claim 6, wherein the extracting step additionally extracts rotational motions, and rotational slides of all five fingertips generate object rotation signals.

9. A method for mapping surface contact gestures as recited in claim 8, wherein the surface is a two-handed, multi-touch surface;

gesture performance by one hand on a portion of the surface causes the scrolling, zooming, and object rotation signals to be directed toward a display background; and

gesture performance by the other hand on another portion of the surface causes the scrolling, zooming, and object rotation signals to be directed toward a foreground object.