System and method for packing multi-touch gestures onto a hand

• US 7,030,861 B1
• Filed: 09/28/2002
• Issued: 04/18/2006
• Est. Priority Date: 02/10/2001
• Status: Expired due to Term

• Current Assignee: Apple Inc.
• Original Assignee: FingerWorks Limited (Apple Inc.)
• Inventors: Westerman, Wayne Carl, Elias, John G.
• Primary Examiner(s): Awad, Amr A.
• Assistant Examiner(s): SHENG, TOM V
• Application Number:
  US10/065,252
• Time in Patent Office: 1,298 Days
• Field of Search: 345/168, 345/173, 345/773, 345/784, 341/22, 382/126, 382/159, 382/160, 382/161, 382/181, 715/812, 715/863, 715/864, 715/865, 715/866, 715/773, 715/784
• US Class Current: 345/173
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

A multi-touch system is disclosed that compares finger arrangements at the beginning of multi-touch gestures and discriminates between neutral- and spread-hand performances. Gestures by the neutral- and spread-hand versions of each chord are then mapped to separate, but intuitively related cursor functions or sets of commands. Less-frequently performed functions and commands are typically assigned to the spread-hand chord, since spreading the hand from neutral takes deliberate effort. With these neutral and spread distinctions, a two-handed gesture set can be packed into a one-handed touch surface, or for two-handed touch surfaces, twice as many intuitively customizable gesture command sets can be accomodated.

Claims

1. A method for doubling the number of distinct multi-touch gestures that can be mapped onto a hand, the method comprising the following steps:

waiting for multiple fingers to touch a surface apparatus;

classifying the combination of fingers that initially touch the surface as a particular chord;

measuring the geometric arrangement of the fingers;

testing whether this geometric arrangement matches that of a neutral, relaxed hand, or of a hand whose fingers have been spread deliberately;

if the finger arrangement tests neutral, selecting a neutral set of gesture functions or commands for the classified chord;

if the finger arrangement tests spread, selecting a spread set of gesture functions or commands for the classified chord that are intuitively related to the neutral set;

generating function or command signals to a host computer from the selected set in response to subsequent finger motions;

and, clearing the selected set and restarting the above steps after liftoff of all the fingers.

2. The method of claim 1, wherein geometric finger arrangement testing further comprises the following sub-steps:

constructing for each classifiable chord a pair of templates modeled upon contact coordinates from typical neutral and maximally spread performances, respectively, of that chord, and storing these templates in a gesture library;

retrieving from the gesture library the template pair associated with the particular chord classified by the combination of fingers initially touching, and storing this template pair as the working templates;

computing the total mismatch between coordinates of the initially touching fingers and the neutral working template;

computing the total mismatch between coordinates of the initially touching fingers and the spread working template;

and, classifying the finger arrangement as spread if the total mismatch to the spread working template is smaller, or neutral if the total mismatch to the neutral working template is smaller.

3. The method of claim 1, wherein both mouse cursor and text cursor manipulation capabilities are packed into one hand by mapping chords of two, three, and four neutrally adjacent fingertips to point, drag, and scroll mouse cursor functions, respectively, while mapping chords of two, three, and four spread fingertips to pointing the text cursor, selection via text cursor, and text paging functions, respectively.

4. The method of claim 1, wherein secondary and tertiary mouse button emulation are packed into a thumb and two fingertip chord by mapping clicks and drags by one of these buttons to taps and translations of a neutral thumb and two fingertip chord, while mapping clicks and drags by the other of these buttons to taps and translations of a spread thumb and two fingertip chord.

5. The method of claim 1, wherein local and global search and replace gestures are packed into a chord consisting of the thumb and a fixed number of fingertips by mapping flicks, pinches, and rotations that start with the hand neutral to local search and replace functions, while mapping equivalent motions that start with the hand spread to corresponding global search and replace functions.

6. The method of claim 1, wherein text styling and alignment gestures are packed into a chord consisting of the thumb and a fixed number of fingertips by intuitively mapping translational slides that start with the hand neutral to text styling functions, while mapping translational slides that start with the hand spread to alignment commands that intuitively match the direction of translation.

7. The method of claim 1, wherein fine and coarse text cursor manipulation gestures are packed into one hand by mapping:

horizontal slides of two neutrally adjacent fingertips to character-wise text cursor motion,

horizontal slides of two spread fingertips to word-wise text cursor motion,

vertical slides of two neutrally adjacent fingertips to line-wise text cursor motion,

vertical slides of two spread fingertips to paragraph-wise text cursor motion,

horizontal slides of three neutrally adjacent fingertips to character-wise text selection,

horizontal slides of three spread fingertips to word-wise text selection,

vertical slides of three neutrally adjacent fingertips to line-wise text selection,

vertical slides of three spread fingertips to paragraph-wise text selection,

horizontal slides of four neutrally adjacent fingertips to beginning and end of line commands,

horizontal slides of four spread fingertips to beginning and end of document commands,

vertical slides of four neutrally adjacent fingertips to page up and down commands,

and vertical slides of four spread fingertips to a special paging function.

8. The method of claim 1 wherein relaxed two-fingertip translations specifically map to mouse pointing commands while spread two-fingertip translations map to text pointing commands, relaxed three-fingertip translations map to mouse dragging while spread three-fingertip translations map to text selection via arrow keys, and wherein relaxed four-fingertip translations map to window panning or scrolling while spread four-fingertip leftward slides map to a cursor to beginning of line command, rightward slides map to a cursor to end of line command, upward slides map to the PageUp command, and downward slides map to the PageDown command.

9. The method of claim 1 wherein both file and application manipulation commands are packed into the multi-touch gesture mappings of one hand by mapping file manipulation commands to the relaxed, neutral version of a chord consisting of the thumb and a fixed number of fingertips, while mapping analogous application manipulation commands to performances of this chord that start with fingers spread.

10. The method of claim 9, wherein a clockwise rotation of the neutral chord fingers maps to the Close document command while a clockwise rotation of the spread chord fingers maps to the Exit application command, and a neutral counter-clockwise rotation maps to the Open document command while a spread counter-clockwise rotation maps to a command helpful in opening other applications.

11. The method of claim 9, wherein a contraction of the chord fingers starting from a relaxed posture maps to the Save document command while a contraction of the chord fingers starting from an outstretched posture maps to the Save As . . . document command, and wherein an expansion of the chord fingers starting from the relaxed posture maps to the New command while an expansion of the chord fingers starting from the outstretched posture maps to the Print command.

12. The method of claim 9, wherein a leftward slide of the relaxed chord fingers maps to a previous document or browse Back command while a leftward slide of the spread chord fingers maps to a switch to previous application command, and wherein a rightward slide of the relaxed chord fingers maps to a next document or browse Forward command while a spread rightward slide maps to a switch to next application command.

13. The method of claim 9, wherein a downward slide of the relaxed chord fingers maps to a document reload or refresh command while a downward slide of the spread chord fingers maps to an application window restore or minimize command, and wherein an upward slide of the relaxed chord fingers maps to a Parent irectory or browse Upward in a hierarchy command while an upward slide by spread fingers maps to an application window maximize command.

14. The method of claim 9, wherein the chord chosen for the document and application command gestures consists of the thumb and exactly three fingertips, and wherein to maximize gesture recognition accuracy, users are instructed to touch the surface with their thumb, index, middle, and ring fingers when performing the document manipulation gestures but touch their thumb, middle, ring and pinky fingers when performing application window manipulations.