Multi-touch system and method for emulating modifier keys via fingertip chords

• US RE40,153 E1
• Filed: 05/27/2005
• Issued: 03/18/2008
• Est. Priority Date: 02/10/2001
• Status: Expired due to Term

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Westerman, Wayne Carl, Elias, John Greer
• Primary Examiner(s): Eisen; Alexander
• Application Number:
  US11/140,529
• Time in Patent Office: 1,026 Days
• Field of Search: 345/156, 345168-179, 341 22- 26
• US Class Current: 345/173
• CPC Class Codes: G06F 17/24 Editing, e.g. insert/delete G06F17/22 takes precedence
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

A multi-touch system is disclosed that recognizes simultaneous touchdown of four fingers on, above, or below the home row of keys as a modifier chord and applies modifiers such as Shift, Ctrl, or Alt to subsequent touch activity until none of the chord fingertips remain touching. Touches by the thumb of the modifier chord hand that occur before any modifiable typing or clicking activity cause the modifier chord to be canceled and reinterpreted as hand resting. The Shift modifier may be released temporarily during thumb keypresses that are intermixed with typing of capitalized characters. Distributing the modifier chord touches across different zones or key rows selects multiple modifiers. In an alternative embodiment, different modifiers can be selected with different arrangements of the fingers relative to one another within the chord, irrespective of absolute hand alignment with the touch surface.

Claims

1. A multi-touch surface apparatus that interprets 4-finger chords performed concurrently with other input activity as modifiers of that input activity, thereby avoiding awkward thumb or pinky reaches for conventional modifier keys, reducing hand strain and reducing the tendency of hands to drift off of home row, the apparatus comprising:

multiple-touch sensing means that reports the locations and times of finger contacts with a surface,

modifier configuration means establishes modifier zones as horizontal bands across the surface and associates each modifier zone with one or more modifier types;

synchronization detection means that scans for modifier chord touchdowns consisting of substantially simultaneous touches by four and no more than four digits of a hand;

modifier zone selection means that selects modifier types for the chord according to which modifier zones said simultaneous touches fall within;

chord cancellation means that cancels a modifier chord if the remaining unsynchronized digit from its hand touches before other modifiable input activity commits the chord;

modifier applicability search means that commits the modifier chord and sends modifier press signals corresponding to the chord'"'"'s selected modifier types to a host computer upon detection of modifiable input activity that occurs substantially between touchdown and liftoff of the modifier chord; and

,

chord liftoff detection means that reports when none of the digits originally constituting the modifier chord are touching the surface and sends modifier release signals to the host computer accordingly.

2. The apparatus of claim 1 wherein the apparatus additionally comprises a typing recognition means and surface key layout with a home row of keys upon which the hands normally rest, wherein the four digits that can constitute a modifier chord do not include the thumb, and wherein Shift modifier zones are configured along the home row of keys and other modifier types are configured for zones substantially above and below the home row of keys.

3. The apparatus of claim 2 wherein any Shift signals arising from a committed modifier chord are temporarily released during transmission to the host of keypress signals from a thumb key touch and then re-pressed if the modifier chord has not yet lifted.

4. The apparatus of claim 1 wherein the modifier zone selection means compensates for the natural arch in a row of relaxed fingers by computing for each touch of the modifier chord a vertical offset from a default location of the finger whose identity a contact identification means has assigned to the touch.

5. The apparatus of claim 4 wherein the modifier zone selection means simultaneously selects the modifier types from each zone within which any of the simultaneous touches fall except if some of the simultaneous touches are bunched straddling the border between two zones, in which case the modifier type associated with the single zone that the average of the vertical offsets lies within is selected.

6. The apparatus of claim 1 wherein the locations of the modifier zones on the surface are indicated to the user via printing on the surface background colors or textures unique to each zone.

7. The apparatus of claim 1 wherein the locations of the modifier zones on the surface are indicated to the user via active surface display of background colors or textures unique to each zone.

8. A multi-touch surface apparatus that interprets certain finger chords performed concurrently with other input activity as modifiers of that input activity, thereby avoiding awkward thumb or pinky reaches for conventional modifier keys, reducing hand strain and reducing the need to reposition the hands, the apparatus comprising:

multiple-touch sensing means that reports the locations and times of finger contacts with a surface;

modifier configuration means that establishes finger arrangement templates and associates each template with one or more modifier types;

synchronization detection means that scans for modifier chord touchdowns consisting of substantially simultaneous touches by a predetermined combination of digits of a hand;

finger arrangement matching means that selects for the chord the modifier types associated with the template that most closely matches the arrangement of the modifier chord touches;

chord cancellation means that cancels a modifier chord if the remaining digits from its hand not included in the predetermined combination touch before other modifiable input activity commits the chord;

modifier applicability search means that commits the modifier chord and sends modifier press signals corresponding to the chord'"'"'s selected modifier types to a host computer upon detection of modifiable input activity that occurs substantially between touchdown and liftoff of the modifier chord; and

modifier chord liftoff detection means that reports when none of the digits originally constituting the modifier chord are touching the surface and sends modifier release signals to the host computer accordingly.

9. The apparatus of claim 8 wherein any Shift signals arising from a committed modifier chord are temporarily released during transmission to the host of keypress signals from thumb key touches and then re-pressed if the modifier chord has not yet lifted.

10. The apparatus of claim 8 wherein four fingertips excluding the thumb is used as the predetermined combination of digits, the neutral four fingertip arrangement of FIG. 16A is associated with the shift modifier type, and the horizontally spread arrangement template of FIG. 16B is associated with the Ctrl modifier type.

11. The apparatus of claim 8 wherein the template matching means computes the matching error as the sum of squared distances between each template point'"'"'s offset from template center and the offset to the corresponding modifier chord touch from chord center, thus making the matching process independent of where the chord is performed on the surface.

12. A method of controlling a modifier on a multi-touch surface having a key layout thereon, the method comprising:

activating the modifier by simultaneously dropping a plurality of fingertips of a hand into a modifier zone on or near a home row of the key layout; and

maintaining activation of the modifier by leaving at least one of the plurality of fingertips in contact with the modifier zone.

13. The method of claim 12 further comprising:

deactivating the modifier by removing each of the plurality of fingertips from contact with the modifier zone.

14. The method of claim 12 further comprising:

deactivating the modifier by placing a thumb of the hand in contact with the key layout.

15. The method of claim 12 wherein the modifier zone on or near a home row of the key layout may be reached through straight flexion or extension of the fingers from their home row position.

16. The method of claim 12 further comprising:

concurrently activating a second modifier key by simultaneously dropping a plurality of fingertips of the hand into two modifier zones on or near a home row of the key layout, wherein the fingertips are clearly distributed into the two modifier zones.

17. A multi-touch surface having embodied thereon a key layout, the key layout comprising at least one modifier zone on or near a home row of the key layout wherein a modifier will be activated by the simultaneous touchdown of a plurality of fingertips within the at least one modifier zone and maintained by leaving at least one of the plurality of fingertips in contact with the modifier zone.

18. The multi-touch surface of claim 17 comprising a plurality of modifier zones on or near a home row of the key layout, wherein each modifier zone corresponds to a different modifier.

19. The multi-touch surface of claim 18 wherein each of the plurality of modifier zones is indicated by a different color.

20. A method of processing incoming signals from a multi-touch surface, the method comprising:

detecting modifier chords by monitoring the incoming touch sequence for simultaneous touchdowns of a plurality of fingertips of a hand within one or more predetermined modifier zones; and

, if a new modifier chord is detected;

allocating a corresponding modifier state data structure; and

appending the modifier state data structure to a modifier chord state chain; and

checking for cancellation of a modifier chord; and

, if cancellation of a modifier chord is detected;

recording either a modifier chord liftoff time or modifier chord cancellation.

21. The method of claim 20 further comprising:

searching a modifier state chain for un-cancelled modifier chords roughly coincident with modifiable input activity; and

, if any applicable modifier chords whose signals have not already been sent to a host computer are identified;

sending the modifier signals to the host computer.

22. A multi-touch surface apparatus comprising:

a multi-touch surface that reports the locations and times of finger contacts with a surface;

a synchronization detector that scans for modifier chord touchdowns consisting of substantially simultaneous touches by a plurality of digits of a hand in a pre-defined modifier zone on the multi-touch surface and sends modifier signals accordingly; and

a chord liftoff detector that reports when none of the digits originally constituting the modifier chord are touching the surface and sends modifier release signals accordingly.

23. The multi-touch surface apparatus of claim 22 wherein there are a plurality of pre-defined modifier zones on the multi-touch surface, each pre-defined modifier zone corresponding to a different modifier, the multi-touch surface apparatus further comprising:

a modifier zone selector that selects modifier types for the chord according to which modifier zones the simultaneous touches fall within.

24. The multi-touch surface apparatus of claim 23 further comprising:

a chord canceller that cancels a modifier chord if a remaining unsynchronized digit from the hand touches the multi-touch surface before other modifiable input activity commits the chord.

25. The multi-touch surface apparatus of claim 22 further comprising:

a chord canceller that cancels a modifier chord if a remaining unsynchronized digit from the hand touches the multi-touch surface before other modifiable input activity commits the chord.

26. The apparatus of any of claims 22-25 further comprising a key layout on the multi-touch surface, the key layout having a home row wherein a first modifier zone is configured along the home row.

27. The apparatus of claim 26 wherein at least one additional modifier zone is configured adjacent the home row.

28. The apparatus of claim 27 wherein the at least one additional modifier zone comprises a second modifier zone above the home row and a third modifier zone below the home row.

29. The apparatus of claim 28 wherein the modifiers corresponding to the first, second, and third modifier zones are selected from the group consisting of Shift, Ctrl, Alt, Windows, Open Apple, or Meta.

30. The apparatus of claim 26 wherein the location of the modifier zone is indicated by printing on the multi-touch surface.

31. The apparatus of claim 30 wherein the printing on the multi-touch surface comprises background color.

32. The apparatus of claim 26 wherein the location of the modifier zone is indicated by texture on the multi-touch surface.

33. The apparatus of any of claims 27-29, wherein the locations of the modifier zones are indicated by printing on the multi-touch surface.

34. The apparatus of claim 33 wherein the printing on the multi-touch surface comprises a unique background color for each zone.

35. The apparatus of claim 27-29 wherein the locations of the modifier zones are indicated by texture on the multi-touch surface.

36. The apparatus of claim 26 wherein the location of the modifier zone is indicated by active surface display of background color.

37. The apparatus of claim 26 wherein the location of the modifier zone is indicated by active surface display of texture.

38. The apparatus of any of claims 27-29 wherein the locations of the modifier zones are indicated by active surface display of background colors unique to each zone.

39. The apparatus of claim 27-29 wherein the locations of the modifier zones are indicated active surface display of textures unique to each zone.

40. A multi-touch surface apparatus comprising:

a multi-touch surface that reports the locations and times of finger contacts with a surface;

a synchronization detector that scans for modifier chord touchdowns consisting of substantially simultaneous touches by a predetermined combination of digits of a hand in a predetermined arrangement;

finger arrangement matcher that selects a modifier corresponding to the modifier chord and sends modifier signals accordingly; and

a chord liftoff detector that reports when none of the digits originally constituting the modifier chord are touching the surface and sends modifier release signals accordingly.

41. The multi-touch surface apparatus of claim 40 further comprising:

a chord canceller that cancels a modifier chord if the remaining digits from its hand not include in the predetermined combination touch before other modifiable input activity commits the chord.

42. The multi-touch surface apparatus of claim 40 or 41 wherein the predetermined arrangement is selected from the group consisting of a neutral fingertip arrangement and a spread fingertip arrangement.

43. The multi-touch surface apparatus of claim 42 wherein the modifiers associated with the predetermined arrangement are selected from the group consisting of:

Shift, Ctrl, Alt, Windows, Open Apple, or Meta.