Resting contacts

• US 9,448,658 B2
• Filed: 07/30/2007
• Issued: 09/20/2016
• Est. Priority Date: 01/26/1998
• Status: Expired due to Term

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Westerman, Wayne, Elias, John G.
• Primary Examiner(s): Olson, Jason
• Assistant Examiner(s): Abebe, Sosina
• Application Number:
  US11/830,815
• Publication Number:
  US 20070268275A1
• Time in Patent Office: 3,340 Days
• Field of Search: 345173-178, 178 1801- 1811
• US Class Current: 1/1
• CPC Class Codes: G06F 18/24137 Distances to cluster centroïds
• CPC Class Codes: G06F 2203/04101 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup
• CPC Class Codes: G06F 2203/04104 Multi-touch detection in digitiser, i.e. details about the simultaneous detection of a plurality of touching locations, e.g. multiple fingers or pen and finger
• CPC Class Codes: G06F 2203/04106 Multi-sensing digitiser, i.e. digitiser using at least two different sensing technologies simultaneously or alternatively, e.g. for detecting pen and finger, for saving power or for improving position detection
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/0235 using chord techniques G06F3/0234 takes precedence
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/0412 Digitisers structurally integrated in a display
• CPC Class Codes: G06F 3/04166 Details of scanning methods, e.g. sampling time, grouping of sub areas or time sharing with display driving Synchronisation with the driving of the display or the backlighting unit to avoid interferences generated internally G06F3/04184
• CPC Class Codes: G06F 3/04186 Touch location disambiguation
• CPC Class Codes: G06F 3/0446 using a grid-like structure of electrodes in at least two directions, e.g. using row and column electrodes
• CPC Class Codes: G06F 3/0447 Position sensing using the local deformation of sensor cells
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06V 30/19173 Classification techniques
• CPC Class Codes: G06V 30/1983 Syntactic or structural pattern recognition, e.g. symbolic string recognition
• CPC Class Codes: G06V 40/107 : Static hand or arm
• CPC Class Codes: H03K 17/9622 : using a plurality of detectors, e.g. keyboard
• CPC Class Codes: H03K 2217/960725 : Charge-transfer

Abstract

Apparatus and methods are disclosed for simultaneously tracking multiple finger and palm contacts as hands approach, touch, and slide across a proximity-sensing, multi-touch surface. Identification and classification of intuitive hand configurations and motions enables unprecedented integration of typing, resting, pointing, scrolling, 3D manipulation, and handwriting into a versatile, ergonomic computer input device.

Claims

1. A method of selecting an input activity of a computing system including a touch sensing surface, the method comprising:

obtaining tracking information of contacts, the contacts corresponding to a plurality of hand parts contacting the touch sensing surface, the tracking information including first tracking information of a first hand part contact, the first tracking information including touchdown information of the touchdown of the first hand part contact, liftoff information of the liftoff of the first hand part contact, and lateral motion information of lateral motion of the first hand part contact between the touchdown and liftoff of the hand part first contact, and the tracking information including second tracking information of a second hand part contact, the second tracking information including touchdown information of the touchdown of the second hand part contact, liftoff information of the liftoff of the second hand part contact, and lateral motion information of lateral motion of the second hand part contact between the touchdown and liftoff of the second hand part contact;

determining whether the first hand part contact based on the touchdown information of the first hand part contact;

determining whether the second hand part contact is a resting contact based on the touchdown information of the second hand part contact; and

periodically processing the tracking information, the processing including selection of input activities of the computing system, the selection of an input activity being based on the tracking information, wherein

in accordance with a determination that the first hand part contact is not a resting contact and the second hand part contact is a resting contact, the periodic processing selects a first input activity based on at least one of the touchdown information, the lateral motion information, and the liftoff information of the first hand part contact and excludes the second tracking information corresponding to the second hand part contact from the selection of input activities; and

in accordance with a determination that the first hand part contact is a resting contact and the second hand part contact is not a resting contact, the periodic processing selects a second input activity based on at least one of the touchdown information, the lateral motion information, and the liftoff information of the second hand part contact and excludes the first tracking information corresponding to the first hand part contact from the selection of input activities.

2. The method of claim 1, wherein the touchdown information of the first hand part contact includes a first value corresponding to a force of the touchdown of the first hand part contact, and determining whether the first hand part contact is a resting contact includes comparing the first value to a predetermined threshold, and determining that the first hand part contact is a resting contact if the first value is less than the predetermined threshold.

3. The method of claim 2, wherein the first value includes a first impulsivity of the touchdown of the first hand part contact, and the predetermined threshold is a threshold impulsivity.

4. The method of claim 3, wherein the periodic processing selects the first input activity if the first impulsivity is greater than the threshold impulsivity.

5. The method of claim 1, wherein the touchdown information of the first hand part contact includes a proximity value of the first hand part contact, and determining whether the first hand part contact is a resting contact includes comparing the proximity value to a predetermined proximity threshold, and determining that the first hand part contact is a resting contact if the proximity value exceeds the predetermined proximity value.

6. The method of claim 1, wherein the touchdown information of the first hand part contact includes a first proximity value at a first time during the touchdown of the first hand part contact, and a second proximity value at a second time, after the first time, during the touchdown of the first hand part contact, and determining whether the first hand part contact is a resting contact includes determining that the first proximity value is less than a predetermined proximity threshold, and determining that the second proximity value exceeds the predetermined proximity threshold, the method further comprising:

cancelling the selected first input activity based on the second proximity value exceeding the predetermined proximity threshold.

7. The method of claim 6, wherein selecting the first input activity includes entering a first input event of the first input activity into an input event queue, and cancelling the selected first input activity includes deleting the first input event from the input event queue.

8. The method of claim 1, wherein the touchdown information of the first hand part contact includes a touchdown time of the first hand part contact, and determining whether the first hand part contact is a resting contact includes determining whether the touchdown time of the first hand part contact is synchronous with a touchdown time of the second hand part contact, the touchdown times being synchronous if a difference between the touchdown times of the first and second hand part contacts is less than a predetermined temporal threshold, and the touchdown times being asynchronous if the difference exceeds the predetermined temporal threshold.

9. The method of claim 8, wherein the first and second hand part contacts are included in a synchronized set of two or more contacts based on a determination that the touchdown times of the first and second hand part contacts are synchronous, and determining whether the first hand part contact is a resting contact further includes determining a third input activity based on the number of contacts in the synchronized set, and determining that the first hand part contact is a resting contact if the third input activity is an ignored input activity, the ignored input activity being an input activity that is predetermined to generate no input events.

10. The method of claim 8, wherein the periodic processing includes selecting a third input activity prior to the touchdown of the first hand part contact and generating input events of the third input activity based on the tracking information of one or more previous contacts that touched down prior to the first hand part contact, wherein determining whether the first hand part contact is a resting contact includes determining whether the generation of the input events of the third input activity is ongoing during the touchdown of the first hand part contact, and determining that the first hand part contact is a resting contact if the generation of the input events of the third input activity is ongoing.

11. The method of claim 10, the selection of the third input activity being based on touchdowns of two or more previous contacts that touched down prior to the first hand part contact, wherein selecting the third input activity includes determining that the two or more previous contacts are included in a synchronized set, and selecting the third input activity based on the number of contacts in the synchronized set, the first hand part contact touching down asynchronously with the contacts in the synchronized set.

12. The method of claim 11, wherein selecting the third input activity based on the number of contacts in the synchronized set further includes selecting the third input activity based on an arrangement of the contacts in the synchronized set.

13. The method of claim 10, wherein generating the input events of the third input activity includes determining, based on the lateral motion information of the one or more previous contacts, one of a translational motion component, a rotational motion component, and a scaling motion component, wherein the generation of the input events is based on the determined motion component.

14. The method of claim 10, wherein the generation of the input events of the third input activity is based on the lateral motion information of each of the one or more previous contacts, and the periodic processing further includes generating the input events of the third input activity based on the lateral motion information of the first hand part contact, such that the generation of the input events is based on the lateral motion information of the one or more previous contacts and the first hand part contact.

15. The method of claim 14, wherein generating the input events of the third input activity includes determining a motion component based on the lateral motion information, wherein the lateral motion of the first contact accesses a greater dynamic range of the motion component than the dynamic range of the motion component that was based on the lateral motion of the one or more previous contacts prior to the touchdown of the first hand part contact.

16. The method of claim 1, wherein the first input activity includes a keypress input activity of a keypress input region corresponding to a touchdown location of the first hand part contact.

17. A computing system including a touch sensing surface, the computing system comprising:

one or more processors;

a non-transitory computer-readable storage medium storing instructions, which when executed by the one or more processors, cause the processor to perform a method of selecting an input activity, the method comprising;

obtaining tracking information of contacts, the contacts corresponding to a plurality of hand parts contacting the touch sensing surface, the tracking information including first tracking information of a first hand part contact, the first tracking information including touchdown information of the touchdown of the first hand part contact, liftoff information of the liftoff of the first hand part contact, and lateral motion information of lateral motion of the first hand part contact between the touchdown and liftoff of the hand part first contact, and the tracking information including second tracking information of a second hand part contact, the second tracking information including touchdown information of the touchdown of the second hand part contact, liftoff information of the liftoff of the second hand part contact, and lateral motion information of lateral motion of the second hand part contact between the touchdown and liftoff of the second hand part contact;

determining whether the first hand part contact is a resting contact based on the touchdown information of the first hand part contact;

determining whether the second hand part contact is a resting contact based on the touchdown information of the second hand part contact; and

periodically processing the tracking information, the processing including selection of input activities of the computing system, the selection of an input activity being based on the tracking information, wherein

in accordance with a determination that the first hand part contact is not a resting contact and the second hand part contact is a resting contact, the periodic processing selects a first input activity based on at least one of the touchdown information, the lateral motion information, and the liftoff information of the first hand part contact and excludes the second tracking information corresponding to the second hand part contact from the selection of input activities, and

in accordance with a determination that the first hand part contact is a resting contact and the second hand part contact is not a resting contact, the periodic processing selects a second input activity based on at least one of the touchdown information, the lateral motion information, and the liftoff information of the second hand part contact and excludes the first tracking information corresponding to the first hand part contact from the selection input activities.

18. The computing system of claim 17, wherein the touchdown information of the first hand part contact includes a first value corresponding to a force of the touchdown of the first hand part contact, and determining whether the first hand part contact is a resting contact includes comparing the first value to a predetermined threshold, and determining that the first hand part contact is a resting contact if the first value is less than the predetermined threshold.

19. The computing system of claim 18, wherein the first value includes a first impulsivity of the touchdown of the first hand part contact, and the predetermined threshold is a threshold impulsivity.

20. The computing system of claim 19, wherein the periodic processing includes selecting the first input activity if the first impulsivity is greater than the threshold impulsivity.

21. The computing system of claim 17, wherein the touchdown information of the first hand part contact includes a proximity value of the first hand part contact, and determining whether the first hand part contact is a resting contact includes comparing the proximity value to a predetermined proximity threshold, and determining that the first hand part contact is a resting contact if the proximity value exceeds the predetermined proximity value.

22. The computing system of claim 17, wherein the touchdown information of the first hand part contact includes a first proximity value at a first time during the touchdown of the first hand part contact, and a second proximity value at a second time, after the first time, during the touchdown of the first hand part contact, and determining whether the first hand part contact is a resting contact includes the first proximity value is less than a predetermined proximity threshold, and determining that the second proximity value exceeds the predetermined proximity threshold, the method further comprising:

cancelling the selected first input activity based on the second proximity value exceeding the predetermined proximity threshold.

23. The computing system of claim 22, wherein selecting the first input activity includes entering a first input event of the first input activity into an input event queue, and cancelling the selected first input activity includes deleting the first input event from the input event queue.

24. The computing of claim 17, wherein the touchdown information of the first hand part contact includes a touchdown time of the first hand part contact, and determining whether the first hand part contact is a resting contact includes determining whether the touchdown time of the first hand part contact is synchronous with a touchdown time of the second hand part contact, the touchdown times being synchronous if a difference between the touchdown times of the first and second hand part contacts is less than a predetermined temporal threshold, and the touchdown times being asynchronous if the difference exceeds the predetermined temporal threshold.

25. The computing system of claim 24, wherein the first and second hand part contacts are included in a synchronized set of two or more contacts based on a determination that the touchdown times of the first and second hand part contacts are synchronous, and determining whether the first hand part contact is a resting contact further includes determining a third input activity based on the number of contacts in the synchronized set, and determining that the first hand part contact is a resting contact if the third input activity is an ignored input activity, the ignored input activity being an input activity that is predetermined to generate no input events.

26. The computing system of claim 24, wherein the periodic processing includes selecting a third input activity prior to the touchdown of the first hand part contact and generating input events of the third input activity based on the tracking information of one or more previous contacts that touched down prior to the first hand part contact, wherein determining whether the first hand part contact is a resting contact includes determining whether the generation of the input events of the third input activity is ongoing during the touchdown of the first hand part contact, and determining that the first hand part contact is a resting contact if the generation of the input events of the third input activity is ongoing.

27. The computing device of claim 26, wherein the selection of the third input activity based on touchdowns of two or more previous contacts that touched down prior to the first hand part contact, and selecting the third input activity includes determining that the two or more previous contacts are included in a synchronized set, and selecting the third input activity based on the number of contacts in the synchronized set, the first hand part contact touching down asynchronously with the contacts in the synchronized set.

28. The computing system of claim 27, wherein selecting the third input activity based on the number of contacts in the synchronized set further includes selecting the third input activity based on an arrangement of the contacts in the synchronized set.

29. The computing system of claim 26, wherein generating the input events of the third input activity includes determining, based on the lateral motion information of the one or more previous contacts, one of a translational motion component, a rotational motion component, and a scaling motion component, wherein the generation of the input events is based on the determined motion component.

30. The computing system of claim 26, wherein the generation of the input events of the third input activity is based on the lateral motion information of each of the one or more previous contacts, and the periodic processing further includes generating the input events of the third input activity based on the lateral motion information of the first hand part contact, such that the generation of the input events is based on the lateral motion information of the one or more previous contacts and the first hand part contact.

31. The computing system of claim 30, wherein generating the input events of the third input activity includes determining a motion component based on the lateral motion information, wherein the lateral motion of the first contact accesses a greater dynamic range of the motion component than the dynamic range of the motion component that was based on the lateral motion of the one or more previous contacts prior to the touchdown of the first hand part contact.

32. The computing system of claim 17, wherein the first input activity includes a keypress input activity of a keypress input region corresponding to a touchdown location of the first hand part contact.

33. A non-transitory computer-readable storage medium storing computer-executable instructions executable to perform a method of selecting an input activity of a computing system including a touch sensing surface, the method comprising:

obtaining tracking information of contacts, the contacts corresponding to a plurality of hand parts contacting the touch sensing surface, the tracking information including first tracking information of a first hand part contact, the first tracking information including touchdown information of the touchdown of the first hand part contact, liftoff information of the liftoff of the first hand part contact, and lateral motion information of lateral motion of the first hand part contact between the touchdown and liftoff of the hand part first contact, and the tracking information including second tracking information of a second hand part contact, the second tracking information including touchdown information of the touchdown of the second hand part contact, liftoff information of the liftoff of the second hand part contact, and lateral motion information of lateral motion of the second hand part contact between the touchdown and liftoff of the second hand part contact;

determining whether the first hand part contact is a resting contact based on the touchdown information of the first hand part contact;

determining whether the second hand part contact is a resting contact based on the touchdown information of the second hand part contact; and

periodically processing the tracking information, the processing including selection of input activities of the computing system, the selection of an input activity being based on the tracking information, wherein

in accordance with a determination that the first hand part contact is not a resting contact and the second hand part contact is a resting contact, the periodic processing selects a first input activity based on at least one of the touchdown information, the lateral motion information, and the liftoff information of the first hand part contact and excludes the second tracking information corresponding to the second hand part contact from the selection of input activities; and

in accordance with a determination that the first hand part contact is a resting contact and the second hand part contact is not a resting contact, the periodic processing selects a second input activity based on at least one of the touchdown information, the lateral motion information, and the liftoff information of the second hand part contact and excludes the first tracking information corresponding to the first hand part contact from the selection of input activities.

34. The non-transitory computer-readable storage medium of claim 33, wherein the touchdown information of the hand part first contact includes a first value corresponding to a force of the touchdown of the first hand part contact, and determining whether the first hand part contact is a resting contact includes comparing the first value to a predetermined threshold, and determining that the first hand part contact is a resting contact if the first value is less than the predetermined threshold.

35. The non-transitory computer-readable storage medium of claim 34, wherein the first value includes a first impulsivity of the touchdown of the first hand part contact, and the predetermined threshold is a threshold impulsivity.

36. The non-transitory computer-readable storage medium of claim 35, wherein the periodic processing selects the first input activity if the first impulsivity is greater than the threshold impulsivity.

37. The non-transitory computer-readable storage medium of claim 33, wherein the touchdown information of the hand part first contact includes a proximity value of the first hand part contact, and determining whether the first hand part contact is a resting contact includes comparing the proximity value to a predetermined proximity threshold, and determining that the first hand part contact is a resting contact if the proximity value exceeds the predetermined proximity value.

38. The non-transitory computer-readable storage medium of claim 33, wherein the touchdown information of the first hand part contact includes a first proximity value at a first time during the touchdown of the first hand part contact, and a second proximity value at a second time, after the first time, during the touchdown of the first hand part contact, and determining whether the first hand part contact is a resting contact includes determining that the first proximity value is less than a predetermined proximity threshold, and determining that the second proximity value exceeds the predetermined proximity threshold, the method further comprising:

cancelling the selected first input activity based on the second proximity value exceeding the predetermined proximity threshold.

39. The non-transitory computer-readable storage medium of claim 38, wherein selecting the first input activity includes entering a first input event of the first input activity into an input event queue, and cancelling the selected first input activity includes deleting the first input event from the input event queue.

40. The non-transitory computer-readable storage medium of claim 33, wherein the touchdown information of the first hand part contact includes a touchdown time of the first hand part contact, and determining whether the first hand part contact is a resting contact includes determining whether the touchdown time of the first hand part contact is synchronous with a touchdown time of the second hand part contact, the touchdown times being synchronous if a difference between the touchdown times of the first and second hand part contacts is less than a predetermined temporal threshold, and the touchdown times being asynchronous if the difference exceeds the predetermined temporal threshold.

41. The non-transitory computer-readable storage medium of claim 40, wherein the first and second hand part contacts are included in a synchronized set of two or more contacts based on a determination that the touchdown times of the first and second hand part contacts are synchronous, and determining whether the first hand part contact is a resting contact further includes determining a third input activity based on the number of contacts in the synchronized set, and determining that the first hand part contact is a resting contact if the third input activity is an ignored input activity, the ignored input activity being an input activity that is predetermined to generate no input events.

42. The non-transitory computer-readable storage medium of claim 40, wherein the periodic processing includes selecting a third input activity prior to the touchdown of the first hand part contact and generating input events of the third input activity based on the tracking information of one or more previous contacts that touched down prior to the first hand part contact, wherein determining whether the first hand part contact is a resting contact includes determining whether the generation of the input events of the third input activity is ongoing during the touchdown of the first hand part contact, and determining that the first hand part contact is a resting contact if the generation of the input events of the third input activity is ongoing.

43. The non-transitory computer-readable storage medium of claim 42, the selection of the third input activity being based on touchdowns of two or more previous contacts that touched down prior to the first hand part contact, wherein selecting the third input activity includes determining that the two or more previous contacts are included in a synchronized set, and selecting the third input activity based on the number of contacts in the synchronized set, the first hand part contact touching down asynchronously with the contacts in the synchronized set.

44. The non-transitory computer-readable storage medium of claim 43, wherein selecting the third input activity based on the number of contacts in the synchronized set further includes selecting the third input activity based on an arrangement of the contacts in the synchronized set.

45. The non-transitory computer-readable storage medium of claim 42, wherein generating the input events of the third input activity includes determining, based on the lateral motion information of the one or more previous contacts, one of a translational motion component, a rotational motion component, and a scaling motion component, wherein the generation of the input events is based on the determined motion component.

46. The non-transitory computer-readable storage medium of claim 42, wherein the generation of the input events of the third input activity is based on the lateral motion information of each of the one or more previous contacts, and the periodic processing further includes generating the input events of the third input activity based on the lateral motion information of the first hand part contact, such that the generation of the input events is based on the lateral motion information of the one or more previous contacts and the first hand part contact.

47. The non-transitory computer-readable storage medium of claim 46, wherein generating the input events of the third input activity includes determining a motion component based on the lateral motion information, wherein the lateral motion of the first contact accesses a greater dynamic range of the motion component than the dynamic range of the motion component that was based on the lateral motion of the one or more previous contacts prior to the touchdown of the first hand part contact.

48. The non-transitory computer-readable storage medium of claim 33, wherein the first input activity includes a keypress input activity of a keypress input region corresponding to a touchdown location of the first hand part contact.