Writing using a touch sensor

• US 9,348,452 B2
• Filed: 04/10/2009
• Issued: 05/24/2016
• Est. Priority Date: 01/26/1998
• Status: Expired due to Fees

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Westerman, Wayne, Elias, John G.
• Primary Examiner(s): Olson, Jason
• Assistant Examiner(s): Abebe, Sosina
• Application Number:
  US12/422,205
• Publication Number:
  US 20100149134A1
• Time in Patent Office: 2,601 Days
• Field of Search: 345156-184
• US Class Current: 1/1
• CPC Class Codes: G06F 18/24137 Distances to cluster centroïds
• CPC Class Codes: G06F 2203/04101 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup
• CPC Class Codes: G06F 2203/04104 Multi-touch detection in digitiser, i.e. details about the simultaneous detection of a plurality of touching locations, e.g. multiple fingers or pen and finger
• CPC Class Codes: G06F 2203/04106 Multi-sensing digitiser, i.e. digitiser using at least two different sensing technologies simultaneously or alternatively, e.g. for detecting pen and finger, for saving power or for improving position detection
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/0235 using chord techniques G06F3/0234 takes precedence
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/0412 Digitisers structurally integrated in a display
• CPC Class Codes: G06F 3/04166 Details of scanning methods, e.g. sampling time, grouping of sub areas or time sharing with display driving Synchronisation with the driving of the display or the backlighting unit to avoid interferences generated internally G06F3/04184
• CPC Class Codes: G06F 3/04186 Touch location disambiguation
• CPC Class Codes: G06F 3/0446 using a grid-like structure of electrodes in at least two directions, e.g. using row and column electrodes
• CPC Class Codes: G06F 3/0447 Position sensing using the local deformation of sensor cells
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06V 30/19173 Classification techniques
• CPC Class Codes: G06V 30/1983 Syntactic or structural pattern recognition, e.g. symbolic string recognition
• CPC Class Codes: G06V 40/107 : Static hand or arm
• CPC Class Codes: H03K 17/9622 : using a plurality of detectors, e.g. keyboard
• CPC Class Codes: H03K 2217/960725 : Charge-transfer

Abstract

Apparatus and methods are disclosed for simultaneously tracking multiple finger and palm contacts as hands approach, touch, and slide across a proximity-sensing, multi-touch surface. Identification and classification of intuitive hand configurations and motions enables unprecedented integration of typing, resting, pointing, scrolling, 3D manipulation, and handwriting into a versatile, ergonomic computer input device.

Claims

1. A method of a computing system that has a surface, the method comprising:

detecting a first set of one or more contacts, each contact corresponding to a hand part touching a touch sensor surface of the computing system, the touch sensor surface being a portion of the surface of the computing system on which the computing system detects contacts, wherein commands of the computing system are selected based on detected contacts;

selecting a first command based on the first set;

detecting a touch down of each contact of a second set of one or more contacts on the touch sensor surface while the first set remains touching the touch sensor surface, each contact of the second set corresponding to a hand part touching the touch sensor surface; and

ignoring the second set by maintaining the first command after the touch down of the second set and by not selecting a command of the computing system based on the second set.

2. The method of claim 1, wherein a hand part includes a finger.

3. The method of claim 1, further comprising:

detecting a liftoff of each contact of the first set from the touch sensor surface, while the second set remains touching the touch sensor surface; and

generating input of the first command based on the second set after the liftoff of the first set.

4. The method of claim 1, further comprising:

obtaining first information of the second set; and

determining, based on the first information, to ignore the second set.

5. The method of claim 4, wherein the first information includes a size of each contact of the second set.

6. The method of claim 5, wherein the second set consists of a single contact and determining to ignore the second set includes determining whether the size of the single contact exceeds a predetermined threshold size.

7. The method of claim 6, wherein the size of the single contact is based on a total proximity of the single contact.

8. The method of claim 4, wherein the first information includes a ratio of a total proximity of each contact of the second set to an eccentricity of each contact of the second set.

9. The method of claim 4, wherein the first information includes a position of each contact of the second set.

10. The method of claim 9, wherein the position of each contact of the second set is a relative position of the contact and another contact touching the touch sensor surface.

11. The method of claim 10, wherein the relative position is based on a distance from the contact to a nearest neighbor contact of the contact.

12. The method of claim 10, wherein the relative position is based on a distance from the contact to a position in a template of contact positions.

13. The method of claim 4, wherein the first information includes a shape parameter of each contact of the second set.

14. The method of claim 13, wherein the shape parameter includes an eccentricity of the contact.

15. The method of claim 4, wherein the first information includes an orientation parameter of each contact of the second set.

16. The method of claim 15, wherein the orientation parameter includes an orientation of an axis of an ellipse corresponding to the contact.

17. The method of claim 4, wherein determining the first information includes determining that each contact of the second set is not a member of the first set.

18. The method of claim 17, wherein the first set includes a plurality of contacts, the method further comprising:

determining that the contacts of the first set touched down synchronously, wherein selecting the first command based on the first set includes selecting the first command based the determination that the contacts of the first set touched down synchronously, and determining that each contact of the second set is not a member of the first set includes determining that at least one contact of the second set touched down asynchronously to at least one contact of the first set.

19. The method of claim 1, wherein the first set consists of a single contact.

20. The method of claim 19, wherein the first command is a key press, selecting the first command based on the first set includes placing the key press in a queue, and the second set consists of a single contact, the method further comprising:

determining that the contact of the second set touched down within a predetermined period of time after the touch down of the contact of the first set, such that the touch downs of the contacts of the first and second sets are synchronized, wherein ignoring the second set includes determining that a size of the contact of the second set exceeds a predetermined size, and maintaining the selection of the first command includes maintaining the key press in the queue.

21. The method of claim 1, wherein the second set includes a plurality of contacts.

22. The method of claim 21, further comprising:

determining that the contacts of the second set touched down asynchronously.

23. The method of claim 21, further comprising:

determining that the contacts of the second set touched down synchronously; and

determining, based on the determination that the contacts of the second set touched down synchronously, that the contacts of the second set are resting.

24. A computing system that has a surface, the computing system comprising:

a touch sensor surface including an array of sensors, each sensor providing an indication of proximity of a hand part to the sensor surface, the touch sensor surface being a portion of the surface of the computing system on which the computing system detects contacts, wherein commands of the computing system are selected based on the detected contacts;

a scanner that scans the array of sensors to determine the indications of proximity and that generates a succession of proximity images based on the indications of proximity;

a image segmentation system that detects a plurality of contacts in each of the succession of proximity images, wherein the image segmentation system detects a first set of one or more contacts, each contact corresponding to a hand part touching the touch sensor surface; and

a command selector that selects a first command based on the first set,

wherein the image segmentation system further detects a touch down of each contact of a second set of one or more contacts on the touch sensor surface while the first set remains touch the touch sensor surface, each contact of the second set corresponding to a hand part touching the touch sensor surface, and

wherein the command selector further ignores the second set by maintaining the first command after the touch down of the second set and by not selecting a command of the computing system based on the second set.

25. The computing system of claim 24, wherein a hand part includes a finger.

26. The computing system of claim 24, further comprising:

a liftoff detector that detects a liftoff of each contact of the first set from the touch sensor surface, while the second set remains touching the touch sensor surface; and

an input generator that generates input of the first command based on the second set after the liftoff of the first set.

27. The computing system of claim 24, further comprising:

a contact tracking and identification system configured to obtain first information of the second set; and

wherein the command selector is configured to determine, based on the first information, to ignore the second set.

28. The computing system of claim 27, wherein the first information includes a size of each contact of the second set.

29. The computing system of claim 28, wherein the second set consists of a single contact and determining to ignore the second set includes determining whether the size of the single contact exceeds a predetermined threshold size.

30. The computing system of claim 29, wherein the size of the single contact is based on a total proximity of the single contact.

31. The computing system of claim 27, wherein the first information includes a ratio of a total proximity of each contact of the second set to an eccentricity of each contact of the second set.

32. The computing system of claim 27, wherein the first information includes a position of each contact of the second set.

33. The computing system of claim 32, wherein the position of each contact of the second set is a relative position of the contact and another contact touching the touch sensor surface.

34. The computing system of claim 33, wherein the relative position is based on a distance from the contact to a nearest neighbor contact of the contact.

35. The computing system of claim 33, wherein the relative position is based on a distance from the contact to a position in a template of contact positions.

36. The computing system of claim 27, wherein the first information includes a shape parameter of each contact of the second set.

37. The computing system of claim 36, wherein the shape parameter includes an eccentricity of the contact.

38. The computing system of claim 27, wherein the first information includes an orientation parameter of each contact of the second set.

39. The computing system of claim 38, wherein the orientation parameter includes an orientation of an axis of an ellipse corresponding to the contact.

40. The computing system of claim 27, wherein determining the first information includes determining that each contact of the second set is not a member of the first set.

41. The computing system of claim 40, wherein the first set includes a plurality of contacts, the computing system further comprising:

a synchronization detector configured to determine that the contacts of the first set touched down synchronously, wherein selecting the first command based on the first set includes selecting the first command based the determination that the contacts of the first set touched down synchronously, and determining that each contact of the second set is not a member of the first set includes determining that at least one contact of the second set touched down asynchronously to at least one contact of the first set.

42. The computing system of claim 24, wherein the first set consists of a single contact.

43. The computing system of claim 42, wherein the first command is a key press, selecting the first command based on the first set includes placing the key press in a queue, and the second set consists of a single contact, the computing system further comprising:

a synchronization detector configured to determine that the contact of the second set touched down within a predetermined period of time after the touch down of the contact of the first set, such that the touch downs of the contacts of the first and second sets are synchronized, wherein ignoring the second set includes determining that a size of the contact of the second set exceeds a predetermined size, and maintaining the selection of the first command includes maintaining the key press in the queue.

44. The computing system of claim 24, wherein the second set includes a plurality of contacts.

45. The computing system of claim 44, further comprising:

a synchronization detector configured to determine that the contacts of the second set touched down asynchronously.

46. The computing system of claim 44, further comprising:

a synchronization detector configured to determine that the contacts of the second set touched down synchronously, wherein the contacts of the second set are determined to be resting based on the determination that the contacts of the second set touched down synchronously.

51. The non-transitory computer-readable storage medium of claim 30, wherein the first information includes a size of each contact of the second set.

54. The non-transitory computer-readable storage medium of claim 51, wherein the second set consists of a single contact and determining to ignore the second set includes determining whether the size of the single contact exceeds a predetermined threshold size.

55. The non-transitory computer-readable storage medium of claim 54, wherein the size of the single contact is based on a total proximity of the single contact.

47. A non-transitory computer-readable storage medium storing computer-executable instructions executable to perform a method of a computing system that has surface, the method comprising:

detecting a first set of one or more contacts, each contact corresponding to a hand part touching a touch sensor surface of the computing system, the touch sensor surface being a portion of the surface of the computing system on which the computing system detects contacts, wherein commands of the computing system are selected based on detected contacts;

selecting a first command based on the first set;

detecting a touch down of each contact of a second set of one or more contacts on the touch sensor surface while the first set remains touching the touch sensor surface, each contact of the second set corresponding to a hand part touching the touch sensor surface; and

ignoring the second set by maintaining the first command after the touch down of the second set and by not selecting a command of the computing system based on the second set.

48. The non-transitory computer-readable storage medium of claim 47, wherein a hand part includes a finger.

49. The non-transitory computer-readable storage medium of claim 47, the method further comprising:

detecting a liftoff of each contact of the first set from the touch sensor surface, while the second set remains touching the touch sensor surface; and

generating input of the first command based on the second set after the liftoff of the first set.

50. The non-transitory computer-readable storage medium of claim 47, the method further comprising:

obtaining first information of the second set; and

determining, based on the first information, to ignore the second set.

52. The non-transitory computer-readable storage medium of claim 50, wherein the first information includes a ratio of a total proximity of each contact of the second set to an eccentricity of each contact of the second set.

53. The non-transitory computer-readable storage medium of claim 50, wherein the first information includes a position of each contact of the second set.

56. The non-transitory computer-readable storage medium of claim 53, wherein the position of each contact of the second set is a relative position of the contact and another contact touching the touch sensor surface.

57. The non-transitory computer-readable storage medium of claim 56, wherein the relative position is based on a distance from the contact to a position in a template of contact positions.

58. The non-transitory computer-readable storage medium of claim 50, wherein the first information includes a shape parameter of each contact of the second set.

59. The non-transitory computer-readable storage medium of claim 58, wherein the shape parameter includes an eccentricity of the contact.

60. The non-transitory computer-readable storage medium of claim 56, wherein the relative position is based on a distance from the contact to a nearest neighbor contact of the contact.

61. The non-transitory computer-readable storage medium of claim 50, wherein the first information includes an orientation parameter of each contact of the second set.

62. The non-transitory computer-readable storage medium of claim 61, wherein the orientation parameter includes an orientation of an axis of an ellipse corresponding to the contact.

63. The non-transitory computer-readable storage medium of claim 50, wherein determining the first information includes determining that each contact of the second set is not a member of the first set.

64. The non-transitory computer-readable storage medium of claim 63, wherein the first set includes a plurality of contacts, the method further comprising:

determining that the contacts of the first set touched down synchronously, wherein selecting the first command based on the first set includes selecting the first command based the determination that the contacts of the first set touched down synchronously, and determining that each contact of the second set is not a member of the first set includes determining that at least one contact of the second set touched down asynchronously to at least one contact of the first set.

65. The non-transitory computer-readable storage medium of claim 47, wherein the first set consists of a single contact.

66. The non-transitory computer-readable storage medium of claim 65, wherein the first command is a key press, selecting the first command based on the first set includes placing the key press in a queue, and the second set consists of a single contact, the method further comprising:

determining that the contact of the second set touched down within a predetermined period of time after the touch down of the contact of the first set, such that the touch downs of the contacts of the first and second sets are synchronized, wherein ignoring the second set includes determining that a size of the contact of the second set exceeds a predetermined size, and maintaining the selection of the first command includes maintaining the key press in the queue.

67. The non-transitory computer-readable storage medium of claim 47, wherein the second set includes a plurality of contacts.

68. The non-transitory computer-readable storage medium of claim 67, the method further comprising:

determining that the contacts of the second set touched down asynchronously.

69. The non-transitory computer-readable storage medium of claim 67, the method further comprising:

determining that the contacts of the second set touched down synchronously; and

determining, based on the determination that the contacts of the second set touched down synchronously, that the contacts of the second set are resting.