Contact tracking and identification module for touch sensing

• US 8,902,175 B2
• Filed: 04/10/2009
• Issued: 12/02/2014
• Est. Priority Date: 01/26/1998
• Status: Expired due to Fees

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Westerman, Wayne Carl, Elias, John Greer
• Primary Examiner(s): Sim, Yong H
• Application Number:
  US12/422,197
• Publication Number:
  US 20090251435A1
• Time in Patent Office: 2,062 Days
• Field of Search: None
• US Class Current: 345/173
• CPC Class Codes: G06F 18/24137 Distances to cluster centroïds
• CPC Class Codes: G06F 2203/04101 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup
• CPC Class Codes: G06F 2203/04104 Multi-touch detection in digitiser, i.e. details about the simultaneous detection of a plurality of touching locations, e.g. multiple fingers or pen and finger
• CPC Class Codes: G06F 2203/04106 Multi-sensing digitiser, i.e. digitiser using at least two different sensing technologies simultaneously or alternatively, e.g. for detecting pen and finger, for saving power or for improving position detection
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/0235 using chord techniques G06F3/0234 takes precedence
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/0412 Digitisers structurally integrated in a display
• CPC Class Codes: G06F 3/04166 Details of scanning methods, e.g. sampling time, grouping of sub areas or time sharing with display driving Synchronisation with the driving of the display or the backlighting unit to avoid interferences generated internally G06F3/04184
• CPC Class Codes: G06F 3/04186 Touch location disambiguation
• CPC Class Codes: G06F 3/0446 using a grid-like structure of electrodes in at least two directions, e.g. using row and column electrodes
• CPC Class Codes: G06F 3/0447 Position sensing using the local deformation of sensor cells
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06V 30/19173 Classification techniques
• CPC Class Codes: G06V 30/1983 Syntactic or structural pattern recognition, e.g. symbolic string recognition
• CPC Class Codes: G06V 40/107 : Static hand or arm
• CPC Class Codes: H03K 17/9622 : using a plurality of detectors, e.g. keyboard
• CPC Class Codes: H03K 2217/960725 : Charge-transfer

Abstract

Apparatus and methods are disclosed for simultaneously tracking multiple finger and palm contacts as hands approach, touch, and slide across a proximity-sensing, multi-touch surface. Identification and classification of intuitive hand configurations and motions enables unprecedented integration of typing, resting, pointing, scrolling, 3D manipulation, and handwriting into a versatile, ergonomic computer input device.

Claims

1. A method of determining a plurality of touches concurrently on or near a touch sensing surface, the method comprising:

obtaining a set of data representing a scan of the touch sensing surface, the set of data including a plurality of proximity values, the proximity values representing measurements of touch object proximity at different locations along first and second axes within a sensing area of the touch sensing surface;

determining a first local maximum of the proximity values in the set of data and a second local maximum of the proximity values in the set of data, including determining that the first local maximum proximity value is greater than both a first neighbor proximity value, nearest in a first direction along the first axis, and a second neighbor proximity value, nearest in a second direction along the second axis;

determining first contact information based on the first local maximum, the first contact information including first surface coordinates of a first contact position within the sensing area of the touch sensing surface; and

determining second contact information based on the second local maximum, the second contact information including second surface coordinates of a second contact position within the sensing area of the touch sensing surface.

2. The method of claim 1, wherein determining the first contact information includes determining a contact edge, the contact edge defining a contact area that corresponds to a plurality of locations of proximity values within the sensing area, wherein the plurality of locations includes the location of the first local maximum.

3. The method of claim 2, wherein determining the first contact information includes determining a center of the contact area.

4. The method of claim 2, wherein determining the contact edge includes determining proximity values that satisfy a first boundary test, wherein the contact edge is determined based on the surface coordinates of the locations within the sensing area corresponding to proximity values that satisfy the first boundary test.

5. The method of claim 4, wherein determining the first contact information includes determining a proximity-weighted center of the contact area.

6. The method of claim 4, wherein determining proximity values that satisfy the first boundary test includes determining whether the proximity value exceeds a predetermined threshold proximity, wherein the contact edge is determined based on the surface coordinates of the locations within the sensing area corresponding to proximity values that exceed the predetermined threshold proximity.

7. The method of claim 6, wherein the predetermined threshold proximity is based on a background proximity in a no-touch condition in which no touch objects are on or near the touch sensing surface.

8. The method of claim 4, wherein determining proximity values that satisfy the first boundary test includes determining whether the proximity value is a partial minimum proximity, wherein the contact edge is determined based on the surface coordinates of the locations within the sensing area corresponding to proximity values that are partial minimum proximities.

9. The method of claim 4, wherein determining proximity values that satisfy the first boundary test includes selecting a first group of the proximity values, determining a proximity value in the first group that satisfies the first boundary test, determining a largest proximity value of the first group, selecting a second group of the proximity values, and determining, based on the location within the sensing area of the largest proximity value of the first group, a proximity value in the second group that satisfies the first boundary test.

10. The method of claim 9, wherein the first group includes proximity values corresponding to a first line of locations within the sensing area, the second group includes proximity values corresponding to a second line of locations within the sensing area, adjacent to the first line of locations, wherein one of the locations in the second line is adjacent to the location of the largest proximity value in the first line, and determining the proximity value in the second group that satisfies the first boundary test includes beginning testing the second group by testing a proximity value in the second group corresponding to the one of the locations in the second line that is adjacent to the location of the largest proximity value in the first line.

11. The method of claim 4, further comprising:

determining, based on one or more local maxima, not including either of the first local maximum and the second local maximum, third contact information of a third contact.

12. The method of claim 11, wherein determining the contact information of the third contact includes determining a contact edge of the third contact, the contact edge of the third contact defining a contact area of the third contact that corresponds to a plurality of locations of proximity values within the sensing area, wherein the plurality of locations includes the location of two or more local maxima.

13. The method of claim 12, wherein determining the contact edge of the third contact includes determining proximity values that satisfy a second boundary test, different than the first boundary test, wherein the contact edge of the third contact is determined based on the proximity values that satisfy the second boundary test.

14. The method of claim 13, wherein the second boundary test is independent of the location of one or more partial minimum proximities of the proximity values that are tested when determining the contact edge of the third contact.

15. The method of claim 13, further comprising:

determining a plurality of non-overlapping regions of the sensing area, including a first region and a second region, wherein a contact edge of each contact in the first region is determined based on the first boundary test, and a contact edge of each contact in the second region is determined based on the second boundary test.

16. The method of claim 15, wherein determining the plurality of regions includes determining the first region based on a portion of the sensing area on which one or more fingers touch, and determining the second region based on a portion of the sensing area on which a palm touches.

17. The method of claim 15, further comprising:

modifying one or more of the regions.

18. The method of claim 17, further comprising:

detecting a no-touch condition in which no touch objects are on or near the touch sensing surface, wherein modifying one or more of the regions includes moving one of the regions to a default position in response to the no-touch condition.

19. The method of claim 17, wherein modifying one or more of the regions includes modifying one or more of the regions based on the contact position of one or more contacts.

20. The method of claim 19, wherein modifying one or more of the regions includes maintaining a minimum separation distance between two of the regions.

21. The method of claim 17, wherein modifying one or more of the regions includes cancelling one of the regions based on a determination that all of the contacts within the region have lifted off of the sensing area.

22. The method of claim 1, wherein each of the first and second surface coordinates consist of a vertical coordinate and a horizontal coordinate.

23. The method of claim 1, wherein obtaining the set of data representing the scan of the touch sensing surface includes receiving a set of proximity measurements of touch sensors of the touch sensing surface, and processing the set of proximity measurements to obtain the set of data.

24. The method of claim 23, wherein processing the set of proximity measurements includes smoothing the set of proximity measurements.

25. The method of claim 24, wherein smoothing the set of proximity measurements includes applying one of a diffusion operator and a Gaussian kernel to the set of proximity measurements.

26. A non-transitory computer-readable storage medium having computer-executable instructions stored therein, the computer-executable instructions executable to perform a method for determining a plurality of touches concurrently on or near a touch sensing surface, the method comprising:

obtaining a set of data representing a scan of the touch sensing surface, the set of data including a plurality of proximity values, the proximity values representing measurements of touch object proximity at different locations along first and second axes within a sensing area of the touch sensing surface;

determining a first local maximum of the proximity values in the set of data and a second local maximum of the proximity values in the set of data, including determining that the first local maximum proximity value is greater than both a first neighbor proximity value, nearest in a first direction along the first axis, and a second neighbor proximity value, nearest in a second direction along the second axis;

determining first contact information based on the first local maximum, the first contact information including first surface coordinates of a first contact position within the sensing area of the touch sensing surface; and

determining second contact information based on the second local maximum, the second contact information including second surface coordinates of a second contact position within the sensing area of the touch sensing surface.

27. The non-transitory computer-readable storage medium of claim 26, wherein determining the first contact information includes determining a contact edge, the contact edge defining a contact area that corresponds to a plurality of locations of proximity values within the sensing area, wherein the plurality of locations includes the location of the first local maximum.

28. The non-transitory computer-readable storage medium of claim 27, wherein determining the first contact information includes determining a center of the contact area.

29. The non-transitory computer-readable storage medium of claim 27, wherein determining the contact edge includes determining proximity values that satisfy a first boundary test, wherein the contact edge is determined based on the surface coordinates of the locations within the sensing area corresponding to proximity values that satisfy the first boundary test.

30. The non-transitory computer-readable storage medium of claim 29, wherein determining the first contact information includes determining a proximity-weighted center of the contact area.

31. The non-transitory computer-readable storage medium of claim 29, wherein determining proximity values that satisfy the first boundary test includes determining whether the proximity value exceeds a predetermined threshold proximity, wherein the contact edge is determined based on the surface coordinates of the locations within the sensing area corresponding to proximity values that exceed the predetermined threshold proximity.

32. The non-transitory computer-readable storage medium of claim 31, wherein the predetermined threshold proximity is based on a background proximity in a no-touch condition in which no touch objects are on or near the touch sensing surface.

33. The non-transitory computer-readable storage medium of claim 29, wherein determining proximity values that satisfy the first boundary test includes determining whether the proximity value is a partial minimum proximity, wherein the contact edge is determined based on the surface coordinates of the locations within the sensing area corresponding to proximity values that are partial minimum proximities.

34. The non-transitory computer-readable storage medium of claim 29, wherein determining proximity values that satisfy the first boundary test includes selecting a first group of the proximity values, determining a proximity value in the first group that satisfies the first boundary test, determining a largest proximity value of the first group, selecting a second group of the proximity values, and determining, based on the location within the sensing area of the largest proximity value of the first group, a proximity value in the second group that satisfies the first boundary test.

35. The non-transitory computer-readable storage medium of claim 34, wherein the first group includes proximity values corresponding to a first line of locations within the sensing area, the second group includes proximity values corresponding to a second line of locations within the sensing area, adjacent to the first line of locations, wherein one of the locations in the second line is adjacent to the location of the largest proximity value in the first line, and determining the proximity value in the second group that satisfies the first boundary test includes beginning testing the second group by testing a proximity value in the second group corresponding to the one of the locations in the second line that is adjacent to the location of the largest proximity value in the first line.

36. The non-transitory computer-readable storage medium of claim 29, the method further comprising:

determining, based on one or more local maxima, not including either of the first local maximum and the second local maximum, third contact information of a third contact.

37. The non-transitory computer-readable storage medium of claim 36, wherein determining the contact information of the third contact includes determining a contact edge of the third contact, the contact edge of the third contact defining a contact area of the third contact that corresponds to a plurality of locations of proximity values within the sensing area, wherein the plurality of locations includes the location of two or more local maxima.

38. The non-transitory computer-readable storage medium of claim 37, wherein determining the contact edge of the third contact includes determining proximity values that satisfy a second boundary test, different than the first boundary test, wherein the contact edge of the third contact is determined based on the proximity values that satisfy the second boundary test.

39. The non-transitory computer-readable storage medium of claim 38, wherein the second boundary test is independent of the location of one or more partial minimum proximities of the proximity values that are tested when determining the contact edge of the third contact.

40. The non-transitory computer-readable storage medium of claim 38, the method further comprising:

determining a plurality of non-overlapping regions of the sensing area, including a first region and a second region, wherein a contact edge of each contact in the first region is determined based on the first boundary test, and a contact edge of each contact in the second region is determined based on the second boundary test.

41. The non-transitory computer-readable storage medium of claim 40, wherein determining the plurality of regions includes determining the first region based on a portion of the sensing area on which one or more fingers touch, and determining the second region based on a portion of the sensing area on which a palm touches.

42. The non-transitory computer-readable storage medium of claim 40, the method further comprising:

modifying one or more of the regions.

43. The non-transitory computer-readable storage medium of claim 42, the method further comprising:

detecting a no-touch condition in which no touch objects are on or near the touch sensing surface, wherein modifying one or more of the regions includes moving one of the regions to a default position in response to the no-touch condition.

44. The non-transitory computer-readable storage medium of claim 42, wherein modifying one or more of the regions includes modifying one or more of the regions based on the contact position of one or more contacts.

45. The non-transitory computer-readable storage medium of claim 44, wherein modifying one or more of the regions includes maintaining a minimum separation distance between two of the regions.

46. The non-transitory computer-readable storage medium of claim 42, wherein modifying one or more of the regions includes cancelling one of the regions based on a determination that all of the contacts within the region have lifted off of the sensing area.

47. The non-transitory computer-readable storage medium of claim 26, wherein each of the first and second surface coordinates consist of a vertical coordinate and a horizontal coordinate.

48. The non-transitory computer-readable storage medium of claim 26, wherein obtaining the set of data representing the scan of the touch sensing surface includes receiving a set of proximity measurements of touch sensors of the touch sensing surface, and processing the set of proximity measurements to obtain the set of data.

49. The non-transitory computer-readable storage medium of claim 48, wherein processing the set of proximity measurements includes smoothing the set of proximity measurements.

50. The non-transitory computer-readable storage medium of claim 49, wherein smoothing the set of proximity measurements includes applying one of a diffusion operator and a Gaussian kernel to the set of proximity measurements.

51. An apparatus for determining a plurality of touches concurrently on or near a touch sensing surface, the apparatus comprising:

a contact determination system that obtains a set of data representing a scan of the touch sensing surface, the set of data including a plurality of proximity values, the proximity values representing measurements of touch object proximity at different locations within a sensing area of the touch sensing surface, determines a first local maximum of the proximity values in the set of data and a second local maximum of the proximity values in the set of data, including determining that the first local maximum proximity value is greater than both a first neighbor proximity value, nearest in a first direction along the first axis, and a second neighbor proximity value, nearest in a second direction along the second axis, determines first contact information based on the first local maximum, the first contact information including first surface coordinates of a first contact position within the sensing area of the touch sensing surface, and determines second contact information based on the second local maximum, the second contact information including second surface coordinates of a second contact position within the sensing area of the touch sensing surface.

52. The apparatus of claim 51, wherein determining the first contact information includes determining a contact edge, the contact edge defining a contact area that corresponds to a plurality of locations of proximity values within the sensing area, wherein the plurality of locations includes the location of the first local maximum.

53. The apparatus of claim 52, wherein determining the first contact information includes determining a center of the contact area.

54. The apparatus of claim 52, wherein determining the contact edge includes determining proximity values that satisfy a first boundary test, wherein the contact edge is determined based on the surface coordinates of the locations within the sensing area corresponding to proximity values that satisfy the first boundary test.

55. The apparatus of claim 54, wherein determining the first contact information includes determining a proximity-weighted center of the contact area.

56. The apparatus of claim 54, wherein determining proximity values that satisfy the first boundary test includes determining whether the proximity value exceeds a predetermined threshold proximity, wherein the contact edge is determined based on the surface coordinates of the locations within the sensing area corresponding to proximity values that exceed the predetermined threshold proximity.

57. The apparatus of claim 56, wherein the predetermined threshold proximity is based on a background proximity in a no-touch condition in which no touch objects are on or near the touch sensing surface.

58. The apparatus of claim 54, wherein determining proximity values that satisfy the first boundary test includes determining whether the proximity value is a partial minimum proximity, wherein the contact edge is determined based on the surface coordinates of the locations within the sensing area corresponding to proximity values that are partial minimum proximities.

59. The apparatus of claim 54, wherein determining proximity values that satisfy the first boundary test includes selecting a first group of the proximity values, determining a proximity value in the first group that satisfies the first boundary test, determining a largest proximity value of the first group, selecting a second group of the proximity values, and determining, based on the location within the sensing area of the largest proximity value of the first group, a proximity value in the second group that satisfies the first boundary test.

60. The apparatus of claim 59, wherein the first group includes proximity values corresponding to a first line of locations within the sensing area, the second group includes proximity values corresponding to a second line of locations within the sensing area, adjacent to the first line of locations, wherein one of the locations in the second line is adjacent to the location of the largest proximity value in the first line, and determining the proximity value in the second group that satisfies the first boundary test includes beginning testing the second group by testing a proximity value in the second group corresponding to the one of the locations in the second line that is adjacent to the location of the largest proximity value in the first line.

61. The apparatus of claim 54, wherein the contact determination system further determines, based on one or more local maxima, not including either of the first local maximum and the second local maximum, third contact information of a third contact.

62. The apparatus of claim 61, wherein determining the contact information of the third contact includes determining a contact edge of the third contact, the contact edge of the third contact defining a contact area of the third contact that corresponds to a plurality of locations of proximity values within the sensing area, wherein the plurality of locations includes the location of two or more local maxima.

63. The apparatus of claim 62, wherein determining the contact edge of the third contact includes determining proximity values that satisfy a second boundary test, different than the first boundary test, wherein the contact edge of the third contact is determined based on the proximity values that satisfy the second boundary test.

64. The apparatus of claim 63, wherein the second boundary test is independent of the location of one or more partial minimum proximities of the proximity values that are tested when determining the contact edge of the third contact.

65. The apparatus of claim 63, wherein the contact determination system further determines a plurality of non-overlapping regions of the sensing area, including a first region and a second region, wherein a contact edge of each contact in the first region is determined based on the first boundary test, and a contact edge of each contact in the second region is determined based on the second boundary test.

66. The apparatus of claim 65, wherein determining the plurality of regions includes determining the first region based on a portion of the sensing area on which one or more fingers touch, and determining the second region based on a portion of the sensing area on which a palm touches.

67. The apparatus of claim 65, wherein the contact determination system further modifies one or more of the regions.

68. The apparatus of claim 67, wherein the contact determination system further detects a no-touch condition in which no touch objects are on or near the touch sensing surface, wherein modifying one or more of the regions includes moving one of the regions to a default position in response to the no-touch condition.

69. The apparatus of claim 67, wherein modifying one or more of the regions includes modifying one or more of the regions based on the contact position of one or more contacts.

70. The apparatus of claim 69, wherein modifying one or more of the regions includes maintaining a minimum separation distance between two of the regions.

71. The apparatus of claim 67, wherein modifying one or more of the regions includes cancelling one of the regions based on a determination that all of the contacts within the region have lifted off of the sensing area.

72. The apparatus of claim 51, wherein each of the first and second surface coordinates consist of a vertical coordinate and a horizontal coordinate.

73. The apparatus of claim 51, wherein obtaining the set of data representing the scan of the touch sensing surface includes receiving a set of proximity measurements of touch sensors of the touch sensing surface, and processing the set of proximity measurements to obtain the set of data.

74. The apparatus of claim 73, wherein processing the set of proximity measurements includes smoothing the set of proximity measurements.

75. The apparatus of claim 74, wherein smoothing the set of proximity measurements includes applying one of a diffusion operator and a Gaussian kernel to the set of proximity measurements.