Multi-touch hand position offset computation

• US 8,674,943 B2
• Filed: 11/14/2006
• Issued: 03/18/2014
• Est. Priority Date: 01/26/1998
• Status: Expired due to Term

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Westerman, Wayne Carl, Elias, John G.
• Primary Examiner(s): Amadiz, Rodney
• Application Number:
  US11/559,833
• Publication Number:
  US 20070078919A1
• Time in Patent Office: 2,681 Days
• Field of Search: 345156-184, 178 1801- 2004
• US Class Current: 345/173
• CPC Class Codes: G06F 18/24137 Distances to cluster centroïds
• CPC Class Codes: G06F 2203/04101 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup
• CPC Class Codes: G06F 2203/04104 Multi-touch detection in digitiser, i.e. details about the simultaneous detection of a plurality of touching locations, e.g. multiple fingers or pen and finger
• CPC Class Codes: G06F 2203/04106 Multi-sensing digitiser, i.e. digitiser using at least two different sensing technologies simultaneously or alternatively, e.g. for detecting pen and finger, for saving power or for improving position detection
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/0235 using chord techniques G06F3/0234 takes precedence
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/0412 Digitisers structurally integrated in a display
• CPC Class Codes: G06F 3/04166 Details of scanning methods, e.g. sampling time, grouping of sub areas or time sharing with display driving Synchronisation with the driving of the display or the backlighting unit to avoid interferences generated internally G06F3/04184
• CPC Class Codes: G06F 3/04186 Touch location disambiguation
• CPC Class Codes: G06F 3/0446 using a grid-like structure of electrodes in at least two directions, e.g. using row and column electrodes
• CPC Class Codes: G06F 3/0447 Position sensing using the local deformation of sensor cells
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06V 30/19173 Classification techniques
• CPC Class Codes: G06V 30/1983 Syntactic or structural pattern recognition, e.g. symbolic string recognition
• CPC Class Codes: G06V 40/107 : Static hand or arm
• CPC Class Codes: H03K 17/9622 : using a plurality of detectors, e.g. keyboard
• CPC Class Codes: H03K 2217/960725 : Charge-transfer

Abstract

Apparatus and methods are disclosed for simultaneously tracking multiple finger and palm contacts as hands approach, touch, and slide across a proximity-sensing, multi-touch surface. Identification and classification of intuitive hand configurations and motions enables unprecedented integration of typing, resting, pointing, scrolling, 3D manipulation, and handwriting into a versatile, ergonomic computer input device.

Claims

1. A method of computing a hand position offset from positions of individual hand parts on or near a touch sensing surface, the method comprising:

obtaining an identity and a current position of each hand part;

for each hand part, determining a difference between the current position of the hand part and a default position of the hand part;

computing a hand position offset based on an average of the differences; and

based on the same default positions and an updated position of each hand part, computing an additional hand position offset.

2. The method of claim 1 further comprising:

establishing an offset weighting for each hand part, wherein computing the hand position offset comprises computing the hand position offset based on a weighted average of the differences.

3. The method of claim 1 further comprising:

computing a hand part position offset based on the default position of the hand part, the hand position offset, and the current position of the hand part.

4. The method of claim 3 further comprising:

computing an average of individual hand part velocities;

predicting current hand and hand part offsets from one or more previous offsets and the computed average velocity; and

updating the predicted hand and hand part offsets to an average of the predicted offset and the computed offsets.

5. The method of claim 4 wherein one or more of the computed averages are weighted averages.

6. The method of claim 4 further comprising:

computing a confidence in hand part identifications; and

giving increasing weight to the computed offsets used in updating the predicted hand and hand part offsets as the computed confidence increases.

7. The method of claim 6 wherein one or more of the computed averages are weighted averages.

8. The method of claim 1 further comprising:

computing an average of individual hand part velocities; and

determining an estimated hand position offset from one or more previously-determined hand position offsets and the computed average velocity,

wherein the computed hand position offset is further based on the estimated hand position offset.

9. The method of claim 8 wherein one or more of the computed averages are weighted averages.

10. The method of claim 8 further comprising:

computing a confidence in hand part identifications; and

giving increasing weight to the computed offset used in updating the predicted hand offset as the computed confidence increases.

11. The method of claim 10 wherein one or more of the computed averages are weighted averages.

12. The method of claim 1, wherein obtaining the identity of each hand part includes determining the identity of a hand part based on a previously-determined hand position offset.

13. The method of claim 1, wherein obtaining the current position of each hand part includes obtaining an estimated position of a hand part based on a previously-determined position of the hand part.

14. The method of claim 13, wherein obtaining the estimated position includes modifying the previously-determined position such that the previously-determined position decays toward the default position of the hand part.

15. The method of claim 14, wherein modifying the previously-determined position includes determining a decay rate from a plurality of decay rates and modifying the previously-determined position to decay toward the default position at the determined decay rate.

16. The method of claim 1, further comprising:

detecting a touchdown of a hand part; and

identifying the detected, touched-down hand part based on the computed hand position offset.

17. The method of claim 16, further comprising:

detecting a liftoff of the hand part from the touch sensing surface, the liftoff occurring prior to the detected touchdown of the hand part; and

computing a hand position offset at a time of the liftoff,

wherein identifying the detected, touched-down hand part includes modifying the hand position offset computed at the time of the liftoff to decay toward a fixed default hand position offset, the decay being based on a difference in the time of the liftoff and a time of the touchdown.

18. A non-transitory computer readable storage medium having stored therein instructions that, when executed by a computing device, cause the computing device to perform a method of computing a hand position offset from positions of individual hand parts on or near a touch sensing surface, the method comprising:

obtaining an identity and a current position of each hand part on or near the touch sensing surface;

for each hand part, determining a difference between the current position of the hand part and a default position of the hand part;

computing a hand position offset based on an average of the differences; and

based on the same default positions and an updated position of each hand part, computing an additional hand position offset.

19. The non-transitory computer readable storage medium of claim 18, wherein the method further comprises:

establishing an offset weighting for each hand part, wherein computing the hand position offset comprises computing the hand position offset based on a weighted average of the differences.

20. The non-transitory computer readable storage medium of claim 18, wherein the method further comprises:

computing a hand part position offset based on the default position of the hand part, the hand position offset, and the current position of the hand part.

21. The non-transitory computer readable storage medium of claim 20, wherein the method further comprises:

computing an average of individual hand part velocities;

predicting current hand and hand part offsets from one or more previous offsets and the computed average velocity; and

updating the predicted hand and hand part offsets to an average of the predicted offset and the computed offsets.

22. The non-transitory computer readable storage medium of claim 21 wherein one or more of the computed averages are weighted averages.

23. The non-transitory computer readable storage medium of claim 21, wherein the method further comprises:

computing a confidence in hand part identifications; and

giving increasing weight to the computed offsets used in updating the predicted hand and hand part offsets as the computed confidence increases.

24. The non-transitory computer readable storage medium of claim 23 wherein one or more of the computed averages are weighted averages.

25. The non-transitory computer readable storage medium of claim 18, wherein the method further comprises:

computing an average of individual hand part velocities; and

determining an estimated hand position offset from one or more previously-determined hand position offsets and the computed average velocity,

wherein the computed hand position offset is further based on the estimated hand position offset.

26. The non-transitory computer readable storage medium of claim 25 wherein one or more of the computed averages are weighted averages.

27. The non-transitory computer readable storage medium of claim 25, wherein the method further comprises:

computing a confidence in hand part identifications; and

giving increasing weight to the computed offset used in updating the predicted hand offset as the computed confidence increases.

28. The non-transitory computer readable storage medium of claim 27 wherein one or more of the computed averages are weighted averages.

29. The non-transitory computer readable storage medium of claim 18, wherein obtaining the identity of each hand part includes determining the identity of a hand part based on a previously-determined hand position offset.

30. The non-transitory computer readable storage medium of claim 29, wherein determining the identity includes obtaining first touch data from a first region of the touch sensing surface, obtaining second touch data from a second region of the touch sensing surface, analyzing the first touch data based on a first rule to determine identities of one or more hand parts on or near the first region, and analyzing the second touch data based on a second rule, different from the first rule, to determine identities of one or more hand parts on or near the second region, the first and second regions being based on the previously-determined hand position offset.

31. The non-transitory computer readable storage medium of claim 30, wherein locations of the first and second regions are translated from first and second default regions, respectively, based on the previously-determined hand position offset.

32. The non-transitory computer readable storage medium of claim 30, wherein the first region includes a strict segmentation region and the first rule includes associating a partial minimum of the first touch data with a group boundary, such that partial maxima on either side of the partial minimum are associated with separate hand parts, and wherein the second region includes a sloppy segmentation region and the second rule includes associating partial maxima on either side of a partial minimum of the second touch data with a same hand part.

33. The non-transitory computer readable storage medium of claim 18, wherein obtaining the current position of each hand part includes obtaining an estimated position of a hand part based on a previously-determined position of the hand part.

34. The non-transitory computer readable storage medium of claim 33, wherein obtaining the estimated position includes modifying the previously-determined position such that the previously-determined position decays toward the default position of the hand part.

35. The non-transitory computer readable storage medium of claim 34, wherein obtaining the current position includes measuring a position of the hand part, and selecting either the measured position or the estimated position based on a level of confidence in the identity of the hand part.

36. The non-transitory computer readable storage medium of claim 35, wherein the method further comprises:

determining that a hand part has lifted off from the touch sensing surface, wherein the estimated position of the hand part is selected as the current position if the hand part is determined to have lifted off.

37. The non-transitory computer readable storage medium of claim 35, wherein the level of confidence is based on a number of hand parts currently touched down on or near the touch sensing surface.

38. The non-transitory computer readable storage medium of claim 35, wherein the level of confidence is based on a determination of whether one or more of the currently touched down hand parts is identified as a hand part associated with substantially strong disambiguating features.

39. The non-transitory computer readable storage medium of claim 38, wherein the hand part associated with substantially strong disambiguating features includes one of a thumb and a palm heel.

40. The non-transitory computer readable storage medium of claim 35, wherein the level of confidence is based on a comparison of a previous confidence level in the identity of the hand part and a current confidence level in the identity of the hand part.

41. The non-transitory computer readable storage medium of claim 34, wherein the previously-determined position is modified to decay toward the default position at one of an exponential decay rate and a linear decay rate.

42. The non-transitory computer readable storage medium of claim 34, wherein modifying the previously-determined position includes determining a decay rate from a plurality of decay rates and modifying the previously-determined position to decay toward the default position at the determined decay rate.

43. The non-transitory computer readable storage medium of claim 42, wherein determining the decay rate is based on a current confidence level in the identification of the hand part.

44. The non-transitory computer readable storage medium of claim 18, wherein the computed hand position offset is further based on one or more previously-determined hand position offsets.

45. The non-transitory computer readable storage medium of claim 44, wherein computing the hand position offset includes averaging a currently-computed hand position offset with the one or more previously-determined hand position offsets.

46. The non-transitory computer readable storage medium of claim 18, wherein the method further comprises:

detecting a touchdown of a hand part; and

identifying the detected, touched-down hand part based on the computed hand position offset.

47. The non-transitory computer readable storage medium of claim 46, wherein the method further comprises:

detecting a liftoff of the hand part from the touch sensing surface, the liftoff occurring prior to the detected touchdown of the hand part; and

computing a hand position offset at a time of the liftoff,

wherein identifying the detected, touched-down hand part includes modifying the hand position offset computed at the time of the liftoff to decay toward a default hand position offset, the decay being based on a difference in the time of the liftoff and a time of the touchdown.

48. A computing system comprising:

a touch sensing surface; and

a hand position offset system that obtains an identity and a current position of each of one or more hand parts on or near the touch sensing surface, determines, for each hand part, a difference between the current position of the hand part and a default position of the hand part, computes a hand position offset based on an average of the differences, and, based on the same default positions and an updated position of each hand part, computes an additional hand position offset.

49. The computing system of claim 48, further comprising:

a hand part identification system that determines the identity of a hand part based on a previously-determined hand position offset.

50. The computing system of claim 48, wherein obtaining the current position of each hand part includes obtaining an estimated position of a hand part based on a previously-determined position of the hand part.

51. The computing system of claim 50, wherein obtaining the estimated position includes modifying the previously-determined position such that the previously-determined position decays toward the default position of the hand part.