Multi-touch contact motion extraction

• US 8,466,880 B2
• Filed: 12/22/2008
• Issued: 06/18/2013
• Est. Priority Date: 01/26/1998
• Status: Expired due to Fees

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Westerman, Wayne Carl, Elias, John Greer
• Primary Examiner(s): Lefkowitz, Sumati
• Assistant Examiner(s): Amadiz, Rodney
• Application Number:
  US12/342,027
• Publication Number:
  US 20090160816A1
• Time in Patent Office: 1,639 Days
• Field of Search: 345156-175, 345/179, 178/18.01, 178/18.03, 178 1805- 1809
• US Class Current: 345/173
• CPC Class Codes: G06F 18/24137 Distances to cluster centroïds
• CPC Class Codes: G06F 2203/04101 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup
• CPC Class Codes: G06F 2203/04104 Multi-touch detection in digitiser, i.e. details about the simultaneous detection of a plurality of touching locations, e.g. multiple fingers or pen and finger
• CPC Class Codes: G06F 2203/04106 Multi-sensing digitiser, i.e. digitiser using at least two different sensing technologies simultaneously or alternatively, e.g. for detecting pen and finger, for saving power or for improving position detection
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/0235 using chord techniques G06F3/0234 takes precedence
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/0412 Digitisers structurally integrated in a display
• CPC Class Codes: G06F 3/04166 Details of scanning methods, e.g. sampling time, grouping of sub areas or time sharing with display driving Synchronisation with the driving of the display or the backlighting unit to avoid interferences generated internally G06F3/04184
• CPC Class Codes: G06F 3/04186 Touch location disambiguation
• CPC Class Codes: G06F 3/0446 using a grid-like structure of electrodes in at least two directions, e.g. using row and column electrodes
• CPC Class Codes: G06F 3/0447 Position sensing using the local deformation of sensor cells
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06V 30/19173 Classification techniques
• CPC Class Codes: G06V 30/1983 Syntactic or structural pattern recognition, e.g. symbolic string recognition
• CPC Class Codes: G06V 40/107 : Static hand or arm
• CPC Class Codes: H03K 17/9622 : using a plurality of detectors, e.g. keyboard
• CPC Class Codes: H03K 2217/960725 : Charge-transfer

Abstract

Apparatus and methods are disclosed for simultaneously tracking multiple finger and palm contacts as hands approach, touch, and slide across a proximity-sensing, multi-touch surface. Identification and classification of intuitive hand configurations and motions enables unprecedented integration of typing, resting, pointing, scrolling, 3D manipulation, and handwriting into a versatile, ergonomic computer input device.

Claims

1. A method for generating control signals based on contacts of a touch sensing surface, each contact having multiple degrees of freedom of an associated touch object on or near the touch sensing surface, the method comprising:

obtaining tracking information of a plurality of contacts, the contacts corresponding to touch objects on or near the touch sensing surface;

computing, based on the tracking information, motion information of the plurality of contacts, the motion information including a first component of motion that is one of translation motion, rotational motion, and scaling motion, and a second component of motion that is a different one of translation motion, rotational motion, and scaling motion than the first component of motion;

modifying the motion information to obtain modified motion information, wherein the modifying includes downscaling the second component of motion relative to the first component of motion; and

transmitting the modified motion information as a control signal of an electronic or electro-mechanical device.

2. The method of claim 1, wherein downscaling the second component of motion includes downscaling the second component of motion based on a speed of the second component of motion and a speed of the first component of motion.

3. The method of claim 2, wherein downscaling the second component of motion further includes downscaling the second component of motion in proportion to a function of an average speed of the second component of motion compared to an average speed of the first component of motion.

4. The method of claim 2, wherein downscaling the second component of motion includes downscaling the second component of motion based on a ratio of the speed of the second component of motion to the speed of the first component of motion.

5. The method of claim 1, further comprising:

selecting an input activity; and

generating, based on the selected input activity and the modified motion information, input of the electronic or electro-mechanical device.

6. The method of claim 5, wherein modifying the motion information is performed in response to the selection of the input activity.

7. The method of claim 5, further comprising:

determining a number of contacts of the plurality of contacts, wherein the input activity is selected based on the number of contacts.

8. The method of claim 5, wherein the input activity includes graphically manipulating a graphical object in multiple degrees of freedom, the multiple degrees of freedom corresponding to at least the first and second components of motion.

9. The method of claim 8, wherein the first component of motion is translation motion and the second component of motion is scaling motion, the translation motion corresponding to a translation manipulation of the input activity, and the scaling motion corresponding to one of a resizing manipulation and a zooming manipulation of the input activity.

10. The method of claim 8, wherein the first component of motion is translation motion and the second component of motion is rotational motion, the translation motion corresponding to a translation manipulation of the input activity, and the rotational motion corresponding to a rotational manipulation of the input activity.

11. The method of claim 5, further comprising:

discretizing the first component of motion into one or more units of motion, each unit of motion corresponding to a single input event of the input activity.

12. The method of claim 11, wherein the generating input includes generating a single input event of the input activity based on one unit of motion, and then disabling input event generation for the input activity.

13. The method of claim 5, wherein modifying the motion information further includes projecting the first component of motion onto a direction of the input activity.

14. The method of claim 13, wherein the first component of motion is translation motion, and projecting the first component of motion includes projecting the translation motion onto a unit vector in the direction of the input activity.

15. The method of claim 14, wherein the input activity includes an arrow key input, and the direction of the input activity corresponds to the direction of the arrow key input.

16. The method of claim 5, wherein modifying the motion information further includes obtaining a motion sensitivity parameter corresponding to the input activity, and modifying the first component of motion based the motion sensitivity parameter.

17. The method of claim 1, wherein computing the motion information includes computing one of the first and the second components of motion based on the tracking information of some, but not all, of the plurality of contacts.

18. The method of claim 17, wherein computing one of the first and the second components of motion based on the tracking information of some, but not all, of the plurality of contacts includes computing one of rotational motion and scaling motion based on an innermost contact and an outermost contact.

19. The method of claim 1, wherein downscaling the second component of motion includes clipping the second component of motion to zero.

20. The method of claim 1, wherein downscaling the second component of motion includes applying a dead-zone filter to the first and second components of motion.

21. The method of claim 1, wherein downscaling the second component of motion consists of applying a dead-zone filter to the first and second components of motion.

22. A non-transitory computer-readable storage medium having computer-executable instructions stored therein, the computer-executable instructions executable to perform a method for generating control signals based on contacts of a touch sensing surface, each contact having multiple degrees of freedom of an associated touch object on or near the touch sensing surface, the method comprising:

obtaining tracking information of a plurality of contacts, the contacts corresponding to touch objects on or near the touch sensing surface;

computing, based on the tracking information, motion information of the plurality of contacts, the motion information including a first component of motion that is one of translation motion, rotational motion, and scaling motion, and a second component of motion that is a different one of translation motion, rotational motion, and scaling motion than the first component of motion;

modifying the motion information to obtain modified motion information, wherein the modifying includes downscaling the second component of motion relative to the first component of motion; and

transmitting the modified motion information as a control signal of an electronic or electro-mechanical device.

23. The non-transitory computer-readable storage medium of claim 22, wherein downscaling the second component of motion includes downscaling the second component of motion based on a speed of the second component of motion and a speed of the first component of motion.

24. The non-transitory computer-readable storage medium of claim 23, wherein downscaling the second component of motion further includes downscaling the second component of motion in proportion to a function of an average speed of the second component of motion compared to an average speed of the first component of motion.

25. The non-transitory computer-readable storage medium of claim 23, wherein downscaling the second component of motion includes downscaling the second component of motion based on a ratio of the speed of the second component of motion to the speed of the first component of motion.

26. The non-transitory computer-readable storage medium of claim 22, the method further comprising:

selecting an input activity; and

generating, based on the selected input activity and the modified motion information, input of the electronic or electro-mechanical device.

27. The non-transitory computer-readable storage medium of claim 26, wherein modifying the motion information is performed in response to the selection of the input activity.

28. The non-transitory computer-readable storage medium of claim 26, the method further comprising:

determining a number of contacts of the plurality of contacts, wherein the input activity is selected based on the number of contacts.

29. The non-transitory computer-readable storage medium of claim 26, wherein the input activity includes graphically manipulating a graphical object in multiple degrees of freedom, the multiple degrees of freedom corresponding to at least the first and second components of motion.

30. The non-transitory computer-readable storage medium of claim 29, wherein the first component of motion is translation motion and the second component of motion is scaling motion, the translation motion corresponding to a translation manipulation of the input activity, and the scaling motion corresponding to one of a resizing manipulation and a zooming manipulation of the input activity.

31. The non-transitory computer-readable storage medium of claim 29, wherein the first component of motion is translation motion and the second component of motion is rotational motion, the translation motion corresponding to a translation manipulation of the input activity, and the rotational motion corresponding to a rotational manipulation of the input activity.

32. The non-transitory computer-readable storage medium of claim 26, the method further comprising:

discretizing the first component of motion into one or more units of motion, each unit of motion corresponding to a single input event of the input activity.

33. The non-transitory computer-readable storage medium of claim 32, wherein the generating input includes generating a single input event of the input activity based on one unit of motion, and then disabling input event generation for the input activity.

34. The non-transitory computer-readable storage medium of claim 26, wherein modifying the motion information further includes projecting the first component of motion onto a direction of the input activity.

35. The non-transitory computer-readable storage medium of claim 34, wherein the first component of motion is translation motion, and projecting the first component of motion includes projecting the translation motion onto a unit vector in the direction of the input activity.

36. The non-transitory computer-readable storage medium of claim 35, wherein the input activity includes an arrow key input, and the direction of the input activity corresponds to the direction of the arrow key input.

37. The non-transitory computer-readable storage medium of claim 26, wherein modifying the motion information further includes obtaining a motion sensitivity parameter corresponding to the input activity, and modifying the first component of motion based the motion sensitivity parameter.

38. The non-transitory computer-readable storage medium of claim 22, wherein computing the motion information includes computing one of the first and the second components of motion based on the tracking information of some, but not all, of the plurality of contacts.

39. The non-transitory computer-readable storage medium of claim 38, wherein computing one of the first and the second components of motion based on the tracking information of some, but not all, of the plurality of contacts includes computing one of rotational motion and scaling motion based on an innermost contact and an outermost contact.

40. The non-transitory computer-readable storage medium of claim 22, wherein downscaling the second component of motion includes clipping the second component of motion to zero.

41. The non-transitory computer-readable storage medium of claim 22, wherein downscaling the second component of motion includes applying a dead-zone filter to the first and second components of motion.

42. The non-transitory computer-readable storage medium of claim 22, wherein downscaling the second component of motion consists of applying a dead-zone filter to the first and second components of motion.

43. An apparatus comprising:

a tracking module that obtains tracking information of a plurality of contacts, the contacts corresponding to touch objects on or near a touch sensing surface; and

a motion component extraction module that computes, based on the tracking information, motion information of the plurality of contacts, the motion information including a first component of motion that is one of translation motion, rotational motion, and scaling motion, and a second component of motion that is a different one of translation motion, rotational motion, and scaling motion than the first component of motion, modifies the motion information to obtain modified motion information, wherein the modifying includes downscaling the second component of motion relative to the first component of motion, and transmits the modified motion information as a control signal of an electronic or electro-mechanical device.

44. The apparatus of claim 43, wherein downscaling the second component of motion includes downscaling the second component of motion based on a speed of the second component of motion and a speed of the first component of motion.

45. The apparatus of claim 44, wherein downscaling the second component of motion further includes downscaling the second component of motion in proportion to a function of an average speed of the second component of motion compared to an average speed of the first component of motion.

46. The apparatus of claim 44, wherein downscaling the second component of motion includes downscaling the second component of motion based on a ratio of the speed of the second component of motion to the speed of the first component of motion.

47. The apparatus of claim 43, further comprising:

an input generation module that selects an input activity, and generates, based on the selected input activity and the modified motion information, input of the electronic or electro-mechanical device.

48. The apparatus of claim 47, wherein modifying the motion information is performed in response to the selection of the input activity.

49. The apparatus of claim 47, wherein the tracking module further determines a number of contacts of the plurality of contacts, wherein the input activity is selected based on the number of contacts.

50. The apparatus of claim 47, wherein the input activity includes graphically manipulating a graphical object in multiple degrees of freedom, the multiple degrees of freedom corresponding to at least the first and second components of motion.

51. The apparatus of claim 50, wherein the first component of motion is translation motion and the second component of motion is scaling motion, the translation motion corresponding to a translation manipulation of the input activity, and the scaling motion corresponding to one of a resizing manipulation and a zooming manipulation of the input activity.

52. The apparatus of claim 50, wherein the first component of motion is translation motion and the second component of motion is rotational motion, the translation motion corresponding to a translation manipulation of the input activity, and the rotational motion corresponding to a rotational manipulation of the input activity.

53. The apparatus of claim 47, wherein the input generation module further discretizes the first component of motion into one or more units of motion, each unit of motion corresponding to a single input event of the input activity.

54. The apparatus of claim 53, wherein the generating input includes generating a single input event of the input activity based on one unit of motion, and then disabling input event generation for the input activity.

55. The apparatus of claim 47, wherein modifying the motion information further includes projecting the first component of motion onto a direction of the input activity.

56. The apparatus of claim 55, wherein the first component of motion is translation motion, and projecting the first component of motion includes projecting the translation motion onto a unit vector in the direction of the input activity.

57. The apparatus of claim 56, wherein the input activity includes an arrow key input, and the direction of the input activity corresponds to the direction of the arrow key input.

58. The apparatus of claim 47, wherein modifying the motion information further includes obtaining a motion sensitivity parameter corresponding to the input activity, and modifying the first component of motion based the motion sensitivity parameter.

59. The apparatus of claim 43, wherein computing the motion information includes computing one of the first and the second components of motion based on the tracking information of some, but not all, of the plurality of contacts.

60. The apparatus of claim 59, wherein computing one of the first and the second components of motion based on the tracking information of some, but not all, of the plurality of contacts includes computing one of rotational motion and scaling motion based on an innermost contact and an outermost contact.

61. The apparatus of claim 43, wherein downscaling the second component of motion includes clipping the second component of motion to zero.

62. The apparatus of claim 43, wherein downscaling the second component of motion includes applying a dead-zone filter to the first and second components of motion.

63. The apparatus of claim 43, wherein downscaling the second component of motion consists of applying a dead-zone filter to the first and second components of motion.