Gesturing with a multipoint sensing device

• US 20080036743A1
• Filed: 01/31/2007
• Published: 02/14/2008
• Est. Priority Date: 01/26/1998
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Computer Incorporated (Apple Inc.)
• Inventors: Parivar, Nima, Haggerty, Myra, Kerr, Duncan, Ording, Bas, Hotelling, Steven, Westerman, Wayne
• Granted Patent: US 9,292,111 B2
• Time in Patent Office: Days
• US Class Current: 345/173
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

Methods and systems for implementing gestures with sensing devices are disclosed. More particularly, methods and systems related to gesturing with multipoint sensing devices are disclosed.

Claims

1. An electronic system, comprising:

a multipoint sensing device that provides a multipoint sensing area for receiving inputs from one or more objects;

a gesture module configured to determine a gesture set for a given input arrangement received by the multipoint sensing area of the multipoint sensing device, to monitor the given input arrangement for one or more gesture events included in the gesture set, and to initiate input actions associated with a gesture event when the gesture event is performed with the input arrangement.

2. The electronic system as recited in claim 1 wherein the multipoint sensing device is a touch sensing device.

3. The electronic system as recited in claim 1 wherein the multipoint sensing device is a proximity sensing device.

4. The electronic system as recited in claim 1 wherein the gesture module is implemented by hardware, software or a combination of hardware and software.

5. The electronic system as recited in claim 1 wherein the electronic system is associated with a computer system including desktops, laptops, tablets or handheld computers.

6. The electronic system as recited in claim 1 wherein the electronic system is associated with handheld consumer electronic devices including cell phones, PDAs, or dedicated media players.

7. The electronic system as recited in claim 1 wherein the gesture set is also determined by one or more factors including location of the input arrangement within the sensing area, an open application, a state of the open application, input functionality, or additional input characteristics.

8. The electronic system as recited in claim 1 wherein the input actions are state commands.

9. The electronic system as recited in claim 1 wherein the input actions are manipulative commands.

10. The electronic system as recited in claim 1 wherein the gesture events are performed by the input arrangement.

11. The electronic system as recited in claim 1 wherein the gesture events are performed by a subset of the input arrangement.

12. The electronic system as recited in claim 1 wherein determining the gesture set includes recognizing a specific input arrangement, and wherein monitoring the given input arrangement includes recognizing gesture events associated with the gesture set.

13. The electronic system as recited in claim 12 wherein recognizing the gesture events includes analyzing characteristics of each input of the input arrangement, identifying a specific pattern associated with the input arrangement, and referring to the gesture set to see if the specific pattern matches a gesture event associated with the gesture set.

14. The electronic system as recited in claim 1 wherein gesture events are associated with touches which can be selected from motions, taps, pressure, and dwells.

15. The electronic system as recited in claim 14 wherein motions are selected from sliding, swiping, flicking, and rotation.

16. The electronic system as recited in claim 14 wherein motions are contraction or expansion between two or more inputs.

17. The electronic system as recited in claim 14 wherein taps are selected from ½

tap, full tap, multiple tap, quick tap and slow tap.

18. The electronic system as recited in claim 14 wherein pressure is associated with hard and light presses.

19. The electronic system as recited in claim 14 wherein dwell is associated with a long or short stationary pause.

20. The electronic system as recited in claim 14 wherein the gesture set is composed of a matrix of different gesture events and different commands.

21. The electronic system as recited in claim 20 wherein the gesture set is composed of a 3×

3 matrix of 9 different gesture events and 9 different commands.

22. The electronic system as recited in claim 20 wherein the 9 different gesture events include rotate CCW, rotate CW, spread, contract, left swipe, right swipe, up swipe, down swipe, and tap.

23. A gestural control method, comprising:

detecting multiple points within a sensing area at the same time;

determining a chord when one or more points are detected within the sensing area, the chord being a specific arrangement of points within the sensing area;

determining a gesture set associating commands to one or more gesture events;

monitoring points for gesture events; and

if a gesture event is recognized, performing command associated with gesture event.

24. The method as recited in claim 23 wherein the points are associated with touches that occur on a touch sensitive surface.

25. The method as recited in claim 23 wherein the points are associated with near touches that occur over a surface

26. The method as recited in claim 23 wherein the chord is made with mechanical features.

27. The method as recited in claim 23 wherein the chord is made with all or portions of one or more hands.

28. The method as recited in claim 23 wherein the chord is based on a number of unknown points.

29. The method as recited in claim 23 wherein the chord is based on a specific arrangement of known points.

30. The method as recited in claim 23 wherein the chord is based on whether the points are close together, in a neutral position or spread apart.

31. The method as recited in claim 23 wherein the chords are based on whether the contacts are adjacent or offset one another.

32. The method as recited in claim 23 wherein the chords are based on whether they are from the left or right hand.

33. The method as recited in claim 23 wherein determining the chord includes analyzing image created by objects touching or near touching the sensing area, and recognizing a specific arrangement of points from the image.

34. The method as recited in claim 23 wherein the gesture set is based on the chord.

35. The method as recited in claim 34 wherein the gesture set is also based on additional factors including location of the chord, open applications, state or more of the application, and/or characteristics of the points.

36. The method as recited in claim 23 wherein the gesture set is a associated with an input functionality.

37. The method as recited in claim 36 wherein the input functionality is selected from navigating operations, file operations, edit operations, viewing operations, formatting operations, tool operations, or web browsing operations.

38. The method as recited in claim 37 wherein the commands are selected from state commands and manipulative commands.

39. The method as recited in claim 23 wherein recognizing gesture events including analyzing characteristics of the points, identifying a specific pattern and referring to the gesture set and its list of gesture events, if a pattern matches a gesture event of the gesture set then the pattern is presumed to be the gesture event.

40. The method as recited in claim 39 wherein the characteristics of the points include first order considerations including motion, tapping, pressure or dwell, and second order considerations including speed, directions, orientation, size, duration, shape, or length.

41. The method as recited in claim 23 wherein gesture events are associated with motions, taps, pressure, and dwells.

42. The method as recited in claim 41 wherein motions are selected from sliding, swiping, flicking, and rotation.

43. The method as recited in claim 41 wherein motions are contraction or expansion between two or more inputs.

44. The method as recited in claim 41 wherein taps are selected from ½

tap, full tap, multiple tap, quick tap and slow tap.

45. The method as recited in claim 41 wherein pressure is associated with hard and light presses.

46. The method as recited in claim 41 wherein dwell is associated with a long or short stationary pause.

47. The method as recited in claim 23 wherein the gesture set is composed of a matrix of different gesture events and different commands.

48. The method as recited in claim 47 wherein the gesture set is composed of a 3×

3 matrix of 9 different gesture events and 9 different commands.

49. The method as recited in claim 48 wherein the 9 different gesture events include rotate CCW, rotate CW, spread, contract, left swipe, right swipe, up swipe, down swipe, and tap.

50. The method as recited in claim 23 further including determining whether or not a switching event has been performed, a switching event refers to an event that rests the operation or initiates a chord change.

51. The method as recited in claim 23 wherein the switching event is implemented by one of the following operations:

a) removing all points from sensing area for a predetermined amount of time, b) changing the chord, c) adding/subtracting points from sensing area, d) holding the points stationary for a predetermined amount of time, e) detecting a specific a key entry or button click, or f) detecting a gesture event.

52. A control operation, comprising:

detecting a touch or near touch;

if a touch is detected, determining a gesture set for the touch, the gesture set including one or more gesture events for provoking or initiating a command;

monitoring the touch for a gesture event associated with the gesture set; and

initiating a command when a gesture event associated with the gesture set is performed.

53. The operation as recited in claim 52 wherein the gesture set depends on touch characteristics, touch location, open application, and/or mode of application.

54. The operation as recited in claim 52 wherein the touch or near touch is detected via a touch device such as a touch pad, touch screen or touch sensitive housing.

55. A gesture operation, comprising:

monitor a touch motion;

differentiating the touch motion between first and second states;

performing a first action if the touch motion is associated with the first state; and

performing a second action if the touch motion is associated with the second state.

56. The gesture operation as recited in claim 55 wherein the touch motion is a sliding motion, and wherein the first state is a flick, and the second state is a swipe.

57. A control operation, comprising:

providing a first input device and a second input device, the first input device including a touch sensitive surface;

monitoring the first input device for input events;

simultaneously monitoring the second input device for input events;

performing input operations in accordance with input events associated with the first input device;

simultaneously performing input operations in accordance with input events associated with the second input device.

58. The control operation as recited in claim 57 wherein the first input device is a mouse.

59. A control operation, comprising:

providing a list of input functions, the input function having commands and gesture events that are linked to the commands, the commands are related to the input function;

assigning input functions to chords; and

linking the input function to chord when the chord is recognized.

60. The control operation as recited in claim 59 wherein the input function is selected from navigating operations, file operations, edit operations, viewing operations, formatting operations, tool operations, or web browsing operations.

61. The control operation as recited in claim 59 wherein the gesture events are actions that are performed by the chord.

62. The control operation as recited in claim 59 wherein the gesture events are selected from rotations, translations as a group, translation relative to one another, and/or taps.

63. The control operation as recited in claim 59 wherein the gesture events are linked to commands in an intuitive manner.

64. The control operation as recited in claim 59 wherein complimentary commands and linked to complimentary gesture events.

65. The control operation as recited in claim 59 wherein a chord is an arrangement of fingers.

66. The control operation as recited in claim 59 wherein assigning input functions to chords includes selecting an input function from a menu of input functions, and then selecting a chord from a menu list of chords.

67. A control panel, comprising:

displaying a menu of input functions;

displaying a gesture set that shows commands associated with a selected input function, and gesture events assigned to the commands;

displaying a list of chords,

wherein in order to set up a gesture map, a user selects desired input function from list of input functions and selects a desired chord from the list of chords.

68. A gesture design process, comprising:

ranking chords from easiest to hardest to perform;

ranking frequency of operations from most frequent to less frequent;

substantially matching the easiest chords with the most frequent operations, and matching the hardest chords with the less frequent operations; and

fine tuning the chord/operation grouping.

69. The process as recited in claim 68 wherein chords are ranked based on one or more attributes selected from speed, reliability, comfort, ease of use, simplicity or size of sensing area.

70. The process as recited in claim 68 wherein fine tuning is based one or more factors including complimentary commands, intuitiveness, cross talk, or organization.

71. A gesture operation, comprising:

detecting a first finger;

determining the state of the finger;

detecting one or more additional fingers;

determining the state of additional fingers; and

implementing different input modes based on timing of states of the first and additional fingers relative to one another.

72. The gesture operation as recited in claim 71 wherein if the first finger is in a first state of two states when the additional fingers are in a first state of two states then a first input mode is implemented, otherwise a second input mode is implemented.

73. The gesture operation as recited in claim 71 wherein the state of the first finger includes moving or stationary, and wherein the state of the additional fingers includes present or not, and wherein if the first finger is moving then a pointing input mode is implemented, and if the first finger is stationary when the additional fingers are present then a dragging mode is implemented.