Detecting and interpreting real-world and security gestures on touch and hover sensitive devices

• US 20080168403A1
• Filed: 06/13/2007
• Published: 07/10/2008
• Est. Priority Date: 01/06/2007
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Haggerty, Myra Mary, Westerman, Wayne Carl
• Granted Patent: US 7,877,707 B2
• Time in Patent Office: Days
• US Class Current: 715/863
• CPC Class Codes: G06F 2203/04101 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup
• CPC Class Codes: G06F 2203/04104 Multi-touch detection in digitiser, i.e. details about the simultaneous detection of a plurality of touching locations, e.g. multiple fingers or pen and finger
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/0486 Drag-and-drop
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

“Real-world” gestures such as hand or finger movements/orientations that are generally recognized to mean certain things (e.g., an “OK” hand signal generally indicates an affirmative response) can be interpreted by a touch or hover sensitive device to more efficiently and accurately effect intended operations. These gestures can include, but are not limited to, “OK gestures,” “grasp everything gestures,” “stamp of approval gestures,” “circle select gestures,” “X to delete gestures,” “knock to inquire gestures,” “hitchhiker directional gestures,” and “shape gestures.” In addition, gestures can be used to provide identification and allow or deny access to applications, files, and the like.

Claims

1. A method for receiving an affirmative gesture formed on or about a sensor panel, comprising:

detecting one or more images at the sensor panel generated from a hand formed in a shape of an OK sign;

determining that the one or more images are arranged in a pattern corresponding to a predetermined OK gesture;

determining a centering parameter from the one or more images;

associating the OK gesture with a user interface (UI) element coincident with the centering parameter; and

performing an affirmative action in accordance with the UI element.

2. The method of claim 1, further comprising determining that the one or more images are arranged in a pattern corresponding to a predetermined OK gesture by:

identifying one or more palm edge and pinky features; and

identifying a thumb and finger feature.

3. A method for receiving a grouping gesture formed on or about a sensor panel, comprising:

detecting one or more images at the sensor panel generated from a hand changing from a palm-down, outstretched shape to a clenched shape;

determining that the one or more images are arranged in space and time in a pattern and sequence corresponding to a predetermined grasp everything gesture;

determining a circumferential boundary from the one or more images;

associating the grasp everything gesture with user interface (UI) elements within the circumferential boundary; and

performing a grouping action in accordance with the UI elements within the circumferential boundary.

4. The method of claim 3, further comprising determining that the one or more images are arranged in space and time in a pattern and sequence corresponding to a predetermined grasp everything gesture by identifying five initial images corresponding to outstretched fingers and thumb and a palm heel image from a single hand;

and

determining that the five initial images move closer to the palm heel image over time.

5. A method for receiving an affirmative gesture formed on or about a sensor panel, comprising:

detecting one or more images at the sensor panel generated from a clenched fist;

determining that the one or more images are arranged in a pattern corresponding to a predetermined stamp of approval gesture;

determining a centering parameter from the one or more images;

associating the stamp of approval gesture with a user interface (UI) element coincident with the centering parameter; and

performing an affirmative action in accordance with the UI element.

6. The method of claim 5, further comprising determining that the one or more images are arranged in a pattern corresponding to a predetermined stamp of approval gesture by:

tracking the one or more images over time; and

determining that the one or more images rapidly appeared and then disappeared.

7. A method for receiving a selection gesture formed on or about a sensor panel, comprising:

detecting an image at the sensor panel generated from a single finger;

tracking movement of the image over time;

determining that the movement of the image is arranged in space and time in a pattern and sequence corresponding to a predetermined circle select gesture;

determining a circumferential boundary from the movement of the image;

associating the circle select gesture with user interface (UI) elements within the circumferential boundary; and

performing a selecting action in accordance with the UI elements within the circumferential boundary.

8. The method of claim 7, further comprising determining that the movement of the image is arranged in space and time in a pattern and sequence corresponding to a predetermined circle select gesture by:

determining that the image substantially returns to its initial position over time.

9. A method for receiving a deletion gesture formed on or about a sensor panel, comprising:

detecting a first image at the sensor panel generated from a single finger and representative of a first touch;

tracking movement of the first image over time;

detecting a second image at the sensor panel generate from the same finger and representative of a second touch;

tracking movement of the second image over time;

determining that the movement of the first and second images are arranged in space and time in a pattern and sequence corresponding to a predetermined X to delete gesture;

determining an intersection of the movements of the first and second images;

associating the X to delete gesture with a user interface (UI) element coincident with the intersection; and

performing a deleting action in accordance with the UI element coincident with the intersection.

10. The method of claim 9, further comprising determining that the movement of the first and second images are arranged in space and time in a pattern and sequence corresponding to a predetermined X to delete gesture by:

determining that the first and second images were detected within a predetermined time interval.

11. A method for receiving an inquiry gesture formed on or about a sensor panel, comprising:

detecting one or more substantially linearly arranged first images at the sensor panel generated from one or more knuckles in a clenched fist and representative of a first knock;

detecting one or more substantially linearly arranged second images at the sensor panel generated from the same knuckles and representative of a second knock;

determining that the first and second images are arranged in space and time in a pattern and sequence corresponding to a predetermined knock to inquire gesture;

determining centering parameter from the first or second images;

associating the knock to inquire gesture with a user interface (UI) element coincident with the centering parameter; and

performing an inquiry action in accordance with the UI element coincident with the centering parameter.

12. The method of claim 11, further comprising determining that the first and second images are arranged in space and time in a pattern and sequence corresponding to a predetermined knock to inquire gesture by:

determining that the first and second images were detected within a predetermined time interval.

13. A method for receiving a directional gesture formed on or about a sensor panel, comprising:

detecting one or more images at the sensor panel generated from a hand with thumb extended and all other fingers curled;

determining that the one or more images are arranged in a pattern corresponding to a predetermined hitchhiker gesture;

determining a directional parameter from the one or more images;

associating the hitchhiker gesture with a user interface (UI) element; and

performing a directional action in accordance with the UI element.

14. The method of claim 13, further comprising determining that the one or more images are arranged in a pattern corresponding to a predetermined hitchhiker gesture by:

identifying four curled finger features;

identifying an extended thumb feature; and

identifying an palm heel feature.

15. A method for receiving a shape gesture formed on or about a sensor panel, comprising:

detecting one or more images at the sensor panel generated from a hand formed in a particular shape;

determining that the one or more images are arranged in a pattern corresponding to a predetermined shape gesture;

associating the shape gesture with a user interface (UI) element; and

performing an action associated with the shape gesture upon the UI element.

16. The method of claim 15, wherein the shape gesture is a triangle shape gesture.

17. The method of claim 15, wherein the shape gesture is a V shape gesture.

18. The method of claim 15, wherein the shape gesture is an inverted V shape gesture.

19. The method of claim 15, wherein the shape gesture is a wing shape gesture.

20. A method for receiving an identification gesture formed on or about a sensor panel, comprising:

detecting one or more images at the sensor panel;

determining that the one or more images represent an attempted identification gesture;

associating the attempted identification gesture with a user interface (UI) element; and

determining whether the movement of the one or more images are arranged in space and time in a pattern and sequence corresponding to a user authorized to access the UI element; and

granting or denying access to the UI element in accordance with the determination of whether the attempted identification gesture corresponds to a user authorized to access the UI element.

21. A computer-readable medium comprising program code for receiving an affirmative gesture formed on or about a sensor panel, the program code for causing performance of a method comprising:

detecting one or more images at the sensor panel-generated from a hand formed in a shape of an OK sign;

determining that the one or more images are arranged in a pattern corresponding to a predetermined OK gesture;

determining a centering parameter from the one or more images;

associating the OK gesture with a user interface (UI) element coincident with the centering parameter; and

performing an affirmative action in accordance with the UI element.

22. The computer-readable medium of claim 21, the method further comprising determining that the one or more images are arranged in a pattern corresponding to a predetermined OK gesture by:

identifying one or more palm edge and pinky features; and

identifying a thumb and finger feature.

23. A computer-readable medium comprising program code for receiving a grouping gesture formed on or about a sensor panel, the program code for causing performance of a method comprising:

detecting one or more images at the sensor panel generated from a hand changing from a palm-down, outstretched shape to a clenched shape;

determining that the one or more images are arranged in space and time in a pattern and sequence corresponding to a predetermined grasp everything gesture;

determining a circumferential boundary from the one or more images;

associating the grasp everything gesture with user interface (UI) elements within the circumferential boundary; and

performing a grouping action in accordance with the UI elements within the circumferential boundary.

24. The computer-readable medium of claim 23, the method further comprising determining that the one or more images are arranged in space and time in a pattern and sequence corresponding to a predetermined grasp everything gesture by:

identifying five initial images corresponding to outstretched fingers and thumb and a palm heel image from a single hand; and

determining that the five initial images move closer to the palm heel image over time.

25. A computer-readable medium comprising program code for receiving an affirmative gesture formed on or about a sensor panel, the program code for causing performance of a method comprising:

detecting one or more images at the sensor panel generated from a clenched fist;

determining that the one or more images are arranged in a pattern corresponding to a predetermined stamp of approval gesture;

determining a centering parameter from the one or more images;

associating the stamp of approval gesture with a user interface (UI) element coincident with the centering parameter; and

performing an affirmative action in accordance with the UI element.

26. The computer-readable medium of claim 25, the method further comprising determining that the one or more images are arranged in a pattern corresponding to a predetermined stamp of approval gesture by:

tracking the one or more images over time; and

determining that the one or more images rapidly appeared and then disappeared.

27. A computer-readable medium comprising program code for receiving a selection gesture formed on or about a sensor panel, the program code for causing performance of a method comprising:

detecting an image at the sensor panel generated from a single finger;

tracking movement of the image over time;

determining that the movement of the image is arranged in space and time in a pattern and sequence corresponding to a predetermined circle select gesture;

determining a circumferential boundary from the movement of the image;

associating the circle select gesture with user interface (UI) elements within the circumferential boundary; and

performing a selecting action in accordance with the UI elements within the circumferential boundary.

28. The computer-readable medium of claim 27, the method further comprising determining that the movement of the image is arranged in space and time in a pattern and sequence corresponding to a predetermined circle select gesture by:

determining that the image substantially returns to its initial position over time.

29. A computer-readable medium comprising program code for receiving a deletion gesture formed on or about a sensor panel, the program code for causing performance of a method comprising:

detecting a first image at the sensor panel generated from a single finger and representative of a first touch;

tracking movement of the first image over time;

detecting a second image at the sensor panel generate from the same finger and representative of a second touch;

tracking movement of the second image over time;

determining that the movement of the first and second images are arranged in space and time in a pattern and sequence corresponding to a predetermined X to delete gesture;

determining an intersection of the movements of the first and second images;

associating the X to delete gesture with a user interface (UI) element coincident with the intersection; and

performing a deleting action in accordance with the UI element coincident with the intersection.

30. The computer-readable medium of claim 29, the method further comprising determining that the movement of the first and second images are arranged in space and time in a pattern and sequence corresponding to a predetermined X to delete gesture by:

determining that the first and second images were detected within a predetermined time interval.

31. A computer-readable medium comprising program code for receiving an inquiry gesture formed on or about a sensor panel, the program code for causing performance of a method comprising:

detecting one or more substantially linearly arranged first images at the sensor panel generated from one or more knuckles in a clenched fist and representative of a first knock;

detecting one or more substantially linearly arranged second images at the sensor panel generated from the same knuckles and representative of a second knock;

determining that the first and second images are arranged in space and time in a pattern and sequence corresponding to a predetermined knock to inquire gesture;

determining centering parameter from the first or second images;

associating the knock to inquire gesture with a user interface (UI) element coincident with the centering parameter; and

performing an inquiry action in accordance with the UI element coincident with the centering parameter.

32. The computer-readable medium of claim 31, the method further comprising determining that the first and second images are arranged in space and time in a pattern and sequence corresponding to a predetermined knock to inquire gesture by:

determining that the first and second images were detected within a predetermined time interval.

33. A computer-readable medium comprising program code for receiving a directional gesture formed on or about a sensor panel, the program code for causing performance of a method comprising:

detecting one or more images at the sensor panel generated from a hand with thumb extended and all other fingers curled;

determining that the one or more images are arranged in a pattern corresponding to a predetermined hitchhiker gesture;

determining a directional parameter from the one or more images;

associating the hitchhiker gesture with a user interface (UI) element; and

performing a directional action in accordance with the UI element.

34. The computer-readable medium of claim 33, the method further comprising determining that the one or more images are arranged in a pattern corresponding to a predetermined hitchhiker gesture by:

identifying four curled finger features;

identifying an extended thumb feature; and

identifying an palm heel feature.

35. A computer-readable medium comprising program code for receiving a shape gesture formed on or about a sensor panel, the program code for causing performance of a method comprising:

detecting one or more images at the sensor panel generated from a hand formed in a particular shape;

determining that the one or more images are arranged in a pattern corresponding to a predetermined shape gesture;

associating the shape gesture with a user interface (UI) element; and

performing an action associated with the shape gesture upon the UI element.

36. The computer-readable medium of claim 35, wherein the shape gesture is a triangle shape gesture.

37. The computer-readable medium of claim 35, wherein the shape gesture is a V shape gesture.

38. The computer-readable medium of claim 35, wherein the shape gesture is an inverted V shape gesture.

39. The computer-readable medium of claim 35, wherein the shape gesture is a wing shape gesture.

40. A computer-readable medium comprising program code for receiving an identification gesture formed on or about a sensor panel, the program code for causing performance of a method comprising:

detecting one or more images at the sensor panel;

determining that the one or more images represent an attempted identification gesture;

associating the attempted identification gesture with a user interface (UI) element; and

determining whether the movement of the one or more images are arranged in space and time in a pattern and sequence corresponding to a user authorized to access the UI element; and

granting or denying access to the UI element in accordance with the determination of whether the attempted identification gesture corresponds to a user authorized to access the UI element.

41. A method for receiving a security gesture formed on or about a sensor panel, comprising:

detecting one or more images at the sensor panel generated from two fingers and two thumbs of two hands touching near a center of the sensor panel;

determining that the one or more images correspond to a predetermined unlock gesture; and

performing an unlocking action.

42. The method of claim 41, further comprising determining that the one or more images correspond to a predetermined unlock gesture by

identifying four initial images corresponding to the two fingers and two thumbs from the two hands;

and

determining that the four initial images move towards the corners of the sensor panel over time.

43. A method for receiving a security gesture formed on or about a sensor panel, comprising:

detecting four images at the sensor panel generated from two fingers and two thumbs of two hands, each image touching near a different corner of the sensor panel;

determining that the four images correspond to a predetermined lock gesture; and

performing an locking action.

44. The method of claim 43, further comprising determining that the one or more images correspond to a predetermined lock gesture by determining that the four initial images move towards a center of the sensor panel over time.

45. A method for receiving a hand edge gesture formed on or about a sensor panel, comprising:

detecting one or more images at a first location on the sensor panel generated from a side of a palm and pinky finger rotating or translating to a second location on the sensor panel;

determining that the one or more images are arranged in space and time in a pattern and sequence corresponding to a predetermined hand edge gesture; and

performing a predetermined action in accordance with the detected hand edge gesture.

46. The method of claim 45, further comprising determining that the one or more images are arranged in space and time in a pattern and sequence corresponding to a predetermined hand edge gesture by

identifying one or more initial images corresponding to the side of the palm and pinky finger from a single hand;

and

determining that the one or more initial images rotate over time.

47. The method of claim 45, further comprising determining that the one or more images are arranged in space and time in a pattern and sequence corresponding to a predetermined hand edge gesture by

identifying one or more initial images corresponding to the side of the palm and pinky finger from a single hand;

and

determining that the one or more initial images translate over time.

48. The method of claim 45, further comprising determining that the one or more images are arranged in space and time in a pattern and sequence corresponding to a predetermined hand edge gesture by

identifying one or more initial images corresponding to the side of the palm and pinky finger from a single hand;

and

determining that the one or more initial images move up and down in a vertical direction over time.

49. The method of claim 45, further comprising determining that the one or more images are arranged in space and time in a pattern and sequence corresponding to a predetermined hand edge gesture by

identifying one or more initial images corresponding to the side of the palm and pinky finger from a single hand;

and

determining that one or more of the initial images corresponding to the pinky finger rotate over time while the one or more initial images corresponding to the palm remain substantially stationary.

50. A method for receiving a framing gesture formed on or about a sensor panel, comprising:

detecting a plurality of images at the sensor panel generated from the palms and pinky fingers of two hands touching the sensor panel in an approximate upside-down U shape;

determining that the one or more images correspond to a predetermined framing gesture;

computing a frame area based on a location of the plurality of detected images; and

performing an action within the frame area.

51. A computer-readable medium comprising program code for receiving a security gesture formed on or about a sensor panel, the program code for causing performance of a method comprising:

detecting one or more images at the sensor panel generated from two fingers and two thumbs of two hands touching near a center of the sensor panel;

determining that the one or more images correspond to a predetermined unlock gesture; and

performing an unlocking action.

52. The computer-readable medium of claim 51, the program code further for causing performance of a method comprising determining that the one or more images correspond to a predetermined unlock gesture by

identifying four initial images corresponding to the two fingers and two thumbs from the two hands;

and

determining that the four initial images move towards the corners of the sensor panel over time.

53. A computer-readable medium comprising program code for receiving a security gesture formed on or about a sensor panel, the program code for causing performance of a method comprising:

detecting four images at the sensor panel generated from two fingers and two thumbs of two hands, each image touching near a different corner of the sensor panel;

determining that the four images correspond to a predetermined lock gesture; and

performing an locking action.

54. The computer-readable medium of claim 53, the program code further for causing performance of a method comprising determining that the one or more images correspond to a predetermined lock gesture by determining that the four initial images move towards a center of the sensor panel over time.

55. A computer-readable medium comprising program code for receiving a hand edge gesture formed on or about a sensor panel, the program code for causing performance of a method comprising:

detecting one or more images at a first location on the sensor panel generated from a side of a palm and pinky finger rotating or translating to a second location on the sensor panel;

determining that the one or more images are arranged in space and time in a pattern and sequence corresponding to a predetermined hand edge gesture; and

performing a predetermined action in accordance with the detected hand edge gesture.

56. The computer-readable medium of claim 55, the program code further for causing performance of a method comprising determining that the one or more images are arranged in space and time in a pattern and sequence corresponding to a predetermined hand edge gesture by

identifying one or more initial images corresponding to the side of the palm and pinky finger from a single hand;

and

determining that the one or more initial images rotate over time.

57. The computer-readable medium of claim 55, the program code further for causing performance of a method comprising determining that the one or more images are arranged in space and time in a pattern and sequence corresponding to a predetermined hand edge gesture by

identifying one or more initial images corresponding to the side of the palm and pinky finger from a single hand;

and

determining that the one or more initial images translate over time.

58. The computer-readable medium of claim 55, the program code further for causing performance of a method comprising determining that the one or more images are arranged in space and time in a pattern and sequence corresponding to a predetermined hand edge gesture by

identifying one or more initial images corresponding to the side of the palm and pinky finger from a single hand;

and

determining that the one or more initial images move up and down in a vertical direction over time.

59. The computer-readable medium of claim 55, the program code further for causing performance of a method comprising determining that the one or more images are arranged in space and time in a pattern and sequence corresponding to a predetermined hand edge gesture by

identifying one or more initial images corresponding to the side of the palm and pinky finger from a single hand;

and

determining that one or more of the initial images corresponding to the pinky finger rotate over time while the one or more initial images corresponding to the palm remain substantially stationary.

60. A computer-readable medium comprising program code for receiving a framing gesture formed on or about a sensor panel, the program code for causing performance of a method comprising:

detecting a plurality of images at the sensor panel generated from the palms and pinky fingers of two hands touching the sensor panel in an approximate upside-down U shape;

determining that the one or more images correspond to a predetermined framing gesture;

computing a frame area based on a location of the plurality of detected images; and

performing an action within the frame area.