Gesture learning

• US 9,311,528 B2
• Filed: 06/15/2007
• Issued: 04/12/2016
• Est. Priority Date: 01/03/2007
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Westerman, Wayne Carl
• Primary Examiner(s): Utama, Robert J
• Application Number:
  US11/763,908
• Publication Number:
  US 20080163130A1
• Time in Patent Office: 3,224 Days
• Field of Search: 434/114, 434/156, 434/185
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Systems and methods that incorporate various techniques for teaching gestures to a user of a multi-touch sensitive device are disclosed herein. Such techniques can include presenting visible feedback of gestures, such as animated motion trails and/or hand motions, along with affirmative feedback for correctly performed gestures and negative feedback for incorrectly performed gestures. Such techniques can be expanded to provide training exercises that present tasks requiring a particular gesture or sequence of gestures to be performed. These training exercises can take the form of games or other engaging activities to encourage use.

Claims

1. A method for teaching gestures performable on a multi touch interface, the method comprising:

presenting a display on a touch sensing touch screen, the display comprising a first display area and a second display area, the second display area comprising a touch monitor window graphically distinct from and in an area of the touch screen non-overlapping with the first display area;

detecting a practice gesture currently being performed on the touch sensing touch screen in the first display area; and

presenting in the touch monitor window an interactive feedback mechanism that indicates an accuracy of the practice gesture currently being performed.

2. The method of claim 1 wherein the touch monitor window is overlaid atop the first display area.

3. The method of claim 2 wherein the touch monitor window incorporates transparency or translucency.

4. The method of claim 1 wherein the touch monitor window is displayed separately from the first display area.

5. The method of claim 1 wherein the interactive feedback mechanism comprises an animated hand together with one or more motion indications.

6. The method of claim 5 wherein the animated hand comprises one or more items selected from the group consisting of:

a line drawing, a three-dimensional rendering, and a translucent shadow of a hand.

7. The method of claim 5 wherein the one or more motion indications comprise motion trails.

8. The method of claim 7 wherein the motion trails are color-coded.

9. The method of claim 5 wherein the interactive feedback mechanism has superimposed thereon an animated display of a gesture so that a difference between the gesture currently being performed and an ideal gesture can be perceived.

10. The method of claim 1 wherein the first display area is used in conjunction with a utility, entertainment, or communication application.

11. The method of claim 1 wherein the first display area is used in conjunction with a gesture learning application.

12. The method of claim 11 wherein the gesture learning application is a game.

13. The method of claim 11 wherein:

a gesture to be performed by the user is presented in the first display area; and

wherein the method further comprises;

comparing the detected gesture to the gesture to be performed; and

if the detected gesture does not correspond to the gesture to be performed providing a negative feedback indicator; and

if the detected gesture does correspond to the gesture to be performed, providing a positive feedback indicator.

14. The method of claim 11 wherein:

a gesture to be performed by the user is presented in the first display area; and

wherein the method further comprises;

comparing the detected gesture to the gesture to be performed; and

if the detected gesture does not correspond to the gesture to be performed providing a negative feedback indicator; and

if the detected gesture does correspond to the gesture to be performed, displaying a next gesture to be performed by the user in the first display area.

16. The graphical user interface of claim 14 wherein the indication of a gesture to be performed displayed in the main window includes one or more items selected from the group consisting of:

an animated hand, one or more motion trails, an iconographic representation, a textual description of the gesture to be performed, and a textual description of a command corresponding to the gesture to be performed.

15. A graphical user interface display for a gesture learning application, the graphical user interface display comprising:

a main window displayed on a touch sensing touch screen including an indication of a gesture to be performed on the touch screen, the main window configured for detecting a practice gesture;

a touch monitor window graphically distinct from and in an area of the touch screen non-overlapping with the main window, the touch monitor window including an interactive feedback mechanism displayed on the touch screen, the interactive feedback mechanism including an indication of an accuracy of the practice gesture currently being performed.

17. The graphical user interface of claim 15 wherein the indication of the accuracy of the practice gesture currently being performed includes one or more items selected from the group consisting of:

an animated hand, one or more motion trails, an iconographic representation, a textual description of the gesture currently being performed, a textual description of a command corresponding to the gesture currently being performed, a positive feedback indicator, and a negative feedback indicator.

18. The graphical user interface of claim 17 wherein at least one of the positive feedback indicator and the negative feedback indicator comprise an audible indication.

19. The graphical user interface of claim 15 wherein the indication of a gesture to be performed in the main window comprises a sequence of gestures.

20. The graphical user interface of claim 19 wherein the sequence of gestures is arranged according to a particular chord or sequence of chords.

21. The graphical user interface of claim 19 wherein the sequence of gestures is arranged according to a particular motion or sequence of motions.

22. The graphical user interface of claim 19 further comprising at least one of a positive feedback indicator and a negative feedback indicator displayed in the main window.

23. A computer system having a touch interface and a graphical user interface, wherein the computer system includes a computer memory encoded with executable instructions for causing the computer system to:

present a display on a touch sensing touch screen, the display comprising a first display area and a second display area, the second display area comprising a touch monitor window graphically distinct from and in an area of the touch screen non-overlapping with the first display area;

detect a practice gesture currently being performed on the touch sensing touch screen in the first display area; and

present in the touch monitor window an interactive feedback mechanism that indicates an accuracy of the practice gesture currently being performed.

24. The computer system of claim 23 wherein the computer system is selected from the group consisting of a desktop computer, a tablet computer, and a notebook computer.

25. The computer system of claim 23 wherein the computer system comprises at least one of a handheld computer, a personal digital assistant, a media player, and a mobile telephone.

26. The computer system of claim 23, wherein the interactive feedback mechanism comprises an animated hand representative of the practice gesture perceived by the computer system together with one or more motion indications.

27. The computer system of claim 26 wherein the computer system is selected from the group consisting of a desktop computer, a tablet computer, and a notebook computer.

28. The computer system of claim 26 wherein the computer system comprises at least one of a handheld computer, a personal digital assistant, a media player, and a mobile telephone.

29. The computer system of claim 26, wherein the interactive feedback mechanism has superimposed thereon an animated display of an ideal gesture so that a difference between the practice gesture currently being performed and the ideal gesture can be perceived.

30. The computer system of claim 29 wherein the computer system is selected from the group consisting of a desktop computer, a tablet computer, and a notebook computer.

31. The computer system of claim 29 wherein the computer system comprises at least one of a handheld computer, a personal digital assistant, a media player, and a mobile telephone.

32. A computer system of claim 23 wherein the first display area is used in conjunction with a gesture learning application.

33. The computer system of claim 32 wherein the computer system is selected from the group consisting of a desktop computer, a tablet computer, and a notebook computer.

34. The computer system of claim 32 wherein the computer system comprises at least one of a handheld computer, a personal digital assistant, a media player, and a mobile telephone.

35. The computer system of claim 32, wherein:

the practice gesture to be performed is presented in the first display area; and

wherein the executable instructions further cause the computer system to;

compare the detected practice gesture to the gesture to be performed; and

if the detected practice gesture does not correspond to the gesture to be performed, provide a negative feedback indicator; and

if the detected gesture does correspond to the gesture to be performed, provide a positive feedback indicator.

36. The computer system of claim 35 wherein the computer system is selected from the group consisting of a desktop computer, a tablet computer, and a notebook computer.

37. The computer system of claim 35 wherein the computer system comprises at least one of a handheld computer, a personal digital assistant, a media player, and a mobile telephone.

38. The computer system of claim 32, wherein:

the gesture to be performed is presented in the first display area; and

wherein the executable instructions further cause the computer system to;

compare the detected practice gesture to the gesture to be performed; and

if the detected practice gesture does not correspond to the gesture to be performed provide a negative feedback indicator; and

if the detected gesture does correspond to the gesture to be performed, display a next gesture to be performed in the first display area.

39. The computer system of claim 38 wherein the computer system is selected from the group consisting of a desktop computer, a tablet computer, and a notebook computer.

40. The computer system of claim 38 wherein the computer system comprises at least one of a handheld computer, a personal digital assistant, a media player, and a mobile telephone.

50. A mobile telephone having a touch interface and a graphical user interface, wherein the mobile telephone includes the computer system of claim 23.

51. The mobile telephone of claim 50 wherein the interactive feedback mechanism comprises an animated hand together with one or more motion indications.

52. The mobile telephone of claim 51 wherein the interactive feedback mechanism has superimposed thereon an animated display of a gesture so that a difference between the gesture currently being performed and an ideal gesture can be perceived.

53. The mobile telephone of claim 50 wherein the first display area is used in conjunction with a gesture learning application.

54. The mobile telephone of claim 53 wherein:

a gesture to be performed is presented in the first display area; and

wherein the executable instructions further cause the mobile telephone to;

compare the detected gesture to the gesture to be performed; and

if the detected gesture does not correspond to the gesture to be performed provide a negative feedback indicator; and

if the detected gesture does correspond to the gesture to be performed, provide a positive feedback indicator.

55. The mobile telephone of claim 53 wherein:

a gesture to be performed is presented in the first display area; and

wherein the executable instructions further cause the mobile telephone to;

compare the detected gesture to the gesture to be performed; and

if the detected gesture does not correspond to the gesture to be performed provide a negative feedback indicator; and

if the detected gesture does correspond to the gesture to be performed, display a next gesture to be performed in the first display area.

59. The computer system of claim 23, wherein the detected practice gesture is a touch gesture.

41. A computer system having a touch interface and a graphical user interface, wherein the computer system includes a computer memory encoded with executable instructions for causing the computer system to generate a graphical user interface, the graphical user interface comprising:

a main window displayed on a touch sensing touch screen including an indication of a gesture to be performed on the touch screen, the main window configured for detecting a practice gesture; and

a touch monitor window graphically distinct from and in an area of the touch screen non-overlapping with the main window, the touch monitor window including an interactive feedback mechanism displayed on the touch screen, the interactive feedback mechanism including an indication of an accuracy of the practice gesture currently being performed.

42. The computer system of claim 41 wherein the computer system is selected from the group consisting of a desktop computer, a tablet computer, and a notebook computer.

43. The computer system of claim 41 wherein the computer system comprises at least one of a handheld computer, a personal digital assistant, a media player, and a mobile telephone.

44. The computer system of claim 41 wherein the indication of a gesture to be performed displayed in the main window includes one or more items selected from the group consisting of:

an animated hand, one or more motion trails, an iconographic representation, a textual description of the gesture to be performed, and a textual description of a command corresponding to the gesture to be performed.

45. The computer system of claim 44 wherein the computer system is selected from the group consisting of a desktop computer, a tablet computer, and a notebook computer.

46. The computer system of claim 44 wherein the computer system comprises at least one of a handheld computer, a personal digital assistant, a media player, and a mobile telephone.

47. The computer system of claim 41, wherein the indication of the accuracy of the practice gesture currently being performed includes one or more items selected from the group consisting of:

an animated hand, one or more motion trails, an iconographic representation, a textual description of the gesture currently being performed, a textual description of a command corresponding to the gesture currently being performed, a positive feedback indicator, and a negative feedback indicator.

48. The computer system of claim 47 wherein the computer system is selected from the group consisting of a desktop computer, a tablet computer, and a notebook computer.

49. The computer system of claim 47 wherein the computer system comprises at least one of a handheld computer, a personal digital assistant, a media player, and a mobile telephone.

56. A mobile telephone having a touch interface and a graphical user interface, wherein the mobile telephone includes the computer system of claim 41.

57. The mobile telephone of claim 56 wherein the indication of a gesture to be performed displayed in the main window includes one or more items selected from the group consisting of:

an animated hand, one or more motion trails, an iconographic representation, a textual description of the gesture to be performed, and a textual description of a command corresponding to the gesture to be performed.

58. The mobile telephone of claim 56 wherein the indication of the accuracy of the practice gesture currently being performed includes one or more items selected from the group consisting of:

an animated hand, one or more motion trails, an iconographic representation, a textual description of the gesture currently being performed, a textual description of a command corresponding to the gesture currently being performed, a positive feedback indicator, and a negative feedback indicator.

60. The computer system of claim 41, wherein the detected practice gesture is a touch gesture.