Enhanced input using recognized gestures

• US 9,507,432 B2
• Filed: 09/17/2015
• Issued: 11/29/2016
• Est. Priority Date: 02/27/2008
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Hildreth, Evan, MacDougall, Francis
• Primary Examiner(s): Nguyen, Kevin
• Application Number:
  US14/857,488
• Publication Number:
  US 20160077594A1
• Time in Patent Office: 439 Days
• Field of Search: 715/863, 715/856, 715701-709
• US Class Current: 1/1
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

A representation of a user can move with respect to a graphical user interface based on input of a user. The graphical user interface comprises a central region and interaction elements disposed outside of the central region. The interaction elements are not shown until the representation of the user is aligned with the central region. A gesture of the user is recognized, and, based on the recognized gesture, the display of the graphical user interface is altered and an application control is outputted.

Claims

1. A computer-implemented method comprising:

causing a display to show interaction elements in a graphical user interface configured in a first mode;

detecting an activity of a user that indicates an engagement with the user interface in the first mode;

transitioning to a second mode in response to the detecting, wherein transitioning to the second mode comprises;

causing the display to show a representation of the user along with the interaction items in the graphical user interface, wherein the representation of the user is not shown in the first mode; and

monitoring for gestures of the user;

detecting a certain gesture of the user while in the second mode;

causing the graphical user interface to be altered based on the certain gesture; and

outputting an application control based on the certain gesture.

2. The computer-implemented method of claim 1, wherein:

detecting the activity of the user that indicates an engagement with the user interface in the first mode comprises recognizing an engagement gesture while the graphical user interface is operating in the first mode.

3. The computer-implemented method of claim 1 wherein the certain gesture is a gesture to select an interaction element.

4. The computer-implemented method of claim 1, wherein the graphical user interface is initially configured in the first mode based on a user configuration.

5. The computer-implemented method of claim 1, further comprising:

recognizing, from a plurality of images, a realignment gesture; and

in response to recognizing the realignment gesture, displaying the representation of the user in a central region of the graphical user interface.

6. The computer-implemented method of claim 1, wherein the representation of the user further comprises a cursor, an image, an icon, or an avatar.

7. The computer-implemented method of claim 1, wherein the interaction elements are disposed outside a central region of the graphical user interface.

8. The computer-implemented method of claim 1, wherein the interaction elements are not displayed until the representation of the user is aligned with a central region of the graphical user interface.

9. The computer-implemented method of claim 1, wherein detecting the certain gesture of the user while in the second mode comprises:

obtaining a plurality of images of the user captured from a camera; and

recognizing, from the plurality of images, the certain gesture of the user.

10. A device comprising:

a display configured to show interaction elements in a graphical user interface configured in a first mode;

a processor communicatively coupled with the display and configured to;

detect an activity of a user that indicates an engagement with the user interface in the first mode;

transition to a second mode in response to the detecting, wherein transitioning to the second mode comprises;

causing the display to show a representation of the user along with the interaction items in the graphical user interface, wherein the representation of the user is not shown in the first mode; and

monitoring for gestures of the user;

detect a certain gesture of the user while in the second mode;

cause the graphical user interface to be altered based on the certain gesture; and

output an application control based on the certain gesture.

11. The device of claim 10, wherein the processor is further configured to:

detect the activity of the user that indicates an engagement with the user interface in the first mode by recognizing an engagement gesture while the graphical user interface is operating in the first mode; and

in response to recognizing the engagement gesture, cause the graphical user interface to switch to the second mode if a subsequent gesture is not recognized within a threshold period of time.

12. The device of claim 10, wherein the processor is further configured to detect the certain gesture by detecting a gesture to select an interaction element.

13. The device of claim 10, wherein the processor is further configured to operate the graphical user interface in either the first mode or the second mode based on a user configuration.

14. The device of claim 10, wherein the processor is further configured to:

recognize, from a plurality of images, a realignment gesture; and

in response to recognizing the realignment gesture, display the representation of the user in a central region of the graphical user interface.

15. The device of claim 10, wherein the processor is further configured to cause the representation of the user to comprise a cursor, an image, an icon, or an avatar.

16. The device of claim 10, wherein the processor is further configured to cause the interaction elements to be disposed outside a central region of the graphical user interface.

17. The device of claim 10, wherein the processor is further configured to not display the interaction elements until the representation of the user is aligned with a central region of the graphical user interface.

18. The device of claim 10, wherein the processor is configured to detect the certain gesture of the user while in the second mode by:

obtaining a plurality of images of the user captured from a camera; and

recognizing, from the plurality of images, the certain gesture of the user.

19. A non-transitory computer-readable media having instructions embedded thereon for implementing the graphical user interface, the instructions including computer code for:

causing a display to show interaction elements in the graphical user interface configured in a first mode;

detecting an activity of a user that indicates an engagement with the user interface in the first mode;

transitioning to a second mode in response to the detecting, wherein transitioning to the second mode comprises;

causing the display to show a representation of the user along with the interaction items in the graphical user interface, wherein the representation of the user is not shown in the first mode; and

monitoring for gestures of the user;

detecting a certain gesture of the user while in the second mode;

causing the graphical user interface to be altered based on the certain gesture; and

outputting an application control based on the certain gesture.

20. The non-transitory computer-readable media of claim 19, wherein:

the computer code for detecting the activity of the user that indicates an engagement with the user interface in the first mode comprises computer code for recognizing an engagement gesture while the graphical user interface is operating in the first mode; and

the instructions further comprise computer code for, in response to recognizing the engagement gesture, causing the graphical user interface to switch to the second mode if a subsequent gesture is not recognized within a threshold period of time.

21. The non-transitory computer-readable media of claim 19, wherein the computer code for detecting the certain gesture comprises computer code for detecting a gesture to select an interaction element.

22. The non-transitory computer-readable media of claim 19, wherein the instructions further comprise computer code for operating the graphical user interface in either the first mode or the second mode based on a user configuration.

23. The non-transitory computer-readable media of claim 19, wherein the instructions further comprise computer code for:

recognizing, from a plurality of images, a realignment gesture; and

in response to recognizing the realignment gesture, displaying the representation of the user in a central region of the graphical user interface.

24. The non-transitory computer-readable media of claim 19, wherein the instructions further comprise computer code for causing the representation of the user to comprise a cursor, an image, an icon, or an avatar.

25. The non-transitory computer-readable media of claim 19, wherein the instructions further comprise computer code for causing the interaction elements to be disposed outside a central region of the graphical user interface.

26. The non-transitory computer-readable media of claim 19, wherein the instructions further comprise computer code for not displaying the interaction elements until the representation of the user is aligned with a central region of the graphical user interface.

27. The non-transitory computer-readable media of claim 19, wherein the computer code for detecting the certain gesture of the user while in the second mode comprises computer code for:

obtaining a plurality of images of the user captured from a camera; and

recognizing, from the plurality of images, the certain gesture of the user.

28. An apparatus comprising:

means for causing a display to show interaction elements in a graphical user interface configured in a first mode;

means for detecting an activity of a user that indicates an engagement with the user interface in the first mode;

means for transitioning to a second mode in response to the detecting, wherein the means for transitioning to the second mode comprises;

means for causing the display to show a representation of the user along with the interaction items in the graphical user interface, wherein the representation of the user is not shown in the first mode; and

means for monitoring for gestures of the user;

means for detecting a certain gesture of the user while in the second mode;

means for causing the graphical user interface to be altered based on the certain gesture; and

means for outputting an application control based on the certain gesture.