Enhanced input using recognized gestures

• US 10,025,390 B2
• Filed: 10/21/2016
• Issued: 07/17/2018
• Est. Priority Date: 02/27/2008
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Hildreth, Evan, MacDougall, Francis
• Primary Examiner(s): Nguyen, Kevin
• Application Number:
  US15/331,438
• Publication Number:
  US 20170038852A1
• Time in Patent Office: 634 Days
• Field of Search: 715839, 715769, 715781, 715764, 715523
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

A representation of a user can move with respect to a graphical user interface based on input of a user. The graphical user interface comprises a central region and interaction elements disposed outside of the central region. The interaction elements are not shown until the representation of the user is aligned with the central region. A gesture of the user is recognized, and, based on the recognized gesture, the display of the graphical user interface is altered and an application control is outputted.

Claims

1. A computer-implemented method of interacting with a user interface comprising:

detecting a plurality of users in a first image obtained from a camera;

selecting a first user of the plurality of users based on a priority of the first user among the plurality of the users, the priority being assigned based on an identity of one or more of the plurality of users;

providing control of the user interface to the first user of the plurality of users based on the selection;

recognizing, from at least a second image obtained from the camera, a gesture of the first user; and

interacting with the user interface based on the recognized gesture.

2. The computer-implemented method of claim 1, wherein the recognized gesture comprises a gesture that transfers control to a second user of the plurality of users, the method further comprising providing control of the user interface to the second user of the plurality of users based on the transfer.

3. The computer-implemented method of claim 2, further comprising:

recognizing, from at least a third image obtained from the camera, a second gesture of the first user; and

providing control of the user interface to the first user of the plurality of users based on the recognized second gesture.

4. The computer-implemented method of claim 2, further comprising providing control of the user interface to the first user of the plurality of users after a threshold period of inactivity has elapsed.

5. The computer-implemented method of claim 1, further comprising causing the user interface to display one or more interaction elements.

6. The computer-implemented method of claim 5, further comprising:

identifying the recognized gesture as an engagement gesture; and

causing the user interface to display the one or more interaction elements, based, at least in part, on the identifying the recognized gesture as the engagement gesture.

7. The computer-implemented method of claim 5, wherein causing the user interface to display the one or more interaction elements comprises:

selecting one or more interaction elements, based, at least in part, on an identity of the first user; and

causing the user interface to display the selected one or more interaction elements.

8. The computer-implemented method of claim 5, wherein causing the user interface to display the one or more interaction elements comprises:

selecting one or more interaction elements, based, at least in part, on a recognized demographic of at least one user of the plurality of users; and

causing the user interface to display the selected one or more interaction elements with the user interface.

9. The computer-implemented method of claim 1, wherein the selecting the first user of the plurality of users based on the priority of the first user among the plurality of the users further comprises determining that the priority of the first user is higher than a priority of a second user of the plurality of users.

10. The computer-implemented method of claim 9, wherein the determination that the priority of the first user is higher than the priority of the second user of the plurality of users comprises:

performing facial recognition to determine an identity of the first user and an identity of the second user; and

determining based, at least in part, on the identity of the first user and the identity of the second user, that the first user has a higher priority than the second user.

11. A device comprising:

a display configured to show a graphical user interface;

a processor communicatively coupled with the display and configured to;

detect a plurality of users in a first image obtained from a camera;

select a first user of the plurality of users based on a priority of the first user among the plurality of the users, the priority being assigned based on an identity of one or more of the plurality of users;

provide control of the graphical user interface to the first user of the plurality of users based on the selection;

recognize, from at least a second image obtained from the camera, a gesture of the first user; and

interact with the graphical user interface based on the recognized gesture.

12. The device of claim 11, wherein the processor is further configured to:

recognize the gesture of the first user as a gesture that transfers control to a second user of the plurality of users; and

provide control of the graphical user interface to the second user of the plurality of users based on the transfer.

13. The device of claim 12, wherein the processor is further configured to:

recognize, from at least a third image obtained from the camera, a second gesture of the first user; and

provide control of the graphical user interface to the first user of the plurality of users based on the recognized second gesture.

14. The device of claim 12, wherein the processor is further configured to provide control of the graphical user interface to the first user of the plurality of users after a threshold period of inactivity has elapsed.

15. The device of claim 11, wherein the processor is further configured to cause the graphical user interface to display one or more interaction elements.

16. The device of claim 15, wherein the processor is further configured to:

identify the recognized gesture as an engagement gesture; and

cause the graphical user interface to display the one or more interaction elements, based, at least in part, on the identifying the recognized gesture as the engagement gesture.

17. The device of claim 15, wherein the processor is further configured to cause the graphical user interface to display the one or more interaction elements, at least in part, by:

selecting one or more interaction elements, based, at least in part, on an identity of the first user; and

causing the graphical user interface to display the selected one or more interaction elements with the graphical user interface.

18. The device of claim 15, wherein the processor is further configured to cause the graphical user interface to display the one or more interaction elements, at least in part, by:

selecting one or more interaction elements, based, at least in part, on a recognized demographic of at least one user of the plurality of users; and

causing the graphical user interface to display the selected one or more interaction elements with the graphical user interface.

19. The device of claim 11, wherein the processor is configured to select the first user of the plurality of users based on the priority of the first user among the plurality of the users by determining that the priority of the first user is higher than a priority of a second user of the plurality of users.

20. The device of claim 19, wherein the processor is configured to determine that the priority of the first user is higher than the priority of the second user of the plurality of users, at least in part, by:

performing facial recognition to determine an identity of the first user and an identity of the second user; and

determining based, at least in part, on the identity of the first user and the identity of the second user, that the first user has a higher priority than the second user.

21. A non-transitory computer-readable medium having instructions embedded thereon for interacting with a user interface, the instructions including computer code for:

detecting a plurality of users in a first image obtained from a camera;

selecting a first user of the plurality of users based on a priority of the first user among the plurality of the users, the priority being assigned based on an identity of one or more of the plurality of users;

providing control of the user interface to the first user of the plurality of users based on the selection;

recognizing, from at least a second image obtained from the camera, a gesture of the first user; and

interacting with the user interface based on the recognized gesture.

22. The non-transitory computer-readable medium of claim 21, further comprising computer code for recognizing the gesture of the first user as a gesture that transfers control to a second user of the plurality of users, and providing control of the user interface to the second user of the plurality of users based on the transfer.

23. The non-transitory computer-readable medium of claim 22, further comprising computer code for:

recognizing, from at least a third image obtained from the camera, a second gesture of the first user; and

providing control of the user interface to the first user of the plurality of users based on the recognized second gesture.

24. The non-transitory computer-readable medium of claim 22, further comprising computer code for providing control of the user interface to the first user of the plurality of users after a threshold period of inactivity has elapsed.

25. The non-transitory computer-readable medium of claim 21, further comprising computer code for causing the user interface to display one or more interaction elements.

26. The non-transitory computer-readable medium of claim 25, further comprising computer code for:

identifying the recognized gesture as an engagement gesture; and

causing the user interface to display the one or more interaction elements, based, at least in part, on the identifying the recognized gesture as the engagement gesture.

27. The non-transitory computer-readable medium of claim 25, wherein the computer code for causing the user interface to display the one or more interaction elements further comprises computer code for:

selecting one or more interaction elements, based, at least in part, on an identity of the first user; and

displaying the selected one or more interaction elements with the user interface.

28. The non-transitory computer-readable medium of claim 25, wherein the computer code for causing the user interface to display the one or more interaction elements further comprises computer code for:

selecting one or more interaction elements, based, at least in part, on a recognized demographic of at least one user of the plurality of users; and

displaying the selected one or more interaction elements with the user interface.

29. The non-transitory computer-readable medium of claim 21, wherein computer code for selecting the first user of the plurality of users based on the priority of the first user among the plurality of the users further comprises computer code for determining that the priority of the first user is higher than a priority of a second user of the plurality of users.

30. The non-transitory computer-readable medium of claim 29, wherein the computer code for determining that the priority of the first user is higher than the priority of the second user of the plurality of users comprises computer code for:

performing facial recognition to determine an identity of the first user and an identity of the second user; and

determining based, at least in part, on the identity of the first user and the identity of the second user, that the first user has a higher priority than the second user.

31. An apparatus comprising:

means for detecting a plurality of users in a first image obtained from image capture means;

means for selecting a first user of the plurality of users based on a priority of the first user among the plurality of the users, the priority being assigned based on an identity of one or more of the plurality of users;

means for providing control of the user interface to the first user of the plurality of users based on the selection;

means for recognizing, from at least a second image obtained from the image capture means, a gesture of the first user; and

means for interacting with the user interface based on the recognized gesture.

32. The apparatus of claim 31, wherein the recognized gesture comprises a gesture that transfers control to a second user of the plurality of users, the apparatus further comprising means for providing control of the user interface to the second user of the plurality of users based on the transfer.

33. The apparatus of claim 32, further comprising:

means for recognizing, from at least a third image obtained from the image capture means, a second gesture of the first user; and

means for providing control of the user interface to the first user of the plurality of users based on the recognized second gesture.

34. The apparatus of claim 32, further comprising means for providing control of the user interface to the first user of the plurality of users after a threshold period of inactivity has elapsed.

35. The apparatus of claim 31, further comprising means for causing the user interface to display one or more interaction elements.

36. The apparatus of claim 35, further comprising:

means for identifying the recognized gesture as an engagement gesture; and

means for causing the user interface to display the one or more interaction elements, based, at least in part, on the identifying the recognized gesture as the engagement gesture.

37. The apparatus of claim 35, wherein the means for causing the user interface to display the one or more interaction elements comprises:

means for selecting one or more interaction elements, based, at least in part, on an identity of the first user; and

means for causing the user interface to display the selected one or more interaction elements.

38. The apparatus of claim 35, wherein the means for causing the user interface to display the one or more interaction elements comprises:

means for selecting one or more interaction elements, based, at least in part, on a recognized demographic of at least one user of the plurality of users; and

means for causing the user interface to display the selected one or more interaction elements with the user interface.

39. The apparatus of claim 31, wherein the means for selecting the first user of the plurality of users based on the priority of the first user among the plurality of the users further comprises means for determining that the priority of the first user is higher than a priority of a second user of the plurality of users.

40. The apparatus of claim 39, wherein the means for determining that the priority of the first user is higher than the priority of the second user of the plurality of users comprises:

means for performing facial recognition to determine an identity of the first user and an identity of the second user; and

means for determining based, at least in part, on the identity of the first user and the identity of the second user, that the first user has a higher priority than the second user.