Gesture based virtual devices

• US 9,874,977 B1
• Filed: 08/07/2012
• Issued: 01/23/2018
• Est. Priority Date: 08/07/2012
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Soyannwo, Olusanya T., Yan, Renwei, Crump, Edward Dietz, Meenakshisundaram, Sankara Narayana Hemanth
• Primary Examiner(s): Mehmood, Jennifer
• Assistant Examiner(s): Adams, Carl
• Application Number:
  US13/568,835
• Time in Patent Office: 1,995 Days
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/042 by opto-electronic means

Abstract

A sequence of 3D hand poses may be used to define a hand gesture, and different hand gestures may be represented by different sequences of poses. One or more hand gestures may be used to generate a virtual input device. Another one or more hand gestures may subsequently be used to provide input via the virtual input device.

Claims

1. A system comprising:

one or more processors;

an imaging sensor;

a projector; and

one or more computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising;

projecting content onto a display area within an environment using the projector;

capturing a sequence of images of the environment using the imaging sensor;

based at least in part on the sequence of images, detecting sequential poses of a hand of a user within the sequence of images;

based at least in part on detecting the sequential poses of the hand of the user, determining that a first gesture of the hand interacts with a physical object different from the user;

determining a first identity of the user;

determining, based at least in part on the first identity of the user, that the user is included within a group of users that are authorized to at least one of generate, customize, or access a virtual input device;

defining the physical object as the virtual input device to receive input from the user at a time in which the user further interacts with the physical object;

determining a second gesture of the hand based at least in part on the user interacting with the physical object;

performing an action based at least in part on the second gesture;

determining a third gesture with respect to the physical object;

determining a second identity of a second user associated with the third gesture;

determining, based at least in part on the second identity of the second user, that the second user is not included within the group of users; and

refraining from performing a second action associated with the third gesture based at least in part on the second user not being included within the group of users.

2. The system of claim 1, wherein:

determining the first gesture comprises identifying movement of a number of fingers of the hand; and

the movement of the number of fingers of the hand defines the boundary of the virtual input device.

3. The system of claim 2, wherein:

determining the second gesture of the hand comprises identifying movement of another number of fingers of the hand within the boundary of the virtual input device; and

performing the action comprises moving a proj ection of a cursor within the display area based at least in part on the movement of the another number of fingers of the hand within the boundary of the virtual input device.

4. The system of claim 1, wherein:

determining the second gesture of the hand comprises determining a tapping by the user within the boundary of the virtual input device; and

performing the action comprises selecting a representation of a mouse button of the virtual input device.

5. The system of claim 1, wherein performing the action comprises at least one of moving a cursor associated with the virtual input device, selecting a representation of a left mouse button of the virtual input device, selecting a representation of a right mouse button of the virtual input device, or scrolling a representation of a mouse wheel of the virtual input device.

6. The system of claim 1, wherein:

determining the first gesture of the hand further comprises identifying a first area and a second area within the boundary of the virtual input device, wherein the first area is associated with a left mouse button of the virtual input device, and wherein the second area is associated with a right mouse button of the virtual input device; and

displaying the virtual input device further comprises displaying the first area and the second area.

7. The system of claim 1, the acts further comprising:

associating the virtual input device with the display area on which the content is projected, such that a movement of at least a portion of the hand within the boundary of the virtual input device causes a corresponding movement of a proj ection of a cursor within the display area.

8. The system of claim 1, wherein displaying the virtual input device further comprises:

illuminating the boundary of the virtual input device.

9. The system of claim 1, wherein the sequential poses of the hand indicate position and orientation of the hand relative to at least three dimensions.

10. One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform acts comprising:

capturing images of an environment;

determining an estimated motion parameter associated with a first gesture of a user that is depicted in at least one image of the images;

determining that the estimated motion parameter associated with the first gesture matches a motion parameter associated with a predefined gesture;

based at least in part on determining that the estimated motion parameter associated with the first gesture matches the motion parameter associated with the predefined gesture, displaying a virtual input device;

determining a first identity of the user;

determining, based at least in part on the first identity of the user, that the user is included within a group of users that are authorized to at least one of generate, customize, or access the virtual input device;

causing a first action to be performed based at least in part on at least one of the first gesture or a second gesture of the user;

determining a third gesture with respect to the virtual input device;

determining a second identity of a second user associated with the third gesture;

determining, based at least in part on the second identity of the second user, that the second user is not included within the group of users; and

refraining from performing a second action associated with the third gesture based at least in part on the second user not being included within the group of users.

11. The one or more non-transitory computer-readable media of claim 10, wherein displaying the virtual input device further comprises:

comparing the first gesture with a plurality of predefined gestures that includes the predefined gesture; and

determining that the first gesture matches the predefined gesture by determining that a match between the first gesture and the predefined gesture meets or exceeds a threshold value.

12. The one or more non-transitory computer-readable media of claim 10, wherein the instructions cause the one or more processors to further perform acts comprising:

at least partly subsequent to displaying the virtual input device, receiving input from the user via the virtual input device.

13. The one or more non-transitory computer-readable media of claim 12, wherein receiving the input further comprises:

determining the second gesture based at least in part on an interaction of the user with the virtual input device; and

based at least in part on the second gesture, receiving the input from the user.

14. The one or more non-transitory computer-readable media of claim 10, wherein the instructions cause the one or more processors to further perform acts comprising:

at least partly subsequent to displaying the virtual input device, determining the second gesture based at least in part on an interaction of the user with the virtual input device; and

based at least in part on the second gesture, configuring a parameter of the virtual input device.

15. The one or more non-transitory computer-readable media of claim 10, wherein displaying the virtual input device further comprises:

defining a boundary of the virtual input device based at least in part on the first gesture; and

displaying the boundary of the virtual input device.

16. The one or more non-transitory computer-readable media of claim 10, wherein displaying the virtual input device further comprises:

projecting light to illuminate a boundary of the virtual input device on a display area.

17. The one or more non-transitory computer-readable media of claim 10, wherein the virtual input device comprises a representation of one of a virtual trackpad, a virtual slider bar, or a virtual remote control.

18. A method comprising:

determining that a first hand gesture of a user interacts with a physical object different from the user;

based at least in part on determining that the first hand gesture interacts with the physical object, defining the physical object as a virtual input device to receive input from the user at a time in which the user further interacts with the physical object;

determining a first identity of the user;

determining, based at least in part on the first identity of the user, that the user is included within a group of users that are authorized to at least one of generate, customize, or access the virtual input device;

causing a first action to be performed based at least in part on at least one of the first gesture or a second gesture of the user;

determining a third gesture of a second user;

determining a second identity of the second user;

determining, based at least in part on the second identity of the second user, that the second user is not included within the group of users; and

refraining from performing a second action associated with the third gesture based at least in part on the second user not being included within the group of users.

19. The method of claim 18, further comprising:

at least partly subsequent to defining the physical object as the virtual input device, tracking a movement of the physical object as the user moves the physical object; and

at least partly in response to tracking the movement of the physical object, correspondingly moving a projection of a cursor within a display area.

20. The method of claim 18, further comprising:

at least partly subsequent to defining the physical object as the virtual input device, identifying a tapping by the user on or using the physical object; and

interpreting the tapping by the user as a selection of a representation of a button of the virtual input device.

21. The method of claim 18, further comprising:

at least partly subsequent to defining the physical object as the virtual input device, detecting a first interaction by the user with the physical object; and

based at least in part on detecting the first interaction by the user with the physical object, performing the first action.

22. The method of claim 18, wherein the first hand gesture comprises one of drawing a circle around the physical object, tapping the physical object, or pointing towards the physical object.

23. The method of claim 18, wherein the physical object is not a physical input device for receiving input from the user.

24. The method of claim 18, wherein determining the first hand gesture of the user further comprises:

capturing images of sequential poses of the hand of the user; and

based at least in part on capturing the images, determining the first hand gesture of the user.

25. The method of claim 18, wherein the virtual input device is one of a virtual pointing device, a virtual trackpad, a virtual mouse, or a virtual slider bar.

26. One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform acts comprising:

capturing images of an environment;

based at least in part on the images, determining that a first gesture of a user matches a predefined gesture;

based at least in part on determining that the first gesture matches the predefined gesture, selecting a virtual input device from a catalog of virtual input devices;

determining a first identity of the user;

determining, based at least in part on the first identity of the user, that the user is included within a group of users that are authorized to at least one of generate, customize, or access the virtual input device;

causing a first action to be performed based at least in part on at least one of the first gesture or a second gesture of the user;

determining a third gesture of a second user;

determining a second identity of the second user;

determining, based at least in part on the second identity of the second user, that the second user is not included within the group of users; and

refraining from performing a second action associated with the third gesture based at least in part on the second user not being included within the group of users.

27. The one or more non-transitory computer-readable media of claim 26, wherein the facilitating further comprises:

determining the second gesture of the user from a set of gestures; and

based at least in part on determining the second gesture, receiving input from the user via the virtual input device.

28. The one or more non-transitory computer-readable media of claim 26, wherein a set of gestures comprise a first set of gestures, and wherein the instructions cause the one or more processors to further perform acts comprising:

facilitating the user to configure the virtual input device through a second set of gestures; and

facilitating the user to save the virtual input device in the catalog of virtual input devices.

29. The one or more non-transitory computer-readable media of claim 26, wherein the instructions cause the one or more processors to further perform acts comprising:

based at least in part on the images, determining the second gesture of the user;

based at least in part on the second gesture, generating a second virtual input device; and

facilitating the user to save the second virtual input device in the catalog of virtual input devices.