Gestural object selection

• US 9,292,089 B1
• Filed: 08/24/2011
• Issued: 03/22/2016
• Est. Priority Date: 08/24/2011
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Sadek, Ramy S.
• Primary Examiner(s): Bost, Dwayne
• Assistant Examiner(s): Michaud, Robert
• Application Number:
  US13/216,953
• Time in Patent Office: 1,672 Days
• Field of Search: 345/156, 345/207, 345/419, 382/103, 463/36
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials

Abstract

A posture for a user is derived and used in conjunction with a known physical location of a tangible object or a presented virtual object to determine when the user selects at least a portion of the object. The derived posture comprises a model having a user origin and a gestural terminus, from which a designation vector intersecting with the selected object is determined. The designation vector may extend from a user origin of the dominant eye through a gestural terminus comprising a tip of an outstretched finger. For example, the user may point to a chair in the room to select the physical chair, or point to a picture of a chair presented via a display device to select the picture.

Claims

1. A system comprising:

a processor;

a projector coupled to the processor and configured to generate structured light within an environment;

a camera coupled to the processor and configured to capture at least a portion of the structured light within the environment;

memory, accessible by the processor;

a spatial analysis module stored in the memory and configured to generate spatial data about one or more objects within the environment; and

a tracking and control module stored in the memory and configured to;

identify an object within the environment as a user based at least in part upon the spatial data;

derive a posture of the user based at least in part upon the spatial data;

determine a physical location of a user origin associated with the user relative to the posture of the user;

determine a location of a gestural terminus of the user based at least in part on the posture of the user;

determine a designation vector based at least in part upon the physical location of the user origin and the location of the gestural terminus, a magnitude of the designation vector being proportionate to a distance between the physical location of the user origin and the location of the gestural terminus;

determine a portion of a representation of a second object that corresponds to the designation vector; and

designate the portion of the representation of the second object as a selection made by the user.

2. The system of claim 1, wherein the designation vector comprises a vector passing through the physical location of the user origin and the location of the gestural terminus.

3. The system of claim 1, wherein the determining the at least the portion of the representation of the second object corresponding to the designation vector comprises determining that the designation vector passes through the at least the portion of the representation of the second object.

4. The system of claim 1, wherein the posture of the user comprises a model of relative positions and orientation of at least a portion of the user'"'"'s anatomy.

5. The system of claim 1, wherein the posture comprises a model of relative positions of the user'"'"'s head, eyes, and hands.

6. The system of claim 5, wherein the posture further comprises a model of orientation of the user'"'"'s head, eyes, and hands.

7. The system of claim 1, wherein the physical location of the user origin is proximate to or coincident with a dominant eye of the user.

8. The system of claim 1, wherein the gestural terminus comprises one or more fingertips of the user.

9. The system of claim 1, wherein the gestural terminus comprises a point on a third object, held by a hand of the user, the point of the third object being distal to the hand of the user.

10. The system of claim 1, wherein the selection made by the user corresponds to the orientation of the gestural terminus.

11. The system of claim 1, wherein the object comprises a virtual object presented via one or more display devices.

12. The system of claim 1, wherein the object comprises a virtual object presented via the projector.

13. The system of claim 1, wherein the object comprises a tangible object.

14. One or more non-transitory computer-readable storage media storing computer-executable instructions that, when executed, cause one or more processors to perform acts comprising:

receiving data about a user and one or more objects within an environment, the data being generated based in part on one or more images of the environment captured by a camera that is separate and spaced from the user;

deriving, based at least in part upon the data, a posture of the user in the environment;

determining a physical location of a user origin associated with the user relative to the posture of the user;

determining a location of a gestural terminus of the user based at least in part on the posture of the user;

determining a designation vector based at least in part upon the physical location of the user origin and the location of the gestural terminus, the designation vector including a magnitude that is proportionate to a distance between the physical location of the user origin and the location of the gestural terminus; and

determining at least a portion of a representation of an object of the one or more objects within the environment that corresponds to the designation vector.

15. The one or more non-transitory computer-readable storage media of claim 14, wherein the designation vector comprises a vector passing through the physical location of the user origin and the location of the gestural terminus.

16. The one or more non-transitory computer-readable storage media of claim 14, wherein the determining the at least the portion of the representation of the object corresponding to the designation vector comprises determining that the designation vector passes through the at least the portion of the representation of the object.

17. The one or more non-transitory computer-readable storage media of claim 14, wherein the deriving the posture of the user comprises determining a dominant eye of the user.

18. The one or more non-transitory computer-readable storage media of claim 17, wherein the determining the dominant eye of the user comprises presenting a calibration object at a known location.

19. The one or more non-transitory computer-readable storage media of claim 14, wherein the deriving the posture of the user comprises:

accessing a baseline human physical arrangement;

acquiring characteristic data describing one or more physical characteristics of the user in the environment;

scaling the baseline human physical arrangement to correspond to the characteristic data to generate a scaled baseline human physical arrangement; and

generating the posture of the user based at least in part upon the scaled baseline human physical arrangement.

20. The one or more non-transitory computer-readable storage media of claim 14, wherein the determining the physical location of the user origin comprises designating a dominant eye of the user as the user origin based at least partly on a determination that the gestural terminus is within a field of view of the user.

21. The one or more non-transitory computer-readable storage media of claim 14, wherein the determining the physical location of the user origin comprises designating a point in the posture of the user other than the dominant eye as the user origin based at least partly on a determination that the gestural terminus is outside a field of view of the user.

22. The one or more non-transitory computer-readable storage media of claim 14, wherein the designation vector extends through the at least the portion of the object to designate a second portion of the object that is non-visible to the user.

23. The one or more non-transitory computer-readable storage media of claim 14, wherein the gestural terminus encompasses one or more fingertips.

24. One or more non-transitory computer-readable storage media storing computer-executable instructions that, when executed, cause one or more processors to perform acts comprising:

receiving data about a user and one or more objects within an environment, the data being generated based in part on one or more images of the environment captured by a camera that is separate and spaced from the user;

determining, based at least in part upon the data, a location of a user origin of the user within the environment, the user origin comprising a reference point associated with an anatomy of the user;

determining a location of a gestural terminus within the environment, the gestural terminus comprising one or more spatial points associated with a gesturing appendage of the user;

determining a designation vector based at least in part upon the location of the user origin of the user and the location of the gestural terminus of the user, a magnitude of the designation vector being proportionate to a distance between the location of the user origin and the location of the gestural terminus; and

determining at least a portion of a physical object that corresponds to the designation vector and that is determined to be selected by the user.

25. The one or more non-transitory computer-readable storage media of claim 24, wherein the location of the user origin comprises a dominant eye of the user.

26. The one or more non-transitory computer-readable storage media of claim 24, wherein the determining the location of the gestural terminus comprises generating a derived posture of the user.

27. The one or more non-transitory computer-readable storage media of claim 24, the acts further comprising selecting the at least the portion of the physical object intersecting the designation vector.

28. The one or more non-transitory computer-readable storage media of claim 27, the acts further comprising selecting the at least the portion of the physical object based at least in part upon an orientation of the gestural terminus.