Obtaining input from a virtual user interface

• US 9,081,418 B1
• Filed: 03/11/2013
• Issued: 07/14/2015
• Est. Priority Date: 03/11/2013
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Rawles, LLC (Amazon.com, Inc.)
• Inventors: Yao, Ning, Chang, Samuel Henry
• Primary Examiner(s): ABDULSELAM, ABBAS I
• Application Number:
  US13/794,296
• Time in Patent Office: 855 Days
• Field of Search: 345/156, 345/619, 345/642, 382/114, 715/852, 358/473
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042

Abstract

A virtual user interface in a 3D environment is projected onto a display object, such as an open hand. An operating object, such as a stylus or a second hand with an index finger extended, may be used to select input features of the projected user interface. In one scenario, for example, a telephone keypad may be projected onto the open hand and the index finger may be used to select a particular number on the keypad. Images of the hands may be segmented to produce contours of the finger and open hand, and these contours are used to determine the location of the finger relative to the open hand.

Claims

1. A system comprising:

one or more processors;

one or more cameras to obtain at least a first 3-dimensional (3D) image and a second 3D image of an environment, the first image and the second image including a display object and an operating object;

one or more computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising;

generating first 2-dimensional (2D) contours of the display object from the first 3D image and second 2D contours of the operating object from the first 3D image;

generating first 1-dimensional (1D) curvature information from the first 2D contours of the display object and second 1D curvature information from the second 2D contours of the operating object;

generating third 2D contours from the second 3D image, wherein the third 2D contours include at least a portion of the first 2D contours of the display object and a first portion of the second 2D contours of the operating object, and a second portion of the second 2D contours of the operating object is absent from the third 2D contours;

generating third 1D curvature information from the third 2D contours; and

determining a location in the second 3D image of the second portion of the second 2D contours of the operating object that is absent from the third 2D contours based, at least in part, on the second 1D curvature information and the third 1D curvature information.

2. The system of claim 1, wherein the display object includes a first hand and the operating object includes a second hand.

3. The system of claim 2, wherein the second portion of the second 2D contours of the operating object that is absent from the third contours includes at least a portion of contours of a finger of the second hand.

4. The system of claim 3, wherein the acts further comprise determining a distance between a peak point of the finger of the second hand and at least one point of a palm portion of the second hand;

and

wherein the determining the location in the second 3D image of the second portion of the second 2D contours of the operating object that is absent from the third 2D contours is based at least in part on the distance between the peak point of the finger of the second hand and the at least one point on the palm portion of the second hand.

5. The system of claim 3, wherein a distance between the finger of the second hand and a palm portion of the first hand is less than a specified distance.

6. The system of claim 1, wherein the first 3D image is obtained at a first time and the second 3D image is obtained at a second time that is subsequent to the first time.

7. The system of claim 1, wherein the acts further comprise:

identifying the at least a portion of the first 2D contours of the display object in the second 2D image based at least in part on the first 1D curvature information and the third 1D curvature information; and

identifying the first portion of the second 2D contours of the operating object based at least in part on the second 1D curvature information and the third 1D curvature information.

8. A method comprising:

obtaining data from an environment, the data including a plurality of images, wherein each of the plurality of images includes one or more objects;

identifying a display object and an operating object in a first image of the plurality of images, wherein the operating object includes an input portion;

projecting a user interface onto the display object;

determining when a portion of the operating object overlaps with a portion of the display object in a second image to produce a merged contour, wherein the portion of the operating object that overlaps with the portion of the display object includes the input portion, and particular contours of the input portion are absent from the merged contour; and

determining a location of the input portion in the second image.

9. The method of claim 8, wherein the display object includes a first hand, the operating object includes a second hand, and the input portion of the operating object includes an index finger of the second hand.

10. The method of claim 8, wherein determining a location of the input portion in the second image is based at least in part on curvature information of the operating object, curvature information of the display object, curvature information of the merged contour, or a combination thereof.

11. The method of claim 8, wherein determining the location of the input portion in the second image includes determining a length of the portion of the operating object that overlaps with the portion of the display object.

12. The method of claim 11, wherein determining the length of the portion of the operating object that overlaps with the portion of the display object is based, at least in part, on 2-dimensional (2D) information of the operating object, 2D information of the display object, 3-dimensional (3D) information of the operating object, 3D information of the display object, or a combination thereof.

13. The method of claim 8, wherein:

identifying the display object in the first image includes comparing contours of at least one object in the first image with a reference group of display object contours; and

identifying the operating object in the first image includes comparing contours of the at least one object in the first image with a reference group of operating object contours.

14. The method of claim 8, further comprising determining a number of peak points of at least one object in the first image;

and

wherein identifying the display object and the operating object in the first image is based at least in part on the number of peak points of the at least one object.

15. The method of claim 8, further comprising receiving a command to project the user interface onto the display object.

16. The method of claim 15, wherein the user interface includes a number of input features, and each of the input features is selectable to indicate a respective input to an augmented reality functional node.

17. The method of claim 16, further comprising:

comparing the location of the input portion of the operating object in the second image with respective locations of the input features of the user interface; and

determining a particular input feature being selected by the input portion of the operating object based, at least in part, on the location of the input portion of the operating object corresponding to a location of the particular input feature.

18. The method of claim 8, wherein the plurality of images are obtained in a sequence over time.

19. A method comprising:

identifying a first hand and a second hand in an image, wherein the first hand functions as a display surface for a projected user interface and the second hand functions as a pointing device to interact with the user interface;

segmenting the image to produce contours of the first and second hands;

determining, from the contours, when the first and second hands overlap such that part of the second hand forming the pointing device is obscured; and

deriving a location of the part of the second hand relative to the first hand to identify an interaction with the user interface.

20. The method of claim 19, wherein the first hand comprises an open hand and the second hand comprises a closed hand with an index finger extended to form the pointing device.

21. The method of claim 19, wherein determining when the first and second hands overlap comprises forming a merged contour from the contours of the first and second hands when the first and second hands are proximally close to satisfy a distance threshold.

22. The method of claim 19, further comprising executing an operation associated with the interaction identified from the location of the part of the second hand relative to the first hand.

23. One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform acts comprising:

identifying a display hand and an operating hand in a first image;

generating first contours of the display hand and second contours of the operating hand from the first image;

generating first curvature information from the first contours and second curvature information from the second contours;

identifying one or more continuous surfaces in a second image;

generating respective contours for each of the one or more continuous surfaces in the second image;

identifying contours of a particular continuous surface in the second image that include at least a portion of the first contours and at least a portion of the second contours; and

determining a location of a particular portion of the second contours that is absent from the contours of the particular continuous surface based at least in part on the second curvature information.

24. The one or more non-transitory computer-readable media of claim 23, wherein the acts further comprise:

determining a first continuous surface in the first image corresponding to the display hand;

determining first boundaries of the first continuous surface;

determining a second continuous surface in the second image corresponding to the operating hand;

determining second boundaries of the second continuous surface; and

wherein generating the first contours of the display hand in the first image is based at least in part on the first boundaries and generating the second contours of operating hand in the first image is based at least in part on the second boundaries.

25. The one or more non-transitory computer-readable media of claim 23, wherein the acts further comprise applying a Gaussian filter to generate at least one of the first contours or the second contours.

26. The one or more non-transitory computer-readable media of claim 23, wherein the location of the particular portion of the second contours that is absent from the contours of the particular continuous surface corresponds at least in part to a particular portion of the display hand.

27. The one or more non-transitory computer-readable media of claim 23, wherein the acts further comprise generating third curvature information for the contours of the particular continuous surface in the second image, and the determining the location of the particular portion of the second contours that is absent from the contours of the particular continuous surface is based at least in part on the third curvature information.