Obtaining input from a virtual user interface

• US 9,703,371 B1
• Filed: 06/29/2015
• Issued: 07/11/2017
• Est. Priority Date: 03/11/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Chang, Samuel Henry, Yao, Ning
• Primary Examiner(s): Abdulselam, Abbas
• Application Number:
  US14/753,745
• Time in Patent Office: 743 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042

Abstract

A virtual user interface in a 3D environment is projected onto a display object, such as an open hand. An operating object, such as a stylus or a second hand with an index finger extended, may be used to select input features of the projected user interface. In one scenario, for example, a telephone keypad may be projected onto the open hand and the index finger may be used to select a particular number on the keypad. Images of the hands may be segmented to produce contours of the finger and open hand, and these contours are used to determine the location of the finger relative to the open hand.

Claims

1. A method comprising:

determining first curvature information associated with a first object;

determining second curvature information associated with a second object;

projecting at least a portion of an image onto the first object;

detecting that a portion of the second object overlaps at least a portion of the first object; and

determining a location of a selected image portion of the image based at least in part on the first curvature information and the second curvature information.

2. The method of claim 1, wherein the second object is a hand.

3. The method of claim 1, further comprising:

forming a merged contour from the first curvature information and the second curvature information; and

wherein determining that the second object overlaps the first object is based at least in part on the merged contour.

4. The method of claim 1, further comprising determining the second object is within a distance threshold of the first object prior to determining the location.

5. The method of claim 1, further comprising applying a Gaussian filter to generate at least one of the first contour information or the second contour information.

6. The method of claim 1, further comprising:

determining a length of the portion of the second object that overlaps with the portion of the first object; and

wherein determining the location is based at least in part on the length.

7. A method comprising:

under control of one or more computer systems configured with executable instructions,

generating first curvature information related to a display object and second curvature information related to an operating object within a first image projected into an environment;

determining one or more continuous surfaces in a second image projected into the environment;

generating contours of surfaces in the second image, the surfaces including at least a portion of the display object and at least a portion of the operating object; and

determining a location based at least in part on the contours.

8. The method of claim 7, wherein determining the location is based at least in part on the first curvature information and the second curvature information.

9. The method of claim 7, further comprising:

detecting a first contour within the first image, the contour associated with the display object;

detecting a second contour within the first image, the contour associated with the display object;

wherein the first curvature information is determined based at least in part on the first contour; and

wherein the second curvature information is determined based at least in part on the second contour.

10. The method of claim 7, further comprising determining at least one boundary associated with the display object and the operating object and wherein determining the location is based at least in part on the at least one boundary.

11. The method of claim 7, further comprising performing at least one operation, the at least one operation associated with the location on the display object.

12. The method of claim 7, wherein the second image is a graphical user interface.

13. The method of claim 7, further comprising:

segmenting the first image to produce contours prior to generating the first curvature information and the second curvature information.

14. A system comprising:

one or more processors;

a component for receiving image data;

one or more computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising;

generating curvature information of a display object and an operating object based at least in part on image data received from the component, the display object and the operating object associated with a projection area; and

determining a location associated with the operating object based at least in part on the curvature information of the display object and the operating object.

15. The system of claim 14, wherein the component for receiving the image data includes a camera.

16. The system of claim 14, further comprising:

a projector for projecting an image into the projection area; and

wherein the image data is received by the component while the image is projected into the projection area.

17. The system of claim 14, wherein the one or more computer-readable media stores computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising determining that the operating object overlaps the display object.

18. The system of claim 14, wherein the one or more computer-readable media stores computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising:

determining contour information associated with the display object and the operating object based on the image data received from the component; and

wherein the curvature information is generated based at least in part on the contour information.

19. The system of claim 14, wherein the one or more computer-readable media stores computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising determining a distance between the operating object and at least one other object within the projection area.

20. The system of claim 14, wherein generating the curvature information of objects associated with the projection area includes

segmenting the image to produce contours.