Providing user feedback in projection environments

• US 8,887,043 B1
• Filed: 01/17/2012
• Issued: 11/11/2014
• Est. Priority Date: 01/17/2012
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Rawles, LLC (Amazon.com, Inc.)
• Inventors: Pollack, Joshua, Crump, Edward Dietz, Coley, Christopher, Meenakshisundaram, Sankara Narayana Hemanth
• Primary Examiner(s): LEE, TING ZHOU
• Application Number:
  US13/351,983
• Time in Patent Office: 1,029 Days
• Field of Search: 715/702, 715/810, 345/168, 345/156, 345/418, 345/108
• US Class Current: 715/702
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042

Abstract

Systems and techniques for providing feedback to users within an environment that interact with user interfaces (UIs) that are projected within the environment. For instance, the systems and techniques may project a UI that includes one or more selectable portions, such as keys, icons, sliders, dials, or any other type of control. After projecting the UI, the systems and techniques may identify the user attempting to interact with the UI. In response, the systems and techniques may provide feedback to the user indicating that the user has engaged the UI. For instance, the systems and techniques may visually alter the projected UI, may output a sound via one or more speakers within the environment, or may provide the feedback in any other manner.

Claims

1. A system comprising:

one or more processors;

a projector, coupled to the one or more processors and configured to project, onto a surface and within an environment, a user interface that includes multiple selectable portions;

a camera, coupled to the one or more processors and configured to capture information for identifying a user within the environment interacting with the user interface; and

one or more computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising;

identifying, using the information captured by the camera, the user interacting with the user interface by moving a selection tool towards the surface;

determining which of the multiple selectable portions the user is in position to select based at least in part on the identifying;

altering the user interface projected by the projector to highlight the determined selectable portion; and

identifying, using the information captured by the camera, the user selecting the determined selectable portion, the identifying of the user selecting the determined selectable portion comprising at least one of;

(1) identifying the user selecting the determined selectable portion in response to the user moving the selection tool toward the surface and past a predefined selection plane without touching the surface, or (2) identifying the user selecting the determined selectable portion in response to the user moving the selection tool toward the surface, without touching the surface, and then back away from the surface.

2. A system as recited in claim 1, wherein the altering comprises causing the projector to enlarge a size of the determined selectable portion.

3. A system as recited in claim 1, wherein the altering comprises causing the projector to alter a color of the determined selectable portion.

4. A system as recited in claim 1, further comprising one or more speakers to output audible content within the environment, and wherein the acts further comprise indicating, using the one or more speakers, that the user is in position to select the determined selectable portion.

5. A system as recited in claim 1, wherein:

the identifying the user interacting with the user interface comprises identifying the user moving the selection tool toward the surface.

6. The system of claim 1, wherein:

the determining comprises determining a trajectory of the selection tool toward the surface and extrapolating the trajectory onto the determined selectable portion.

7. A system as recited in claim 1, the acts further comprising:

instructing the projector to visually alter the determined selectable portion to indicate acceptance of the selecting.

8. A system as recited in claim 7, wherein:

the identifying that the user is interacting with the user interface comprises identifying the user moving the selection tool toward the surface;

the determining comprises determining a trajectory of the selection tool toward the surface and extrapolating the trajectory onto the determined selectable portion; and

the identifying of the user selecting the determined selectable portion comprises identifying the user moving the selection tool towards the surface and along at least a portion of the trajectory after the projector has visually altered the user interface to highlight the determined selectable portion.

9. A system as recited in claim 1, the acts further comprising:

causing performance of an action associated with the determined selectable portion.

10. One or more computer-readable storage media storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform acts comprising:

projecting, with a projector, a user interface onto a surface within an environment;

capturing, with a camera, images of a user within the environment moving a selection tool toward the surface;

identifying a trajectory of the selection tool toward the surface using the captured images;

highlighting a portion of multiple different portions of the user interface that corresponds to a position of the selection tool based at least in part on the identified trajectory; and

identifying, using the captured images, the user selecting the portion of the user interface that corresponds to the position of the selection tool by at least one of;

(1) identifying the user selecting the portion of the user interface that corresponds to the position of the selection tool in response to the user moving the selection tool toward the surface and past a predefined selection plane without touching the surface, or (2) identifying the user selecting the portion of the user interface that corresponds to the position of the selection tool in response to the user moving the selection tool toward the surface, without touching the surface, and then back away from the surface.

11. One or more computer-readable media as recited in claim 10, wherein the selection tool comprises one or more of a hand of the user, a finger of the user, or an object held by the user.

12. One or more computer-readable media as recited in claim 10, the acts further comprising mapping the identified trajectory to the portion of the user interface that corresponds to the position of the selection tool.

13. One or more computer-readable media as recited in claim 10, wherein the indicating comprises altering the user interface as projected onto the surface.

14. One or more computer-readable media as recited in claim 13, wherein the visually altering the user interface comprises one or more of enlarging the portion of the user interface that corresponds to the position of the selection tool or altering a color of the portion of the user interface that corresponds to the position of the selection tool.

15. One or more computer-readable media as recited in claim 10, wherein the indicating comprises audibly outputting an indication that the selection tool is in position to select the portion of the user interface.

16. One or more computer-readable media as recited in claim 10, wherein the portion that corresponds to the position of the selection tool includes multiple sub-portions, and the acts further comprise:

continuing to identify the trajectory of the selection tool toward the surface using captured images; and

indicating which sub-portion of the multiple sub-portions corresponds to the position of the selection tool based at least in part on the identified trajectory.

17. One or more computer-readable media as recited in claim 16, the acts further comprising:

continuing to identify the trajectory of the selection tool after indicating that the position of the selection tool corresponds to the sub-portion;

identifying a selection of the sub-portion of the portion of the user interface based at least in part on the identified trajectory; and

indicating that the selection of the sub-portion has been accepted.

18. One or more computer-readable media as recited in claim 17, wherein the indicating that the sub-portion has been selected comprises one or more of altering the user interface as projected onto the surface or outputting an audible indication that the sub-portion has been selected.

19. One or more computer-readable media as recited in claim 17, the acts further comprising causing performance of an action associated with the sub-portion at least partly in response to identifying the selection.

20. One or more computer-readable media as recited in claim 10, the acts further comprising:

determining that the position of the selection tool corresponds to a different portion of the multiple different portions of the user interface; and

indicating that the position of the selection tool corresponds to the different portion of the user interface at least partly in response to the determining.

21. A method, comprising:

projecting, with a projector, a user interface that includes multiple selectable portions onto a surface within an environment;

identifying, with a camera, a user within the environment attempting to interact with the user interface projected onto the surface by identifying the user moving a selection tool towards the surface;

identifying a trajectory of the selection tool toward the surface using the captured images;

visually highlighting a portion of multiple selectable portions of the user interface, the portion selected based at least in part on the trajectory of the selection tool; and

after visually highlighting the portion, identifying a selection of a key of the portion in response to the user continuing to move the selection tool towards the surface and past a selection plane without touching the surface.

22. A method as recited in claim 21, wherein:

the user interface comprises a virtual keyboard projected onto the surface and the one or more selectable portions comprise keys of the virtual keyboard; and

the feedback comprises visually highlighting one of the keys of the virtual keyboard that the user is in position to select.

23. A method as recited in claim 21, wherein:

the user interface comprises a virtual keyboard projected onto the surface and the one or more selectable portions comprise keys of the virtual keyboard; and

the feedback comprises visually highlighting a block of the keys of the virtual keyboard.

24. A method as recited in claim 21, wherein:

the user interface comprises a slider bar projected onto the surface and the one or more selectable portions comprises a slider of the slider bar; and

the feedback comprises visually highlighting the slider to indicate that the user is in position to move the slider along the slider bar.

25. A method as recited in claim 21, further comprising:

after providing the feedback, identifying a selection of one of the one or more selectable portions by the user; and

providing feedback to the user indicating that the selection has been accepted.

26. A method as recited in claim 25, wherein:

the identifying of the selection by the user comprises identifying that the user has continued moving the selection tool towards the surface and then back away from the surface.

27. A method as recited in claim 21, wherein the selection plane is defined relative to the user.

28. A method as recited in claim 21, wherein the selection plane is defined relative to the surface.

29. A method as recited in claim 21, wherein the user defines the selection plane.

30. A method as recited in claim 21, further comprising selecting, prior to the projecting, a location within the environment in which to project the user interface.

31. A method as recited in claim 30, wherein the location is selected relative to a location of the user within the environment.

32. A method as recited in claim 30, wherein the location is selected explicitly by the user.