Augmented reality environment with secondary sensory feedback

• US 9,478,067 B1
• Filed: 04/08/2011
• Issued: 10/25/2016
• Est. Priority Date: 04/08/2011
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Worley, William Spencer III, Coley, Christopher, Selker, Edwin Joseph, Weatherford, William Thomas
• Primary Examiner(s): AMIN, JWALANT B
• Application Number:
  US13/083,326
• Time in Patent Office: 2,027 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0426 tracking fingers with respect to a virtual keyboard projected or printed on the surface virtual keyboards on touch screens G06F3/04886
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: H04N 2201/3256 colour related metadata, e.g. colour, ICC profiles for use made of colour related metadata see the appropriate place, e.g. H04N1/60, H04N1/60 and subgroups

Abstract

An architecture includes a system to create an augmented reality environment in which images are projected onto a scene and user movement within the scene is captured. In addition to primary visual stimuli, the architecture further includes introduction of a secondary form of sensory feedback into the environment to enhance user experience. The secondary sensory feedback may be tactile feedback and/or olfactory feedback. The secondary sensory feedback is provided to the user in coordination with the visual based activity occurring within the scene.

Claims

1. A system comprising:

an augmented reality functional node configured to be positioned within an environment and independent of a user, the augmented reality functional node having a projector and a camera for generating an augmented reality environment, the projector projecting one or more images on one or more surfaces of one or more physical objects within the environment and the camera capturing interaction of a user with the one or more images, the images providing primary stimuli for the user;

a computing device configured to;

determine a plurality of different types of secondary sensory devices available in the environment,

select a secondary sensory device of a first type of the plurality of different types, wherein the secondary sensory device is separate from the augmented reality functional node and separate from the user, and

facilitate coordination between the augmented reality functional node and the secondary sensory device so that secondary sensory feedback is provided from the secondary sensory device to the user in synchronization with the projecting of the one or more images; and

wherein the secondary sensory device is configured to generate the secondary sensory feedback within the augmented reality environment in response to the user interacting with the one or more images projected on the one or more surfaces of the one or more physical objects.

2. The system of claim 1, wherein the augmented reality functional node generates an augmented reality environment through use of structured light.

3. The system of claim 1, wherein the secondary sensory device comprises an air engine device to generate haptically-perceptible puffs of air.

4. The system of claim 1, wherein the secondary sensory device comprises an acoustic device to generate non-audible acoustical waves that provide tactile feedback to the user.

5. The system of claim 1, wherein the secondary sensory device comprises a heat generation device to create zones of focused heat that provide tactile feedback to the user.

6. The system of claim 1, wherein the secondary sensory device comprises an air cannon device to generate concentrated air turbulence that provides tactile feedback to the user.

7. The system of claim 1, wherein the secondary sensory device comprises a vibration device to provide vibration forces as a form of tactile feedback.

8. The system of claim 1, wherein the secondary sensory device comprises a liquid propellant device to release droplets of liquid as a form of tactile feedback.

9. The system of claim 1, wherein the secondary sensory device comprises an olfactory device to release an olfactory-perceptible whiff as a form of olfactory feedback.

10. The system of claim 1, wherein the computing device is associated with the augmented reality functional node.

11. The system of claim 1, wherein the computing device comprises a remote computing system remote from the augmented reality environment.

12. An architecture comprising:

an augmented reality generation system having projection and image capture capabilities to generate an augmented reality environment by projecting structured light on one or more physical objects in an environment and capturing user movement of a user within the structured light in the environment, the augmented reality generation system being positioned in a first location independent of the user;

a computing device configured to;

determine a plurality of different types of sensory devices available in the environment, and select one or more sensory devices of a tactile type of the plurality of different types, the one or more tactile sensory devices being separate from the user and positioned in a second location different from the first location of the augmented reality generation system and the one or more tactile sensory devices being operable in cooperation with the augmented reality generation system to provide tactile feedback to the user as the user interacts with the structured light projected on the one or more physical objects within the augmented reality environment; and

a remote computing system having processors and data storage separate from the augmented reality generation system, the remote computing system providing instructions to at least one tactile sensory device to provide the tactile feedback in coordination with the augmented reality environment.

13. The architecture of claim 12, wherein the one or more tactile sensory devices are selected from a group of devices comprising an acoustic device, a pneumatic device, a heat generating device, a liquid propellant device, and a vibration device.

14. The architecture of claim 12, wherein the remote computing system comprises cloud services.

15. The architecture of claim 12, further comprising a user device independent from the one or more tactile sensory devices that is accessible by the remote computing system over a network, the user device being configured to communicate with the remote computing system and provide tactile feedback to the user in response to the user device receiving an instruction from the remote computing system.

16. A method comprising:

projecting, from a first location independent of a user, visually-perceptible images onto surfaces in an environment;

determining a plurality of different types of secondary sensory devices available in the environment;

selecting a secondary sensory device of a first type of the plurality of different types;

capturing interaction of the user with one or more of the visually-perceptible images; and

generating, from a second location different than the first location and independent of the user, a secondary sensory feedback from the secondary sensory device to provide sensory feedback to the user in coordination with the user interacting with one or more of the visually-perceptible images projected onto the surfaces in the environment.

17. The method of claim 16, further comprising generating an augmented reality environment by projecting structured light onto surroundings within the environment.

18. The method of claim 16, wherein generating a secondary sensory feedback comprises producing haptically-perceptible puffs of air.

19. The method of claim 16, wherein generating a secondary sensory feedback comprises producing non-audible acoustical waves.

20. The method of claim 16, wherein generating a secondary sensory feedback comprises producing focused heat.

21. The method of claim 16, wherein generating a secondary sensory feedback comprises producing vibrations.

22. The method of claim 16, wherein generating a secondary sensory feedback comprises producing droplets of liquid.

23. A method comprising:

projecting, from a first location independent of a user, structured light onto one or more surfaces of physical surroundings within a scene;

determining a plurality of different types of sensory devices available in the scene;

selecting a sensory device of a tactile type of the plurality of different types;

capturing movement of the user within the scene by detecting deformation of the structured light projected onto the one or more surfaces of the physical surroundings;

receiving input as a result of capturing the movement of the user; and

delivering, from a second location different than the first location and independent of the user, tactile feedback from the sensory device to the user in response to receipt of the input, the tactile feedback being delivered to the user while the user is within the scene.

24. The method of claim 23, wherein delivering tactile feedback comprises producing haptically-perceptible bursts of air.

25. The method of claim 23, wherein delivering tactile feedback comprises producing regions of focused heat.

26. The method of claim 23, wherein delivering tactile feedback comprises producing one of non-audible acoustical waves or vibrations.

27. The method of claim 23, wherein delivering tactile feedback comprises producing droplets of liquid.

28. The method of claim 23, wherein delivering tactile feedback comprises producing tactile feedback through use of an independent device associated with the user.