Augmented reality environment with environmental condition control

• US 9,462,262 B1
• Filed: 08/29/2011
• Issued: 10/04/2016
• Est. Priority Date: 08/29/2011
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Worley, William Spencer III, Crump, Edward Dietz, Weatherford, William Thomas, Coley, Christopher
• Primary Examiner(s): Zhai, Kyle
• Application Number:
  US13/220,410
• Time in Patent Office: 1,863 Days
• Field of Search: 345/633
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/012 Walk-in-place systems for allowing a user to walk in a virtual environment while constraining him to a given position in the physical environment
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 5/003 Details of a display terminal, the details relating to the control arrangement of the display terminal and to the interfaces thereto specific for a CRT G09G1/165; for a flat panel G09G3/2092
• CPC Class Codes: H04N 13/363 using image projection screens volumetric display H04N13/388

Abstract

An architecture includes a system to create an augmented reality environment in which images are projected onto a scene and user movement within the scene is captured. The augmented reality environment is hosted within a surrounding area in which various ambient conditions, external to the augmented reality environment, persist. The architecture monitors the external conditions and controls secondary devices that selectively modify the conditions as desired in cooperation with operation of the augmented reality environment to effectively enhance user experience. Alternatively or additionally, the architecture may inform the user of what changes to make, and the user manually makes the adjustments.

Claims

1. A system comprising:

an augmented reality functional node having a projector and a camera for generating an augmented reality environment within an area, the projector projecting one or more images within the environment and the camera capturing interaction of a user with the one or more images, the augmented reality functional node determining a task to perform based at least in part on the captured interaction of the user;

a secondary device in communication with the augmented reality functional node, the secondary device being configured to control at least one environmental condition within the area; and

the augmented reality functional node being configured to;

determine, based at least in part on the task, a predetermined range for an environmental condition that physically persists in the area;

identify that the environmental condition within the area is outside of the predetermined range;

display, via the projector projecting at least one image of the one or more images within the environment, a notification of a proposed environmental change to the environmental condition, the proposed environmental change to physically alter the area; and

determine, based at least in part on user interaction with the at least one image, a user response to the notification of the proposed environmental change to the environmental condition.

2. The system of claim 1, wherein the secondary device is external to and free from formation of the augmented reality.

3. The system of claim 1, wherein the augmented reality functional node generates an augmented reality environment at least partly through use of one or more of structured light, light detection and ranging (LIDAR), optical time-of-flight, ultrasonic ranging, or radar.

4. The system of claim 1, wherein the augmented reality functional node is configured to sense ambient temperature within the area and the secondary device comprises an ambient temperature regulator configured to initiate a change in the ambient temperature of the area in response to direction from the augmented reality functional node.

5. The system of claim 1, wherein the augmented reality functional node is configured to sense ambient light within the area and the secondary device is configured to alter the ambient light within the area in response to direction from the augmented reality functional node.

6. The system of claim 5, wherein the light-modifying device is selected from a group of devices comprising a lighting apparatus, a window blind, and a light emitting display.

7. The system of claim 1, wherein the augmented reality functional node is configured to sense humidity within the area and the secondary device comprises a humidity regulator configured to alter the ambient humidity of the area in response to direction from the augmented reality functional node.

8. The system of claim 1, wherein the augmented reality functional node is configured to sense an activity within the area and the secondary device comprises an olfactory device configured to emit one or more fragrances in the area in response to sensing the activity by the augmented reality functional node.

9. The system of claim 1, wherein the augmented reality functional node is configured to detect air flow within the area and the secondary device comprises a fan configured to alter air movement within the area in response to direction from the augmented reality functional node.

10. The system of claim 1, wherein the secondary device comprises an oxygen supply device configured to add oxygen to the area in response to direction from the augmented reality functional node.

11. The system of claim 1, wherein the augmented reality functional node is configured to detect sounds and the secondary device comprises a noise cancellation system configured to reduce an audible impact of the external sounds within the area in response to direction from the augmented reality functional node.

12. The system of claim 1, wherein the augmented reality functional node is configured to detect whether head movement of the user exhibits a characteristic of sleepiness, and in response, the augmented reality functional node directs the secondary device to modify the environmental condition within the area to render the environmental condition more suitable for a user rest state.

13. The system of claim 12, wherein the augmented reality functional node detects that the user intends to rest by monitoring the head movement of the user and determining that movement has fallen below a predetermined threshold.

14. The system of claim 1, wherein the augmented reality functional node is configured to detect whether the user exhibits a trend towards sleepiness, and in response, the augmented reality functional node directs the secondary device to power off one or more devices within the area to render the environmental condition more suitable for a user rest state.

15. The system of claim 1, wherein the augmented reality functional node is configured to detect whether the user exhibits a trend towards sleepiness, and in response, the augmented reality functional node directs the secondary device to provide additional audio within the area to render the environmental condition more suitable for a user rest state.

16. The system of claim 1, wherein:

the augmented reality functional node is configured to detect user movement and determine whether the user has entered a rest state, and in response, the augmented reality functional node determines whether user should be awakened; and

the secondary device is configured to awaken the user in an event the augmented reality functional node determines the user should be awakened.

17. The system of claim 16, wherein the secondary device comprises a haptic device to provide haptic feedback to the user to awaken the user.

18. The system of claim 16, wherein the secondary device comprises a vibration mechanism associated with furniture, the vibration mechanism being configured to vibrate the furniture to awaken the user.

19. The system of claim 16, wherein the secondary device adjusts one or more conditions within the area to awaken the user.

20. The system of claim 1, wherein:

the augmented reality functional node is configured to detect user movement and predict whether the user is about to enter a rest state, and in response, the augmented reality functional node determines whether the user should be awakened; and

the secondary device is configured to awaken the user in an event the augmented reality functional node determines the user should be awakened.

21. The system of claim 20, wherein the secondary device comprises a haptic device to provide haptic feedback to the user to awaken the user.

22. The system of claim 20, wherein the secondary device comprises a vibration mechanism associated with furniture, the vibration mechanism being configured to vibrate the furniture to awaken the user.

23. The system of claim 20, wherein the secondary device adjusts one or more conditions within the area to awaken the user.

24. The system of claim 1, wherein:

the augmented reality functional node is configured to create an augmented reality environment that engages the user in a physical activity; and

the secondary device is configured to adjust one or more conditions within the area based at least in part on detection of the user performing the physical activity.

25. The system of claim 24, wherein the one or more conditions is selected from a group of conditions including temperature, humidity, fragrance, airflow, and ambient noise.

26. The system of claim 1, wherein the augmented reality functional node is further configured to:

direct the secondary device to initiate the proposed environmental change in the environmental condition.

27. The system of claim 1, wherein the augmented reality functional node communicates with the secondary device via a remote computing system having processors and data storage separate from the augmented reality system.

28. A system comprising:

an augmented reality generation system having projection and image capture capabilities to generate an augmented reality environment by projecting structured light and capturing user movement within the structured light, the augmented reality generation system generating the augmented reality environment within a defined area, the augmented reality generation system determining (1) a task to perform based at least in part on the captured user movement within the structured light and (2) a predetermined range associated with one or more environmental conditions used to alter the defined area, wherein the predetermined range is based at least in part on the task;

one or more devices, operable in cooperation with the augmented reality generation system, to controllably alter the one or more environmental conditions in accordance with the predetermined range; and

a remote computing system having processors and data storage separate from the augmented reality generation system, the remote computing system to;

output, within the augmented reality environment, a notification of a proposed environmental change to the one or more environmental conditions;

determine, based at least in part on user movement captured within the augmented reality environment, a user response to the notification of the proposed environmental change to the one or more environmental conditions; and

provide instructions to at least one device to alter the one or more environmental conditions in accordance with the response.

29. The system of claim 28, wherein the one or more devices are selected from a group of devices comprising an ambient temperature regulator, a lighting apparatus, a window blind, a light emitting display, a humidity regulator, an acoustic device, an olfactory device, an air movement mechanism, a noise cancellation system, a pneumatic device, a haptic device, and a vibration mechanism.

30. The system of claim 28, wherein the remote computing system comprises cloud services.

31. The system of claim 28, wherein the augmented reality generation system comprises one or more sensors to detect the one or more environmental conditions within the area.

32. A method comprising:

facilitating an augmented reality environment within an area;

projecting visually-perceptible images onto objects as part of the augmented reality environment;

determining an activity to perform based at least in part on user interaction with the visually-perceptible images;

determining, based at least in part on the activity, a predetermined range associated one or more environmental conditions used to alter the area;

output, within the augmented reality environment, a notification of a proposed environmental change to the one or more environmental conditions;

determine, based at least in part on user interaction captured within the augmented reality environment, a user response to the notification of the proposed environmental change to the one or more environmental conditions; and

coordinate control of the one or more environmental conditions in accordance with the response, wherein the one or more environmental conditions are external to generation of the augmented reality environment.

33. The method of claim 32, wherein facilitating an augmented reality environment projecting structured light onto surroundings within the scene.

34. The method of claim 32, wherein the coordinating control of one or more environmental conditions of the area comprises modifying ambient temperature of the area.

35. The method of claim 32, wherein the coordinating control of one or more environmental conditions of the area comprises adjusting ambient lighting of the area.

36. The method of claim 32, wherein the coordinating control of one or more environmental conditions of the area comprises altering ambient humidity of the area.

37. The method of claim 32, wherein the coordinating control of one or more environmental conditions of the area comprises changing air flow within the area.