DYNAMIC ADJUSTMENTS FOR AUGMENTED, MIXED AND VIRTUAL REALITY PRESENTATIONS

• US 20160350967A1
• Filed: 06/01/2016
• Published: 12/01/2016
• Est. Priority Date: 06/01/2015
• Status: Abandoned Application

• Current Assignee: Cable Television Laboratories Incorporated
• Original Assignee: Cable Television Laboratories Incorporated
• Inventors: Klassen, Eric
• Application Number:
  US15/170,209
• Publication Number:
  US 20160350967A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 15/60 Shadow generation
• CPC Class Codes: G06T 2215/12 Shadow map, environment map

Abstract

Dynamically adjusting presentations associated with augmented, mixed and/or virtual reality is contemplated. The dynamic adjustments may be made to augment sensory influences on a user, such as but not necessarily limited to dynamically adjusting lighting effects of the presentation to compensate for changes in direct and/or indirect luminosity or other lighting related affects.

Claims

1. A non-transitory computer-readable medium having a plurality of non-transitory instructions executable with a processor of an augmented reality (AR) device to facilitate presenting an AR view of a real area, the AR device having a housing shaped to position a see-through screen in front of one or more eyes of a person, the AR device having a light sensor configured to sense luminosity within the real area proximate the person, the AR device having an image generation device sufficient to facilitate displaying or more image frames on the screen, the non-transitory instructions being sufficient to facilitate:

generating a first lighting map of the real area based on a first luminosity data determined for at least a portion of the real area with the light sensor at a first instance in time;

determining an AR object to be displayed on the screen with the image generation device in order to present the AR view of the real area at a second instance in time occurring after the first instance; and

determining a first lighting effect to be displayed on the screen with the image generation device for augmenting the AR object in the AR view at the second instance based at least in part on the first lighting map.

2. The non-transitory computer-readable medium of claim 1 further comprising non-transitory instructions sufficient for:

determining the first lighting effect to include a first shadow; and

determining a first shape and/or a first intensity of the first shadow based at least in part on the first lighting map.

3. The non-transitory computer-readable medium of claim 2 further comprising non-transitory instructions sufficient for:

determining the real area to include at least a first light source and a second light source at the first instance based at least in part on the first lighting map;

determining a first angle of incidence with the AR object for a first light emitted from the first light source based at least in part on the first lighting map;

determining a second angle of incidence with the AR object for a second light emitted from the first light source based at least in part on the first lighting map; and

determining the first shape of the first shadow based at least in part on the first and second angles such that the first shadow represents how the first and second light sources would influence a real shadow for the AR object if the AR object was real.

4. The non-transitory computer-readable medium of claim 3 further comprising non-transitory instructions sufficient for:

generating a second lighting map of the real area based on a second luminosity data determined for at least a portion of the real area with the light sensor at a third instance in time occurring after the second instance;

determining the AR view at a fourth instance in time occurring after the third instance to require the AR object to be appear stationary at least in so far as the AR object appearing in the AR view at the second and fourth instances without changing position; and

determining a second lighting effect to be displayed on the screen with the image generation device for augmenting the AR object at the fourth instance based at least in part on the second lighting map;

determining the second lighting effect to include a second shadow;

determining a second shape of the second shadow based at least in part on the second lighting map;

determining the real area to include the first and second light sources at the third instance based at least in part on the second lighting map;

determining a third angle of incidence with the AR object for a third light emitted from the first light source based at least in part on the second lighting map, including determining the third angle to match the first angle due to a position of the first light source being the same at the first and third instances;

determining a fourth angle of incidence with the AR object for a fourth light emitted from the second light source based at least in part on the second lighting map, including determining the fourth angle to differ from the second angle due to a position of the second light source being different at the first and third instances; and

determining the second shape of the second shadow based at least in part on the third and fourth angles such that the second shape differs from the first shape by an amount sufficient to cause the second shadow to appear differently in the AR view than the first shadow.

5. The non-transitory computer-readable medium of claim 4 further comprising non-transitory instructions sufficient for:

generating a first image frame to be displayed on the screen with the image generation device at the second instance to facilitate presenting the AR view to include at least the AR object and the first shadow; and

generating a second image frame to be displayed on the screen with the image generation device at the fourth instance to facilitate presenting the AR view to include at least the AR object and the second shadow.

6. The non-transitory computer-readable medium of claim 5 further comprising non-transitory instructions sufficient for:

determining the first and second lighting effects to respectively include a first coloring and a second coloring, the first coloring being determined from the first lighting map to represent a coloring influence of the first and second light sources on the AR object at the first instance, the second coloring being determined from the second lighting map to represent a coloring influence of the first and second light sources on the AR object at the second instance, including determining the second coloring to be different from the first coloring due to the position of the second light source being different at the first and third instances;

generating the first image frame to include the first coloring being applied to the AR object; and

generating the second image frame to include the second coloring being applied to the AR object.

7. The non-transitory computer-readable medium of claim 1 further comprising non-transitory instructions sufficient for:

determining the first lighting effect to be a first shadow associated with a light source positioned at a first location within the real area at the first instance;

generating a first image frame to be displayed on the screen with the image generation device at the second instance to facilitate presenting the AR view to include at least the AR object and the first shadow;

generating a second lighting map of the real area based on a second luminosity data determined for at least a portion of the real area with the light sensor at a third instance in time occurring after the first instance and after the light source has moved from the first position to a second position;

determining a second shadow to be displayed on the screen with the image generation device for augmenting the AR object at a fourth instance occurring after the third instance based at least in part on the second lighting map, the second shadow being different in shape, color and/or intensity than the first shadow;

determining a rate of change of the light source associated with moving from the first position to the second position;

generating a second image frame to be displayed on the screen with the image generation device at a fifth instance occurring after the fourth instance to facilitate presenting the AR view to include;

i) the AR object and the first shadow when the rate of change is greater than a threshold; and

ii) the AR object and the second shadow when the rate of change is equal to or less than the threshold.

8. The non-transitory computer-readable medium of claim 7 further comprising non-transitory instructions sufficient for generating the first and second image frames such that a position of the AR object appears unchanged in the AR view at the second and fifth instances.

9. The non-transitory computer-readable medium of claim 1 further comprising non-transitory instructions sufficient for:

determining the first lighting effect to be a first shadow associated with a light source positioned at a first location within the real area at the first instance;

generating a first image frame to be displayed on the screen with the image generation device at the second instance to facilitate presenting the AR view to include at least the AR object and the first shadow;

generating a second lighting map of the real area based on a second luminosity data determined for at least a portion of the real area with the light sensor at a third instance in time occurring after the first instance and after the light source has moved from the first position to a second position;

determining a second shadow to be displayed on the screen with the image generation device for augmenting the AR object at a fourth instance occurring after the third instance based at least in part on the second lighting map, the second shadow being different in shape, color and/or intensity than the first shadow;

determining a rate of change of the light source associated with moving from the first position to the second position;

determining a plurality of buffered shadows stored in a memory of the AR device, each of the plurality of buffered shadows having been previously generated for the AR object from a corresponding one of a plurality of lighting maps generated for at least a portion of the real area prior to the second instance, the plurality of buffered shadows including the first shadow;

generating a second image frame to be displayed on the screen with the image generation device at a fifth instance occurring after the fourth instance to facilitate presenting the AR view to include;

i) the AR object and the second shadow when the rate of change is less than a threshold; and

ii) the AR object and a third shadow selected from the plurality of buffered shadows when the rate of change is greater than the threshold.

10. The non-transitory computer-readable medium of claim 1 further comprising non-transitory instructions sufficient for:

determining the first lighting effect to be a first shadow associated with a light source positioned at a first location within the real area at the first instance;

generating a first image frame to be displayed on the screen with the image generation device at the second instance to facilitate presenting the AR view to include at least the AR object and the first shadow;

determining a movement of the light source from a second location to the first location, the second location corresponding with positioning of the light source within the real area at a third instance in time occurring prior to the first instance;

adjusting the first shadow based at least in part on the movement to facilitate determining a second shadow to be displayed on the screen with the image generation device for augmenting the AR object at a fourth instance occurring after the third instance; and

generating a second image frame to be displayed on the screen with the image generation device at a fifth instance occurring after the fourth instance to facilitate presenting the AR view to include the AR object and the second shadow.

11. The non-transitory computer-readable medium of claim 10 further comprising non-transitory instructions sufficient for:

generating the first shadow to darken a first surface area proximate the AR object; and

generating the second shadow to darken a second surface area proximate the AR object, the second surface area being less than the first surface area by an amount proportional to the movement.

12. The non-transitory computer-readable medium of claim 1 further comprising non-transitory instructions sufficient for:

generating a spatial coordinate system for the real area using a spatial device of the AR device, the spatial coordinate system being sufficient to facilitate determining real object positioning in the real area relative to an X, Y and Z coordinate system centered at the AR device;

generating the first lighting map as a color-temperature graph sufficient for indicating a color and an intensity of light within the real area reaching the light sensor;

determining AR object coordinates within the X, Y and Z coordinate system for the AR object associated with positioning of the AR object within the real area;

estimating a lighting influence on the AR object based an offset or other relationship between the AR object coordinates and the first lighting map, the lighting influence relating the color and the intensity of the color-temperature map to the AR object coordinates; and

generating the first lighting effect to account for the lighting influence.

13. A non-transitory computer-readable medium having a plurality of non-transitory instructions executable with a processor of an augmented reality (AR) device to facilitate presenting an AR view of a real area, the non-transitory instructions being sufficient to facilitate:

generating a first lighting map for at least a portion of the real area;

determining an AR object to be displayed in order to present the AR view of the real area; and

determining a first lighting effect to be displayed with the AR object in the AR view based at least in part on extrapolating the first lighting map to determine lighting influences on the AR object.

14. The non-transitory computer-readable medium of claim 13 further comprising non-transitory instructions sufficient for:

generating the first lighting map to represent luminosity measured for the real area at a first position proximate the AR device;

determining a second position for the AR object within the AR view; and

determining the first lighting influence by extrapolating the first lighting map based at least in part on a positional offset between the first and second positions.

15. The non-transitory computer-readable medium of claim 13 further comprising non-transitory instructions sufficient for:

determining the first lighting influence to be a first shadow;

generating the first lighting map to at least in part represent luminosity measured for the real area proximate the AR device; and

determining a first shape of the first shadow to be proportional to the luminosity measured in the first lighting map.

16. The non-transitory computer-readable medium of claim 13 further comprising non-transitory instructions sufficient for:

determining the first lighting effect to be a first shadow associated with a light source positioned at a first location within the real area at a first instance;

generating a first image frame to be displayed on a screen of the AR device with at a second instance occurring after the first instance to facilitate presenting the AR view to include at least the AR object and the first shadow;

determining a movement of the light source from a second location to the first location, the second location corresponding with positioning of the light source within the real area at a third instance in time occurring prior to the first instance;

adjusting the first shadow based at least in part on the movement to facilitate determining a second shadow to be displayed on the screen for augmenting the AR object at a fourth instance occurring after the third instance; and

generating a second image frame to be displayed on the screen at a fifth instance occurring after the fourth instance to facilitate presenting the AR view to include the AR object and the second shadow.

17. The non-transitory computer-readable medium of claim 13 further comprising non-transitory instructions sufficient for:

generating a spatial coordinate system for the real area, the spatial coordinate system being sufficient to facilitate determining real object positioning in the real area relative to an X, Y and Z coordinate system relative to the AR device;

generating the first lighting map as a color-temperature graph sufficient for indicating a color and an intensity of light within the real area;

determining AR object coordinates within the X, Y and Z coordinate system for the AR object within the real area;

estimating a lighting influence on the AR object based an offset or other relationship between the AR object coordinates and the first lighting map, the lighting influence relating the color and the intensity of the color-temperature map to the AR object coordinates; and

generating the first lighting effect to account for the lighting influence.

18. The non-transitory computer-readable medium of claim 13 further comprising non-transitory instructions sufficient for:

determining the real area to include at least a first light source and a second light source at a first instance based at least in part on the first lighting map;

determining a first angle of incidence with the AR object for a first light emitted from the first light source based at least in part on the first lighting map;

determining a second angle of incidence with the AR object for a second light emitted from the first light source based at least in part on the first lighting map; and

determining a first shape of the first lighting effect at a second instance occurring after the first instance based at least in part on the first and second angles such that the first shape represents how the first and second light sources would influence a real shadow for the AR object if the AR object was real;

generating a second lighting map of the real area for at least a portion of the real area at a third instance occurring after the second instance;

determining the AR view at a fourth instance occurring after the third instance to require the AR object to be appear stationary at least in so far as the AR object appearing in the AR view at the second and fourth instances without changing position;

determining a second lighting effect to be displayed for augmenting the AR object at the fourth instance based at least in part on the second lighting map;

determining the second lighting effect to include a second shadow;

determining a second shape of the second shadow based at least in part on the second lighting map;

determining the real area to include the first and second light sources at the third instance based at least in part on the second lighting map;

determining a third angle of incidence with the AR object for a third light emitted from the first light source based at least in part on the second lighting map, including determining the third angle to match the first angle due to a position of the first light source being the same at the first and third instances;

determining a fourth angle of incidence with the AR object for a fourth light emitted from the second light source based at least in part on the second lighting map, including determining the fourth angle to differ from the second angle due to a position of the second light source being different at the first and third instances; and

determining the second shape of the second shadow based at least in part on the third and fourth angles such that the second shape differs from the first shape by an amount sufficient to cause the second shadow to appear differently in the AR view than the first shadow.

19. A method for presenting an augmented reality (AR) view of a real area comprising:

generating a first lighting map for at least a portion of the real area with a mapping device associated with an AR device;

determining an AR object to be projected on a screen of AR device with a sensory device in order to present the AR view of the real area; and

determining a first shadow to be displayed with the AR object in the AR view based at least in part on extrapolating the first lighting map to determine lighting influences on the AR object.

20. The method of claim 19 further comprising:

generating a second lighting map for at least a portion of the real area with the mapping device after detecting movement of at least on light source in the real area; and

determining a second shadow to be displayed with the AR object in the AR view in place of the first shadow after detecting the movement based at least in part on extrapolating the second lighting map to determine lighting influences on the AR object.