THEME-BASED AUGMENTATION OF PHOTOREPRESENTATIVE VIEW

• US 20120229508A1
• Filed: 03/10/2011
• Published: 09/13/2012
• Est. Priority Date: 03/10/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Tedesco, Megan, Wigdor, Daniel J.
• Granted Patent: US 10,972,680 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 5/262 Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects

Abstract

On a display configured to provide a photorepresentative view from a user'"'"'s vantage point of a physical environment in which the user is located, a method is provided comprising receiving, from the user, an input selecting a theme for use in augmenting the photorepresentative view. The method further includes obtaining, optically and in real time, environment information of the physical environment and generating a spatial model of the physical environment based on the environment information. The method further includes identifying, via analysis of the spatial model, one or more features within the spatial model that each corresponds to one or more physical features in the physical environment. The method further includes based on such analysis, displaying, on the display, an augmentation of an identified feature, the augmentation being associated with the theme.

Claims

1. On a display configured to provide a photorepresentative view from a user'"'"'s vantage point of a physical environment in which the user is located, a method of providing theme-based augmenting of the photorepresentative view, the method comprising:

receiving, from the user, an input selecting a theme for use in augmenting the photorepresentative view;

obtaining, optically and in real time, environment information of the physical environment;

generating a spatial model of the physical environment based on the environment information;

identifying, via analysis of the spatial model, one or more features within the spatial model that each corresponds to one or more physical features in the physical environment; and

based on such analysis, displaying, on the display, an augmentation of an identified feature, the augmentation being associated with the theme.

2. The method of claim 1, further comprising, upon analysis of the spatial model, selecting the augmentation from a plurality of augmentations associated with the theme, such selection being based on the one or more features identified.

3. The method of claim 1, wherein the display is a head-up display.

4. The method of claim 1, wherein the display is a head-mounted display.

5. The method of claim 1, wherein displaying the augmentation comprises displaying an image which, from the user'"'"'s vantage point, overlays the physical feature in the physical environment corresponding to the identified feature.

6. The method of claim 1, wherein displaying the augmentation comprises applying a virtual skin to the identified feature of the spatial model and displaying an image of the virtual skin which, from the user'"'"'s vantage point, overlays the corresponding physical feature in the physical environment.

7. The method of claim 1, wherein the display is an optical see-through display configured to provide the photorepresentative view via one or more sufficiently transparent portions of the display through which the physical environment is visible to the user.

8. The method of claim 7, wherein displaying the augmentation comprises overlaying a virtual object onto the identified feature of the spatial model and displaying an image of the virtual object which, from the user'"'"'s vantage point, overlays the corresponding physical feature in the physical environment visible to the user through the display.

9. The method of claim 1, wherein the display is an immersive display configured to provide the photorepresentative view by displaying a fully rendered image of the spatial model.

10. The method of claim 9, wherein displaying the augmentation comprises modifying one or more features identified within the spatial model, and in response, displaying a fully rendered image of the spatial model which reflects such modification of the spatial model.

11. The method of claim 1, wherein obtaining the environment information comprises detecting the environment information via one or more sensors associated with the display.

12. The method of claim 1, further comprising maintaining displaying of the augmentation as the user moves through the physical environment.

13. A display system for providing theme-based augmentation via display output, comprising:

a display configured to provide a photorepresentative view from a user'"'"'s vantage point of a physical environment in which the user is located;

one or more sensors configured to obtain, optically and in real time, environment information of the physical environment;

a data-holding subsystem operatively coupled with the display and the one or more sensors, the data-holding subsystem containing instructions executable by a logic subsystem to;

receive, from the user, an input selecting a theme for use in augmenting the photorepresentative view;

generate a spatial model of the physical environment based on the environment information;

identify, via analysis of the spatial model, one or more features within the spatial model that each corresponds to one or more physical features in the physical environment; and

based on such analysis, display, on the display, an augmentation of an identified feature, the augmentation being associated with the theme.

14. The display system of claim 13, wherein the display is a head-up display.

15. The display system of claim 13, wherein the display is a head-mounted display.

16. The display system of claim 13, wherein the display is an optical see-through display configured to provide the photorepresentative view via one or more sufficiently transparent portions of the display through which the physical environment is visible to the user.

17. The display system of claim 13, wherein the display is an immersive display configured to provide the photorepresentative view by displaying a fully rendered image of the spatial model.

18. The display system of claim 13, wherein the one or more sensors comprise an image capture device configured to obtain one or more depth images of the physical environment.

19. On a display, a method of providing theme-based augmentation via display output, the method comprising:

receiving, from a user, an input selecting a theme for augmenting a photorepresentative view, the photorepresentative view being from a vantage point of the user and of a physical environment in which the user is located;

detecting, optically and in real time, environment information of the physical environment via one or more sensors positioned proximal to the display;

generating a spatial model of the physical environment based on the environment information;

identifying, via analysis of the spatial model, one or more features within the spatial model that each corresponds to one or more physical features in the physical environment;

selecting an augmentation from a plurality of augmentations associated with the theme based on the one or more features identified; and

in response, displaying, on the display, the augmentation of an identified feature while still providing at least some of the photorepresentative view of the physical environment.

20. The method of claim 19, wherein displaying the augmentation comprises dynamically displaying the augmentation as the user moves within the physical environment to cause consequent change in the photorepresentative view.