VIRTUAL LIGHT IN AUGMENTED REALITY

• US 20130141434A1
• Filed: 12/01/2011
• Published: 06/06/2013
• Est. Priority Date: 12/01/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Robert Crocco Jr
• Inventors: Sugden, Ben, Bennett, Darren, Mount, Brian, Tomlin, Arthur, Hastings, Ryan, McCulloch, Daniel, Geisner, Kevin, Crocco, Robert Jr., Sylvan, Sebastian
• Granted Patent: US 8,872,853 B2
• Time in Patent Office: Days
• US Class Current: 345/426
• CPC Class Codes: G02B 2027/0112 comprising device for genereting colour display
• CPC Class Codes: G02B 2027/0118 comprising devices for improving the contrast of the display / brillance control visibility
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 15/60 Shadow generation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/73 using feature-based methods

Abstract

A head-mounted display system includes a see-through display that is configured to visually augment an appearance of a physical environment to a user viewing the physical environment through the see-through display. Graphical content presented via the see-through display is created by modeling the ambient lighting conditions of the physical environment.

Claims

1. A method for a computing device, comprising:

receiving optical sensor information output by an optical sensor system observing a physical environment, the optical sensor system forming a sensory component of a head-mounted display system;

receiving position sensor information output by a position sensor system indicating a perspective of the optical sensor system within the physical environment, the position sensor system forming another sensory component of the head-mounted display system;

creating an ambient lighting model from the optical sensor information and the position sensor information, the ambient lighting model describing ambient lighting conditions of the physical environment;

modeling the physical environment from the optical sensor information and the position sensor information to create a virtual environment;

applying the ambient lighting model to the virtual environment including an added virtual object not present in the physical environment to obtain an illuminated virtual object; and

rendering a graphical representation of the illuminated virtual object for presentation via a see-through display of the head-mounted display system, the see-through display configured to visually augment an appearance of the physical environment to a user viewing the physical environment through the see-through display.

2. The method of claim 1, further comprising:

applying the ambient lighting model to the virtual environment to obtain a virtual shadow of the illuminated virtual object projected on a virtual surface within the virtual environment; and

rendering a graphical representation of a non-shadow region of the virtual surface bordering the virtual shadow for presentation via the see-through display.

3. The method of claim 1, wherein the virtual shadow is rendered with the graphical representation of the non-shadow region as one or more pixels omitted from the graphical representation.

4. The method of claim 1, further comprising:

applying the ambient lighting model to the virtual environment to obtain a virtual shadow of the illuminated virtual object projected on a virtual surface within the virtual environment; and

rendering a graphical representation of the virtual shadow for presentation via the see-through display, the graphical representation of the virtual shadow configured to reduce an intensity of light transmitted through the see-through display corresponding to a location of the virtual shadow in the physical environment as viewed by the user through the see-through display.

5. The method of claim 1, wherein the ambient lighting model describes one or more of a light vector, a light intensity, and/or a light color of the ambient lighting conditions.

6. The method of claim 5, wherein applying the ambient lighting model to the virtual object includes adding a virtual light source at a location within the virtual environment that approximates the light vector, the light intensity, and/or the light color of the ambient lighting conditions within the physical environment.

7. The method of claim 1, further comprising:

directly observing a light source within the physical environment via the optical sensor system;

identifying a location of the light source within the physical environment from the position sensor information; and

adding a virtual light source to the virtual environment at a location that approximates the location of the light source identified within the physical environment.

8. The method of claim 1, wherein the position sensor information indicates the perspective of the optical sensor system as a measured change in a location and/or orientation of the head-mounted display system within the physical environment.

9. The method of claim 1, wherein the position sensor information indicates the perspective of the optical sensor system within the physical environment as a location of the head-mounted display system measured in three-dimensional space along three orthogonal axes, and an orientation of the head-mounted display system about the three orthogonal axes.

10. The method of claim 1, wherein the optical sensor information includes time-based optical sensor information output by the optical sensor system observing the physical environment as the perspective of the optical sensor system within the physical environment changes over time;

and

wherein the ambient lighting model and the virtual environment are based on multiple time-series frames of the time-based optical sensor information; and

wherein the graphical representation of the illuminated virtual object is rendered in real-time as the perspective of the optical sensor system within the physical environment changes over time.

11. The method of claim 10, wherein the position sensor information includes time-based position sensor information output by the position sensor system as the perspective of the optical sensor system within the physical environment changes over time;

and

wherein the ambient lighting model and the virtual environment are based on multiple time-series measurements of the time-based position sensor information.

12. The method of claim 1, wherein the optical sensor system is a first optical sensor system of a plurality of optical sensor systems within the physical environment;

wherein the method further comprises receiving optical sensor information output by a second optical sensor system observing the physical environment, the second optical sensor system forming a sensory component of a second head-mounted display system; and

wherein the ambient lighting model and virtual environment are further based on the optical sensor information output by the second optical sensor system.

13. A data-holding subsystem holding instructions executable by a logic subsystem to:

receive optical sensor information output by an optical sensor system observing a physical environment, the optical sensor system forming a sensory component of a head-mounted display system;

receive position sensor information output by a position sensor system indicating a perspective of the optical sensor system within the physical environment, the position sensor system forming another sensory component of the head-mounted display system;

create an ambient lighting model from the optical sensor information and the position sensor information describing ambient lighting conditions of the physical environment;

apply the ambient lighting model to a virtual environment modeling the physical environment to obtain an illuminated virtual environment; and

render a graphical representation of the illuminated virtual environment for presentation via a see-through display of the head-mounted display system, the see-through display configured to visually augment an appearance of the physical environment to a user viewing the physical environment through the see-through display.

14. The data-holding subsystem of claim 13, wherein the instructions are further executable by the logic subsystem to:

add a virtual object not present in the physical environment to the virtual environment;

apply the ambient lighting model to the virtual object of the virtual environment to obtain an illuminated virtual object; and

render a graphical representation of the illuminated virtual object for presentation via the see-through display.

15. The data-holding subsystem of claim 14, wherein the ambient lighting model is applied to the virtual environment to obtain a virtual shadow of the illuminated virtual object projected on a virtual surface within the virtual environment;

and

wherein the illuminated virtual environment includes a non-shadow region of the virtual surface bordering the virtual shadow of the illuminated virtual object.

16. The data-holding subsystem of claim 13, wherein the ambient lighting model describes one or more of a light vector, light intensity, and/or a light color of the ambient lighting conditions;

and

wherein applying the ambient lighting model to the virtual object includes adding a virtual light source at a location within the virtual environment that approximates the light vector, light intensity, and light color of the ambient lighting conditions within the physical environment.

17. The data-holding subsystem of claim 13, wherein the instructions are further executable by the logic subsystem to:

directly observe a light source within the physical environment via the optical sensor system;

identify a location of the light source within the physical environment from the position sensor information; and

add a virtual light source to the virtual environment at a location that approximates the location of the light source identified within the physical environment.

18. The data-holding subsystem of claim 13, wherein the instructions are further executable by the logic subsystem to:

create the virtual environment modeling the physical environment from the position sensor information and the optical sensor information as a wireframe mesh; and

render the graphical representation as a surface mapped to the wireframe mesh.

19. A computing system, comprising:

a head-mounted display system including;

a see-through display configured to visually augment an appearance of the physical environment to a user viewing the physical environment through the see-through display,

an optical sensor system configured to observe a physical environment, and

a position sensor system configured to indicate a perspective of the optical sensor system within the physical environment; and

a computing device configured to;

receive optical sensor information output by the optical sensor system observing the physical environment;

receive position sensor information output by the position sensor system indicating the perspective of the optical sensor system;

create an ambient lighting model from a combination of the optical sensor information and the position sensor information describing ambient lighting conditions of the physical environment;

model the physical environment from the combination of the optical sensor information and the position sensor information to create a virtual environment;

apply the ambient lighting model to the virtual environment to obtain an illuminated virtual environment; and

render a graphical representation of the illuminated virtual environment in real-time for presentation via the see-through display as the perspective of the optical sensor system within the physical environment changes over time.

20. The computing system of claim 19, wherein the computing device is further configured to:

add a virtual object not present in the physical environment to the virtual environment;

apply the ambient lighting model to the virtual object of the virtual environment to obtain an illuminated virtual object; and

render a graphical representation of the illuminated virtual object for presentation via the see-through display.