Virtual environment generating system

• US 9,734,633 B2
• Filed: 01/27/2012
• Issued: 08/15/2017
• Est. Priority Date: 01/27/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Bennett, Darren, Hastings, Ryan, Steed, Jonathan, Scott, Jason, Tomlin, Arthur, Sylvan, Sebastian, Lamb, Mathew, Mount, Brian, Latta, Stephen, Kipman, Alex, McCulloch, Daniel
• Primary Examiner(s): McDowell, Jr., Maurice L
• Application Number:
  US13/360,492
• Publication Number:
  US 20130194259A1
• Time in Patent Office: 2,027 Days
• Field of Search: 345420, 345633
• CPC Class Codes: G05B 2219/35503 Eye tracking associated with head mounted display to detect eye position
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: H04M 2203/359 Augmented reality

Abstract

A system and related methods for visually augmenting an appearance of a physical environment as seen by a user through a head-mounted display device are provided. In one embodiment, a virtual environment generating program receives eye-tracking information, lighting information, and depth information from the head-mounted display. The program generates a virtual environment that models the physical environment and is based on the lighting information and the distance of a real-world object from the head-mounted display. The program visually augments a virtual object representation in the virtual environment based on the eye-tracking information, and renders the virtual object representation on a transparent display of the head-mounted display device.

Claims

1. A method for visually augmenting an appearance of a physical environment including a real-world object as seen by a user through a transparent display in a head-mounted display device, comprising:

receiving eye-tracking information from an eye-tracking system in the head-mounted display device;

determining from the eye-tracking information that the user is looking at a virtual object representation;

receiving lighting information from an optical sensor system in the head-mounted display device;

receiving depth information from the optical sensor system, the depth information including a distance of the real-world object from the head-mounted display device;

generating a virtual environment that models the physical environment and is based on the lighting information and the distance of the real-world object from the head-mounted display device, the virtual environment including the virtual object representation and a real-world object representation;

visually augmenting the virtual object representation in the virtual environment based on the eye-tracking information that the user is looking at the virtual object representation;

rendering the virtual object representation on the transparent display of the head-mounted display device;

determining from the eye-tracking information that the user is looking at the real-world object;

based on determining that the user is looking at the real-world object, blurring the virtual object representation as seen by the user through the transparent display;

analyzing a contrast pattern behind the virtual object representation in the physical environment; and

adjusting one or more of a color and a brightness of the virtual object representation to enhance contrast of the virtual object representation with respect to the contrast pattern.

2. The method of claim 1, further comprising:

based on the lighting information, rendering a virtual shadow representation on the transparent display in an area adjacent to the virtual object representation.

3. The method of claim 2, wherein rendering the virtual shadow representation further comprises filtering light that enters the transparent display from the physical environment and/or rendering color in the area adjacent to the virtual object representation.

4. The method of claim 1, further comprising:

determining from the depth information that the user is looking at the virtual object representation; and

based on the lighting information, rendering a virtual shadow representation on the transparent display in an area adjacent to the virtual object representation.

5. The method of claim 1, further comprising visually augmenting an appearance of the real-world object to render a reflected image of the virtual object representation.

6. The method of claim 1, further comprising:

determining a blurring amount based on one or more of the depth information, a focal point distance, and a pupil dilation amount.

7. The method of claim 6, further comprising progressively increasing the blurring amount along segments of the virtual object representation as a distance of the segments from a virtual position of the head-mounted display device in the virtual environment increases.

8. The method of claim 1, further comprising:

generating one or more of a light originating from the virtual object representation and a shadow cast by the virtual object representation; and

when the virtual object representation is not within a current viewing perspective of the user, visually augmenting the real-world object as seen by the user through the transparent display with the one or more of the light originating from the virtual object representation and the shadow cast by the virtual object representation.

9. The method of claim 1, further comprising:

rendering the virtual object representation in a fixed location on the transparent display regardless of an orientation of the head-mounted display device.

10. A virtual environment generating system including a head-mounted display device operatively connected to a computing device, the head-mounted display device including a transparent display through which a user may view a physical environment that includes a real-world object, the virtual environment generating system comprising:

a virtual environment generating program executed by a processor of the computing device, the virtual environment generating program configured to;

receive eye-tracking information from an eye-tracking system in the head-mounted display device;

determine from the eye-tracking information that the user is looking at a virtual object representation;

receive lighting information from an optical sensor system in the head-mounted display device;

receive depth information from the optical sensor system, the depth information including a distance of the real-world object from the head-mounted display device;

generate a virtual environment that models the physical environment and is based on the lighting information and the distance of the real-world object from the head-mounted display device, the virtual environment including the virtual object representation and a real-world object representation;

visually augment the virtual object representation in the virtual environment based on the eye-tracking information that the user is looking at the virtual object representation;

render the virtual object representation on the transparent display of the head-mounted display device;

determine from the eye-tracking information that the user is looking at the real-world object;

based on determining that the user is looking at the real-world object, blur the virtual object representation as seen by the user through the transparent display;

analyze a contrast pattern behind the virtual object representation in the physical environment; and

adjust one or more of a color and a brightness of the virtual object representation to enhance contrast of the virtual object representation with respect to the contrast pattern.

11. The virtual environment generating system of claim 10, wherein the virtual environment generating program is further configured to visually augment an appearance of the real-world object to render a reflected image of the virtual object representation.

12. The virtual environment generating system of claim 10, wherein the virtual environment generating program is further configured to determine a blurring amount based on one or more of the depth information, a focal point distance, and a pupil dilation amount.

13. The virtual environment generating system of claim 12, wherein the virtual environment generating program is further configured to progressively increase the blurring amount along segments of the virtual object representation as a distance of the segments from a virtual position of the head-mounted display device in the virtual environment increases.

14. The virtual environment generating system of claim 10, wherein the virtual environment generating program is further configured to:

generate one or more of a light originating from the virtual object representation and a shadow cast by the virtual object representation; and

when the virtual object representation is not within a current viewing perspective of the user, visually augment the real-world object as seen by the user through the transparent display with the one or more of the light originating from the virtual object representation and the shadow cast by the virtual object representation.

15. A nonvolatile computer-readable storage medium comprising instructions stored thereon and executable by a computing device to visually augment an appearance of a physical environment including a real-world object as seen by a user through a transparent display in a head-mounted display device, the instructions being executable to:

receive eye-tracking information from an eye-tracking system in the head-mounted display device;

receive lighting information from an optical sensor system in the head-mounted display device;

receive depth information from the optical sensor system, the depth information including a distance of the real-world object from the head-mounted display device;

generate a virtual environment that models the physical environment and is based on the lighting information and the distance of the real-world object from the head-mounted display device, the virtual environment including a virtual object representation and a real-world object representation;

determine from the eye-tracking information that the user is looking at the real-world object;

based on the eye-tracking information that the user is looking at the real-world object, blur an appearance of the virtual object representation as seen by the user through the transparent display;

analyze a contrast pattern behind the virtual object representation in the physical environment;

adjust one or more of a color and a brightness of the virtual object representation to enhance contrast of the virtual object representation with respect to the contrast pattern; and

render the virtual object representation on the transparent display of the head-mounted display device.

16. The nonvolatile computer-readable storage medium of claim 15, wherein the instructions are executable by the computing device to:

determine a blurring amount based on one or more of the depth information, a focal point distance, and a pupil dilation amount.