Virtual reality environment with real world objects

• US 9,865,089 B2
• Filed: 04/16/2015
• Issued: 01/09/2018
• Est. Priority Date: 07/25/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Burns, Aaron, Salter, Tom, Sugden, Ben, Sutherland, Jeff
• Primary Examiner(s): Du, Haixia
• Application Number:
  US14/688,695
• Publication Number:
  US 20160027215A1
• Time in Patent Office: 999 Days
• CPC Class Codes: G02B 2027/0123 comprising devices increasing the field of view
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0174 holographic
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0103 comprising holographic elements
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts

Abstract

An HMD device renders a virtual reality environment in which areas of the real world are masked out so that real world objects such as computer monitors, doors, people, faces, and the like appear visible to the device user and no holographic or virtual reality content is rendered over the visible objects. The HMD device includes a sensor package to support application of surface reconstruction techniques to dynamically detect edges and surfaces of the real world objects and keep objects visible on the display as the user changes position or head pose or when the real world objects move or their positions are changed. The HMD device can expose controls to enable the user to select which real world objects are visible in the virtual reality environment.

Claims

1. A method performed by a head mounted display (HMD) device that supports rendering of a mixed-reality environment including holographic content from a virtual world and objects from a real world, comprising:

obtaining sensor data for one or more objects in the real world included within a physical environment that adjoins a user of the HMD device;

reconstructing a geometry for the one or more objects from the sensor data;

using the reconstructed geometry, masking selected areas of the real world for inclusion in the mixed-reality environment in which the masked areas include real world objects, the real world objects being viewable by the user as the mixed-reality environment is rendered on the HMD device;

rendering the mixed-reality environment on a display in the HMD device including portions of the virtual world and the masked areas of the real world; and

using the reconstructed geometry, creating a holographic geometry in which virtual content is prevented from being rendered in front of the masked areas regardless of the user'"'"'s head pose or position relative to the virtual world.

2. The method of claim 1 in which the sensor data includes depth data and further including generating the sensor data using a depth sensor and applying surface reconstruction techniques to reconstruct the geometry.

3. The method of claim 1 further including generating depth data using depth-from-stereo imaging analyses.

4. The method of claim 1 further including tracking the user'"'"'s head in the physical environment using reconstructed geometry of the physical environment to determine a current field of view.

5. The method of claim 4 further performing the showing in a dynamic manner responsively to the current field of view.

6. The method of claim 1 further including exposing controls to the user for selecting objects in the real world that are included within the masked areas.

7. The method of claim 1 further including using the reconstructed geometry to generate a holographic geometry in which the HMD device prevents holographic content from being rendered on the display either in front of the holographic geometry or behind the holographic geometry.

8. The method of claim 1 further including providing a control to the user to select areas of the real world to be masked.

9. The method of claim 1 further including using the reconstructed geometry to identify real world objects to the user that are selectable for masking.

10. The method of claim 1 in which the masked areas of the real world are configured to be decoupled or non-interactive with virtual world content.

11. A head mounted display (HMD) device configured to render a mixed-reality environment including holographic content from a virtual world and objects from a real world, and operable by a user in the real world, comprising:

one or more processors;

a see-through display having areas onto which a virtual reality environment is rendered to the user, a field of view of the rendered virtual reality environment being variable depending at least in part on a pose of the user'"'"'s head in the real world;

a sensor package; and

one or more memory devices storing computer-readable instructions which, when executed by the one or more processors, perform;

generating surface reconstruction data for at least a portion of the real world using the sensor package,

using the surface reconstruction data to create a holographic geometry, in which selected areas of the real world are masked for inclusion in the mixed-reality environment, the masked areas including real world objects, the real world objects being viewable by the user as the mixed-reality environment is rendered on the see-through display, and

operating the HMD device to use the holographic geometry when rendering the masked areas of the real world and the virtual reality environment within the field of view of the see-through display,

the holographic geometry created so that virtual content is prevented from being rendered in front of the masked areas regardless of the user'"'"'s head pose or position relative to the virtual world.

12. The HMD device of claim 11 further including providing one or more controls to enable the user to select portions of the real world that are masked.

13. The HMD device of claim 11 further including using the surface reconstruction data to detect edges or surfaces of real world objects in the real world environment when creating the holographic geometry.

14. The HMD device of claim 11 further including using the surface reconstruction data to track a head position of the user and adjusting the field of view based on the head position while maintaining the holographic geometry.

15. The HMD device of claim 11 further including modeling the real world environment using a surface reconstruction data pipeline that implements a volumetric method creating multiple overlapping surfaces that are integrated.

16. The HMD device of claim 11 further including rendering special holographic content into masked areas, the special holographic content being characterized by one or more of priority, importance, or urgency.

17. One or more computer readable memories storing computer-executable instructions for rendering a mixed-reality environment including holographic content from a virtual world and objects from a real world within a variable field of view of a head mounted display (HMD) device located in the real world, the method comprising the steps of:

using data from a sensor package incorporated into the HMD device to dynamically generate a surface reconstruction model of the real world including real world objects included therein;

enabling the user to select real world objects to be visible on a display of the HMD device;

masking the selected areas of the real world for inclusion in the mixed-reality environment in which masked areas include real world objects, the real world objects being viewable by the user as the mixed-reality environment is rendered on the display; and

using the surface reconstruction model to render the mixed-reality environment including portions of the virtual world and the masked real world objects, and

using the reconstructed geometry, creating a holographic geometry in which virtual content is prevented from being rendered in front of the masked real world objects regardless of the user'"'"'s head pose or position relative to the virtual world.

18. The one or more computer readable memories of claim 17 further including tracking a location of an HMD device user'"'"'s head in the real world environment using the surface reconstruction model of the real world environment to determine a field of view of the virtual reality environment.

19. The one or more computer-readable memories of claim 17 further including refreshing the rendered virtual reality environment as the field of view changes.

20. The one or more computer-readable memories of claim 17 further including enabling the user to select the real world objects by tracking the user'"'"'s projected gaze and receiving a selection instruction from the user comprising one of a control actuation, voice command, natural language interaction, or gesture.