Anti-trip when immersed in a virtual reality environment

• US 10,311,638 B2
• Filed: 01/20/2015
• Issued: 06/04/2019
• Est. Priority Date: 07/25/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Da Veiga, Alexandre, Sugden, Ben, Massey, Laura
• Primary Examiner(s): Patel, Jitesh
• Application Number:
  US14/600,774
• Publication Number:
  US 20160027212A1
• Time in Patent Office: 1,596 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0487 using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G06T 15/08 Volume rendering
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/332 Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

An HMD device with a see-through display and depth sensing capability is configured to selectively dim or fade out a display of a virtual reality environment to enable a user to see the real world without obstruction by the virtual world when a distance between the user and a real world object is determined to be less than a threshold distance. The current height of the user'"'"'s head (i.e., the distance from head to ground) may be utilized when performing the dimming/fading so that different threshold distances can be used depending on whether the user is standing or seated.

Claims

1. A method performed by a head mounted display (HMD) device supporting rendering of a virtual reality environment, comprising:

obtaining sensor data describing a physical space adjoining a user of the HMD device;

using the sensor data, reconstructing geometries of real world objects located in the physical space;

representing the user with a volumetric shape that is subject to dynamic variations in size or shape;

setting a predetermined threshold distance between the user and the real world objects by which to trigger an operation by the HMD device, wherein the predetermined threshold distance is defined as extending from an outer boundary of the volumetric shape to a surface of a respective real world object;

dynamically adjusting the predetermined threshold distance responsive to one or more variations in the size or shape of the volumetric shape;

determining a current distance between the user and the real world objects using the reconstructed geometries and the volumetric shape; and

when the determined current distance is equal to or less than the predetermined threshold distance, triggering the HMD device to operate so that the user can see the real world objects without obstruction from the virtual reality environment.

2. The method of claim 1 in which the sensor data includes depth data and further including generating the sensor data using a depth sensor and applying surface reconstruction techniques to reconstruct the real world object geometries.

3. The method of claim 1 further including generating depth data using one or more depth-from-stereo imaging analyses.

4. The method of claim 1 further including dimming at least a portion of a see-through display incorporated into the HMD device during the triggered operation.

5. The method of claim 1 further including causing at least a portion of a see-through display incorporated into the HMD device to become substantially transparent.

6. The method of claim 1 further including sizing the volumetric shape according to a height of the user'"'"'s head above a ground of the physical space.

7. The method of claim 6 further including configuring the predetermined threshold distance to be relatively smaller when the user is seated as compared to when the user is standing.

8. The method of claim 6 further including using the sensor data to determine a current height of the user.

9. A head mounted display (HMD) device operable by a user in a physical environment, comprising:

one or more processors;

a see-through display employing additive blending between real world and virtual world display elements; and

one or more memory devices storing computer-readable instructions which, when executed by the one or more processors, perform a method comprising the steps of;

representing the user with a volumetric shape subject to variations,

generating surface reconstruction data for real world objects using a depth sensor or camera system,

setting a predetermined threshold distance between the user and the real world objects by which to trigger an operation by the HMD device, wherein the predetermined threshold distance is defined as extending from an outer boundary of the volumetric shape to a surface of a respective real world object,

dynamically adjusting the predetermined threshold distance responsive to one or more variations in a size or shape of the volumetric shape, wherein the dynamically adjusted predetermined threshold distance is affected by, but independent of, the variations in size or shape of the volumetric shape,

determining a current distance to a real world object in the physical environment from the volumetric shape, and

when the current distance is equal to or less than the predetermined threshold distance, triggering the HMD device to operate so that at least a portion of the see-through display is seen through by the user.

10. The HMD device of claim 9 in which the triggered operation further includes operating the see-through display by rendering dark pixels for display of the virtual world so that at least a portion of the virtual world does not obscure display of the real world object.

11. The HMD device of claim 9 in which the depth sensor is arranged for collecting the surface reconstruction data.

12. The HMD device of claim 9 further including varying the predetermined threshold distance depending on a determination as to whether the user is standing or sitting.

13. The HMD device of claim 9 further including modeling the physical environment using a surface reconstruction data pipeline that implements a volumetric method creating multiple overlapping surfaces that are integrated.

14. The HMD device of claim 9 further including a sealed protective visor that substantially encases the see-through display.

15. One or more computer readable memories storing computer-executable instructions for implementing a method for providing an anti-trip feature for an immersive virtual environment rendered for a user of a head mounted display (HMD) device in a physical environment, the method comprising the steps of:

configuring display optics incorporated into the HMD device for selectively transparent operation by dimming the rendering of the virtual environment;

dynamically modeling the physical environment and real world objects contained therein using surface reconstruction;

determining at least the user'"'"'s height in the physical environment using the surface reconstruction model;

representing the user with a volumetric shape, wherein a size or shape of the volumetric shape is at least based on the determined user'"'"'s height;

determining a distance from the volumetric shape to a real world object in the surface reconstructed model; and

dimming the display optics for transparent operation to enable the user to see the real world object through the display optics when the distance is equal to or less than a predetermined threshold, wherein the predetermined threshold is varied according to the size, shape, or other characteristic of the volumetric shape, and wherein the predetermined threshold is defined as extending from an outer boundary of the volumetric shape to a surface of the real world object.

16. The one or more computer readable memories of claim 15 further including iteratively performing the steps of dynamically modeling the physical environment and determining the distance from the volumetric shape to the real world object.

17. The one or more computer-readable memories of claim 16 further including undimming the display optics to resume rendering of the virtual environment when the distance exceeds the predetermined threshold.