Ground plane adjustment in a virtual reality environment

• US 10,649,212 B2
• Filed: 08/17/2017
• Issued: 05/12/2020
• Est. Priority Date: 07/25/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Burns, Aaron, Sugden, Ben, Massey, Laura, Da Veiga, Alexandre, Salter, Tom, Alt, Greg
• Primary Examiner(s): Chow, Jeffrey J
• Application Number:
  US15/679,997
• Publication Number:
  US 20180003982A1
• Time in Patent Office: 999 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0167 Emergency system, e.g. to prevent injuries
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

An HMD device is configured to vertically adjust the ground plane of a rendered virtual reality environment that has varying elevations to match the flat real world floor so that the device user can move around to navigate and explore the environment and always be properly located on the virtual ground and not be above it or underneath it. Rather than continuously adjust the virtual reality ground plane, which can introduce cognitive dissonance discomfort to the user, when the user is not engaged in some form of locomotion (e.g., walking), the HMD device establishes a threshold radius around the user within which virtual ground plane adjustment is not performed. The user can make movements within the threshold radius without the HMD device shifting the virtual terrain. When the user moves past the threshold radius, the device will perform an adjustment as needed to match the ground plane of the virtual reality environment to the real world floor.

Claims

1. A method for rendering a virtual reality environment having a virtual ground plane on a head mounted display (HMD) device located in a physical space having a floor, comprising:

rendering, to a user of the HMD device, one or more virtual images in a stream of frames in which the virtual images are rendered with respect to the virtual ground plane, the virtual ground plane having variable elevations in the virtual reality environment;

using a depth sensor to generate surface reconstruction data of the physical space;

determining a position of the user'"'"'s head within the physical space using the surface reconstruction data;

dynamically adjusting the virtual ground plane to align with the floor of the physical space on a frame-by-frame basis, or group of frames basis; and

during transitions between chunks of the virtual ground plane, subjecting the dynamic adjustment of the alignment of the virtual ground plane to smoothing on the frame-by-frame basis or group of frames basis,

wherein the dynamic adjustment of the alignment of the virtual ground plane is performed according to the determined position of the user'"'"'s head within the physical space.

2. The method of claim 1 including using the surface reconstruction data to determine a location of the user'"'"'s head in the physical space, establishing a threshold radius around the location, maintaining a current alignment of the ground plane when the location is within the threshold radius, and performing the adjustment of the ground plane when the location is beyond the threshold radius.

3. The method of claim 1 in which the surface reconstruction data is generated using one or more depth-from-stereo imaging analyses.

4. The method of claim 1 further including configuring the HMD device to expose a user interface (UI) to the user and performing the adjusting of the virtual ground plane responsively to user input to the UI.

5. The method of claim 1 further including performing the virtual ground plane adjusting depending on a traversal path of the user in the virtual reality environment, the path being one of a path having increasing virtual elevation or a path having decreasing virtual elevation.

6. The method of claim 5 further including performing the adjusting for each of one or more discrete chunks of the virtual reality environment along the traversal path.

7. The method of claim 1 further including determining the height using surface reconstruction data obtained when the user looks down at the virtual reality environment ground plane.

8. A head mounted display (HMD) device operable by a user in a physical environment having a real world floor, comprising:

one or more processors;

a display for rendering a virtual reality environment to the user, the virtual reality environment including a ground plane with different elevations at different locations within the virtual reality environment; and

one or more memory devices storing computer-readable instructions which, when executed by the one or more processors, perform a method comprising the steps of;

generating surface reconstruction data using a depth sensor or camera system,

using the surface reconstruction data, determining a position of the user'"'"'s head within the physical environment,

operating the HMD device to render the virtual reality environment on the display to align a ground plane of the virtual reality environment with the real world floor based on the determined height, wherein the alignment is performed dynamically based on a time-based metric, and

subjecting the dynamic alignment of the virtual ground plane based on the time-based metric to smoothing, wherein the dynamic alignment is performed according to the determined position of the user'"'"'s head within the physical environment.

9. The HMD device of claim 8 in which the time-based metric is one of rate of a frame or group of frames.

10. The HMD device of claim 8 further including using the surface reconstruction data to determine whether the user is traversing the virtual reality environment and establishing a threshold radius when the user is not traversing the virtual reality environment.

11. The HMD device of claim 10 further including maintaining a current alignment of the ground plane when the location is within the threshold radius, and performing the alignment of the ground plane when the location is beyond the threshold radius.

12. The HMD device of claim 11 further including varying the threshold radius when the user is proximate to a portion of the virtual reality environment that includes abrupt changes in elevation.

13. The HMD device of claim 11 further including modeling the physical environment using a surface reconstruction data pipeline that implements a volumetric method creating multiple overlapping surfaces that are integrated.

14. One or more computer readable memory devices not consisting of propagating signals storing computer-executable instructions for rendering a virtual reality environment having variable virtual elevations on a head mounted display (HMD) device located in a physical space having a floor, the method comprising the steps of:

using one or more sensors in the HMD device to dynamically generate and update a model of the physical space as the user traverses the virtual reality environment;

determining a position of the user'"'"'s head within the physical space using the model;

displaying a portion of the virtual reality environment in a display of the HMD device using a succession of video frames; and

using the determined position, matching an elevation of a ground plane of the virtual reality environment with the floor of the physical space in one or more frames of video,

wherein the matching is performed on a frame-by-frame basis or a group of frames basis according to the determined position of the user'"'"'s head within the physical space.

15. The one or more computer-readable memory devices of claim 14 in which the matching is performed continuously.

16. The one or more computer-readable memory devices of claim 14 further including suppressing the matching when the user remains at a stationary location in the virtual reality environment.

17. The one or more computer-readable memory devices of claim 14 further including suppressing the matching when the user is located within a predetermined threshold to a point in the virtual reality environment.

18. The one or more computer-readable memory devices of claim 17 further including resuming the matching when the user moves past the threshold.