Use of surface reconstruction data to identify real world floor

• US 9,645,397 B2
• Filed: 04/27/2015
• Issued: 05/09/2017
• Est. Priority Date: 07/25/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: da Veiga, Alexandre, Sugden, Ben, Salter, Tom
• Primary Examiner(s): Okebato, Sahlu
• Application Number:
  US14/697,227
• Publication Number:
  US 20160027217A1
• Time in Patent Office: 743 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0167 Emergency system, e.g. to prevent injuries
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

In a virtual reality or mixed reality environment, an HMD device is configured to use surface reconstruction data points obtained with a sensor package to identify a location of a floor of a real world environment in which the device operates by sorting the data points by height into respective buckets where each bucket holds a different range of heights. A bucket having the greatest number of data points that are below the height of a user of the HMD device is used to identify the height of the real world floor, for example, by calculating an average of height values of data points in that bucket. A floor for the virtual reality environment may then be aligned to the identified height of the real world floor.

Claims

1. A method performed by a head mounted display (HMD) device to identify a height of a real world floor, the HMD device supporting rendering of a virtual or mixed reality environment, the method comprising:

obtaining surface reconstruction data associated with a real world environment adjoining a user of the HMD device;

classifying surface reconstruction data points by height;

sorting the classified data points into respective buckets;

selecting a bucket having a greatest number of data points that are below a height of the user; and

identifying a height of the real world floor relative to the user based on the data points in the selected bucket.

2. The method of claim 1 in which the surface reconstruction data includes depth data and further including generating the surface reconstruction data using a depth sensor and applying surface reconstruction techniques to reconstruct the real world environment geometry.

3. The method of claim 1 further including generating depth data using one or more depth-from-stereo imaging analyses.

4. The method of claim 1 further including aligning a height of a virtual world floor with the height of the real world floor.

5. The method of claim 1 further including aligning a virtual world object to the real world floor.

6. The method of claim 1 in which the surface reconstruction data is captured using a sensor package in the HMD device, the sensor package including a depth sensor or depth camera.

7. The method of claim 1 further including configuring the HMD device to expose a user interface (UI) to the user and prompting the user, through the UI, to stand and look down.

8. The method of claim 1 further including determining the height of the user utilizing surface reconstruction data that is captured when the user looks down.

9. The method of claim 1 further including selecting a bucket that is below the user'"'"'s height by some predetermined threshold.

10. The method of claim 1 further including calculating an average of height values of data points in the selected bucket to determine the height of the real world floor.

11. A head mounted display (HMD) device operable by a user in a physical environment having a real world floor, comprising:

one or more processors;

a sensor package;

a display for rendering a virtual reality environment to the user; and

one or more memory devices storing computer-readable instructions which, when executed by the one or more processors, perform a method comprising the steps of;

i) generating surface reconstruction data for the physical environment using the sensor package,

ii) using the surface reconstruction data to identify a location of the real world floor relative to the HMD device by

a) sorting surface reconstruction data points into a group among a plurality of groups according to vertical distance from the HMD device, each group in the plurality of groups covering a different range of vertical distances,

b) selecting a group having a greatest number of data points that exceed a threshold vertical distance from a current position of the HMD device,

c) identifying a location of the real world floor relative to the HMD device based on the data points in the selected group, and

iii) operating the HMD device to render the virtual reality environment on the display so that the virtual world floor is aligned with the identified location of the real world floor.

12. The HMD device of claim 11 further including calculating an average of values of data points in the selected group to identify the location of the real world floor relative to the HMD device.

13. The HMD device of claim 11 further including modeling the physical environment using a surface reconstruction data pipeline that implements a volumetric method creating multiple overlapping surfaces that are integrated.

14. The HMD device of claim 11 further including a sealed protective visor that substantially encases the display.

15. The HMD device of claim 11 further including dynamically identifying the location of the real world floor relative to the HMD device as the user traverses the physical environment.

16. The HMD device of claim 15 further including realigning the virtual world floor with the real world floor responsively to the dynamic identifying.

17. One or more computer readable memories not consisting of propagated signals storing computer-executable instructions for rendering a virtual or mixed reality environment having variable elevations of virtual ground on a head mounted display (HMD) device located in a physical environment having a floor, the method comprising the steps of:

obtaining a point cloud of surface reconstruction data;

generating a mesh of the physical environment from the point cloud, the mesh including a plurality of vertices and faces;

selecting a vertex in the mesh having a height that is lowest relative to the HMD device; and

establishing a location of the floor of the physical environment at the height of the selected vertex.

18. The one or more computer readable memories of claim 17 further including performing an adjustment of the virtual ground to align with the location of the floor of the physical environment.

19. The one or more computer readable memories of claim 18 in which the adjustment is performed dynamically as a vertical adjustment.

20. The one or more computer-readable memories of claim 19 further including filtering or smoothing the dynamic vertical adjustment.