Space carving based on human physical data

• US 10,330,931 B2
• Filed: 06/28/2013
• Issued: 06/25/2019
• Est. Priority Date: 06/28/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Ambrus, Anthony J., Park, Jea Gon, Poulos, Adam G., Clark, Justin Avram, Gourlay, Michael Jason, Mount, Brian J., McCulloch, Daniel J., Tomlin, Arthur C.
• Primary Examiner(s): Ge, Jin
• Application Number:
  US13/931,507
• Publication Number:
  US 20150002507A1
• Time in Patent Office: 2,188 Days
• Field of Search: 345419
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering

Abstract

Technology is described for (3D) space carving of a user environment based on movement through the user environment of one or more users wearing a near-eye display (NED) system. One or more sensors on the near-eye display (NED) system provide sensor data from which a distance and direction of movement can be determined. Spatial dimensions for a navigable path can be represented based on user height data and user width data of the one or more users who have traversed the path. Space carving data identifying carved out space can be stored in a 3D space carving model of the user environment. The navigable paths can also be related to position data in another kind of 3D mapping like a 3D surface reconstruction mesh model of the user environment generated from depth images.

Claims

1. A method for three dimensional (3D) space carving of a user environment based on movement through the user environment by one or more users wearing respective near-eye display (NED) systems, comprising:

identifying, by one or more processors, one or more user navigable paths respectively traversed by the one or more of the users wearing the respective NED systems in the user environment based on sensor data from one or more respective position indicating sensors of the respective NED systems;

merging overlapping portions of the one or more user navigable paths traversed by the one or more users;

storing position and spatial dimensions for the one or more navigable paths as constituting carved out space within stored human space carving data configured for use by a 3D modeling of the user environment; and

generating and displaying a 3D space carved mapping of the user environment by;

detecting, by the one or more processors, one or more object boundaries by distinguishing the carved out space and un-carved space based on the human space carving data;

determining a position relationship between the one or more object boundaries and a path position stored in the human space carving data; and

based on the position relationship for the path position and the one or more object boundaries determined based on the human space carving data, the one or more processors generating the 3D space carved mapping relating the carved out space to the one or more object boundaries in the user environment.

2. The method of claim 1, further comprising:

retrieving a stored 3D mapping of the user environment and relating positions of the carved out space to the retrieved 3D mapping.

3. The method of claim 1, further comprising:

updating a 3D surface reconstruction model of the user environment.

4. The method of claim 3, further comprising:

relating position data of the carved out space to position data in the 3D surface reconstruction model of the user environment; and

responsive to differences in the position data of the carved out space and the position data in the 3D surface reconstruction model, updating the position data in the 3D surface reconstruction model to align with the position data of the carved out space.

5. The method of claim 1, wherein the identifying the one or more user navigable paths based on the sensor data comprises:

representing spatial dimensions of a carved out space portion of a navigable path of the one or more user navigable paths at a path position as a substantially vertical 3D tile having dimensions corresponding to at least one of user height data or user width data.

6. A method for three dimensional (3D) space carving of a user environment based on movement through the user environment by one or more users wearing respective near-eye display (NED) systems, comprising:

identifying, by one or more processors, one or more user navigable paths respectively traversed by the one or more of the users wearing the respective NED systems in the user environment based on sensor data from one or more respective position indicating sensors of the respective NED systems;

representing spatial dimensions of a carved out space portion of the one or more navigable paths at a path position as a substantially vertical three-dimensional tile having dimensions corresponding to at least one of user height data and user width data;

merging overlapping portions of the one or more user navigable paths traversed by the one or more users;

associating a set of time stamped sensor data with a starting path position;

setting the starting path position as the previous path position;

selecting a next in time set of time stamped sensor data as a current path position;

determining a position vector from the previous path position to the current path position based on the respective sets of time stamped sensor data associated with the previous path position and current path position;

interpolating a height offset curve between the previous path position and the current path position based at least on accelerometer data captured in a time period between time stamps for sets of time stamped sensor data for the previous path position and the current path position;

identifying a path segment volume between the current path position and the previous path position based on the position vector and corresponding points along a boundary of each tile representing the previous and current path positions;

storing the position vector and position data for the corresponding boundary points for the path segment volume in space carving data;

linking any path segments with overlapping volumes with a same path identifier in memory; and

storing position and spatial dimensions for the one or more navigable paths as constituting carved out space within stored human space carving data configured for use by a 3D modeling of the user environment.

7. A near-eye display (NED) system comprising:

a near-eye support structure;

a NED supported by the near-eye support structure and having a field of view approximating a user field of view; and

one or more processors communicatively coupled to one or more position indicating sensors supported by the near-eye support structure, the one or more processors configured to;

determine a distance and a direction of movement of a user wearing the NED system based on time stamped sensor data from the one or more position indicating sensors;

identify one or more three dimensional (3D), space carved navigable paths traversed by the user based on the determined distance and the determined direction of movement, wherein the identified one or more 3D, space carved navigable paths including at least one of width dimensions or height dimensions associated with the user;

store, in an accessible memory, space carving data representing the identified one or more 3D, space carved navigable paths as part of a 3D space carving based model of a user environment; and

generate and display a 3D space carved mapping of the user environment by;

detecting one or more object boundaries by distinguishing the carved out space and un-carved space based on the human space carving data;

determining a position relationship between the one or more object boundaries and a path position stored in the human space carving data; and

based on the position relationship for the path position and the one or more object boundaries determined based on the human space carving data, generating the 3D space carved mapping relating the carved out space to the one or more object boundaries in the user environment.

8. The NED system of claim 7, wherein:

the one or more processors are further configured to determine spatial dimensions of the one or more 3D, space carved navigable paths based on user physical features including a user height and a user body width.

9. The NED of claim 7, wherein the one or more position indicating sensors include an accelerometer.

10. The NED system of claim 7, wherein:

the one or more processors are communicatively coupled over a network for sending the 3D space carving based model to a 3D surface reconstruction unit of a network accessible computer system and for receiving a 3D surface reconstruction model of the user environment updated based on the 3D space carving based model from the 3D surface reconstruction unit.

11. The NED system of claim 10, wherein:

the one or more processors are further configured to receive the 3D surface reconstruction model of the user environment, wherein the 3D surface reconstruction model is further updated based on space carving data generated for the user environment due to traversal of the user environment by at least one other user.

12. The NED system of claim 8, wherein:

the one or more processors are further configured to represent a respective path position of the one or more 3D, space carved navigable paths as a respective, substantially vertical three-dimensional tile, the tile having spatial dimensions determined by the user height and the user body width.

13. The NED system of claim 12 wherein:

the one or more position indicating sensors include an accelerometer; and

the one or more processors are further configured to determine a height offset between two tiles representing two respective path positions and a position vector between path positions based on data from the accelerometer.

14. One or more processor readable storage devices storing processor readable instructions that cause one or more processors to execute a method for sensing a three dimensional (3D) space carving of a user environment due to movement through the user environment by one or more users wearing respective near-eye display (NED) systems each having at least one respective position indicating sensor, the instructions comprising code for:

identifying one or more 3D navigable paths traversed by the one or more users wearing the respective NED systems in the user environment, the identifying being based on user physical features including at least one of user height or user body width, and further based on sensor data from at least one respective position indicating sensor of each respectively used NED systems;

merging overlapping portions of the one or more 3D navigable paths traversed by the one or more users;

storing position and spatial dimensions for the one or more 3D navigable paths as carved out space as human space carving data in a memory of a 3D space carving modeling unit;

generating a 3D space carved mapping of the user environment by;

detecting one or more object boundaries by distinguishing the carved out space and un-carved space based on the human space carving data;

determining a position relationship between the one or more object boundaries and a path position stored in the human space carving data; and

based on the position relationship for the path position and the one or more object boundaries determined based on the human space carving data, generating a 3D space carved mapping relating the carved out space to the one or more object boundaries in the user environment; and

displaying a visual representation of the state of space carving thus far carved out in the user environment as feedback to a user of the one or more users.

15. The one or more processor readable storage devices of claim 14, further comprising code for:

representing spatial dimensions of a navigable path of the one or more 3D navigable paths at a path position as a substantially vertical tile having dimensions based on the at least one of the user height or the user body width of one or more of the users.

16. The one or more processor readable storage devices of claim 14, further comprising code for:

determining a position vector from a previous path position to a current path position based on time stamped accelerometer data associated with the previous path and the current path, and a user stride length.

17. The one or more processor readable storage devices of claim 14, further comprising code for:

displaying an un-carved space overlay so as to appear below a height criteria determined based on a height of a respective user of the one or more users for a designated portion of a field of view of the NED system even when space in the designated portion of the field of view is un-carved.

18. The one or more processor readable storage devices of claim 14, further comprising code for:

providing a game based incentive for a user wearing a respective one of the NED systems to approach at least one of physical object boundaries in the user environment.