SPACE CARVING BASED ON HUMAN PHYSICAL DATA

• US 20150002507A1
• Filed: 06/28/2013
• Published: 01/01/2015
• Est. Priority Date: 06/28/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Justin Avram Clark
• Inventors: Ambrus, Anthony J., Poulos, Adam G., Mount, Brian J., McCulloch, Daniel J., Tomlin, Arthur C., Park, Jea Gon, Clark, Justin Avram, Gourlay, Michael Jason
• Granted Patent: US 10,330,931 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering

Abstract

Technology is described for (3D) space carving of a user environment based on movement through the user environment of one or more users wearing a near-eye display (NED) system. One or more sensors on the near-eye display (NED) system provide sensor data from which a distance and direction of movement can be determined. Spatial dimensions for a navigable path can be represented based on user height data and user width data of the one or more users who have traversed the path. Space carving data identifying carved out space can be stored in a 3D space carving model of the user environment. The navigable paths can also be related to position data in another kind of 3D mapping like a 3D surface reconstruction mesh model of the user environment generated from depth images.

Claims

1. A method for three dimensional (3D) space carving of a user environment based on movement through the user environment of one or more users wearing a near-eye display (NED) system comprising:

identifying by one or more processors one or more navigable paths traversed by one or more users wearing the NED system in a user environment based on sensor data from one or more sensors on the near-eye display (NED) system;

merging overlapping portions of the one or more navigable paths traversed by the one or more users; and

storing position and spatial dimensions for the one or more navigable paths as carved out space in human space carving data in a 3D space carving model of the user environment.

2. The method of claim 1 further comprising retrieving a stored 3D mapping of the user environment and relating positions of the carved out space to the retrieved 3D mapping.

3. The method of claim 1 further comprising generating a 3D space carved mapping of the user environment.

4. The method of claim 3 wherein generating a 3D space carved mapping of the user environment further comprises:

detecting one or more object boundaries by the one or more processors by distinguishing carved out space and uncarved space based on the human space carving data;

determining a position relationship between the one more object boundaries and a path position stored in the human space carving data; and

based on the respective position relationships for each path position and respective object boundaries determined based on the human space carving data, the one or more processors generating a 3D space carved mapping relating the carved out space to the one or more object boundaries in the user environment.

5. The method of claim 4 further comprising updating a 3D surface reconstruction model of the user environment.

6. The method of claim 5 further comprising

relating position data of the carved out space to position data in the 3D surface reconstruction model of the user environment;

and

responsive to differences in position data between the carved out space and the related position data in the 3D surface reconstruction model, updating the related position data in the 3D surface reconstruction model to align with the position data of the carved out space.

7. The method of claim 1 wherein identifying by one or more processors one or more navigable paths traversed by one or more users wearing the NED system in a user environment based on sensor data from one or more sensors on the near-eye display (NED) system further comprises:

representing spatial dimensions of a navigable path at a path position as a tile having dimensions based on user height data and user width data.

8. The method of claim 7 further comprising:

associating a set of time stamped sensor data with a starting path position;

setting the starting path position as the previous path position;

selecting a next in time set of time stamped sensor data as a current path position;

determining a position vector from the previous path position to the current path position based on the respective sets of time stamped sensor data associated with the previous path position and current path position;

interpolating a height offset curve between the previous path position and the current path position based at least on accelerometer data captured in a time period between time stamps for sets of time stamped sensor data for the previous path position and the current path position;

identifying a path segment volume between the current path position and the previous path position based on the position vector and corresponding points along a boundary of each tile representing the previous and current path positions;

storing the position vector and position data for the corresponding boundary points for the path segment volume in space carving data; and

linking any path segments with overlapping volumes with a same path identifier in memory.

9. A near-eye display (NED) system comprising:

a near-eye support structure;

a near-eye display (NED) supported by the near-eye support structure and having a field of view approximating a user field of view;

one or more processors are communicatively coupled to one or more sensors supported by the near-eye support structure;

the one or more processors determining a distance and a direction of movement of a user wearing the NED system based on time stamped sensor data from the one or more sensors and identifying one or more navigable paths traversed by the user based on the determined distance and direction; and

the one or more processors storing in an accessible memory space carving data representing the identified one or more navigable paths in a 3D space carving model of the user environment.

10. The near-eye display (NED) system of claim 9 further comprising determining spatial dimensions of the one or more navigable paths based on user physical features including a user height and a user body width for the one or more users.

11. The near-eye display (NED) system of claim 9 wherein the one or more sensors include an accelerometer.

12. The NED system of claim 9 further comprising:

the one or more processors are communicatively coupled over a network for sending the 3D space carving model to a 3D surface reconstruction module of a network accessible computer system and for receiving a 3D surface reconstruction model of the user environment updated based on the 3D space carving model from the 3D surface reconstruction module.

13. The NED system of claim 12 further comprising the one or more processors receiving the 3D surface reconstruction model of the user environment updated based also on space carving data generated for the user environment by at least one other user.

14. The NED system of claim 10 further comprising:

the one or more processors representing a respective path position of the one or more navigable paths as a respective tile of at least two dimensions, the tile having spatial dimensions determined by the user height and the user width.

15. The NED system of claim 14 further comprising:

the one or more sensors include an accelerometer; and

the one or more processors determining a height offset between two tiles representing two respective path positions and a position vector between path positions based on data from the accelerometer.

17. The one or more processor readable storage devices of claim 15 wherein identifying by one or more processors one or more navigable paths traversed by one or more users wearing the NED system in a user environment based on user physical features including user height and user body width, and sensor data from one or more sensors on the near-eye display (NED) system further comprises:

representing spatial dimensions of a navigable path at a path position as a tile having dimensions based on the user height and the user width.

19. The one or more processor readable storage devices of claim 15

wherein displaying a visual representation of the state of space carving in the user environment by the NED as feedback to a user further comprises displaying an uncarved space overlay so as to appear below a height criteria determined based on a height of the viewer for a designated portion of a field of view of the NED display even when space in the designated portion of the field of view is uncarved.

20. The one or more processor readable storage devices of claim 15 wherein the method further comprises:

providing a game based incentive for a user wearing the near-eye display (NED) system to approach object boundaries in a user environment.

16. One or more processor readable storage devices comprising instructions which cause one or more processors to execute a method for three dimensional (3D) space carving of a user environment based on movement through the user environment of one or more users wearing a near-eye display (NED) system, the method comprising:

identifying by one or more processors one or more navigable paths traversed by one or more users wearing the NED system in a user environment based on user physical features including user height and user body width, and sensor data from one or more sensors on the near-eye display (NED) system;

merging overlapping portions of the one or more navigable paths traversed by the one or more users;

storing position and spatial dimensions for the one or more navigable paths as carved out space in human space carving data in a 3D space carving model of the user environment; and

displaying a visual representation of the state of space carving in the user environment by the NED as feedback to a user.

18. The one or more processor readable storage devices of claim 16 wherein the method further comprises

determine a position vector from a previous path position to a current path position based on time stamped accelerometer data associated with the previous path and the current path, and a user stride length.