SMART PLACEMENT OF VIRTUAL OBJECTS TO STAY IN THE FIELD OF VIEW OF A HEAD MOUNTED DISPLAY

• US 20160025981A1
• Filed: 02/03/2015
• Published: 01/28/2016
• Est. Priority Date: 07/25/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Burns, Aaron, Sugden, Ben, Salter, Tom
• Granted Patent: US 9,904,055 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0123 comprising devices increasing the field of view
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0101 characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G09G 2320/0261 in the context of movement of objects on the screen or movement of the observer relative to the screen

Abstract

An HMD device is configured to check the placement of newly introduced objects in a virtual reality environment such as interactive elements like menus, widgets, and notifications to confirm that the objects are significantly present within the user'"'"'s field of view. If the intended original placement would locate the object outside the field of view, the HMD device relocates the object so that a portion of the object is viewable at the edge of the HMD display closest to its original placement. Such smart placement of virtual objects enables the user to readily discover new objects when they are introduced into the virtual reality environment, and then interact with the objects within a range of motions and/or head positions that is comfortable to support a more optimal interaction and user experience.

Claims

1. A method performed by a head mounted display (HMD) device that supports rendering of a virtual reality environment within a field of view, comprising:

obtaining sensor data describing a physical space adjoining a user of the HMD device;

using the sensor data, reconstructing a geometry of the physical space;

tracking the user'"'"'s head in the physical space using the reconstructed geometry to determine a current field of view;

when a new virtual object is introduced into the virtual reality environment, checking its original location; and

relocating the new virtual object if the original location is outside the current field of view so that at least a portion of the new virtual object is within the current field of view when relocated.

2. The method of claim 1 in which the sensor data includes depth data and further including generating the sensor data using a depth sensor and applying surface reconstruction techniques to reconstruct the physical space geometry.

3. The method of claim 1 further including generating depth data using depth-from-stereo imaging analyses.

4. The method of claim 1 further including relocating the new virtual object along an edge of the field of view in a position that is closest to the original location in the virtual reality environment.

5. The method of claim 1 further including rendering the new virtual object substantially in its entirety when the original location is within the field of view.

6. The method of claim 1 further including rendering the new virtual object substantially in its entirety when an orientation of the head is within a range of comfortable orientations for the user.

7. The method of claim 6 in which the range of comfortable orientations includes a neutral head orientation.

8. The method of claim 1 further including detecting a gaze direction of the user when determining the current field of view.

9. The method of claim 1 in which the new virtual object is an interactive element supporting a user interface or a user experience.

10. The method of claim 9 in which the interactive element is one of menu, widget, or notification.

11. A head mounted display (HMD) device operable by a user in a physical environment, comprising:

one or more processors;

a display for rendering a virtual reality environment to the user, a field of view of the rendered virtual reality environment being variable depending at least in part on a pose of the user'"'"'s head in the physical environment;

a sensor package; and

one or more memory devices storing computer-readable instructions which, when executed by the one or more processors, perform a method comprising the steps of;

generating surface reconstruction data for at least a portion of the physical environment using the sensor package,

dynamically tracking the user'"'"'s field of view of the virtual reality environment using the surface reconstruction data,

determining if an original location for a new virtual object introduced into the virtual reality environment enables the new virtual object to be viewable within the user'"'"'s current field of view, and

operating the HMD device to relocate the new virtual object from the original location in response to the determination so that at least a portion of the new virtual object is viewable within the user'"'"'s current field of view.

12. The HMD device of claim 11 further including changing a location of the new virtual object within the current field of view in response to movement from the user, the movement including head movement or eye movement.

13. The HMD device of claim 12 in which the location is changed so as to guide the user to the original location of the new virtual object and further including rendering the new virtual object in full view when the current field of view includes the original location.

14. The HMD device of claim 12 in which the location is changed so as to guide the user to assume a head pose that falls within a predetermined range and further including rendering the new virtual object in full view when the head pose is assumed.

15. The HMD device of claim 11 further including modeling the physical environment using a surface reconstruction data pipeline that implements a volumetric method creating multiple overlapping surfaces that are integrated.

16. The HMD device of claim 11 further including a sealed protective visor that substantially encases the display.

17. One or more computer readable memories storing computer-executable instructions for rendering a virtual reality environment within a variable field of view of a head mounted display (HMD) device located in a real world space, the method comprising the steps of:

using data from a sensor package incorporated into the HMD device to dynamically generate a surface reconstruction model of the real world space;

determining a current field of view of the virtual reality environment using the model;

enabling discovery by the user of a virtual object introduced into the virtual reality environment by locating at least a portion of the virtual object at an edge of the current field of view; and

adjusting the location of the virtual object as the current field of view changes with movement by the user.

18. The one or more computer readable memories of claim 17 in which the adjusting comprises moving the location of the virtual object along the edge of the current field of view.

19. The one or more computer-readable memories of claim 17 further including locating the virtual object to be viewable substantially in full within the current field of view when a pose of the user'"'"'s head is within a threshold range.

20. The one or more computer-readable memories of claim 17 further including locating the virtual object to be viewable substantially in full within the current field of view when an intended original location for the virtual object is within the current field of view.