Smart placement of virtual objects to stay in the field of view of a head mounted display

• US 9,904,055 B2
• Filed: 02/03/2015
• Issued: 02/27/2018
• Est. Priority Date: 07/25/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Burns, Aaron, Sugden, Ben, Salter, Tom
• Primary Examiner(s): Chang, Kent
• Assistant Examiner(s): Morales Fernande, Benjamin
• Application Number:
  US14/612,850
• Publication Number:
  US 20160025981A1
• Time in Patent Office: 1,120 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0123 comprising devices increasing the field of view
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0101 characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G09G 2320/0261 in the context of movement of objects on the screen or movement of the observer relative to the screen

Abstract

An HMD device is configured to check the placement of newly introduced objects in a virtual reality environment such as interactive elements like menus, widgets, and notifications to confirm that the objects are significantly present within the user'"'"'s field of view. If the intended original placement would locate the object outside the field of view, the HMD device relocates the object so that a portion of the object is viewable at the edge of the HMD display closest to its original placement. Such smart placement of virtual objects enables the user to readily discover new objects when they are introduced into the virtual reality environment, and then interact with the objects within a range of motions and/or head positions that is comfortable to support a more optimal interaction and user experience.

Claims

1. A method performed by a head mounted display (HMD) device that supports rendering of a virtual reality environment within a field of view, comprising:

obtaining sensor data describing a physical space adjoining a user of the HMD device;

using the sensor data, reconstructing a geometry of the physical space;

tracking the user'"'"'s head in the physical space using the reconstructed geometry to determine a current field of view of the user;

when a new virtual object is being introduced into the virtual reality environment, checking its original intended location; and

relocating the new virtual object when and while the original intended location is outside the user'"'"'s current field of view so that at least a portion of the new virtual object is within the user'"'"'s current field of view when relocated.

2. The method of claim 1 in which the sensor data includes depth data and further including generating the sensor data using a depth sensor and applying surface reconstruction techniques to reconstruct the physical space geometry.

3. The method of claim 1 further including generating depth data using depth-from-stereo imaging analyses.

4. The method of claim 1 further including relocating the new virtual object along an edge of the user'"'"'s current field of view in a position that is closest to the original intended location in the virtual reality environment.

5. The method of claim 1 further including rendering the new virtual object substantially in its entirety when the original intended location is within the user'"'"'s current field of view.

6. The method of claim 1 further including rendering the new virtual object substantially in its entirety when an orientation of the head is within a range of comfortable orientations for the user.

7. The method of claim 6 in which the range of comfortable orientations includes a neutral head orientation.

8. The method of claim 1 further including detecting a gaze direction of the user when determining the current field of view.

9. The method of claim 1 in which the new virtual object is an interactive element supporting a user interface or a user experience.

10. The method of claim 9 in which the interactive element is one of menu, widget, or notification.

11. A head mounted display (HMD) device operable by a user in a physical environment, comprising:

one or more processors;

a display for rendering a virtual reality environment to the user, a field of view of the rendered virtual reality environment being variable depending at least in part on a pose of the user'"'"'s head in the physical environment;

a sensor package; and

one or more memory devices storing computer-readable instructions which, when executed by the one or more processors, cause the processors to;

generate surface reconstruction data for at least a portion of the physical environment using the sensor package,

dynamically track the user'"'"'s field of view of the virtual reality environment using the surface reconstruction data,

determine if an original intended location for a new virtual object being introduced into the virtual reality environment enables the new virtual object to be viewable within the user'"'"'s current field of view, and

operate the HMD device to relocate the new virtual object from the original intended location in response to the determination so that at least a portion of the new virtual object is viewable within the user'"'"'s current field of view, the determination being that the new virtual object is not within the user'"'"'s current field of view.

12. The HMD device of claim 11, the computer-readable instructions further including instructions that cause the processors to change a location of the new virtual object within the current field of view in response to movement from the user, the movement including head movement or eye movement.

13. The HMD device of claim 12, the computer-readable instructions further including instructions that cause the processors to change the location so as to guide the user to the original intended location of the new virtual object and render the new virtual object in full view when the current field of view includes the original intended location.

14. The HMD device of claim 12, the computer-readable instructions further including instructions that cause the processors to change the location so as to guide the user to assume a head pose that falls within a predetermined range and render the new virtual object in full view when the head pose is assumed.

15. The HMD device of claim 11, the computer-readable instructions further including instructions that cause the processors to model the physical environment using a surface reconstruction data pipeline that implements a volumetric method creating multiple overlapping surfaces that are integrated.

16. The HMD device of claim 11 further including a sealed protective visor that substantially encases the display.

17. One or more computer readable memories storing computer-executable instructions that cause one or more processors of a head mounted display (HMD) device located in a real world space and rendering a virtual reality environment within a variable field of view to:

use data from a sensor package incorporated into the HMD device to dynamically generate a surface reconstruction model of the real world space;

determine a user'"'"'s current field of view of the virtual reality environment using the model;

determine that an intended original position of a virtual object lies outside the current field of view;

when and while the virtual object lies outside the current field of view, enable discovery by the user of the virtual object introduced into the virtual reality environment by locating at least a portion of the virtual object at an edge of the current field of view; and

adjust the location of the virtual object as the current field of view changes with movement by the user.

18. The one or more computer readable memories of claim 17 wherein the instructions to adjust the location of the virtual object further comprise instructions that cause the processors to move the location of the virtual object along the edge of the current field of view.

19. The one or more computer-readable memories of claim 17 further including instructions that cause the processors to locate the virtual object to be viewable substantially in full within the current field of view when a pose of the user'"'"'s head is within a threshold range.

20. The one or more computer-readable memories of claim 19 further including instructions that cause the processors to enable the user to interact with the virtual object when the pose of the user'"'"'s head is within the threshold range.