Gaze-based object placement within a virtual reality environment

• US 10,416,760 B2
• Filed: 04/27/2015
• Issued: 09/17/2019
• Est. Priority Date: 07/25/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Burns, Aaron, Sugden, Ben, Massey, Laura, Salter, Tom
• Primary Examiner(s): Wu, Chong
• Application Number:
  US14/697,109
• Publication Number:
  US 20160026242A1
• Time in Patent Office: 1,604 Days
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 9/31 Projection devices for colour picture display , e.g. using electronic spatial light modulators [ESLM]

Abstract

A head mounted display (HMD) device operating in a real world physical environment is configured with a sensor package that enables determination of an intersection of a device user'"'"'s projected gaze with a location in a virtual reality environment so that virtual objects can be placed into the environment with high precision. Surface reconstruction of the physical environment can be applied using data from the sensor package to determine the user'"'"'s view position in the virtual world. A gaze ray originating from the view position is projected outward and a cursor or similar indicator is rendered on the HMD display at the ray'"'"'s closest intersection with the virtual world such as a virtual object, floor/ground, etc. In response to user input, such as a gesture, voice interaction, or control manipulation, a virtual object is placed at the point of intersection between the projected gaze ray and the virtual reality environment.

Claims

1. A method performed by a head mounted display (HMD) device that supports rendering of a virtual reality environment that excludes real world objects and includes existing virtual objects, comprising:

obtaining sensor data describing a real world physical environment adjoining a user of the HMD device;

using the sensor data, reconstructing a geometry of the real world physical environment;

tracking the user'"'"'s head and gaze in the real world physical environment using the reconstructed geometry to determine a field of view and view position within the virtual reality environment;

projecting a gaze ray outward from the view position;

identifying a point at which the projected gaze ray intersects with the virtual reality environment;

operating the HMD device to place a virtual object at the identified point;

in response to the placed virtual object being clipped along an edge of the field of view, rotating the projected gaze ray independent of movement and positioning of the user'"'"'s head, eyes, or body, such that the rotating of the projected gaze ray is performed by the HMD device irrespective of the user'"'"'s actions;

locating a point at which the rotated gaze ray intersects with the virtual reality environment; and

repositioning the virtual object at the located point for the rotated gaze ray so that the virtual object is no longer clipped along the edge of the field of view.

2. The method of claim 1 in which the sensor data includes depth data and further including generating the sensor data using a depth sensor and applying surface reconstruction techniques to reconstruct the physical environment geometry.

3. The method of claim 1 further including generating depth data using depth-from-stereo imaging analyses.

4. The method of claim 1 further including identifying a point at which the projected gaze ray intersects with the virtual reality environment that is closest to the HIVID device.

5. The method of claim 1 in which the virtual object is placed responsive to receiving user input, and further including exposing a user interface (UI) for receiving the user input, the UI providing user controls or supporting gesture recognition or voice recognition.

6. The method of claim 1 further including performing the rotating so that the gaze ray is projected through a bottom portion of the field of view, such that the bottom portion is at least on a lower half of the field of view.

7. The method of claim 1 further including performing the rotating when the placed virtual object cannot be rendered to fit entirely within the field of view.

8. The method of claim 1 in which the placed virtual object is an interactive element supporting a user interface or a user experience in which the interactive element is one of menu, widget, or notification.

9. The method of claim 1 further including using one or more inward facing sensors located in the HMD device to determine the gaze direction and projecting the gaze ray from the view position along a direction of the user'"'"'s gaze.

10. The method of claim 1, in which the reconstructed geometry from the sensor data is at least partially used to determine the point at which the gaze ray intersects with the virtual reality environment.

11. A head mounted display (HMD) device operable by a user in a physical environment, comprising:

one or more processors;

a display for rendering a virtual reality environment to the user, a field of view of the rendered virtual reality environment being variable depending at least in part on a pose of the user'"'"'s head in the physical environment;

a sensor package; and

one or more memory devices storing computer-readable instructions which, when executed by the one or more processors, perform a method comprising the steps of;

generating surface reconstruction data for at least a portion of the physical environment using the sensor package,

dynamically tracking a view position of the user for the virtual reality environment using the surface reconstruction data,

locating a point at which a gaze ray projected from the view position along the user'"'"'s gaze direction intersects with the virtual reality environment within a current field of view,

operating the HMD device to place a virtual object at the located point;

in response to the placed virtual object being clipped along an edge of the field of view, rotating the projected gaze ray independent of movement and positioning of the user'"'"'s head, eyes, or body, such that the rotating of the projected gaze ray is performed by the HMD device irrespective of the user'"'"'s actions;

locating a point at which the rotated gaze ray intersects with the virtual reality environment; and

repositioning the virtual object at the located point for the rotated gaze ray so that the virtual object is no longer clipped along the edge of the field of view.

12. The HMD device of claim 11 in which the method further comprises a step of using the sensor package to dynamically track the direction.

13. The HMD device of claim 11 in which the method further comprises a step of modeling the physical environment using a surface reconstruction data pipeline that implements a volumetric method creating multiple overlapping surfaces that are integrated.

14. The HMD device of claim 11 in which the projected gaze ray is rotated downward so that the placed virtual object, when rendered on the display, is not clipped by an upper edge of the field of view.

15. One or more hardware-based computer readable memories not consisting of propagated signals storing computer-executable instructions for rendering a virtual reality environment within a variable field of view of a head mounted display (HMD) device located in a real world environment, the instructions when executed by one or more processors disposed in the HMD device cause the HMD device to:

use data from a sensor package incorporated into the HMD device to a) dynamically generate a surface reconstruction model of the real world environment and b) generate a gaze ray that is projected from a view position of a user of the HMD device within the virtual reality environment using the surface reconstruction model of the real world environment;

determine a field of view of the virtual reality environment using the surface reconstruction model;

receive an input to the HMD device from the user;

place a virtual object within the field of view at a point at which the gaze ray intersects with the virtual reality environment in response to the received user input;

in response to the virtual object being clipped along an edge of the field of view within the virtual reality environment, rotate the virtual object independent of movement and positioning of the user'"'"'s head, eyes, or body, such that the rotation of the projected gaze ray is performed by the HMD device irrespective of the user'"'"'s actions;

locate a point at which the rotated gaze ray intersects with the virtual reality environment; and

reposition the virtual object at the located point for the rotated gaze ray so that the virtual object is no longer clipped along the edge of the field of view.

16. The one or more hardware-based computer readable memories of claim 15 in which the executed instructions further cause the HMD device to track a gaze direction of the user, project the gaze ray from the view position along the gaze direction, and render a cursor within the field of view at the point which the gaze ray intersects with the virtual reality environment.

17. The one or more hardware-based computer-readable memories of claim 15 in which the executed instructions further cause the HMD device to rotate the gaze ray at an angle that is sufficient to provide a portion of the field of view that is unobstructed by the placed virtual object.