GAZE-BASED OBJECT PLACEMENT WITHIN A VIRTUAL REALITY ENVIRONMENT

• US 20160026242A1
• Filed: 04/27/2015
• Published: 01/28/2016
• Est. Priority Date: 07/25/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Burns, Aaron, Sugden, Ben, Massey, Laura, Salter, Tom
• Granted Patent: US 10,416,760 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 9/31 Projection devices for colour picture display , e.g. using electronic spatial light modulators [ESLM]

Abstract

A head mounted display (HMD) device operating in a real world physical environment is configured with a sensor package that enables determination of an intersection of a device user'"'"'s projected gaze with a location in a virtual reality environment so that virtual objects can be placed into the environment with high precision. Surface reconstruction of the physical environment can be applied using data from the sensor package to determine the user'"'"'s view position in the virtual world. A gaze ray originating from the view position is projected outward and a cursor or similar indicator is rendered on the HMD display at the ray'"'"'s closest intersection with the virtual world such as a virtual object, floor/ground, etc. In response to user input, such as a gesture, voice interaction, or control manipulation, a virtual object is placed at the point of intersection between the projected gaze ray and the virtual reality environment.

Claims

1. A method performed by a head mounted display (HMD) device that supports rendering of a virtual reality environment, comprising:

obtaining sensor data describing a real world physical environment adjoining a user of the HMD device;

using the sensor data, reconstructing a geometry of the physical environment;

tracking the user'"'"'s head and gaze in the physical environment using the reconstructed geometry to determine a field of view and view position;

projecting a gaze ray outward from the view position;

identifying an intersection between the projected gaze ray and the virtual reality environment; and

placing a virtual object at the intersection within the current field in response to user input.

2. The method of claim 1 in which the sensor data includes depth data and further including generating the sensor data using a depth sensor and applying surface reconstruction techniques to reconstruct the physical environment geometry.

3. The method of claim 1 further including generating depth data using depth-from-stereo imaging analyses.

4. The method of claim 1 further identifying an intersection between the projected gaze ray and the virtual reality environment that is closest to the HMD device.

5. The method of claim 1 further including exposing a user interface (UI) for receiving the user input, the UI providing user controls or supporting gesture recognition or voice recognition.

6. The method of claim 1 further including rotating the projected gaze ray so that a placed virtual object is not clipped along an edge of the field of view.

7. The method of claim 6 further including performing the rotating so that the gaze ray is projected through a bottom portion of the field of view.

8. The method of claim 6 further including performing the rotating when the placed object cannot be rendered to fit entirely within the field of view.

9. The method of claim 1 in which the placed virtual object is an interactive element supporting a user interface or a user experience in which the interactive element is one of menu, widget, or notification.

10. The method of claim 1 further including using one or more inward facing sensors located in the HMD device to determine the gaze direction and projecting the gaze ray from the view position along a direction of the user'"'"'s gaze.

11. A head mounted display (HMD) device operable by a user in a physical environment, comprising:

one or more processors;

a display for rendering a virtual reality environment to the user, a field of view of the rendered virtual reality environment being variable depending at least in part on a pose of the user'"'"'s head in the physical environment;

a sensor package; and

one or more memory devices storing computer-readable instructions which, when executed by the one or more processors, perform a method comprising the steps of;

generating surface reconstruction data for at least a portion of the physical environment using the sensor package,

dynamically tracking a view position of the user for the virtual reality environment using the surface reconstruction data,

locating an intersection between a ray projected from the view position along the user'"'"'s gaze direction and a point of the virtual reality environment within a current field of view, and

operating the HMD device to render a cursor at the point of intersection.

12. The HMD device of claim 11 further including using the sensor package to dynamically track the gaze direction.

13. The HMD device of claim 11 further including a user interface and operating the HMD device to place a virtual object at the point of intersection responsively to a user input to the UI.

14. The HMD device of claim 11 further including dynamically repositioning the cursor as the field of view or gaze direction changes.

15. The HMD device of claim 11 further including modeling the physical environment using a surface reconstruction data pipeline that implements a volumetric method creating multiple overlapping surfaces that are integrated.

16. The HMD device of claim 11 further including rotating the projected ray downward so that a placed virtual object, when rendered on the display, is not clipped by an upper edge of the current field of view.

17. One or more computer readable memories storing computer-executable instructions for rendering a virtual reality environment within a variable field of view of a head mounted display (HMD) device located in a real world environment, the method comprising the steps of:

using data from a sensor package incorporated into the HMD device to a) dynamically generate a surface reconstruction model of the real world environment and b) generate a gaze ray that is projected from a view position of a user of the HMD device;

determining a field of view of the virtual reality environment using the model;

receiving an input to the HMD device from the user; and

placing a virtual object within the field of view at a point of intersection between the gaze ray and the virtual reality environment in response to the received user input.

18. The one or more computer readable memories of claim 17 further including tracking a gaze direction of the user, projecting the gaze ray from the view position along the gaze direction, and rendering a cursor within the field of view at a point of intersection of the gaze ray and the virtual reality environment.

19. The one or more computer-readable memories of claim 17 further including determining when a placed virtual object extends outside the field of view and rotating the gaze ray in response to the determining.

20. The one or more computer-readable memories of claim 17 further including rotating the gaze ray at an angle that is sufficient to provide a portion of the field of view that is unobstructed by the placed virtual object.