Three-dimensional mixed-reality viewport

• US 9,858,720 B2
• Filed: 04/16/2015
• Issued: 01/02/2018
• Est. Priority Date: 07/25/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: da Veiga, Alexandre, Sugden, Ben, Christen, Jon
• Primary Examiner(s): Du, Haixia
• Application Number:
  US14/688,817
• Publication Number:
  US 20160027216A1
• Time in Patent Office: 992 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0174 holographic
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G03H 1/2294 Addressing the hologram to an active spatial light modulator
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/0486 Drag-and-drop
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09B 23/04 for geometry, trigonometry, projection or perspective for surveying G09B25/06
• CPC Class Codes: H04N 13/204 : using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/344 : with head-mounted left-right displays
• CPC Class Codes: H04N 13/383 : for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes

Abstract

An application running on a computing platform that employs three-dimensional (3D) modeling is extended using a virtual viewport into which 3D holograms are rendered by a mixed-reality head mounted display (HMD) device. The HMD device user can position the viewport to be rendered next to a real world 2D monitor and use it as a natural extension of the 3D modeling application. For example, the user can interact with modeled objects in mixed-reality and move objects between the monitor and the viewport. The 3D modeling application and HMD device are configured to exchange scene data for modeled objects (such as geometry, lighting, rotation, scale) and user interface parameters (such as mouse and keyboard inputs). The HMD device implements head tracking to determine where the user is looking so that user inputs are appropriately directed to the monitor or viewport.

Claims

1. A head mounted display (HMD) device operable by a user in a physical environment, comprising:

one or more processors;

a sensor package;

a display configured for rendering a mixed reality environment to the user, a view position of the user for the rendered mixed-reality environment being variable depending at least in part on a pose of the user'"'"'s head in the physical environment; and

one or more memory devices storing computer-readable instructions which, when executed by the one or more processors, cause the HMD device to;

implement a three-dimensional (3D) virtual viewport on the display,

provide extensibility of the viewport to a 3D modeling application executing on a computing platform in the physical environment, the application supporting a 3D model that is rendered on a two-dimensional (2D) desktop on a monitor coupled to the computing platform,

render the 3D model as a hologram in the viewport, and

based on the head pose, enable user interaction with either the 3D model rendered on the desktop or the 3D model rendered in the viewport,

wherein the 3D model on the 2D desktop on the monitor is independently rendered from the hologram in the viewport on the HMD device, such that the 3D model rendered on the 2D desktop and the 3D model rendered as the hologram are generated by separate components.

2. The HMD device of claim 1 further including a network interface and receiving extensibility data from the computing platform over the network interface, the extensibility data describing the 3D model and user inputs at the computing platform.

3. The HMD device of claim 2 further including dynamically updating the rendered hologram in the viewport responsively to the user inputs.

4. The HMD device of claim 1 further including

obtaining sensor data from the sensor package, the sensor data associated with the physical environment;

using the sensor data, reconstructing a geometry of the physical environment including any real world object located therein; and

using the reconstructed geometry to determine a location of the monitor that is coupled to the computing platform within the physical environment.

5. The HMD device of claim 4 in which the sensor data includes depth data and further including generating the sensor data using a depth sensor and applying surface reconstruction techniques to reconstruct the physical environment geometry.

6. The HMD device of claim 4 further including tracking the user'"'"'s head in the physical environment using the reconstructed geometry to determine the view position, rendering a mouse cursor in the viewport when user input causes the mouse cursor to move off a desktop supported by the monitor, and consuming keyboard inputs when a ray projected from the view position intersects with the viewport.

7. The HMD device of claim 6 further including discontinuing the rendering of the mouse cursor in the viewport when the projected ray indicates that the mouse cursor has transitioned to the desktop supported by the monitor.

8. The HMD device of claim 6 further including providing a control to the user to control viewport characteristics including at least one of viewport location in the mixed-reality environment, viewport size, or viewport shape.

9. The HMD device of claim 1 further including clipping the 3D model to constrain the 3D model to an extent of the viewport.

10. The HMD device of claim 1 in which the separate components include respective graphic processing units associated with the computing platform and the HMD device.

11. A method performed by a head mounted display (HMD) device supporting a mixed-reality environment including virtual objects and real objects, the method comprising:

implementing a virtual three-dimensional (3D) viewport on a display of the HMD device;

receiving extensibility data from a computing platform over a network connection, the extensibility data including scene data describing a 3D model supported by an application executing on the computing platform coupled to a monitor wherein the monitor provides a two-dimensional (2D) desktop on which the 3D model is rendered, and the extensibility data further including user interface (UI) data describing user inputs to the computing platform; and

dynamically rendering the 3D model in the viewport as a hologram using the received extensibility data,

wherein the 3D model on the 2D desktop on the monitor is independently rendered from the hologram in the viewport on the HMD device, such that the 3D model rendered on the 2D desktop and the 3D model rendered as the hologram are generated by separate components.

12. The method of claim 11 further including rendering a mouse cursor in the viewport based on mouse messages included in the UI data.

13. The method of claim 11 further including controlling rendering of the 3D model in the viewport using keyboard messages included in the UI data.

14. The method of claim 11 in which the scene data includes at least one of 3D model data, environmental data, or camera parameters.

15. The method of claim 12 further including utilizing sensor data to determine a view position of a user of the HMD device and transitioning the cursor back to a desktop supported by a monitor coupled to the computing platform when a ray projected from the view position intersects the monitor.

16. The method of claim 15 further including modeling a physical environment in which the HMD device is located using a surface reconstruction data pipeline that implements a volumetric method creating multiple overlapping surfaces that are integrated and using the modeled physical environment at least in part to determine the view position or to determine a location of the monitor that is coupled to the computing platform within the physical environment.

17. A computing device, comprising:

one or more processors;

an interface to a monitor, the monitor displaying a two-dimensional (2D) desktop configured to render a three-dimensional (3D) model provided by an application executing on the computing device;

a mouse interface for connecting to a mouse and receiving signals from the mouse indicating mouse movement and inputs to mouse controls from a user of the computing device;

a keyboard interface for connecting to a keyboard and receiving signals from the keyboard indicating keyboard inputs from the user;

a network interface for communicating with a head mounted display (HMD) device over a network connection, the HMD device including a display and configured to implement a 3D virtual viewport on the display in which the 3D model is rendered as a hologram; and

one or more memory devices storing computer-readable instructions which, when executed by the one or more processors implement the application providing the 3D model and a user interface (UI) server configured to

track mouse messages that describe the mouse movements and inputs,

track keyboard messages that describe the keyboard inputs,

when a mouse movement indicates that a cursor associated with the mouse is moving beyond an edge of the monitor, take control of the mouse messages and prevent propagation of the mouse messages to systems operating on the computing device,

send the mouse messages to the HMD device over the network connection, and

send the keyboard messages to the HMD device over the network connection,

wherein the mouse and keyboard messages sent over the network connection are utilized by the HMD device as user inputs to the 3D model when rendered in the viewport,

wherein the 3D model on the 2D desktop on the monitor is independently rendered from the hologram in the viewport on the HMD device, such that the 3D model rendered on the 2D desktop and the 3D model rendered as the hologram are generated by separate components.

18. The computing device of claim 17 in which the HMD device is configured for i) identifying a location of the monitor within a physical environment using one of surface reconstruction or observing a pre-determined, trackable image that is displayed on the monitor, and ii) supporting a mixed-reality environment on an optical display, the mixed-reality environment including a virtual 3D viewport into which hologram-based models from the 3D modeling application may be rendered.

19. The computing device of claim 18 further including tracking the mouse messages and keyboard messages by interacting with an operating system executing on the computing device.

20. The computing device of claim 18 further including receiving a message from the HMD device that the mouse cursor has transitioned to the desktop and calculating an initial cursor position on the desktop using a last reported position of the mouse cursor in the viewport.