Three-dimensional mixed-reality viewport

• US 10,096,168 B2
• Filed: 12/13/2017
• Issued: 10/09/2018
• Est. Priority Date: 07/25/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Da Veiga, Alexandre, Sugden, Ben, Christen, Jon
• Primary Examiner(s): Du, Haixia
• Application Number:
  US15/841,047
• Publication Number:
  US 20180101994A1
• Time in Patent Office: 300 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0174 holographic
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G03H 1/2294 Addressing the hologram to an active spatial light modulator
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/0486 Drag-and-drop
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09B 23/04 for geometry, trigonometry, projection or perspective for surveying G09B25/06
• CPC Class Codes: H04N 13/204 : using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/344 : with head-mounted left-right displays
• CPC Class Codes: H04N 13/383 : for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes

Abstract

An application running on a computing platform that employs three-dimensional (3D) modeling is extended using a virtual viewport into which 3D holograms are rendered by a mixed-reality head mounted display (HMD) device. The HMD device user can position the viewport to be rendered next to a real world 2D monitor and use it as a natural extension of the 3D modeling application. For example, the user can interact with modeled objects in mixed-reality and move objects between the monitor and the viewport. The 3D modeling application and HMD device are configured to exchange scene data for modeled objects (such as geometry, lighting, rotation, scale) and user interface parameters (such as mouse and keyboard inputs). The HMD device implements head tracking to determine where the user is looking so that user inputs are appropriately directed to the monitor or viewport.

Claims

1. A head mounted display (HMID) device operable by a user in a physical environment, comprising:

one or more processors;

a sensor package configured to dynamically provide sensor data used to determine a pose of the user'"'"'s head in the physical environment;

a network connection configured to support an exchange of extensibility data between the HID device and a remote computing device, in which the remote computing device executes an application that renders a three-dimensional (3D) model on a two-dimensional (2D) desktop;

a see-through display configured for rendering a mixed-reality environment to the user, a view position of the user for the rendered mixed-reality environment being variable depending at least in part on the determined user head pose; and

one or more memory devices storing computer-readable instructions which, when executed by the one or more processors, cause the HMID device to;

determine user head pose in the physical environment using the sensor data,

implement a 3D virtual viewport on the see-through display,

exchange extensibility data between the HMID device and the remote computing device, the extensibility data including scene data for the 3D model and user interface data for the 3D model, and wherein the extensibility data is configured to support multiple different rendering modes including 2D rendering on the desktop and 3D rendering on the viewport, in which the desktop and viewport each utilize the exchanged extensibility data to facilitate rendering of the 3D model in the different respective 2D rendering and 3D rendering supported by the desktop and the viewport, respectively,

selectively utilize one or more of the scene data or interface data to render the 3D model as a hologram in the viewport based on the view position,

enable user interaction with the hologram in the viewport, and

operate the see-through display based on the view position to enable user interaction with the 3D model on the desktop.

2. The HMD device of claim 1 further including providing a control to the user to control viewport characteristics including at least one of viewport location in the mixed-reality environment, viewport size, or viewport shape.

3. The HMD device of claim 1 further including clipping the 3D model to constrain the 3D model to an extent of the viewport.

4. The HMD device of claim 1 further including a rendering engine configured to render the hologram in the viewport, and in which the executed instructions further cause the HMD device to utilize the exchanged extensibility data in the rendering engine.

5. The HMD device of claim 1 in which the executed instructions further cause the HMD device to consume user inputs to the remote computing device based on the view position.

6. The HMD device of claim 1 in which the executed instructions further cause the HMD device to update the extensibility data based on user interactions with the hologram in the viewport, in which the updated extensibility data is utilized by the remote computing device to render the 3D model on the desktop.

7. The HMD of claim 1 further including

obtaining sensor data from the sensor package, the sensor data associated with a physical environment adjoining a user of the HMD device;

using the sensor data, reconstructing a geometry of the physical environment including any real world object located therein; and

using the reconstructed geometry to determine a location of a monitor that is coupled to the remote computing device within the physical environment, wherein the monitor is configured to render the 2D desktop.

8. The HMD device of claim 7 further including tracking the user'"'"'s head in the physical environment using the reconstructed geometry to determine the view position, rendering a mouse cursor in the viewport when user input causes the mouse cursor to move off the 2D desktop supported by the monitor coupled to the remote computing device, and consuming keyboard inputs when a ray projected from the view position intersects with the viewport.

9. The HMD device of claim 8 further including discontinuing the rendering of the mouse cursor in the viewport when the projected ray indicates that the mouse cursor has transitioned to the desktop supported by the monitor.

10. The HMD device of claim 1 further including enabling the 3D model to be transferred between the 2D desktop and the viewport using a mouse or keyboard that is coupled to the remote computing device.

11. The HMD device of claim 1 in which the sensor data includes depth data and further including generating the sensor data using a depth sensor and wherein the executed instructions further cause the HMD to apply surface reconstruction techniques to determine the user head pose.

12. A method performed by a head mounted display (IMD) device supporting a mixed-reality environment including virtual objects and real objects, the method comprising:

implementing a three-dimensional (3D) virtual viewport on a display of the HMD device;

receiving extensibility data from a remote computing device over a network connection, the extensibility data including scene data describing a 3D model supported by an application executing on the computing device, the computing device being associated with a monitor that supports a desktop, and further including user interface (UI) data describing user inputs to the computing device, and wherein the extensibility data is configured to support multiple different rendering modes including two-dimensional (2D) rendering on the desktop and 3D rendering on the viewport, in which the desktop and viewport each utilize the exchanged extensibility data to facilitate rendering of the 3D model in the different respective 2D rendering and 3D rendering supported by the desktop and the viewport, respectively; and

dynamically rendering the 3D model in the viewport using the received extensibility data.

13. The method of claim 12 further including rendering a mouse cursor in the 3D virtual viewport based on mouse messages included in the UI data.

14. The method of claim 12 further including controlling rendering of the 3D model in the viewport using keyboard messages included in the UI data.

15. The method of claim 14 further including utilizing sensor data to determine a view position of a user of the HMD device and transitioning a cursor back to the desktop supported by a monitor coupled to the computing device when a ray projected from the view position intersects the monitor.

16. The method of claim 15 further including modeling a physical environment in which the HMD device is located using a surface reconstruction data pipeline that implements a volumetric method creating multiple overlapping surfaces that are integrated and using the modeled physical environment at least in part to determine the view position or to determine a location of the monitor that is coupled to the computing device within the physical environment.

17. The method of claim 12 in which the scene data includes at least one of 3D model data, environmental data, or camera parameters.

18. A computing device, comprising:

one or more processors;

an interface to a monitor, the monitor displaying a two-dimensional (2D) desktop;

a mouse interface for connecting to a mouse and receiving signals from the mouse indicating mouse movement and inputs to mouse controls from a user of the computing device;

a keyboard interface for connecting to a keyboard and receiving signals from the keyboard indicating keyboard inputs from the user;

a network interface for communicating with a remote head mounted display (HMD) device over a network connection, the HMD device being configured to support a three-dimensional (3D) virtual viewport; and

one or more memory devices storing computer-readable instructions which, when executed by the one or more processors implement a three-dimensional (3D) modeling application and a user interface (UI) server configured for

tracking mouse messages that describe the mouse movements and inputs,

tracking keyboard messages that describe the keyboard inputs,

when a mouse movement indicates that a cursor associated with the mouse is moving beyond an edge of the monitor, taking control of the mouse messages and preventing propagation of the mouse messages to systems operating on the computing device,

sending the mouse messages to the HMD device over the network connection,

sending the keyboard messages to the HMD device over the network connection, and

sending extensibility data to the HMD device over the network, the extensibility data including scene data describing a 3D model supported by an application executing on the computing device, and wherein the extensibility data is configured to support multiple different rendering modes including 2D rendering on the desktop and 3D rendering on the viewport, in which the desktop and viewport each utilize the exchanged extensibility data to facilitate rendering of the 3D model in the different respective 2D rendering and 3D rendering supported by the desktop and the viewport, respectively.

19. The computing device of claim 18 further including tracking the mouse messages and keyboard messages by interacting with an operating system executing on the computing device.

20. The computing device of claim 18 further including receiving a message from the HMD device that the mouse cursor has transitioned to the desktop and calculating an initial cursor position on the desktop using a last reported position of the mouse cursor in the viewport.