MOUSE SHARING BETWEEN A DESKTOP AND A VIRTUAL WORLD

• US 20160027214A1
• Filed: 02/23/2015
• Published: 01/28/2016
• Est. Priority Date: 07/25/2014
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Sugden, Ben, Salter, Tom, Memmott, Robert
• Application Number:
  US14/628,539
• Publication Number:
  US 20160027214A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0123 comprising devices increasing the field of view
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/03543 Mice or pucks G06F3/03541 takes precedence
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/0486 Drag-and-drop
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A mixed-reality head mounted display (HMD) device supports a three dimensional (3D) virtual world application with which a real world desktop displayed on a monitor coupled to a personal computer (PC) may interact and share mouse input. A mouse input server executing on the PC tracks mouse movements on the desktop displayed on a monitor. When movement of the mouse takes it beyond the edge of the monitor screen, the mouse input server takes control of the mouse and stops mouse messages from propagating through the PC'"'"'s system. The mouse input server communicates over a network connection to a mouse input client exposed by the application to inform the client that the mouse has transitioned to operating in the virtual world and passes mouse messages describing movements and control operation such as button presses.

Claims

1. A head mounted display (HMD) device operable by a user in a physical environment, comprising:

one or more processors;

a see-through display configured for rendering a mixed reality environment to the user, a view position of the user for the rendered mixed reality environment being variable depending at least in part on a pose of the user'"'"'s head in the physical environment; and

one or more memory devices storing computer-readable instructions which, when executed by the one or more processors, perform a method comprising the steps of;

rendering the mixed reality environment within a field of view of the HMD device, the mixed reality environment including objects supported in a virtual world and objects supported in a real world,

receiving mouse messages over a network connection from a mouse input server running on a remote computing device, the mouse messages describing movements of a mouse that is operatively connected to the computing device, the mouse controlling a cursor displayable in the virtual world and on a monitor in the real world,

when movement of the mouse causes the cursor to move beyond a border of the monitor, calculating an initial position of the cursor in the virtual world,

using the mouse messages to calculate subsequent positions of the cursor in the virtual world, and

rendering the cursor in the virtual world using the calculated initial and subsequent positions.

2. The HMD of claim 1 further including determining deltas between mouse movements from the mouse messages and using the deltas to calculate a subsequent position for the cursor in the virtual world.

3. The HMD of claim 1 further including receiving button push events in the mouse messages, and using the button push events as inputs when rendering the mixed reality environment.

4. The HMD of claim 1 further including

obtaining sensor data describing a physical space adjoining a user of the HMD device;

using the sensor data, reconstructing a geometry of the physical space; and

tracking the user'"'"'s head in the physical space using the reconstructed geometry to determine the view position.

5. The HMD of claim 4 in which the sensor data includes depth data and further including generating the sensor data using a depth sensor and applying surface reconstruction techniques to reconstruct the physical space geometry.

6. The HMD of claim 4 further including determining if the cursor is transitioning to the desktop by calculating a ray between the next position of the cursor and the view position and, if the ray intersects the real world monitor, informing the computing device that the cursor has transitioned to a desktop supported on the monitor.

7. The HMD of claim 6 further including discontinuing the rendering of the cursor in the virtual world when the cursor has transitioned to the desktop.

8. The HMD of claim 6 further including a network interface over which the mouse messages are communicated from the computing device and over which the computing device is informed that the cursor has transitioned to the desktop.

9. The HMD of claim 6 further including enabling an object to be moved from the desktop to the virtual world using the mouse.

10. The HMD of claim 1 further including a sensor package for detecting a gaze direction of the user when determining the view position.

11. The HMD of claim 1 further including enabling interactions with one or more virtual objects using the cursor.

12. The HMD of claim 1 further including enabling collisions between the cursor and real world objects.

13. A method for communicating mouse information between a computing device and an application executing on a head mounted display (HMD) device, the application supporting a mixed reality environment on the HMD device including a virtual world and a real world, the method comprising:

operating a mouse input client in the application;

receiving mouse messages over a network connection from a mouse input server executing on the computing device, the mouse messages describing movements of a mouse that is operatively coupled to the computing device having an associated monitor, the mouse input server sending the mouse messages when a movement of the mouse causes a mouse cursor to move past an edge of the monitor to exit the real world and enter the virtual world;

determining an initial position of the mouse cursor in the virtual world using a position of exit from the real world; and

utilizing movements of the mouse to determine subsequent mouse cursor positions in the virtual world.

14. The method of claim 13 further including rendering the mouse cursor in the virtual world at the initial position and at the subsequent mouse cursor positions on the HMD device.

15. The method of claim 13 further including utilizing sensor data to determine a view position of a user of the HMD device and transitioning the cursor back to a desktop supported by the monitor when a ray projected from the view position intersects the monitor.

16. The method of claim 15 further including modeling a physical environment in which the HMD device is located using a surface reconstruction data pipeline that implements a volumetric method creating multiple overlapping surfaces that are integrated and using the modeled physical environment at least in part to determine the view position.

17. A computing device, comprising:

one or more processors;

an interface to a monitor, the monitor displaying a desktop;

a mouse interface for connecting to a mouse and receiving signals from the mouse indicating mouse movement and inputs to mouse controls from a user of the computing device;

a network interface for communicating with a remote head mounted display (HMD) device over a network connection; and

one or more memory devices storing computer-readable instructions which, when executed by the one or more processors implement a mouse input server configured for

tracking mouse messages that describe the mouse movements and inputs,

when a mouse movement indicates that a cursor associated with the mouse is moving beyond and edge of the monitor, taking control of the mouse messages and preventing propagation of the mouse messages to systems operating on the computing device, and

sending the mouse messages to the HMD device over the network connection.

18. The computing device of claim 17 in which the HMD device is configured for rendering a mixed reality environment on an optical display, the mixed reality environment including objects in a virtual world and objects in a real world, the mouse messages being utilized by the HMD device to at least render the cursor in the virtual world.

19. The computing device of claim 17 further including tracking the mouse messages by interacting with an operating system executing on the computing device.

20. The computing device of claim 17 further including receiving a message from the HMD device that the mouse cursor has transitioned to the desktop and calculating an initial cursor position on the desktop using a last reported position of the mouse cursor in the virtual world.