REMOTE RENDERING FOR VIRTUAL IMAGES

• US 20170115488A1
• Filed: 10/26/2015
• Published: 04/27/2017
• Est. Priority Date: 10/26/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Egbert, Cameron Quinn, McGlynn, Jeffrey Wallace, Ey, Michael, Ambrus, Anthony, Kroymann, Dan
• Granted Patent: US 10,962,780 B2
• Time in Patent Office: Days
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0147 comprising a device modifying the resolution of the displayed image
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 15/005 General purpose rendering architectures
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

One or more sensors of a virtual reality device track a pose of the virtual reality device. The virtual reality device requests a virtual image having a perspective corresponding to a future pose from a remote computer. After receiving the requested virtual image, the virtual reality device adjusts the virtual image to an adjusted virtual image having an updated perspective corresponding to an updated tracked pose of the virtual reality device. Then, a virtual reality display displays the adjusted virtual image.

Claims

1. A virtual reality device, comprising:

one or more sensors configured to track a pose of the virtual reality device;

a pose prediction machine configured to predict a future pose of the virtual reality device at a future time;

a communications machine configured to send to a remote computer a request for a virtual image having a perspective corresponding to the future pose of the virtual reality device and to receive from the remote computer the virtual image;

a late stage reprojection machine configured to adjust the virtual image to an adjusted virtual image having an updated perspective corresponding to an updated tracked pose of the virtual reality device; and

a virtual reality display configured to display the adjusted virtual image.

2. The virtual reality device of claim 1, wherein the communications machine is further configured to determine a communications latency by measuring an elapsed time between the request being sent and the virtual image being received.

3. The virtual reality device of claim 2, wherein the pose prediction machine is configured to dynamically adjust a buffer period between a current time and the future time for which the future pose is predicted, the buffer period dynamically increased with increasing communications latency and dynamically decreased with decreasing communications latency.

4. The virtual reality device of claim 1, wherein the one or more sensors are configured to detect a motion of the virtual reality device in one or more dimensions.

5. The virtual reality device of claim 4, wherein the pose prediction machine is configured to predict the future pose by extrapolating the detected motion forward from a current time to the future time.

6. The virtual reality device of claim 5, wherein extrapolating the detected motion comprises extrapolating the detected motion forward with biomechanical constraints.

7. The virtual reality device of claim 5, wherein extrapolating the detected motion comprises extrapolating the detected motion forward with environmental constraints.

8. The virtual reality device of claim 4, wherein the virtual reality device is configured to display an image having a set resolution, and wherein the virtual image is oversized relative to the set resolution.

9. The virtual reality device of claim 8, wherein an amount of oversizing is proportional to a speed of the detected motion.

10. The virtual reality device of claim 9, wherein an amount of oversizing in a first dimension of the virtual image is proportional to a speed of the detected motion in the first dimension, and wherein an amount of oversizing in a second dimension of the virtual image is independently proportional to a speed of the detected motion in the second dimension.

11. The virtual reality device of claim 10, wherein adjusting the virtual image comprises cropping the virtual image to the set resolution.

12. The virtual reality device of claim 1, wherein adjusting the virtual image comprises shifting the virtual image by an amount that is proportional to a difference between the predicted future pose and the updated tracked pose.

13. The virtual reality device of claim 12, wherein the virtual image is a three-dimensional image, adjusting the virtual image comprises independently shifting different aspects of the three-dimensional virtual image by an amount that is proportional to a difference between the predicted future pose and the updated tracked pose, and further comprises independently shifting each particular aspect of the three-dimensional virtual image by an amount that is inversely proportional to a virtual depth of the particular aspect.

14. The virtual reality device of claim 1, further comprising a local virtual image renderer configured to locally render one or more virtual objects, wherein the virtual reality display is configured to display the one or more virtual objects rendered by the local virtual image renderer with the adjusted virtual image.

15. The virtual reality device of claim 1, wherein the virtual reality device and the remote computer communicate over a computer network, and each of the virtual reality device and the remote computer occupy a different network node.

16. The virtual reality device of claim 1, wherein the virtual image received from the remote computer includes additional content provided by the remote computer.

17. A computing device, comprising:

a remote communications machine configured to receive from a virtual reality device a request for a virtual image having a perspective corresponding to a predicted future pose of the virtual reality device, and send to the virtual reality device a rendered virtual image; and

a virtual image renderer configured to render the virtual image for the virtual reality device as specified by the request received by the remote communications machine.

18. A virtual reality device, comprising:

one or more sensors configured to track a pose of the virtual reality device;

a communications machine configured to send to a remote computer a request for a virtual image having a perspective corresponding to a future pose of the virtual reality device and to receive from the remote computer the virtual image;

a late stage reprojection machine configured to adjust the virtual image to an adjusted virtual image having an updated perspective corresponding to an updated tracked pose of the virtual reality device; and

a virtual reality display configured to display the adjusted virtual image.

19. The virtual reality device of claim 18, wherein a future time of the future pose is dynamically predicted based on communication latency.

20. The virtual reality device of claim 19, wherein the future pose is predicted by extrapolating detected motion forward from a current time to the future time.