Depth-aware reprojection

• US 10,129,523 B2
• Filed: 06/22/2016
• Issued: 11/13/2018
• Est. Priority Date: 06/22/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Michail, Ashraf Ayman, Klein, Georg, Pearson, Andrew Martin, Mathe, Zsolt, Grossman, Mark S., Xu, Ning
• Primary Examiner(s): Nguyen, Phu K
• Application Number:
  US15/190,085
• Publication Number:
  US 20170374341A1
• Time in Patent Office: 874 Days
• Field of Search: 345419
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06T 15/06 Ray-tracing
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling
• CPC Class Codes: G06T 3/60 Rotation of whole images or parts thereof
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/324 Colour aspects
• CPC Class Codes: H04N 13/332 Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
• CPC Class Codes: H04N 13/366 using viewer tracking

Abstract

Examples are disclosed that relate to depth-aware late-stage reprojection. One example provides a computing system configured to receive and store image data, receive a depth map for the image data, processing the depth map to obtain a blurred depth map, and based upon motion data, determine a translation to be made to the image data. Further, for each pixel, the computing system is configured to translate an original ray extending from an original virtual camera location to an original frame buffer location to a reprojected ray extending from a translated camera location to a reprojected frame buffer location, determine a location at which the reprojected ray intersects the blurred depth map, and sample a color of a pixel for display based upon a color corresponding to the location at which the reprojected ray intersects the blurred depth map.

Claims

1. A computing system comprising a display, a sensor subsystem, and a reprojection subsystem configured to:

receive image data to be displayed via the display;

receive a depth map for the image data;

store the image data in a frame buffer;

process the depth map to obtain a blurred depth map;

based upon motion data received from the sensor subsystem, determine a translation to be made to the image data for display; and

for each pixel of a plurality of pixels in the frame buffer, based upon the translation determined,

translate an original ray extending from an original virtual camera location to a corresponding original frame buffer location to a reprojected ray extending from a translated camera location to a reprojected frame buffer location,

determine a location at which the reprojected ray intersects the blurred depth map, and

sample a color of a pixel for display based upon a color in the frame buffer corresponding to the location at which the reprojected ray intersects the blurred depth map.

2. The computing system of claim 1, wherein the computing system comprises a head-mounted display device.

3. The computing system of claim 1, wherein the reprojection subsystem is further configured to display via the display the image data after translating the original ray to the reprojected ray.

4. The computing system of claim 1, wherein the reprojection subsystem is further configured to make a rotation to the image data based upon motion data received from the sensor subsystem.

5. The computing system of claim 1, wherein the reprojection subsystem is configured to process the depth map to obtain the blurred depth map by omitting background values and weights beyond threshold values during a depth map blur convolution.

6. The computing system of claim 1, wherein the reprojection subsystem is further configured to determine the location at which the reprojected ray intersects the blurred depth map by

determining an initial depth value on the blurred depth map corresponding to the original frame buffer location,

then determining an initial location on the reprojected ray corresponding to the initial depth value,

then determining an updated depth value on the blurred depth map corresponding to the initial location on the reprojected ray, and

then determining an updated location on the reprojected ray corresponding to the updated depth value.

7. The computing system of claim 6, wherein the reprojection subsystem is further configured to determine the location at which the reprojected ray intersects the blurred depth map by iteratively repeating the determining of the updated depth value and the determining of the updated location on the reprojected ray.

8. The computing system of claim 6, wherein the reprojection subsystem is further configured to iteratively repeat the determining of the updated depth value and the determining of the updated location on the reprojected ray a fixed number of times for each pixel in the frame buffer.

9. The computing system of claim 1, wherein the reprojection subsystem is further configured to reduce a resolution of the depth map.

10. On a computing system comprising a display and a sensor subsystem, a method comprising:

receiving image data to be displayed via the display;

receiving a depth map for the image data;

storing the image data in a frame buffer;

processing the depth map to obtain a blurred depth map;

based upon motion data received from the sensor subsystem, determining a translation to be made to the image data prior to display; and

for each pixel of a plurality of pixels in the frame buffer, based upon the translation determined,

translating an original ray extending from an original virtual camera location to a corresponding original frame buffer location to a reprojected ray extending from a translated camera location to a reprojected frame buffer location,

determining a location at which the reprojected ray intersects the blurred depth map, and

sampling a color of a pixel in the frame buffer for display at the reprojected frame buffer location based upon a color in the frame buffer corresponding to the location at which the reprojected ray intersects the blurred depth map.

11. The method of claim 10, further comprising displaying via the display the image data after translating the original ray to the reprojected ray for each pixel in the frame buffer.

12. The method of claim 10, wherein the plurality of pixels comprises each pixel in the frame buffer.

13. The method of claim 10, wherein determining the location at which the reprojected ray intersects the blurred depth map comprises

determining an initial depth value on the blurred depth map corresponding to the original frame buffer location,

then determining an initial location on the reprojected ray corresponding to the initial depth value,

then determining an updated depth value on the blurred depth map corresponding to the initial location on the reprojected ray, and

then determining an updated location on the reprojected ray corresponding to the updated depth value.

14. The method of claim 13, wherein determining the location at which the reprojected ray intersects the blurred depth map comprises iteratively repeating the determining of the updated depth value and the determining of the updated location on the reprojected ray.

15. The method of claim 14, further comprising iteratively repeating the determining of the updated depth value and the determining of the updated location on the reprojected ray a same number of times for each pixel in the frame buffer.

16. The method of claim 10, further comprising reducing a resolution of the depth map.

17. A head-mounted display device comprising a display, a sensor subsystem, and a reprojection subsystem configured to:

receive image data to be displayed via the display;

receive depth data for the image data;

store the image data in a frame buffer;

process the depth map to obtain a blurred depth map;

based upon motion data received from the sensor subsystem, determine a translation to be made to the image data prior to display; and

for each pixel of a plurality of pixels in the frame buffer, based upon the translation determined,

translate an original ray extending from an original virtual camera location to a corresponding original frame buffer location to a reprojected ray extending from a translated camera location to a reprojected frame buffer location,

determine a location at which the reprojected ray intersects the blurred depth map using successive approximation, and

sample a color of a pixel in the frame buffer for display at the reprojected frame buffer location based upon a color in the frame buffer corresponding to the location at which the reprojected ray intersects the blurred depth map.

18. The head-mounted display device of claim 17, wherein the reprojection subsystem is configured to determine the location at which the reprojected ray intersects the blurred depth map using successive approximation by

determining an initial depth value on the blurred depth map corresponding to the original frame buffer location,

then determining an initial location on the reprojected ray corresponding to the initial depth value,

then determining an updated depth value on the blurred depth map corresponding to the initial location on the reprojected ray, and

then determining an updated location on the reprojected ray corresponding to the updated depth value.

19. The head-mounted display device of claim 17, wherein the reprojection subsystem is further configured to iteratively repeat the determining of the updated depth value and the determining of the updated location on the reprojected ray a fixed number of times for each pixel in the frame buffer.

20. The head-mounted display device of claim 17, wherein the reprojection subsystem is further configured to determine the translation to be made to the image data based on one or more additional frame buffers and corresponding blurred depth maps.