DISCONTINUITY-AWARE REPROJECTION

• US 20170374344A1
• Filed: 01/17/2017
• Published: 12/28/2017
• Est. Priority Date: 06/22/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Boulton, Michael George, Michail, Ashraf Ayman
• Granted Patent: US 10,237,531 B2
• Time in Patent Office: Days
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06T 15/06 Ray-tracing
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/15 for colour aspects of image signals
• CPC Class Codes: H04N 13/324 Colour aspects
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof

Abstract

In various embodiments, methods and systems reprojecting three-dimensional (3D) virtual scenes using discontinuity depth late stage reprojection are provided. A reconstruction point, that indicates camera pose information, is accessed. The reconstruction point is associated with a plurality of sample points of a three-dimensional (3D) virtual scene. One or more closest sample points, relative to the reconstruction point, are identified, from the plurality of sample points. Each of the one or more closest sample points is associated with a cube map of color data and depth data. A relative convergence score is determined for each of the one or more closest sample points based on performing a depth-aware cube map late stage reprojection operation in relation to the reconstruction point. A subset of the one or more closest sample points is identified based on the relative convergence score. A reconstructed 3D virtual image is generated using the subset.

Claims

1. A system for reprojecting three-dimensional (3D) virtual scenes using discontinuity depth late stage reprojection, the system comprising:

one or more processors and memory configured for providing computer program instructions to the one or more processors;

a reprojection engine configured to;

access a reconstruction point that indicates camera pose information associated with a 3D virtual scene, the 3D virtual scene is associated with a plurality of sample points each having a cube map of color data and depth data;

identify one or more closest sample points relative to the reconstruction point, the one or more closest sample points is identified from the plurality of sample points;

determine a relative convergence score generated for each of the one or more closest sample points based on performing a depth-aware cube map late stage projection operation for each of the one or more closest sample points in relation to the reconstruction point, wherein the relative convergence score is an indicator of a capacity of a sample point to reconstruct at least portions of the 3D virtual scene from the reconstruction point;

identifying a subset of the one or more closest sample points based on eliminating one or more of the one or more closest sample points based on the relative convergence score; and

generating a reconstructed 3D virtual scene using the subset of the one or more closest sample points.

2. The system of claim 1, wherein the reprojection engine is further configured to perform the depth-aware cube map late stage projection operation based on:

adjusting an original ray extending from an original virtual camera location associated with a selected sample point to a corresponding original frame buffer location to a reprojected ray extending from a translated camera location associated with the reconstruction point to a reprojected frame buffer location; and

determining a location at which the reprojected ray intersects the depth data comprising a blurred depth map.

3. The system of claim 2, wherein the reprojection engine is further configured to determine the location at which the reprojected ray intersects the blurred depth map by iteratively repeating determining of updated depth values and determining of updated locations on the reprojected ray;

and

sample a color of a pixel for display based upon a color in a frame buffer corresponding to the location at which the reprojected ray intersects the blurred depth map.

4. The system of claim 2, wherein the reprojection engine is further configured to generate the original frame buffer and the reprojected frame buffer based on the cube map color data and depth data.

5. The system of claim 1, wherein the reprojection engine is further configured to generate the reconstructed 3D virtual scene using a first stereoscopic view and a second stereoscopic view of a stereoscopic projection of the 3D virtual scene from the reconstruction point, using the subset of the plurality of sample points.

6. The system of claim 1, wherein selecting locations for plurality of sample points in the 3D virtual scene is based on a sample point selection scheme that includes selecting more sample points in locations of the 3D virtual scene with more content detail and selecting less sample points in locations of the 3D virtual scene with less content detail.

7. The system of claim 6, wherein a distance between a selected sample point and a closest sample point is greater than an intra-ocular distance for stereoscopic reprojection of the reconstructed 3D virtual scene.

8. A computer-implemented method for reprojecting three-dimensional (3D) virtual scenes using discontinuity depth late stage reprojection, the method comprising:

accessing a reconstruction point that indicates camera pose information associated with a 3D virtual scene, the 3D virtual scene is associated with a plurality of sample points each having a cube map of color data and depth data;

identifying a first selected sample point from the plurality of sample points, wherein the first selected sample point is a closest sample point relative to the reconstruction point;

determining a dis-occlusion associated with generating a reconstructed 3D virtual scene from the reconstruction point, wherein the dis-occlusion is determined based on performing a depth-aware cube map late stage projection operation for first selected sample point in relation to reference point;

identifying a second selected sample point, from the plurality of sample points, for resolving the dis-occlusion, wherein the second selected sample point is a second closest sample point relative to the reconstruction point; and

generating a corresponding first portion and second portion of the reconstructed 3D virtual scene using the first selected sample point and the second selected sample point based on performing the depth-aware cube map late stage projection operation on the first selected sample point and the second selected sample point in relation to the reference point.

9. The method of claim 8, wherein performing a depth-aware cube map late stage projection operation on a selected sample point is based on:

adjusting an original ray extending from an original virtual camera location associated with a selected sample point to a corresponding original frame buffer location to a reprojected ray extending from a translated camera location associated with the reconstruction point to a reprojected frame buffer location; and

determining a location at which the reprojected ray intersects the depth data comprising a blurred depth map.

10. The method of claim 9, wherein determining the location at which the reprojected ray intersects the blurred depth map is based on iteratively repeating determining of updated depth values and determining of updated locations on the reprojected ray.

11. The method of claim 10, further comprising sampling a color of a pixel for display based upon a color in a frame buffer corresponding to the location at which the reprojected ray intersects the blurred depth map.

12. The method of claim 8, wherein the reconstruction point is associated with a first stereoscopic view and a second stereoscopic view such that generating the reconstructed 3D virtual scene further comprises generating the first stereoscopic view and the second stereoscopic view.

13. The method of claim 12, wherein the first stereoscopic view is associated with a first original frame buffer and a first reprojected frame buffer based on a cube map color data and depth data of a selected sample point;

and wherein the second stereoscopic view is associated with a second original frame buffer and a second reprojected frame buffer based on the cube map color data and depth data of the selected sample point.

14. The method of claim 8, wherein a distance between the reconstruction point and the selected sample point is greater than an intra-ocular distance for stereoscopic reprojection of the reconstructed 3D virtual scene.

15. One or more computer storage media having computer-executable instructions embodied thereon that, when executed, by one or more processors, causes the one or more processors to perform a method for reprojecting images based on three-dimensional (3D) virtual images using discontinuity and depth late stage reprojection, the method comprising:

referencing a reconstruction point that indicates camera pose information associated with a 3D virtual scene, the 3D virtual scene is associated with a plurality of sample points each having a cube map of color data and depth data;

identifying a first closest sample point of the reconstruction point, the first closest sample point is selected from the plurality of sample points;

determining a dis-occlusion associated with generating a reconstructed 3D virtual scene from the reconstruction point, wherein the dis-occlusion is determined based on performing a depth-aware cube map late stage projection operation for first closest sample point in relation to reference point, wherein the dis-occlusion is associated with one of a first stereoscopic view or a second stereoscopic view of the reconstruction point;

identifying a second closest sample point of the reconstruction point, the second closest sample point is selected from the plurality of sample points; and

reconstructing the dis-occlusion using the second closest sample point based on performing a depth-aware cube map late stage projection process based on the second closest sample point in relation to the reference point.

16. The media of claim 15, wherein performing a depth-aware cube map late stage projection operation on the selected sample point is based on:

adjusting an original ray extending from an original virtual camera location associated with a selected sample point to a corresponding original frame buffer location to a reprojected ray extending from a translated camera location associated with the reconstruction point to a reprojected frame buffer location; and

determining a location at which the reprojected ray intersects the depth data comprising a blurred depth map.

17. The media of claim 15, wherein determining the location at which the reprojected ray intersects the blurred depth map is based on iteratively repeating determining of updated depth values and determining of updated locations on the reprojected ray.

18. The media of claim 17, further comprising sampling a color of a pixel for display based upon a color in a frame buffer corresponding to the location at which the reprojected ray intersects the blurred depth map.

19. The media of claim 15, further comprising generating the reconstructed 3D virtual scene using the first closest sample point and the second closest sample point based on performing the depth-aware cube map late stage projection operation on the first selected sample point and the second selected sample point in relation to the reference point.

20. The media of claim 15, wherein determining the dis-occlusion is based on the depth-aware cube map late stage reprojection operation failure to converge to a corresponding location of the reference point relative to the first closest sample point.