INTERLEAVED APPROACH TO DEPTH-IMAGE-BASED RENDERING OF STEREOSCOPIC IMAGES

• US 20140125650A1
• Filed: 11/02/2012
• Published: 05/08/2014
• Est. Priority Date: 11/02/2012
• Status: Active Grant

• Current Assignee: NVIDIA Corporation
• Original Assignee: NVIDIA Corporation
• Inventors: NEILL, Patrick
• Granted Patent: US 9,241,146 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 2213/003 Aspects relating to the "2D+depth" image format

Abstract

Techniques are disclosed for generating stereoscopic images. The techniques include receiving a first image frame associated with a first eye, and receiving a first depth frame associated with the first eye. The techniques further include reprojecting the first image frame based on the first depth frame to create a second image frame associated with a second eye. The techniques further include identifying a first pixel in the second image frame that remains unwritten as a result of reprojecting the first image frame, and determining a value for the first pixel based on a corresponding pixel in a prior image frame associated with the second eye. One advantage of the disclosed techniques is that DIBR reprojected image frames have a more realistic appearance where gaps are filled using pixels from a prior image for the same eye.

Claims

1. A method for generating stereoscopic images, the method comprising:

receiving a first image frame associated with a first eye;

receiving a first depth frame associated with the first eye;

reprojecting the first image frame based on the first depth frame to create a second image frame associated with a second eye;

identifying a first pixel in the second image frame that remains unwritten as a result of reprojecting the first image frame; and

determining a value for the first pixel based on a corresponding pixel in a prior image frame associated with the second eye.

2. The method of claim 1, wherein determining a value for the first pixel comprises:

retrieving the corresponding pixel from a memory buffer that includes the prior image frame; and

setting the value for the first pixel to the corresponding pixel in the prior image frame.

3. The method of claim 1, further comprising applying a filter to the first image frame based on the value determined for the first pixel.

4. The method of claim 3, wherein the filter comprises a blurring filter.

5. The method of claim 1, further comprising:

receiving a third image frame associated with the second eye;

receiving a second depth frame associated with the second eye;

reprojecting the third image frame based on the second depth frame to create a fourth image frame associated with the first eye;

identifying a second pixel in the fourth image frame that remains unwritten as a result of reprojecting the third mage frame; and

determining a value for the second pixel based on a corresponding pixel in the first image frame.

6. The method of claim 1, wherein reprojecting the first image frame comprises:

determining that a second pixel associated with the first image frame comprises a plurality of samples, wherein each sample of the plurality of samples includes image and depth information;

reprojecting each sample of the plurality of samples to a corresponding pixel in the second image frame;

blending the reprojected samples to determine a final value for the corresponding pixel; and

storing the corresponding pixel in the second image frame.

7. The method of claim 6, wherein the depth information for a first sample included in the plurality of samples comprises a depth value associated with a second sample included in the plurality of samples that is located closest to a screen surface of a display device and a depth offset comprising a distance between the first sample and the second sample.

8. A computer-readable storage medium including instructions that, when executed by a processing unit, cause the processing unit to perform an operation for generating stereoscopic images, the operation comprising:

receiving a first image frame associated with a first eye;

receiving a first depth frame associated with the first eye;

reprojecting the first image frame based on the first depth frame to create a second image frame associated with a second eye;

identifying a first pixel in the second image frame that remains unwritten as a result of reprojecting the first image frame; and

determining a value for the first pixel based on a corresponding pixel in a prior image frame associated with the second eye.

9. The computer-readable storage medium of claim 8, wherein determining a value for the first pixel comprises:

retrieving the corresponding pixel from a memory buffer that includes the prior image frame; and

setting the value for the first pixel to the corresponding pixel in the prior image frame.

10. The computer-readable storage medium of claim 8, wherein the processing unit is further configured to perform the step of comprising applying a filter to the first image frame based on the value determined for the first pixel.

11. The computer-readable storage medium of claim 10, wherein the filter comprises a blurring filter.

12. The computer-readable storage medium of claim 8, wherein the processing unit is further configured to perform the steps of:

receiving a third image frame associated with the second eye;

receiving a second depth frame associated with the second eye;

reprojecting the third image frame based on the second depth frame to create a fourth image frame associated with the first eye;

identifying a second pixel in the fourth image frame that remains unwritten as a result of reprojecting the third mage frame; and

determining a value for the second pixel based on a corresponding pixel in the first image frame.

13. The computer-readable storage medium of claim 8, wherein reprojecting the first image frame comprises:

determining that a second pixel associated with the first image frame comprises a plurality of samples, wherein each sample of the plurality of samples includes image and depth information;

reprojecting each sample of the plurality of samples to a corresponding pixel in the second image frame;

blending the reprojected samples to determine a final value for the corresponding pixel; and

storing the corresponding pixel in the second image frame.

14. The computer-readable storage medium of claim 13, wherein the depth information for a first sample included in the plurality of samples comprises a depth value associated with a second sample included in the plurality of samples that is located closest to a screen surface of a display device and a depth offset comprising a distance between the first sample and the second sample.

15. A subsystem, comprising:

a processing unit; and

a memory containing instructions, that, when executed by the processing unit, cause the processing to generate stereoscopic images, by;

receiving a first image frame associated with a first eye;

receiving a first depth frame associated with the first eye;

reprojecting the first image frame based on the first depth frame to create a second image frame associated with a second eye;

identifying a first pixel in the second image frame that remains unwritten as a result of reprojecting the first image frame; and

determining a value for the first pixel based on a corresponding pixel in a prior image frame associated with the second eye.

16. The subsystem of claim 15, wherein determining a value for the first pixel comprises:

retrieving the corresponding pixel from a memory buffer that includes the prior image frame; and

setting the value for the first pixel to the corresponding pixel in the prior image frame.

17. The subsystem of claim 15, wherein the processing unit is further configured to perform the step of applying a filter to the first image frame based on the value determined for the first pixel.

18. The subsystem of claim 17, wherein the filter comprises a blurring filter.

19. The subsystem of claim 15, wherein the processing unit is further configured to perform the steps of:

receiving a third image frame associated with the second eye;

receiving a second depth frame associated with the second eye;

reprojecting the third image frame based on the second depth frame to create a fourth image frame associated with the first eye;

identifying a second pixel in the fourth image frame that remains unwritten as a result of reprojecting the third mage frame; and

determining a value for the second pixel based on a corresponding pixel in the first image frame.

20. The subsystem of claim 15, wherein reprojecting the first image frame comprises:

determining that a second pixel associated with the first image frame comprises a plurality of samples, wherein each sample of the plurality of samples includes image and depth information;

reprojecting each sample of the plurality of samples to a corresponding pixel in the second image frame;

blending the reprojected samples to determine a final value for the corresponding pixel; and

storing the corresponding pixel in the second image frame.