Interleaved approach to depth-image-based rendering of stereoscopic images

• US 9,241,146 B2
• Filed: 11/02/2012
• Issued: 01/19/2016
• Est. Priority Date: 11/02/2012
• Status: Active Grant

• Current Assignee: NVIDIA Corporation
• Original Assignee: NVIDIA Corporation
• Inventors: Neill, Patrick
• Primary Examiner(s): Perromat, Carlos
• Application Number:
  US13/668,055
• Publication Number:
  US 20140125650A1
• Time in Patent Office: 1,173 Days
• Field of Search: 348/42, 348/43, 348/51, 348/E13.071, 348/E13.02, 348/E13.068, 348/E13.07, 348/E13.003, 345/419, 345/422, 352/58, 352/60, 352/62, 352/86
• US Class Current: 1/1
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 2213/003 Aspects relating to the "2D+depth" image format

Abstract

Techniques are disclosed for generating stereoscopic images. The techniques include receiving a first image frame associated with a first eye, and receiving a first depth frame associated with the first eye. The techniques further include reprojecting the first image frame based on the first depth frame to create a second image frame associated with a second eye. The techniques further include identifying a first pixel in the second image frame that remains unwritten as a result of reprojecting the first image frame, and determining a value for the first pixel based on a corresponding pixel in a prior image frame associated with the second eye. One advantage of the disclosed techniques is that DIBR reprojected image frames have a more realistic appearance where gaps are filled using pixels from a prior image for the same eye.

Claims

1. A method for generating stereoscopic images, the method comprising:

receiving a first image frame associated with a first eye;

receiving a first depth frame associated with the first eye;

reprojecting the first image frame based on the first depth frame to create a second image frame associated with a second eye;

identifying a first pixel in the second image frame that remains unwritten as a result of reprojecting the first image frame; and

determining a value for the first pixel based on a corresponding pixel in a prior image frame associated with the second eye, wherein the prior image frame is not reconstructed based on depth information.

2. The method of claim 1, wherein determining a value for the first pixel comprises:

retrieving the corresponding pixel from a memory buffer that includes the prior image frame; and

setting the value for the first pixel to the corresponding pixel in the prior image frame.

3. The method of claim 1, further comprising applying a filter to the first image frame based on an amount of motion associated with the first pixel.

4. The method of claim 3, wherein the filter comprises a blurring filter.

5. The method of claim 1, further comprising:

receiving a third image frame associated with the second eye;

receiving a second depth frame associated with the second eye;

reprojecting the third image frame based on the second depth frame to create a fourth image frame associated with the first eye;

identifying a second pixel in the fourth image frame that remains unwritten as a result of reprojecting the third mage frame; and

determining a value for the second pixel based on a corresponding pixel in the first image frame.

6. The method of claim 1, wherein reprojecting the first image frame comprises:

determining that a second pixel associated with the first image frame comprises a plurality of samples, wherein each sample of the plurality of samples includes image and depth information;

reprojecting each sample of the plurality of samples to a corresponding pixel in the second image frame;

blending the reprojected samples to determine a final value for the corresponding pixel; and

storing the corresponding pixel in the second image frame.

7. The method of claim 6, wherein the depth information for a first sample included in the plurality of samples comprises a depth value associated with a second sample included in the plurality of samples that is located closest to a screen surface of a display device and a depth offset comprising a distance between the first sample and the second sample.

8. A non-transitory computer-readable storage medium including instructions that, when executed by a processing unit, cause the processing unit to perform an operation for generating stereoscopic images, the operation comprising:

receiving a first image frame associated with a first eye;

receiving a first depth frame associated with the first eye;

reprojecting the first image frame based on the first depth frame to create a second image frame associated with a second eye;

identifying a first pixel in the second image frame that remains unwritten as a result of reprojecting the first image frame; and

determining a value for the first pixel based on a corresponding pixel in a prior image frame associated with the second eye, wherein the prior image frame is not reconstructed based on depth information.

9. The non-transitory computer-readable storage medium of claim 8, wherein determining a value for the first pixel comprises:

retrieving the corresponding pixel from a memory buffer that includes the prior image frame; and

setting the value for the first pixel to the corresponding pixel in the prior image frame.

10. The non-transitory computer-readable storage medium of claim 8, wherein the processing unit is further configured to perform the step of comprising applying a filter to the first image frame based on an amount of motion associated with the first pixel.

11. The non-transitory computer-readable storage medium of claim 10, wherein the filter comprises a blurring filter.

12. The non-transitory computer-readable storage medium of claim 8, wherein the processing unit is further configured to perform the steps of:

receiving a third image frame associated with the second eye;

receiving a second depth frame associated with the second eye;

reprojecting the third image frame based on the second depth frame to create a fourth image frame associated with the first eye;

identifying a second pixel in the fourth image frame that remains unwritten as a result of reprojecting the third mage frame; and

determining a value for the second pixel based on a corresponding pixel in the first image frame.

13. The non-transitory computer-readable storage medium of claim 8, wherein reprojecting the first image frame comprises:

determining that a second pixel associated with the first image frame comprises a plurality of samples, wherein each sample of the plurality of samples includes image and depth information;

reprojecting each sample of the plurality of samples to a corresponding pixel in the second image frame;

blending the reprojected samples to determine a final value for the corresponding pixel; and

storing the corresponding pixel in the second image frame.

14. The non-transitory computer-readable storage medium of claim 13, wherein the depth information for a first sample included in the plurality of samples comprises a depth value associated with a second sample included in the plurality of samples that is located closest to a screen surface of a display device and a depth offset comprising a distance between the first sample and the second sample.

15. The non-transitory computer-readable storage medium of claim 8, wherein the second image frame is reconstructed based on depth information.

16. The non-transitory computer-readable storage medium of claim 8, wherein image frames associated with the first eye and image frames associated with the second eye are received in an alternating manner.

17. A subsystem, comprising:

a processing unit; and

a memory containing instructions, that, when executed by the processing unit, cause the processing to generate stereoscopic images, by;

receiving a first image frame associated with a first eye;

receiving a first depth frame associated with the first eye;

reprojecting the first image frame based on the first depth frame to create a second image frame associated with a second eye;

identifying a first pixel in the second image frame that remains unwritten as a result of reprojecting the first image frame; and

determining a value for the first pixel based on a corresponding pixel in a prior image frame associated with the second eye, wherein the prior image frame is not reconstructed based on depth information.

18. The subsystem of claim 17, wherein determining a value for the first pixel comprises:

retrieving the corresponding pixel from a memory buffer that includes the prior image frame; and

setting the value for the first pixel to the corresponding pixel in the prior image frame.

19. The subsystem of claim 17, wherein the processing unit is further configured to perform the step of applying a filter to the first image frame based on an amount of motion associated with the first pixel.

20. The subsystem of claim 19, wherein the filter comprises a blurring filter.

21. The subsystem of claim 17, wherein the processing unit is further configured to perform the steps of:

receiving a third image frame associated with the second eye;

receiving a second depth frame associated with the second eye;

reprojecting the third image frame based on the second depth frame to create a fourth image frame associated with the first eye;

identifying a second pixel in the fourth image frame that remains unwritten as a result of reprojecting the third mage frame; and

determining a value for the second pixel based on a corresponding pixel in the first image frame.

22. The subsystem of claim 17, wherein reprojecting the first image frame comprises:

determining that a second pixel associated with the first image frame comprises a plurality of samples, wherein each sample of the plurality of samples includes image and depth information;

reprojecting each sample of the plurality of samples to a corresponding pixel in the second image frame;

blending the reprojected samples to determine a final value for the corresponding pixel; and

storing the corresponding pixel in the second image frame.