RENDERING MULTIVIEW CONTENT IN A 3D VIDEO SYSTEM

• US 20110058021A1
• Filed: 09/09/2009
• Published: 03/10/2011
• Est. Priority Date: 09/09/2009
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Corporation
• Inventors: Chen, Ying, Hannuksela, Miska Matias
• Granted Patent: US 8,284,237 B2
• Time in Patent Office: Days
• US Class Current: 348/46
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 2213/003 Aspects relating to the "2D+depth" image format

Abstract

There is disclosed methods and apparatuses for multi-view video encoding, decoding and display. A depth map is provided for each of the available views. The depth maps of the available views are used to synthesize a target view for rendering an image from the perspective of the target view based on images of the available views.

Claims

1. A method comprising:

receiving at least one reference image having a corresponding depth image;

synthesizing a set of pixels, each associated with a warped depth value, based on the at least one reference image and the corresponding depth image;

segmenting the warped depth values of the set of pixels into a plurality of depth ranges; and

transforming the at least one reference image into a synthesized image based on at least one depth range.

2. The method according to claim 1, the method further comprising

creating a histogram on the basis of the warped depth values of the set of pixels;

and

performing said segmenting on the basis of the histogram.

3. The method according to claim 1, the method further comprising

determining a depth range for visible pixels for a pixel position of the synthesized picture;

selecting candidate pixels from the set of pixels within said depth range for visible pixels; and

determining one or more candidate pixels which are visible at said pixel position of the synthesized picture.

4. The method according to claim 3, the method further comprising

selecting the depth range such that the depth range is the minimum of all the depth ranges that cover the candidate pixels.

5. The method according to claim 3, wherein in response to having more than one candidate pixel,

reconstructing at least one reconstructed pixel on the basis of the warped depth values of the visible candidate pixels.

6. The method according to claim 1, the method further comprising

determining a pinhole pixel position in the synthesized image;

and

determining a pixel value of the pinhole pixel position based on a selected depth range.

7. The method according to claim 6, the method further comprising

selecting a second set of candidate pixels from the set of pixels within the selected depth range that are in the proximity of the pinhole pixel position;

and

deriving a pinhole pixel value from the second set of candidate pixels.

8. The method according to claim 7, wherein

the depth range is selected in a way that the depth range is the minimum of all the ranges that cover the second set of candidate pixels.

9. The method according to claim 7, wherein

the depth range is selected to comprise a maximum number of candidate pixels among the second set of candidate pixels.

10. The method according to claim 7, wherein in response to having more than one candidate pixel,

reconstructing at least one reconstructed pixel on the basis of the warped depth values of the second set of candidate pixels.

11. An apparatus comprising:

a view synthesizer unit for creating a synthesized image;

the view synthesizer unit being arranged to

receive at least one reference image having a corresponding depth image;

synthesize a set of pixels, each associated with a warped depth value, based on the at least one reference image and the corresponding depth image;

segment the warped depth values of the set of pixels into a plurality of depth ranges; and

transform the at least one reference image into a synthesized image based on at least one depth range.

12. The apparatus according to claim 11, wherein the view synthesizer unit is further arranged to

create a histogram on the basis of the warped depth values of the set of pixels;

and

perform said segmenting on the basis of the histogram.

13. The apparatus according to claim 11, wherein the view synthesizer unit is further arranged to

determine a depth range for visible pixels for a pixel position of the synthesized picture;

select candidate pixels from the set of pixels within said depth range for visible pixels; and

determine one or more candidate pixels which are visible at said pixel position of the synthesized picture.

14. The apparatus according to claim 13, wherein the view synthesizer unit is further arranged to

select the depth range such that the depth range is the minimum of all the depth ranges that cover the candidate pixels.

15. The apparatus according to claim 13, wherein in response to having more than one candidate pixel, the view synthesizer unit is further arranged to

reconstruct at least one reconstructed pixel on the basis of the warped depth values of the visible candidate pixels.

16. The apparatus according to claim 11, wherein the view synthesizer unit is further arranged to

determine a pinhole pixel position in the synthesized image;

and

determine a pixel value of the pinhole pixel position based on a selected depth range.

17. The apparatus according to claim 16, wherein the view synthesizer unit is further arranged to

select a second set of candidate pixels from the set of pixels within the selected depth range that are in the proximity of the pinhole pixel position;

and

derive a pinhole pixel value from the second set of candidate pixels.

18. The apparatus according to claim 17, wherein the view synthesizer unit is further arranged to

select the depth range such that the depth range is the minimum of all the ranges that cover the second set of candidate pixels.

19. The apparatus according to claim 17, wherein the view synthesizer unit is further arranged to

select the depth range to comprise a maximum number of candidate pixels among the second set of candidate pixels.

20. The apparatus according to claim 17, wherein in response to having more than one candidate pixel, the view synthesizer unit is further arranged to

reconstruct at least one reconstructed pixel on the basis of the warped depth values of the second set of candidate pixels.

21. A decoder comprising:

a receiving unit for receiving and decoding at least one encoded reference image having a corresponding depth image;

a view synthesizer unit for creating a synthesized image;

the view synthesizer unit being arranged to

synthesize a set of pixels, each associated with a warped depth value, based on the at least one reference image and the corresponding depth image;

segment the warped depth values of the set of pixels into a plurality of depth ranges; and

transform the at least one reference image into a synthesized image based on at least one depth range.

22. A computer readable medium comprising a computer program stored therein, said computer program being executable in a data processing device, the computer program comprising:

a computer program code section for receiving at least one reference image having a corresponding depth image;

a computer program code section for synthesizing a set of pixels, each associated with a warped depth value, based on the at least one reference image and the corresponding depth image;

a computer program code section for segmenting the warped depth values of the set of pixels into a plurality of depth ranges; and

a computer program code section for transforming the at least one reference image into a synthesized image based on at least one depth range.

23. A chipset comprising:

a view synthesizer functionality for creating a synthesized image;

the view synthesizer functionality being arranged to

receive at least one reference image having a corresponding depth image;

synthesize a set of pixels, each associated with a warped depth value, based on the at least one reference image and the corresponding depth image;

segment the warped depth values of the set of pixels into a plurality of depth ranges; and

transform the at least one reference image into a synthesized image based on at least one depth range.