VISION-BASED QUALITY METRIC FOR THREE DIMENSIONAL VIDEO

• US 20120014456A1
• Filed: 04/26/2011
• Published: 01/19/2012
• Est. Priority Date: 07/16/2010
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Martinez Bauza, Judit, Solh, Mashhour M.
• Granted Patent: US 9,406,132 B2
• Time in Patent Office: Days
• US Class Current: 375/240.25
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 19/42 characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation H04N19/635 takes precedence
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding

Abstract

In general, techniques are described for determining a vision-based quality metric for three-dimensional (3D) video. A device (12, 14) comprising a 3D analysis unit (24, 48) may implement these techniques to compute a 3D objective metric (36, 60). The 3D analysis unit (24, 48) estimates an ideal depth map (62, 70) that would generate a distortion limited image view using DIBR-based 3D video data (34) and then derives one or more distortion metrics (64, 72) based on quantitative comparison of the ideal depth map (62, 70) and a depth map (28, 58) used in the generation of the DIBR-based 3D video data (34). Based on the derived one or more distortion metrics (64, 72), the 3D analysis unit (24, 48) computes the objective metric (36, 60) to quantify visual quality of DIBR-based video data (34). The 3D objective metric (36, 60) may be used to alter or otherwise adjust depth estimation and encoding/decoding parameters to correct for expected visual discomfort.

Claims

1. A method for obtaining an objective metric (36, 60) to quantify the visual quality of depth-image-based rendering (DIBR)-based three-dimensional (3D) video data, the method comprising:

estimating an ideal depth map (62, 70) that would generate a distortion limited image view using DIBR-based 3D video data (138, 156);

deriving one or more distortion metrics (64, 72) based on quantitative comparison of the ideal depth map (62, 70) to a depth map (28, 58) used in the generation of the DIBR-based 3D video data (34, 140, 158); and

computing the objective metric (36, 60) to quantify visual quality of the DIBR-based 3D video data based on the derived one or more distortion metrics (64, 142, 160).

2. The method of claim 1, wherein each of the distortion metrics indicates a measurement of 3D viewing visual discomfort expected when viewing the DIBR-based 3D video data.

3. The method of claim 1, wherein estimating the ideal depth map includes estimating the ideal depth map in accordance with the following equation:

4. The method of claim 1, wherein estimating the ideal depth map includes estimating the ideal depth map in accordance with the following equation:

5. The method of claim 1, wherein estimating the ideal depth map includes estimating the ideal depth map in accordance with the following equation:

6. The method of claim 1, wherein deriving the one or more distortion metrics comprises deriving one or more of a spatial error outliers (SO) distortion metric and a temporal error outliers (TO) distortion metric.

7. The method of claim 1,

wherein deriving the one or more distortion metrics comprises:

deriving a spatial error outliers (SO) distortion metric as standard deviation of a difference between an ideal depth map and the depth map of the DIBR-based 3D video data; and

deriving a temporal error outliers (TO) distortion metric as a standard deviation of a difference between the ideal depth map and the depth map of the DIBR-based 3D video data for two consecutive frames of the 3D video,

wherein the method further comprises deriving a temporal inconsistency (TI) distortion metric as a standard deviation of the difference between depth maps used to generate DIBR-based 3D video data and a successive DIBR-based 3D video data,

wherein computing the objective metric comprises;

normalizing each of the SO, TO and TI distortion metrics to a range of zero to one; and

computing the objective metric in accordance with the following equation;

3VQM=K(1−

SO(SO∩

TO))^a(1−

TI)^b(1−

TO)^c,

wherein 3VQM represents the objective metric, SO, TO, and SI represent the normalized SO, TO and TI distortion metrics and a, b, and c represent constants determined by training sequences and k represents a scaling factor.

8. The method of claim 1,

wherein the DIBR-based 3D video data further includes a provided view, and

wherein the method further comprises:

estimating the depth map from 3D video data in accordance with one or more parameters, wherein the 3D video data includes a left view and a right view, wherein the provided view is selected from one of the left and right views; and

revising the one or more parameters used to estimate the depth map for the 3D video data so as to correct for deficiencies identified by the objective metric.

9. The method of claim 1,

wherein the DIBR-based 3D video data further includes a provided view, and

wherein the method further comprises:

encoding the depth map and the provided view of the DIBR-based 3D video data in accordance with one or more parameters to generate encoded DIBR-based 3D video data; and

revising the one or more parameters used to encode one or more of the depth map and the provided view to correct for deficiencies identified by the objective metric.

10. The method of claim 1,

wherein the 3D video data further includes a provided view, and

wherein the method further comprises:

decoding an encoded version of the depth map and an encoded version of the provided view in accordance with one or more parameters to generate the depth map and the provided view; and

revising the one or more parameters used to decode the encoded version of the depth map and the encoded version of the provided view so as to correct for deficiencies identified by the objective metric.

11. The method of claim 1,

wherein the DIBR-based 3D video data further includes a provided view, and

wherein the method further comprises:

reconstructing either a left view or a right view of 3D video data from the depth map and the provided view in accordance with one or more parameters to generate the 3D video data, wherein the provided view is selected as either the left view or the right view of the 3D video data; and

revising the one or more parameters used to reconstruct either the left view or the right view to correct for deficiencies identified by the objective metric.

17. The device of claim 1,

wherein the distortion metric computation unit derives a spatial error outliers (SO) distortion metric as standard deviation of a difference between an ideal depth map and the depth map of the DIBR-based 3D video data, a temporal error outliers (TO) distortion metric as a standard deviation of a difference between the ideal depth map and the depth map of the DIBR-based 3D video data for two consecutive frames of the 3D video and a temporal inconsistency (TI) distortion metric as a standard deviation of the difference between depth maps used to generate DIBR-based 3D video data and a successive DIBR-based 3D video data,

wherein an objective metric computation unit normalizes each of the SO, TO and TI distortion metrics to a range of zero to one and computing the objective metric in accordance with the following equation:

3VQM=K(1−

SO(SO∩

TO))^a(1−

TI)^b(1−

TO)^c,

wherein 3VQM represents the objective metric, SO, TO, and SI represent the normalized SO, TO and TI distortion metrics and a, b, and c represent constants determined by training sequences and k represents a scaling factor.

12. A device that obtains an objective metric (36, 60) to quantify the visual quality of depth-image-based rendering (DIBR)-based three-dimensional (3D) video data, the device (12, 14) comprising:

a 3D analysis unit (24, 48) that computes the 3D objective metric (36, 60), wherein the 3D analysis unit (24, 48) includes;

an ideal depth estimation unit (82) that estimates an ideal depth map (62, 70) that would generate a distortion limited image view using DIBR-based 3D video data (34);

a distortion metric computation unit (84) that derives one or more distortion metrics (64, 72) based on quantitative comparison of the ideal depth (62, 70) and a depth map (28, 58) used in the generation of the DIBR-based 3D video data (34); and

an objective metric computation unit (86) that computes the objective metric (36, 60) to quantify visual quality of DIBR-based video data (34) based on the derived one or more distortion metrics (64, 72).

13. The device of claim 12, wherein each of the distortion metrics indicates a measurement of 3D viewing visual discomfort expected when viewing the DIBR-based 3D video data.

14. The device of claim 12, wherein the ideal depth map estimation unit estimates the ideal depth map in accordance with the following equation:

15. The device of claim 12, wherein the ideal depth map estimation unit estimates the ideal depth map in accordance with the following equation:

16. The device of claim 12, wherein the ideal depth map estimation unit estimates the ideal depth map in accordance with the following equation:

18. The device of claim 12,

wherein the DIBR-based 3D video data further includes a provided view, and

wherein the device further includes a depth estimation unit that estimates the depth map from 3D video data in accordance with one or more parameters, wherein the 3D video data includes a left view and a right view, wherein the provided view is selected from one of the left and right views and revises the one or more parameters used to estimate the depth map for the 3D video data so as to correct for deficiencies identified by the objective metric.

19. The device of claim 12,

wherein the DIBR-based 3D video data further includes a provided view, and

wherein the device further includes a 3D video encoder that encodes the depth map and the provided view of the DIBR-based 3D video data in accordance with one or more parameters to generate encoded DIBR-based 3D video data and revises the one or more parameters used to encode one or more of the depth map and the provided view to correct for deficiencies identified by the objective metric.

20. The device of claim 12,

wherein the 3D video data further includes a provided view, and

wherein the device further comprises a 3D video decoder that decodes an encoded version of the depth map and an encoded version of the provided view in accordance with one or more parameters to generate the depth map and the provided view and revises the one or more parameters used to decode the encoded version of the depth map and the encoded version of the provided view so as to correct for deficiencies identified by the objective metric.

21. The device of claim 12,

wherein the DIBR-based 3D video data further includes a provided view, and

wherein the device further comprises a view reconstruction unit that reconstructs either a left view or a right view of 3D video data from the depth map and the provided view in accordance with one or more parameters to generate the 3D video data, wherein the provided view is selected as either the left view or the right view of the 3D video data and revises the one or more parameters used to reconstruct either the left view or the right view to correct for deficiencies identified by the objective metric.

22. An apparatus that obtains an objective metric (36, 60) to quantify the visual quality of depth-image-based rendering (DIBR)-based three-dimensional (3D) video data, the apparatus (12, 14) comprising:

means for estimating (82) an ideal depth map (62, 70) that would generate a distortion limited image view using DIBR-based 3D video data (34);

means for deriving (84) one or more distortion metrics (64, 72) based on quantitative comparison of the ideal depth map to a depth map (62, 70) used in the generation of the DIBR-based 3D video data (34); and

means for computing (86) the objective metric (36, 60) to quantify visual quality of the DIBR-based 3D video data (34) based on the derived one or more distortion metrics (64, 72).

23. The apparatus of claim 22, wherein each of the distortion metrics indicates a measurement of 3D viewing visual discomfort expected when viewing the DIBR-based 3D video data.

24. The apparatus of claim 22, further comprising means for estimating the ideal depth map in accordance with the following equation:

25. The apparatus of claim 22, further comprising means for estimating the ideal depth map in accordance with the following equation:

26. The apparatus of claim 22, further comprising means for estimating the ideal depth map in accordance with the following equation:

27. The apparatus of claim 22, further comprising means for deriving one or more of a spatial error outliers (SO) distortion metric and a temporal error outliers (TO) distortion metric.

28. The apparatus of claim 22, further comprising:

means for deriving a spatial error outliers (SO) distortion metric as standard deviation of a difference between an ideal depth map and the depth map of the DIBR-based 3D video data; and

means for deriving a temporal error outliers (TO) distortion metric as a standard deviation of a difference between the ideal depth map and the depth map of the DIBR-based 3D video data for two consecutive frames of the 3D video;

means for deriving a temporal inconsistency (TI) distortion metric as a standard deviation of the difference between depth maps used to generate DIBR-based 3D video data and a successive DIBR-based 3D video data;

means for normalizing each of the SO, TO and TI distortion metrics to a range of zero to one; and

means for computing the objective metric in accordance with the following equation;

3VQM=K(1−

SO(SO∩

TO))^a(1−

TI)^b(1−

TO)^c,

wherein 3VQM represents the objective metric, SO, TO, and SI represent the normalized SO, TO and TI distortion metrics and a, b, and c represent constants determined by training sequences and k represents a scaling factor.

29. The apparatus of claim 22,

wherein the DIBR-based 3D video data further includes a provided view, and

wherein the apparatus further comprises:

means for estimating the depth map from 3D video data in accordance with one or more parameters, wherein the 3D video data includes a left view and a right view, wherein the provided view is selected from one of the left and right views; and

means for revising the one or more parameters used to estimate the depth map for the 3D video data so as to correct for deficiencies identified by the objective metric.

30. The apparatus of claim 22,

wherein the DIBR-based 3D video data further includes a provided view, and

wherein the apparatus further comprises:

means for encoding the depth map and the provided view of the DIBR-based 3D video data in accordance with one or more parameters to generate encoded DIBR-based 3D video data; and

means for revising the one or more parameters used to encode one or more of the depth map and the provided view to correct for deficiencies identified by the objective metric.

31. The apparatus of claim 22,

wherein the 3D video data further includes a provided view, and

wherein the apparatus further comprises:

means for decoding an encoded version of the depth map and an encoded version of the provided view in accordance with one or more parameters to generate the depth map and the provided view; and

means for revising the one or more parameters used to decode the encoded version of the depth map and the encoded version of the provided view so as to correct for deficiencies identified by the objective metric.

32. The method of claim 22,

wherein the DIBR-based 3D video data further includes a provided view, and

wherein the apparatus further comprises:

means for reconstructing either a left view or a right view of 3D video data from the depth map and the provided view in accordance with one or more parameters to generate the 3D video data, wherein the provided view is selected as either the left view or the right view of the 3D video data; and

means for revising the one or more parameters used to reconstruct either the left view or the right view to correct for deficiencies identified by the objective metric.

33. A non-transitory computer-readable medium comprising instructions that, when executed, cause one or more processor to:

estimate an ideal depth map (62, 70) that would generate a distortion limited image view using DIBR-based 3D video data (34);

derive one or more distortion metrics (64, 72) based on quantitative comparison of the ideal depth map (62, 70) to a depth map (28, 58) used in the generation of the DIBR-based 3D video data (34); and

compute the objective metric (36, 60) to quantify visual quality of the DIBR-based 3D video data (34) based on the derived one or more distortion metrics (64, 72).

34. The non-transitory computer-readable medium of claim 33, wherein each of the distortion metrics indicates a measurement of 3D viewing visual discomfort expected when viewing the DIBR-based 3D video data.

35. The non-transitory computer-readable medium of claim 33, further comprising instructions that cause the one or more processors to estimate the ideal depth map includes estimating the ideal depth map in accordance with the following equation:

36. The non-transitory computer-readable medium of claim 33, further comprising instructions that cause the one or more processors to estimate the ideal depth map in accordance with the following equation:

37. The non-transitory computer-readable medium of claim 33, non-transitory computer-readable medium the ideal depth map in accordance with the following equation:

38. The non-transitory computer-readable medium of claim 33, further comprising instructions that cause the one or more processors to derive one or more of a spatial error outliers (SO) distortion metric and a temporal error outliers (TO) distortion metric.

39. The non-transitory computer-readable medium of claim 33, further comprising instructions that cause the one or more processors to estimate

derive a spatial error outliers (SO) distortion metric as standard deviation of a difference between an ideal depth map and the depth map of the DIBR-based 3D video data;

derive a temporal error outliers (TO) distortion metric as a standard deviation of a difference between the ideal depth map and the depth map of the DIBR-based 3D video data for two consecutive frames of the 3D video;

derive a temporal inconsistency (TI) distortion metric as a standard deviation of the difference between depth maps used to generate DIBR-based 3D video data and a successive DIBR-based 3D video data,

normalize each of the SO, TO and TI distortion metrics to a range of zero to one; and

compute the objective metric in accordance with the following equation;

3VQM=K(1−

SO(SO∩

TO))^a(1−

TI)^b(1−

TO)^c,

wherein 3VQM represents the objective metric, SO, TO, and SI represent the normalized SO, TO and TI distortion metrics and a, b, and c represent constants determined by training sequences and k represents a scaling factor.

40. The non-transitory computer-readable medium of claim 33,

wherein the DIBR-based 3D video data further includes a provided view, and

further comprising instructions that cause the one or more processors to:

estimate the depth map from 3D video data in accordance with one or more parameters, wherein the 3D video data includes a left view and a right view, wherein the provided view is selected from one of the left and right views; and

revise the one or more parameters used to estimate the depth map for the 3D video data so as to correct for deficiencies identified by the objective metric.

41. The non-transitory computer-readable medium of claim 33,

wherein the DIBR-based 3D video data further includes a provided view, and

further comprising instructions that cause the one or more processors to:

encode the depth map and the provided view of the DIBR-based 3D video data in accordance with one or more parameters to generate encoded DIBR-based 3D video data; and

revise the one or more parameters used to encode one or more of the depth map and the provided view to correct for deficiencies identified by the objective metric.

42. The non-transitory computer-readable medium of claim 33,

wherein the 3D video data further includes a provided view, and

further comprising instructions that cause the one or more processors to:

decode an encoded version of the depth map and an encoded version of the provided view in accordance with one or more parameters to generate the depth map and the provided view; and

revise the one or more parameters used to decode the encoded version of the depth map and the encoded version of the provided view so as to correct for deficiencies identified by the objective metric.

43. The non-transitory computer-readable medium of claim 33,

wherein the DIBR-based 3D video data further includes a provided view, and

further comprising instructions that cause the one or more processors to:

reconstruct either a left view or a right view of 3D video data from the depth map and the provided view in accordance with one or more parameters to generate the 3D video data, wherein the provided view is selected as either the left view or the right view of the 3D video data; and

revise the one or more parameters used to reconstruct either the left view or the right view to correct for deficiencies identified by the objective metric.