TRANSFORMING VIDEO DATA IN ACCORDANCE WITH HUMAN VISUAL SYSTEM FEEDBACK METRICS

• US 20110032328A1
• Filed: 05/13/2010
• Published: 02/10/2011
• Est. Priority Date: 08/06/2009
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Soliman, Samir S., Martinez Bauza, Judit, Raveendran, Vijayalakshmi R.
• Granted Patent: US 8,629,899 B2
• Time in Patent Office: Days
• US Class Current: 348/43
• CPC Class Codes: H04N 13/10 Processing, recording or transmission of stereoscopic or multi-view image signals
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof

Abstract

In general, techniques are described for transforming video data in accordance with human visual system feedback metrics. For example, an apparatus comprising a transformation module, a parameter discovery module and a human visual system (HVS) feedback module implements these techniques. The parameter discovery module configures the transformation module to generate three-dimensional (3D) video data in accordance with parameters defining capabilities supported by a 3D display device. The transformation module transforms video data to generate the 3D video data. The HVS feedback module determines, while the transformation module transforms the video data, one or more metrics using an HVS model that reflects a quality of 3D visualization of the generated 3D video data with respect to a human visual system and reconfigures the one or more modules based on the determined one or more metrics to refine the generation of the 3D video data.

Claims

1. A method comprising:

configuring a video processing device to generate three-dimensional (3D) video data in accordance with parameters defining capabilities supported by a 3D display device;

transforming, with the configured video processing device, video data to generate the 3D video data;

forwarding the 3D video data to the 3D display device;

while transforming the video data, determining one or more metrics using a Human Visual System (HVS) model that reflects a quality of 3D visualization of the generated 3D video data with respect to a human visual system; and

while transforming the video data, reconfiguring the one or more modules based on the determined one or more metrics to refine the generation of the 3D video data.

2. The method of claim 1, wherein determining one or more metrics comprises:

communicating with the 3D display device to receive one or more of the transformed 3D video data and decoded 3D video data from the 3D display device, wherein the decoded 3D video data comprises a decoded version of the transformed 3D video data forwarded by the device to the 3D display device; and

analyzing one or more of the transformed 3D video data and the decoded 3D video data to determine one or more qualitative metrics using the HVS model, wherein the qualitative metrics describe the quality of 3D visualization of the generated 3D video data with respect to the human visual system.

3. The method of claim 2, wherein the qualitative metrics comprise visual discomfort measured in term of at least one of a binocular geometric asymmetry, a binocular filter asymmetry, excessive binocular disparity, fast motion in depth, a 3D artifact, and an amount and consistency of blurriness,

wherein the binocular geometric asymmetry includes one or more of different lens distortion, different focal length that generates different magnification, and un-rectified images,

wherein the binocular filter asymmetry includes one or more of a different luminance, a different contrast, a color difference, a cross-talk, and a different quality related to compression artifacts, and

wherein the 3D artifact comprises spatial and temporal inconsistencies in depth cues.

4. The method of claim 2, further comprising:

presenting a user interface that receives input data defining one or more user preferences; and

configuring the video processing device to transform the video data into the 3D video data so as to generate the 3D video data in accordance with the defined user preferences.

5. The method of claim 4, further comprising configuring the HVS model in accordance with the defined user preferences so as to generate configuration data,

wherein determining one or more metrics comprises determining the one or more metrics using the configured Human Visual System (HVS) model that reflect a quality of 3D visualization of the generated 3D video data with respect to a human visual system, and

wherein reconfiguring the video processing device comprises reconfiguring the video processing device using the generated configuration data to reconfigure the video processing device based at least partially on these user preferences.

6. The method of claim 4, wherein the user preferences include at least one of a desired sharpness, a desired contrast, a desired display format, a desired color temperature, desired color settings, a desired brightness, a desired maximum depth range, a desired location of the convergence plane, a desired degree of smoothness of background objects, a desired eye-distance configuration, a desired dominant-eye configuration, a desired number of views, a desired viewing distance, a desired viewing angles, and a desired display location and resolution of the 3D video content within the screen.

7. The method of claim 1, further comprising:

receiving, with the HVS feedback model, the 3D video data generated by the one or more modules; and

determining, with the HVS feedback model, quantitative metrics based on the received 3D video data, wherein the quantitative metrics include at least one of metrics related to a relationship between a depth and a size of an object defined by the 3D video data, a relationship between a depth discontinuity and a mean region color discontinuity of the 3D video data, a filter efficacy metric, an interpolation efficacy metric, metrics useful in normalization of depth, and metrics related to measuring discontinuities of depth along time,

wherein reconfiguring the one or more modules comprises reconfiguring the one or more modules based at least partially on the quantitative metrics.

8. The method of claim 1, further comprising automatically determining the parameters that define the capabilities supported by the 3D display device, wherein the one or more parameters include an input format supported by the 3D display device, and

wherein configuring the video processing device comprises configuring the video processing device to generate three-dimensional (3D) video data in accordance with the determined input file format supported by the 3D display device.

9. The method of claim 8, wherein the input format comprises one of a 2D-plus-depth input file format, a 2D-plus-depth-occlusion-and-global-effects input file format, and a multiview stream input file format.

10. The method of claim 1, wherein forwarding the 3D video data comprising forwarding the 3D video data to the 3D display device as the 3D video data is generated.

11. The method of claim 1, further comprising receiving one or more qualitative metrics determined by another HVS model included within the 3D display device, wherein the qualitative metrics describe the quality of 3D visualization of the generated 3D video data with respect to the human visual system,

wherein the qualitative metrics comprise visual discomfort measured in term of at least one of a binocular geometric asymmetry, a binocular filter asymmetry, excessive binocular disparity, fast motion in depth, a 3D artifact, and an amount and consistency of blurriness,

wherein the binocular geometric asymmetry includes one or more of different lens distortion, different focal length that generates different magnification, and un-rectified images,

wherein the binocular filter asymmetry includes one or more of a different luminance, a different contrast, a color difference, a cross-talk, and a different quality related to compression artifacts, and

wherein the 3D artifact comprises spatial and temporal inconsistencies in depth cues.

12. The method of claim 1, wherein the device comprises one of a mobile cellular phone, a mobile computing device, a portable device and a handset device.

13. An apparatus comprising:

a transformation module;

a parameter discovery module that configures the transformation module to generate three-dimensional (3D) video data in accordance with parameters defining capabilities supported by a 3D display device,

wherein the transformation module transforms video data to generate the 3D video data;

at least one interface that forwards the 3D video data to the 3D display device; and

a human visual system (HVS) feedback module that determines, while the transformation module transforms the video data, one or more metrics using an HVS model that reflects a quality of 3D visualization of the generated 3D video data with respect to a human visual system and, again while the transformation module transforms the video data, reconfigures the one or more modules based on the determined one or more metrics to refine the generation of the 3D video data.

14. The apparatus of claim 13, wherein the HVS feedback module communicates with the 3D display device to receive one or more of the transformed 3D video data and decoded 3D video data from the 3D display device, wherein the decoded 3D video data comprises a decoded version of the transformed 3D video data forwarded by the device to the 3D display device, and analyzes one or more of the transformed 3D video data and the decoded 3D video data to determine one or more qualitative metrics using the HVS model, wherein the qualitative metrics describe the quality of 3D visualization of the generated 3D video data with respect to the human visual system.

15. The apparatus of claim 14, wherein the qualitative metrics comprise visual discomfort measured in term of at least one of a binocular geometric asymmetry, a binocular filter asymmetry, excessive binocular disparity, fast motion in depth, a 3D artifact, and an amount and consistency of blurriness,

wherein the binocular geometric asymmetry includes one or more of different lens distortion, different focal length that generates different magnification, and un-rectified images,

wherein the binocular filter asymmetry includes one or more of a different luminance, a different contrast, a color difference, a cross-talk, and a different quality related to compression artifacts, and

wherein the 3D artifact comprises spatial and temporal inconsistencies in depth cues.

16. The apparatus of claim 14, further comprising a user interface module that presents a user interface to receive input data defining one or more user preferences,

wherein the HVS feedback module configures the transformation module to transform the video data into the 3D video data so as to generate the 3D video data in accordance with the defined user preferences.

17. The apparatus of claim 16,

wherein the HVS feedback module configures the HVS model in accordance with the defined user preferences so as to generate configuration data,

wherein the HVS feedback module determines the one or more metrics using a Human Visual System (HVS) model that reflect a quality of 3D visualization of the generated 3D video data with respect to a human visual system, and

wherein the HVS feedback module reconfigures the video processing device using the generated configuration data to reconfigure the video processing device based at least partially on these user preferences.

18. The apparatus of claim 16, wherein the user preferences include at least one of a desired sharpness, a desired contrast, a desired display format, a desired color temperature, desired color settings, a desired brightness, a desired maximum depth range, a desired location of the convergence plane, a desired degree of smoothness of background objects, a desired eye-distance configuration, a desired dominant-eye configuration, a desired number of views, a desired viewing distance, a desired viewing angles, and a desired display location and resolution of the 3D video content within the screen.

19. The apparatus of claim 13,

wherein the HVS feedback model includes a quantitative evaluation module that receives the 3D video data generated by the transformation module and determines quantitative metrics based on the received 3D video data, wherein the quantitative metrics include at least one of metrics related to a relationship between a depth and a size of an object defined by the 3D video data, a relationship between a depth discontinuity and a mean region color discontinuity of the 3D video data, a filter efficacy metric, an interpolation efficacy metric, metrics useful in normalization of depth, and metrics related to measuring discontinuities of depth along time,

wherein the HVS feedback module reconfigures the one or more modules based at least partially on the quantitative metrics.

20. The apparatus of claim 13, further comprising a parameter discovery module that automatically determines the parameters that define the capabilities supported by the 3D display device, wherein the one or more parameters include an input format supported by the 3D display device and configures the video processing device to generate three-dimensional (3D) video data in accordance with the determined input file format supported by the 3D display device.

21. The apparatus of claim 20, wherein the input format comprises one of a 2D-plus-depth input file format, a 2D-plus-depth-occlusion-and-global-effects input file format, and a multiview stream input file format.

22. The apparatus of claim 13, wherein the at least one interface forwards the 3D video data to the 3D display device as the 3D video data is generated.

23. The apparatus of claim 13, wherein the HVS feedback module receives one or more qualitative metrics determined by another HVS model included within the 3D display device, wherein the qualitative metrics describe the quality of 3D visualization of the generated 3D video data with respect to the human visual system,

wherein the qualitative metrics comprise visual discomfort measured in term of at least one of a binocular geometric asymmetry, a binocular filter asymmetry, excessive binocular disparity, fast motion in depth, a 3D artifact, and an amount and consistency of blurriness,

wherein the binocular geometric asymmetry includes one or more of different lens distortion, different focal length that generates different magnification, and un-rectified images,

wherein the binocular filter asymmetry includes one or more of a different luminance, a different contrast, a color difference, a cross-talk, and a different quality related to compression artifacts, and

wherein the 3D artifact comprises spatial and temporal inconsistencies in depth cues.

24. The apparatus of claim 13, wherein the apparatus comprises one of a video processing device, a mobile cellular phone, a mobile computing device, a portable device and a handset device.

25. A computer-readable storage medium comprising instructions that cause a processor to:

configure a video processing device to generate three-dimensional (3D) video data in accordance with parameters defining capabilities supported by a 3D display device;

transform, with the configured video processing device, video data to generate the 3D video data;

forward the 3D video data to the 3D display device;

while transforming the video data, determine one or more metrics using a Human Visual System (HVS) model that reflects a quality of 3D visualization of the generated 3D video data with respect to a human visual system; and

while transforming the video data, reconfigure the one or more modules based on the determined one or more metrics to refine the generation of the 3D video data.

26. The computer-readable storage medium of claim 25, wherein the instructions also cause the processor to:

communicate with the 3D display device to receive one or more of the transformed 3D video data and decoded 3D video data from the 3D display device, wherein the decoded 3D video data comprises a decoded version of the transformed 3D video data forwarded by the device to the 3D display device; and

analyze one or more of the transformed 3D video data and the decoded 3D video data to determine one or more qualitative metrics using the HVS model, wherein the qualitative metrics describe the quality of 3D visualization of the generated 3D video data with respect to the human visual system.

27. The computer-readable storage medium of claim 26, wherein the qualitative metrics comprise visual discomfort measured in term of at least one of a binocular geometric asymmetry, a binocular filter asymmetry, excessive binocular disparity, fast motion in depth, a 3D artifact, and an amount and consistency of blurriness,

wherein the binocular geometric asymmetry includes one or more of different lens distortion, different focal length that generates different magnification, and un-rectified images,

wherein the binocular filter asymmetry includes one or more of a different luminance, a different contrast, a color difference, a cross-talk, and a different quality related to compression artifacts, and

wherein the 3D artifact comprises spatial and temporal inconsistencies in depth cues.

28. The computer-readable storage medium of claim 26, wherein the instruction further cause the processor to:

present a user interface that receives input data defining one or more user preferences; and

configure the video processing device to transform the video data into the 3D video data so as to generate the 3D video data in accordance with the defined user preferences.

29. The computer-readable storage medium of claim 28, wherein the instructions further cause the processor to:

configure the HVS model in accordance with the defined user preferences so as to generate configuration data;

determine the one or more metrics using the configured Human Visual System (HVS) model that reflect a quality of 3D visualization of the generated 3D video data with respect to a human visual system; and

reconfigure the video processing device using the generated configuration data to reconfigure the video processing device based at least partially on these user preferences.

30. The computer-readable storage medium of claim 28, wherein the user preferences include at least one of a desired sharpness, a desired contrast, a desired display format, a desired color temperature, desired color settings, a desired brightness, a desired maximum depth range, a desired location of the convergence plane, a desired degree of smoothness of background objects, a desired eye-distance configuration, a desired dominant-eye configuration, a desired number of views, a desired viewing distance, a desired viewing angles, and a desired display location and resolution of the 3D video content within the screen.

31. The computer-readable storage medium of claim 25, wherein the instructions further cause the processor to:

receive, with the HVS feedback model, the 3D video data generated by the one or more modules;

determine, with the HVS feedback model, quantitative metrics based on the received 3D video data, wherein the quantitative metrics include at least one of metrics related to a relationship between a depth and a size of an object defined by the 3D video data, a relationship between a depth discontinuity and a mean region color discontinuity of the 3D video data, a filter efficacy metric, an interpolation efficacy metric, metrics useful in normalization of depth, and metrics related to measuring discontinuities of depth along time; and

reconfigure the one or more modules based at least partially on the quantitative metrics.

32. The computer-readable storage medium of claim 25, wherein the instructions further cause the processor to:

automatically determine the parameters that define the capabilities supported by the 3D display device, wherein the one or more parameters include an input format supported by the 3D display device; and

configure the video processing device to generate three-dimensional (3D) video data in accordance with the determined input file format supported by the 3D display device.

33. The computer-readable storage medium of claim 32, wherein the input format comprises one of a 2D-plus-depth input file format, a 2D-plus-depth-occlusion-and-global-effects input file format, and a multiview stream input file format.

34. The computer-readable storage medium of claim 25, wherein the instructions further cause the processor to forward the 3D video data to the 3D display device as the 3D video data is generated.

35. The computer-readable storage medium of claim 25, wherein the instructions further cause the processor to receive one or more qualitative metrics determined by another HVS model included within the 3D display device, wherein the qualitative metrics describe the quality of 3D visualization of the generated 3D video data with respect to the human visual system,

wherein the qualitative metrics comprise visual discomfort measured in term of at least one of a binocular geometric asymmetry, a binocular filter asymmetry, excessive binocular disparity, fast motion in depth, a 3D artifact, and an amount and consistency of blurriness,

wherein the binocular geometric asymmetry includes one or more of different lens distortion, different focal length that generates different magnification, and un-rectified images,

wherein the binocular filter asymmetry includes one or more of a different luminance, a different contrast, a color difference, a cross-talk, and a different quality related to compression artifacts, and

wherein the 3D artifact comprises spatial and temporal inconsistencies in depth cues.

36. The computer-readable storage medium of claim 25, wherein the device comprises one of a mobile cellular phone, a mobile computing device, a portable device and a handset device.

37. An apparatus comprising:

means for configuring a video processing device to generate three-dimensional (3D) video data in accordance with parameters defining capabilities supported by a 3D display device;

means for transforming, with the configured video processing device, video data to generate the 3D video data;

means for forwarding the 3D video data to the 3D display device;

means for determining, while transforming the video data, one or more metrics using a Human Visual System (HVS) model that reflects a quality of 3D visualization of the generated 3D video data with respect to a human visual system; and

means for reconfiguring, while transforming the video data, the one or more modules based on the determined one or more metrics to refine the generation of the 3D video data.

38. The apparatus of claim 37, wherein the means for determining one or more metrics comprises:

means for communicating with the 3D display device to receive one or more of the transformed 3D video data and decoded 3D video data from the 3D display device, wherein the decoded 3D video data comprises a decoded version of the transformed 3D video data forwarded by the device to the 3D display device; and

means for analyzing one or more of the transformed 3D video data and the decoded 3D video data to determine one or more qualitative metrics using the HVS model, wherein the qualitative metrics describe the quality of 3D visualization of the generated 3D video data with respect to the human visual system.

39. The apparatus of claim 38, wherein the qualitative metrics comprise visual discomfort measured in term of at least one of a binocular geometric asymmetry, a binocular filter asymmetry, excessive binocular disparity, fast motion in depth, a 3D artifact, and an amount and consistency of blurriness,

wherein the binocular geometric asymmetry includes one or more of different lens distortion, different focal length that generates different magnification, and un-rectified images,

wherein the binocular filter asymmetry includes one or more of a different luminance, a different contrast, a color difference, a cross-talk, and a different quality related to compression artifacts, and

wherein the 3D artifact comprises spatial and temporal inconsistencies in depth cues.

40. The apparatus of claim 38, further comprising:

means for presenting a user interface that receives input data defining one or more user preferences; and

means for configuring the video processing device to transform the video data into the 3D video data so as to generate the 3D video data in accordance with the defined user preferences.

41. The apparatus of claim 40, further comprising means for configuring the HVS model in accordance with the defined user preferences so as to generate configuration data,

wherein the means for determining one or more metrics comprises means for determining the one or more metrics using the configured Human Visual System (HVS) model that reflect a quality of 3D visualization of the generated 3D video data with respect to a human visual system, and

wherein the means for reconfiguring the video processing device comprises means for reconfiguring the video processing device using the generated configuration data to reconfigure the video processing device based at least partially on these user preferences.

42. The apparatus of claim 40, wherein the user preferences include at least one of a desired sharpness, a desired contrast, a desired display format, a desired color temperature, desired color settings, a desired brightness, a desired maximum depth range, a desired location of the convergence plane, a desired degree of smoothness of background objects, a desired eye-distance configuration, a desired dominant-eye configuration, a desired number of views, a desired viewing distance, a desired viewing angles, and a desired display location and resolution of the 3D video content within the screen.

43. The apparatus of claim 37, further comprising:

means for receiving, with the HVS feedback model, the 3D video data generated by the one or more modules; and

means for determining, with the HVS feedback model, quantitative metrics based on the received 3D video data, wherein the quantitative metrics include at least one of metrics related to a relationship between a depth and a size of an object defined by the 3D video data, a relationship between a depth discontinuity and a mean region color discontinuity of the 3D video data, a filter efficacy metric, an interpolation efficacy metric, metrics useful in normalization of depth, and metrics related to measuring discontinuities of depth along time,

wherein the means for reconfiguring the one or more modules comprises means for reconfiguring the one or more modules based at least partially on the quantitative metrics.

44. The apparatus of claim 37, further comprising means for automatically determining the parameters that define the capabilities supported by the 3D display device, wherein the one or more parameters include an input format supported by the 3D display device, and

wherein the means for configuring the video processing device comprises means for configuring the video processing device to generate three-dimensional (3D) video data in accordance with the determined input file format supported by the 3D display device.

45. The apparatus of claim 44, wherein the input format comprises one of a 2D-plus-depth input file format, a 2D-plus-depth-occlusion-and-global-effects input file format, and a multiview stream input file format.

46. The apparatus of claim 37, wherein the means for forwarding the 3D video data comprising means for forwarding the 3D video data to the 3D display device as the 3D video data is generated.

47. The apparatus of claim 37, further comprising means for receiving one or more qualitative metrics determined by another HVS model included within the 3D display device, wherein the qualitative metrics describe the quality of 3D visualization of the generated 3D video data with respect to the human visual system,

wherein the qualitative metrics comprise visual discomfort measured in term of at least one of a binocular geometric asymmetry, a binocular filter asymmetry, excessive binocular disparity, fast motion in depth, a 3D artifact, and an amount and consistency of blurriness,

wherein the binocular geometric asymmetry includes one or more of different lens distortion, different focal length that generates different magnification, and un-rectified images,

wherein the binocular filter asymmetry includes one or more of a different luminance, a different contrast, a color difference, a cross-talk, and a different quality related to compression artifacts, and

wherein the 3D artifact comprises spatial and temporal inconsistencies in depth cues.

48. The apparatus of claim 37, wherein the apparatus comprises one of a mobile cellular phone, a mobile computing device, a portable device and a handset device.