SYSTEM AND METHOD FOR TRANSMISSION, PROCESSING, AND RENDERING OF STEREOSCOPIC AND MULTI-VIEW IMAGES

• US 20120008672A1
• Filed: 09/13/2010
• Published: 01/12/2012
• Est. Priority Date: 07/07/2010
• Status: Active Grant

• Current Assignee: Cheytec Technologies, LLC
• Original Assignee: A2Zlogix, Inc.
• Inventors: Gaddy, William L., Seran, Vidhya
• Granted Patent: US 8,774,267 B2
• Time in Patent Office: Days
• US Class Current: 375/240.01
• CPC Class Codes: G06T 1/0028 Adaptive watermarking, e.g. Human Visual System [HVS]-based watermarking
• CPC Class Codes: G06T 2201/0202 whereby the quality of watermarked images is measured; Measuring quality or performance of watermarking methods; Balancing between quality and robustness
• CPC Class Codes: H04N 13/106 Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/467 characterised by the embedded information being invisible, e.g. watermarking
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 2213/003 Aspects relating to the "2D+depth" image format

Abstract

A digital image processing system takes color plus Z channel data as input, preprocesses, decimates, and codes the Z channel in-band as digital watermark data embedded within the color data prior to encoding and transmission. A second digital image processing system receives, decodes, and extracts the decimated Z channel data before applying statistical regularization to restore a full-resolution Z channel prior to depth-image-based rendering.

Claims

1. A computer implemented method for transmitting three-dimensional (3D) imagery, comprising the steps of:

extracting a depth map and color data from at least one 3D image;

reducing bandwidth of the depth map to produce reduced bandwidth depth map;

inserting the reduced bandwidth depth map into the color data to produce a reduced bandwidth 3D image; and

transmitting the reduced bandwidth 3D image into a transmission channel for delivery to a display.

2. The method of claim 1, wherein the step of reducing bandwidth of the depth map comprises retaining at least one region of the depth map comprising at least one discontinuity that corresponds to at least one object boundary in the color data.

3. The method of claim 1, wherein the step of reducing bandwidth of the depth map further comprises removing depth values associated with a modality in a distribution representing depth values in the depth map.

4. The method of claim 3, further comprising identifying the modality by:

generating a histogram of depth values in the depth map, and

performing a modal analysis on the histogram.

5. The method of claim 4, further comprising applying a grayscale morphological closing operation on the histogram to remove 0.0 or 1.0 depth values.

6. The method of claim 3, further comprising filtering the depth map with a median image processing filter.

7. The method of claim 6, wherein the median image processing filter is a 5×

5 median filter.

8. The method of claim 3, further comprising:

performing a spatial decimation operation on the depth map, and

applying a lossless method of statistical coding on the depth map.

9. The method of claim 8, wherein the step of performing a spatial decimation operation on the depth map comprises at least one of applying a cubic reduction filter to the depth map and performing a repeated succession of one-octave bicubic reductions.

10. The method of claim 8, wherein the step of applying a lossless method of statistical coding on the depth map further comprises at least one of:

transforming the depth map to a 7-bit-per-pixel representation and encoding statistically with a Huffmann encoding scheme,

applying arithmetic coding to the depth map, and

applying a two-dimensional codebook encoding scheme to the depth map.

11. The method of claim 1, wherein the step of inserting the reduced bandwidth depth map into the color data comprises adding the depth map to color data as a watermark.

12. A computer implemented method for receiving three-dimensional (3D) imagery, comprising the steps of:

receiving a reduced bandwidth 3D image comprising a reduced bandwidth depth map and color data; and

applying a filter that employs a statistical domain of the color data to restore bandwidth of the reduced bandwidth depth map.

13. The method of claim 12, wherein the filter is configured to restore discontinuities in the reduced bandwidth depth map by matching discontinuities of the color data containing at least one object boundary.

14. The method of claim 12, further comprising the step of applying a depth-image-based rendering (DIBR) method to warp the restored depth map and the color image to construct at least one view suitable for stereoscopic or auto-stereoscopic 3D displays.

15. The method of claim 12, wherein the at least one filter is a RGBZO filter.

16. The method of claim 15, wherein the RGBZO filter employs two radiosity weighting functions provided by the reduced bandwidth depth map as a first domain, the color data as a second domain, and a spatial weighting function.

17. The method of claim 15, further comprising the step of repeatedly subjecting the reduced bandwidth depth map to iterations of one-octave bicubic up-scaling followed by application of the RGBZO filter until a first octave up-scaled depth map is produced.

18. The method of claim 17, wherein the step of repeatedly subjecting the reduced bandwidth depth map to iterations of one-octave bicubic up-scaling followed by application of the RGBZO filter minimizes an error function comprising a difference between an edge gradient of the depth map and a color edge gradient of the color data to regularize output of the up-scaled depth map.

19. The method of claim 17, further comprising the step of applying a lossless decoding method to the reduced bandwidth depth map before said step of repeatedly subjecting the reduced bandwidth depth map to iterations of one-octave bicubic up-scaling followed by application of the RGBZO filter.

20. The method of claim 14, wherein the step of applying a DIBR method further comprises the steps of:

applying an optical flow method to the color data;

applying motion compensation and image warping to the color to produce a table of motion compensated pixels;

applying one of temporal predictions and spatial predictions of candidate occluded pixels from the table of motion compensated pixels;

applying a statistical in-painting procedure to the candidate occluded pixels; and

warping pixels obtained from the statistical in-painting procedure to obtain left and right eye views of images for display.

21. The method of claim 20, further comprising the steps of:

classifying disocclusions from the depth map to inform spatial predictions of candidate occluded pixels; and

applying a Z smoothing method to the depth map to produce a processed depth map.

22. The method of claim 21, wherein the step of warping is informed by the processed depth map.

23. A computer-readable medium comprising computer-executable instructions for processing three-dimensional (3D) imagery, the instructions configured to:

receive a reduced bandwidth 3D image comprising a reduced bandwidth depth map and color data; and

apply a filter that employs a statistical domain of the color data to restore bandwidth of the reduced bandwidth depth map.

24. The computer-readable medium of claim 23, wherein the instructions are further configured to:

apply a depth-image-based rendering (DIBR) method to warp the restored depth map and the color image to construct at least one view suitable for stereoscopic or auto-stereoscopic 3D displays.

25. The computer-readable medium of claim 23, wherein the at least one filter is operable to employ a statistical domain of the color data to restore the bandwidth of the reduced bandwidth depth map.

26. The computer-readable medium of claim 23, wherein the filter is a RGBZO filter.

27. The computer-readable medium of claim 26, wherein the RGBZO employs two radiosity weighting functions provided by the reduced bandwidth depth map as a first domain, the color data as a second domain, and a spatial weighting function.

28. The computer-readable medium of claim 26, wherein the instructions are further configured to:

repeatedly subject the reduced bandwidth depth map to iterations of one-octave bicubic up-scaling followed by application of the RGBZO filter until a 1st octave up-scaled depth map is produced.

29. The computer-readable medium of claim 28, wherein repeatedly subjecting the reduced bandwidth depth map to iterations of one-octave bicubic up-scaling followed by application of the RGBZO filter minimizes an error function comprising a difference between an edge gradient of the depth map and a color edge gradient of the color data to regularize output of the up-scaled depth map.

30. A receiver for receiving and restoring at least one reduced bandwidth image comprising a reduced bandwidth depth map, the receiver configured to:

receive a reduced bandwidth 3D image comprising a reduced bandwidth depth map and color data; and

apply a filter that employs a statistical domain of the color data to restore bandwidth of the reduced bandwidth depth map.

31. The receiver of claim 30, wherein restoring bandwidth of the reduced bandwidth depth map is based on restoring discontinuities in the reduced bandwidth depth map by matching discontinuities of the color data containing at least one object boundary.

32. The receiver of claim 30, further comprising a depth-image-based rendering (DIBR) module for warping the restored depth map and the color data to construct at least one view suitable for stereoscopic or auto-stereoscopic 3D displays.

33. The receiver of claim 30, wherein the filter is a RGBZO filter.

34. The receiver of claim 33, wherein the RGBZO employs two radiosity weighting functions provided by the depth map as a first domain, the color data as a second domain, and a spatial weighting function.

35. The receiver of claim 33, further comprising:

a module for repeatedly subjecting the reduced bandwidth depth map to iterations of one-octave bicubic up-scaling followed by application of the RGBZO filter until a first octave up-scaled depth map is produced.

37. The receiver of claim 30, wherein the receiver is implemented using an application-specific integrated circuit (ASIC).

36. The receiver of claim 36, wherein the module for repeatedly subjecting the depth map to iterations of one-octave bicubic up-scaling followed by application of the RGBZO filter minimizes an error function comprising a difference between an edge gradient of the depth map and a color edge gradient of the color data to regularize output of the up-scaled depth map.

38. A system for transmitting, receiving, and rendering 3D imagery, comprising:

a transmitter for;

extracting a depth map and color data from at least one 3D image,

reducing bandwidth of the depth map to produce a reduced bandwidth depth map,

inserting the reduced bandwidth depth map into the color data to produce a reduced bandwidth 3D image, and

transmitting the reduced bandwidth 3D image into a transmission channel; and

a receiver for;

receiving a reduced bandwidth 3D image comprising a reduced bandwidth depth map and color data from the transmission channel, and

applying a filter that employs a statistical domain of the color data to restore bandwidth of the reduced bandwidth depth map, and

rendering the restored depth map and color data on a display.

39. The system of claim 38, wherein reducing bandwidth of the depth map comprises retaining at least one region of the depth map comprising at least one discontinuity that corresponds to at least one object boundary in the color data.

40. The system of claim 38, wherein the filter is configured to restore discontinuities in the reduced bandwidth depth map by matching discontinuities of the color data containing at least one object boundary.

41. A RGBZO filter configured to employ a statistical domain of the color data to restore bandwidth of a reduced bandwidth depth map.

42. The RGBZO filter of claim 41, wherein the RGBZO filter employs two radiosity weighting functions provided by the reduced bandwidth depth map as a first domain, the color data as a second domain, and a spatial weighting function.