Frame-compatible full-resolution stereoscopic 3D video delivery with symmetric picture resolution and quality

• US 20140286397A1
• Filed: 09/27/2012
• Published: 09/25/2014
• Est. Priority Date: 09/29/2011
• Status: Active Grant

• Current Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Original Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Inventors: Chen, Tao, Lakshminarayanan, Gopi, Yin, Peng, Lu, Taoran, Husak, Walter J., Ganapathy, I, Hariharan, Hulyalkar, Samir N.
• Granted Patent: US 10,097,820 B2
• Time in Patent Office: Days
• US Class Current: 375/240.02
• CPC Class Codes: H04N 13/106 Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 19/119 Adaptive subdivision aspects, e.g. subdivision of a picture into rectangular or non-rectangular coding blocks
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding

Abstract

A high resolution 3D image may be encoded into a first multiplexed image frame and a second multiplexed image frame in a base layer (BL) video signal and an enhancement layer (EL) video signal. The first multiplexed image frame may comprise horizontal high resolution image data for both eyes, while the second multiplexed image frame may comprise vertical high resolution image data for both eyes. Encoded symmetric-resolution image data for the 3D image may be distributed to a wide variety of devices for 3D image processing and rendering. A recipient device may reconstruct reduced resolution 3D image from one of the first multiplexed image frame or the second multiplexed image frame. A recipient device may also reconstruct high resolution 3D image by combining high resolution image data from both of the first multiplexed image frame and the second multiplexed image frame.

Claims

1-35. -35. (canceled)

36. A method using a processor, the method comprising:

receiving an input 3D image, the input 3D image comprising a left eye (LE) input image frame and a right eye (RE) input image frame;

generating, based on the LE input image frame and the RE input image frame, a first multiplexed image frame comprising first high spatial frequency content in a vertical direction and first reduced resolution content in a horizontal direction;

generating, based on the LE input image frame and the RE input image frame, a second multiplexed image frame comprising second high spatial frequency content in the horizontal direction and second reduced resolution content in the vertical direction;

wherein one of the first multiplexed image frame or the second multiplexed image frame comprises residual image data, wherein the residual image data is generated by subtracting reference image data generated based on the other of the first multiplexed image frame or the second multiplexed image frame from input image data derived from the LE input image frame and the RE input image frame; and

encoding and outputting the first multiplexed image frame and the second multiplexed image frame to represent the input 3D image.

37. The method as recited in claim 36, wherein the 3D input image is a first 3D input image in a sequence of 3D input images comprising a second different 3D input image having a second LE input image frame and a second RE input image frame;

and the method further comprising;

generating, based on the second LE input image frame and the second RE input image frame, a third multiplexed image frame comprising third high spatial frequency content in the vertical direction and third reduced resolution content in the horizontal direction;

generating, based on the second LE input image frame and the second RE input image frame, a fourth multiplexed image frame comprising fourth high spatial frequency content in the horizontal direction and fourth reduced resolution content in the vertical direction; and

encoding and outputting the third multiplexed image frame and the fourth multiplexed image frame to represent the second input 3D image.

38. The method as recited in claim 36, wherein the first multiplexed image frame comprises a first LE image data portion and a first RE image data portion;

wherein the first LE image data portion and the first RE image data portion are of a same spatial resolution along both horizontal and vertical directions;

wherein the second multiplexed image frame comprises a second LE image data portion and a second RE image data portion; and

wherein the second LE image data portion and the second RE image data portion are of a same spatial resolution along both horizontal and vertical directions.

39. The method as recited in claim 38, wherein each of the first LE image data portion and the first RE image data portion represents a subsampled version of a full resolution image frame;

wherein the first multiplexed image frame adopts a side-by-side (SbS) format to carry the first LE image data portion and the first RE image data portion;

wherein each of the second LE image data portion and the second RE image data portion represents a subsampled version of a full resolution image frame; and

wherein the second multiplexed image frame adopts a top-and-bottom (TaB) format to carry the second LE image data portion and the second RE image data portion.

40. The method as recited in claim 36, wherein the first multiplexed image frame adopts a first multiplexing format that preserves the high spatial frequency content in the vertical direction, and wherein the second multiplexed image frame adopts a second multiplexing format that preserves the high spatial frequency content in the horizontal direction.

41. The method as recited in claim 36, wherein one of the first multiplexed image frame or the second multiplexed image frame is outputted in a base layer bitstream in a plurality of bit streams, while the other of the first multiplexed image frame or the second multiplexed image frame is outputted in an enhancement layer bitstream in the plurality of bit streams.

42. The method as recited in claim 36, further comprising:

generating, based at least in part on the first multiplexed image frame, prediction reference image data; and

encoding an enhancement layer video signal based on differences between the prediction reference image data and the input 3D image.

43. The method as recited in claim 36, further comprising:

applying one or more first operations comprising at least one of (a) spatial frequency filtering operations or (b) spatial subsampling operations in the second direction to the first input image frame and the second input image frame in generating the first multiplexed image frame, wherein the one or more first operations removes high spatial frequency content in the second direction and preserves high spatial frequency content in the first direction; and

applying one or more second operations comprising at least one of (a) spatial frequency filtering operations or (b) spatial subsampling operations in the first direction to the first input image frame and the second input image frame in generating the second multiplexed image frame, wherein the one or more second operations removes high spatial frequency content in the first direction and preserves high spatial frequency content in the second direction.

44. The method as recited in claim 36, further comprising converting one or more 3D input images represented, received, transmitted, or stored with one or more input video signals into one or more 3D output images represented, received, transmitted, or stored with one or more output video signals.

45. The method as recited in claim 36, wherein the input 3D image comprises image data encoded in one of a high dynamic range (HDR) image format, a RGB color space associated with the Academy Color Encoding Specification (ACES) standard of the Academy of Motion Picture Arts and Sciences (AMPAS), a P3 color space standard of the Digital Cinema Initiative, a Reference Input Medium Metric/Reference Output Medium Metric (RIMM/ROMM) standard, an sRGB color space, or a RGB color space associated with the BT.709 Recommendation standard of the International Telecommunications Union (ITU).

46. A method using a processor, the method comprising:

receiving a 3D image represented by a first multiplexed image frame and second multiplexed image frame, the first multiplexed image frame comprising first high spatial frequency content in a vertical direction and first reduced resolution content in a horizontal direction, and the second multiplexed image frame comprising second high spatial frequency content in the horizontal direction and second reduced resolution content in the vertical direction;

wherein one of the first multiplexed image frame or the second multiplexed image frame comprises residual image data, wherein the residual image data has been generated by subtracting reference image data generated based on the other of the first multiplexed image frame or the second multiplexed image frame from input image data derived from a LE input image frame and a RE input image frame;

generating, based on the first multiplexed image frame and the second multiplexed image frame, a left eye (LE) image frame and a right eye (RE) image frame, the LE image frame comprising LE high spatial frequency content in both horizontal and vertical directions, and the RE image frame comprising RE high spatial frequency content in both horizontal and vertical directions; and

rendering the 3D image by rendering the LE image frame and the RE image frame.

47. The method as recited in claim 46, wherein the 3D image is a first 3D image in a sequence of 3D images comprising a second different 3D image having third multiplexed image frame and a fourth multiplexed image frame, the third multiplexed image frame comprising third high spatial frequency content in the vertical direction and third reduced resolution content in the horizontal direction, and the fourth multiplexed image frame comprising fourth high spatial frequency content in the horizontal direction and fourth reduced resolution content in the vertical direction;

and the method further comprising;

generating a second LE image frame and a second RE image frame, the second LE image frame comprising high spatial frequency content in both horizontal and vertical directions, and the second LE image frame comprising high spatial frequency content in both horizontal and vertical directions; and

rendering the second 3D image by rendering the second LE image frame and the second RE image frame.

48. The method as recited in claim 46, wherein at least one of the first multiplexed image frame or the second multiplexed image frame comprises an LE image data portion and an RE image data portion;

and wherein the LE image data portion and the RE image data portion are of a same spatial resolution.

49. The method as recited in claim 48, wherein each of the LE image data portion and the RE image data portion represents a subsampled version of a full resolution image frame;

and wherein the LE image data portion and the RE image data portion forms a single image frame in one of a side-by-side format or a top-and-bottom format.

50. The method as recited in claim 46, wherein one of the first multiplexed image frame or the second multiplexed image frame is decoded from a base layer bitstream in a plurality of bit streams, while the other of the first multiplexed image frame or the second multiplexed image frame is decoded from an enhancement layer bitstream in the plurality of bit streams.

51. The method as recited in claim 46, further comprising:

generating, based at least in part on one of the first multiplexed image frame or the second multiplexed image frame, prediction reference image data; and

generating, based on enhancement layer (EL) data decoded from an EL video signal and the prediction reference image data, one of the LE image frame or the RE image frame.

52. The method as recited in claim 46, further comprising:

applying one or more first operations comprising at least one of (a) spatial frequency filtering operations or (b) demultiplexing operations in generating the LE image frame, wherein the one or more first operations combine LE high spatial frequency content, as derived from the first multiplexed image frame and the second multiplexed image frame, of both horizontal and vertical directions into the LE image frame; and

applying one or more second operations comprising at least one of (a) spatial frequency filtering operations or (b) demultiplexing operations in generating the RE image frame, wherein the one or more second operations combine RE high spatial frequency content, as derived from the first multiplexed image frame and the second multiplexed image frame, of both horizontal and vertical directions into the RE image frame.

53. The method as recited in claim 52, wherein the one or more first operations and the one or more second operations comprise at least a high pass filtering operation.

54. The method as recited in claim 52, wherein the one or more first operations and the one or more second operations comprise a processing sub-path that replaces at least one high pass filtering operation;

and wherein the processing sub-path comprises at least one subtraction operation and no high pass filtering operation.

55. The method as recited in claim 46, the method further comprising:

decoding and processing enhancement layer image data without generating prediction reference data from the other of the first multiplexed image frame or the second multiplexed image frame.

56. The method as recited in claim 46, further comprising processing one or more 3D images represented, received, transmitted, or stored with one or more input video signals.

57. The method as recited in claim 46, wherein the 3D image comprises image data encoded in one of a high dynamic range (HDR) image format, a RGB color space associated with the Academy Color Encoding Specification (ACES) standard of the Academy of Motion Picture Arts and Sciences (AMPAS), a P3 color space standard of the Digital Cinema Initiative, a Reference Input Medium Metric/Reference Output Medium Metric (RIMM/ROMM) standard, an sRGB color space, or a RGB color space associated with the BT.709 Recommendation standard of the International Telecommunications Union (ITU).

58. A system, comprising:

an encoder configured to

receive an input 3D image, the input 3D image comprising a left eye (LE) input image frame and a right eye (RE) input image frame;

generate, based on the LE input image frame and the RE input image frame, a first multiplexed image frame comprising first high spatial frequency content in a vertical direction and first reduced resolution content in a horizontal direction;

generate, based on the LE input image frame and the RE input image frame, a second multiplexed image frame comprising second high spatial frequency content in the horizontal direction and second reduced resolution content in the vertical direction;

wherein one of the first multiplexed image frame or the second multiplexed image frame comprises residual image data, wherein the residual image data is generated by subtracting reference image data generated based on the other of the first multiplexed image frame or the second multiplexed image frame from input image data derived from the LE input image frame and the RE input image frame; and

encode and outputting the first multiplexed image frame and the second multiplexed image frame to represent the input 3D image;

a decoder configured to;

receive the first multiplexed image frame and the second multiplexed image frame in a plurality of video bitstreams;

generate, based on the first multiplexed image frame and the second multiplexed image frame, a left eye (LE) image frame and a right eye (RE) image frame, the LE image frame comprising LE high spatial frequency content in both horizontal and vertical directions, and the RE image frame comprising RE high spatial frequency content in both horizontal and vertical directions; and

rendering the LE image frame and the RE image frame.

59. A method for encoding 3D frame compatible full resolution (FCFR) images using a processor, the method comprising:

receiving an input 3D image, the input 3D image comprising a left eye (LE) input image frame and a right eye (RE) input image frame;

generating, based on the LE input image frame and the RE input image frame, a first multiplexed image frame comprising first high spatial frequency content in a first direction and first reduced resolution content in a second direction, the second direction being orthogonal to the first direction;

generating, based on the first multiplexed image frame reference image data and carrier image data;

subtracting the reference image data from the input 3D image data to generate residual image data;

generating, based on the residual image data and the carrier image data, a second multiplexed image frame comprising second high spatial frequency content in the second direction and second reduced spatial frequency content in the first direction; and

encoding and outputting the first multiplexed image frame and the second multiplexed image frame to represent the input 3D image, wherein generating the carrier image data comprises;

applying a first spatial filtering in a first direction of a base layer image frame to down-sample the base layer image frame and generate an intermediate image, the base layer image based at least in part on the first multiplexed image frame; and

applying a second spatial filtering in a second direction to the intermediate image to up-sample the intermediate image to generate the carrier image data, wherein the second direction is orthogonal to the first direction.

60. The method of claim 59, wherein all the carrier image data comprise pixel values of the same fixed value.

61. A method using a processor for decoding 3D signals coded in an FCFR format, the method comprising:

receiving a 3D image represented by a first multiplexed image frame and second multiplexed image frame, the first multiplexed image frame comprising first high spatial frequency content in a first spatial direction and first reduced resolution content in a second spatial direction, and the second multiplexed image frame comprising second high spatial frequency content in the second spatial direction and second reduced resolution content in the first spatial direction, wherein the second spatial direction is orthogonal to the first spatial direction;

generating based on the first multiplexed image a frame-compatible pair of image frames (FC-L, FC-R) comprising high spatial frequency content in the first spatial direction and reduced spatial frequency content in the second spatial direction; and

generating based on the first multiplexed image and the second multiplexed image frame a full-resolution pair of frames (FR-LE, FR-RE) comprising high spatial frequency content in both the first spatial direction and the second spatial direction, wherein the second multiplexed image frame comprises a residual image frame combined with a carrier image frame and wherein the carrier image frame is generated based on the first multiplexed image frame and then subtracted from the second multiplexed image frame to generate the residual image frame.

62. The method of claim 61, further comprising applying a high-pass filter in the second spatial direction of the second multiplexed image frame to generate the residual image frame.