Stereoscopic video encoding and decoding methods and apparatus

• US 9,313,474 B1
• Filed: 12/16/2013
• Issued: 04/12/2016
• Est. Priority Date: 08/24/2009
• Status: Active Grant

• Current Assignee: Nevermind Capital LLC
• Original Assignee: NextVR, Inc. (Apple Inc.)
• Inventors: Cole, David Michael, Roller, David Jr., McKay Moss, Alan
• Primary Examiner(s): Pe, Geepy
• Application Number:
  US14/107,852
• Time in Patent Office: 848 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 19/00 Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
• CPC Class Codes: H04N 19/583 Motion compensation with overlapping blocks
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/82 involving filtering within a prediction loop
• CPC Class Codes: H04N 19/91 Entropy coding, e.g. variable length coding [VLC] or arithmetic coding
• CPC Class Codes: H04N 2213/007 Aspects relating to detection of stereoscopic image format, e.g. for adaptation to the display format

Abstract

Methods and apparatus for stereoscopic image encoding and decoding are described. Left and right eye images are encoded following an entropy reduction operation being applied to one of the eye images when there is a difference between the left and right images of an image pair. Information about regions of negative parallax within the entropy reduced image of an image pair is encoded along with the images. Upon decoding a sharpening filter is applied to the image in an image pair which was subjected to the entropy reduction operation. In addition edge enhancement filtering is performed on the regions of the recovered entropy reduced image which are identified in the encoded image data as regions of negative parallax. Interleaving of left and right eye images at the input of the encoder combined with entropy reduction allows for efficient encoding, storage, and transmission of 3D images.

Claims

1. A method of processing encoded frames of stereoscopic image data, the method comprising:

decoding first and second encoded frames representing first and second eye images of an image pair to recover decoded first and second eye image data;

determining if one of said first and second eye images was subject to an entropy reduction operation prior to encoding;

when it is determined that that one of said first and second eye images was subject to an entropy reduction operation prior to encoding;

i) performing a sharpening operation, using decoded eye image data corresponding to the one of the first and second eye images which was subject to an entropy reduction operation, to produce a sharpened version of the entropy reduced image, said entropy reduced image being the one of the first and second eye images which was subject to an entropy reduction operation; and

ii) rendering a 3-D image from said sharpened version of the entropy reduced image and the decoded one of the first and second eye images which was not subject to an entropy reduction operation.

2. The method of claim 1, wherein said sharpening operation increases edge prominence within the image being subject to said sharpening.

3. The method of claim 1, wherein determining if one of said first and second eye images was subject to an entropy reduction operation prior to encoding includes:

determining if there is a difference between said first and second eye images.

4. The method of claim 1, wherein determining if one of said first and second eye images was subject to an entropy reduction operation prior to encoding includes:

determining if a difference indicator value associated with at least one of said first and second encoded frames indicates a difference between the first and second eye images.

5. The method of claim 4, wherein when it is determined that that one of said first and second eye images was subject to an entropy reduction operation, the method further comprises:

controlling the strength of a sharpening filter used to perform said sharpening operation as a function of said difference indicator value, a higher amount of sharpening being performed when said difference indicator value indicates a smaller difference than when said difference indicator value indicates a larger difference.

6. The method of claim 2, further comprising:

when it is determined that that one of said first and second eye images was subject to an entropy reduction operation prior to encoding;

performing an edge enhancement operation on said sharpened version of the entropy reduced image prior to using said sharpened version of the entropy reduced image in said rendering step.

7. The method of claim 6, wherein said edge enhancement operation is performed on image portions identified as negative parallax regions, by information associated with said entropy reduced image.

8. The method of claim 7, further comprises:

receiving a file including said encoded frames of stereoscopic image data, said file including a plurality of frame pairs which were interleaved to form a left and right eye image sequence prior to encoding.

9. The method of claim 8, wherein information is included in said file for at least one pair of frames indicating an amount of difference between the images represented by said at least one pair of frames and information identifying at least one region of negative parallax in one of said frames which was generated from an image subjected to an entropy reduction operation.

10. The method of claim 1, wherein when said determining indicates that one of said first and second eye images was not subject to an entropy reduction operation, the method further comprises:

rendering a 3-D image from said first and second eye image data without performing a sharpening operation.

11. An apparatus for processing encoded frames of stereoscopic image data, the apparatus comprising:

a decoder for decoding first and second encoded frames representing first and second eye images of an image pair to recover decoded first and second eye image data;

an entropy reduction determinator for determining if one of said first and second eye images was subject to an entropy reduction operation prior to encoding;

a sharpening filter for performing a sharpening operation, using decoded eye image data corresponding to the one of the first and second eye images which was subject to an entropy reduction operation, to produce a sharpened version of the entropy reduced image when it is determined that that one of said first and second eye images was subject to an entropy reduction operation prior to encoding, said entropy reduced image being the one of the first and second eye images which was subject to an entropy reduction operation; and

a renderer for rendering a 3-D image from said sharpened version of the entropy reduced image and the decoded one of the first and second eye images which was not subject to an entropy reduction operation.

12. The apparatus of claim 11, wherein said sharpening operation increases edge prominence within the image being subject to said sharpening.

13. The apparatus of claim 11, wherein said entropy reduction determination module includes:

a difference determinator for determining if there is a difference between said first and second eye images.

14. The apparatus of claim 11, wherein said entropy reduction determinator is further configured to determine if a difference indicator value associated with at least one of said first and second encoded frames indicates a difference between the first and second eye images.

15. The apparatus of claim 14, further comprising:

a controller for controlling the strength of a sharpening filter used to perform said sharpening operation as a function of said difference indicator value when it is determined that that one of said first and second eye images was subject to an entropy reduction operation, said controller controlling a higher amount of sharpening to be performed when said difference indicator value indicates a smaller difference than when said difference indicator value indicates a larger difference.

16. The apparatus of claim 12, further comprising:

an edge enhancement operator for performing an edge enhancement operation on said sharpened version of the entropy reduced image.

17. The apparatus of claim 16, wherein said edge enhancement operation is performed on image portions identified as negative parallax regions, by information associated with said entropy reduced image.

18. The apparatus of claim 17, further comprising:

an interface for receiving a file including said encoded frames of stereoscopic image data, said file including a plurality of frame pairs which were interleaved to form a left and right eye image sequence prior to encoding.