3D video encoding and decoding methods and apparatus

• US 9,485,494 B1
• Filed: 06/26/2012
• Issued: 11/01/2016
• Est. Priority Date: 04/10/2011
• Status: Active Grant

• Current Assignee: Nevermind Capital LLC
• Original Assignee: NextVR, Inc. (Apple Inc.)
• Inventors: Cole, David Michael, McKay Moss, Alan
• Primary Examiner(s): Rahaman, Mohammed
• Assistant Examiner(s): Lee, Jimmy S
• Application Number:
  US13/533,465
• Time in Patent Office: 1,589 Days
• Field of Search: 348/42, 348/43, 382/100, 382/154, 382/275, 382/285
• US Class Current: 1/1
• CPC Class Codes: H04N 13/106 Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression

Abstract

Methods and apparatus relating to encoding and decoding stereoscopic (3D) image data, e.g., left and right eye images, are described. Various pre-encoding and post-decoding operations are described in conjunction with difference based encoding and decoding techniques. In some embodiments left and right eye image data is subject to scaling, transform operation(s) and cropping prior to encoding. In addition, in some embodiments decoded left and right eye image data is subject to scaling, transform operations(s) and filling operations prior to being output to a display device. Transform information and/or scaling information may be included in a bitstream communicating encoded left and right eye images. The amount of scaling can be the same for an entire scene and/or program.

Claims

1. A method of processing stereoscopic image data including a left eye image and a right eye image of a first frame pair, comprising:

comparing the left eye image and right eye images of the first frame pair to determine a left eye image shift and a right eye image shift which can reduce a difference between the left eye image and right eye image of the first frame pair;

generating a scaled and cropped left eye image based on the left eye image shift determined by said comparing;

generating a scaled and cropped right eye image based on the right eye image shift determined by said comparing;

encoding said scaled and cropped left eye image and said scaled and cropped right eye image using an encoder which performs difference encoding on at least one of said scaled and cropped left and right eye images using the other one of said scaled and cropped left and right eye images as a reference image to generate encoded image data; and

storing transform information with said encoded image data, said transform information providing scaling and shift information relating to the encoded image data.

2. The method of claim 1, wherein generating a scaled and cropped left eye image and generating a scaled and cropped right eye image both include:

performing a scaling operation using a scaling parameter which is left unchanged for multiple frame pairs, said left and right eye images being one of said multiple frame pairs.

3. The method of claim 2, wherein said difference encoding includes motion compensated prediction encoding.

4. The method of claim 2, where said left eye image shift includes a shift in both a vertical and a horizontal direction.

5. The method of claim 4, wherein the area of said cropped left eye image is equal in size to the area of the left eye image, and

wherein the area of the cropped right eye image is equal in size to the area of the right eye image.

6. The method of claim 4, wherein the area of the scaled left eye image is equal to the area of the scaled right eye image.

7. The method of claim 6, wherein the area of the scaled left eye image is larger than the area of the left eye image.

8. The method of claim 5, wherein the area of the left eye image is equal to the area of the right eye image.

9. The method of claim 1, wherein generating a scaled and cropped left eye image and generating a scaled and cropped right eye image further includes performing a preliminary transform including at least one of a-keystone transform or a warping transform.

10. An apparatus including a processor configured to process stereoscopic image data including a left eye image and a right eye image of a first frame pair, said processor being configured to:

compare the left eye image and right eye images of the first frame pair to determine a left eye image shift and a right eye image shift which can reduce a difference between the left eye image and right eye image of the first frame pair;

generate a scaled and cropped left eye image based on the left eye image shift determined by said comparing;

generate a scaled and cropped right eye image based on the right eye image shift determined by said comparing,

encode said scaled and cropped left eye image and said scaled and cropped right eye image using an encoder which performs difference encoding on at least one of said scaled and cropped left and right eye images using the other one of said scaled and cropped left and right eye images as a reference image to generate encoded image data; and

store transform information with said encoded image data, said transform information providing scaling and shift information relating to the encoded image data.

11. A method comprising:

receiving i) encoded stereoscopic image data including an encoded left eye image of a frame pair which was generated by performing difference encoding on a pair of images including at least one of a scaled and cropped left eye image and a scaled and cropped right eye image using the other one of said scaled and cropped left eye image and scaled and cropped right eye image as a reference image and ii) transform information providing scaling and shift information relating to the encoded image data which provides information on a shift operation performed prior to encoding to reduce a difference between the pair of images prior to encoding;

decoding said encoded left eye image to produce a decoded left eye image;

scaling said decoded left eye image by an amount based on the scaling information included in said received transform information to generate a scaled decoded left eye image; and

performing an inverse left eye transform on said scaled decoded left eye image based on at least some of said received transform information to generate a left eye output image, wherein performing an inverse left eye transform includes performing a first fill operation.

12. The method of claim 11, wherein said frame pair includes an encoded right eye image, the method further comprising:

decoding said encoded right eye image to produce a decoded right eye image;

scaling said decoded right eye image by an amount based on the scaling information included in said received transform information to generate a scaled decoded right eye image; and

performing an inverse right eye transform on said scaled decoded right eye image to generate a right eye output image based on at least some of said received transform information, wherein performing an inverse right eye transform includes performing a second fill operation.

13. The method of claim 12, further comprising:

transmitting said left and right eye output images to a display device.

14. The method of claim 11,

wherein said step of scaling said decoded left eye image to generate a scaled decoded left eye image includes performing a size reduction operation on said decoded left eye image.

15. The method of claim 12,

wherein said step of scaling said decoded right eye image to generate a scaled decoded right eye image includes performing a size reduction operation on said decoded right eye image;

wherein said first fill operation fills in a portion of said left eye output image which is not occupied by the scaled decoded left eye image;

wherein said left eye output image is the same size as said encoded left eye image, and

wherein said right eye output image is the same size as said encoded right eye image.

16. The method of claim 12,

wherein performing an inverse left eye transform further includes:

performing a preliminary inverse left eye transform operation prior to performing said first fill operation; and

wherein performing an inverse right eye transform further includes;

performing a preliminary inverse right eye transform operation prior to performing said second fill operation.

17. The method of claim 16, wherein said preliminary inverse left eye transform operation includes one of:

a null transform, shifting transform, anamorphic scaling transform, keystone transform or warping transform which is opposite a left eye transform operation performed to generate the encoded left eye image.

18. The method of claim 12,

wherein said scaling of said decoded left eye image and said decoded right eye image is by a first amount, said first amount being an amount indicated by information supplied along with at least one of said encoded left eye image and said encoded right eye image or prior to said encoded left and right eye images being supplied;

wherein said inverse left eye transform includes a shift in a direction and amount which is based on left eye transform information supplied with said encoded left eye image; and

wherein said inverse right eye transform includes a shift in a direction and amount which is based on right eye transform information supplied with said encoded right eye image.

19. An apparatus comprising a processor configured to:

receive i) encoded stereoscopic image data including an encoded left eye image of a frame pair which was generated by performing difference encoding on a pair of images including at least one of a scaled and cropped left eye image and a scaled and cropped right eye image using the other one of said scaled and cropped left eye image and scaled and cropped right eye image as a reference image and ii) transform information providing scaling and shift information relating to the encoded image data which provides information on a shift operation performed prior to encoding to reduce a difference between the pair of images prior to encoding;

decode said encoded left eye image to produce a decoded left eye image;

scale said decoded left eye image to generate a scaled decoded left eye image which is smaller than the decoded left eye image by an amount based on the scaling information included in said received transform information; and

perform an inverse left eye transform on said scaled decoded left eye image based on at least some of said received transform information to generate a left eye output image, wherein performing an inverse left eye transform includes performing a first fill operation.

20. An apparatus configured to process encoded stereoscopic image data including an encoded left eye image of a frame pair, the apparatus comprising:

a receiver configured to receive i) encoded stereoscopic image data including an encoded left eye image of a frame pair which was generated by performing difference encoding on a pair of images including at least one of a scaled and cropped left eye image and a scaled and cropped right eye image using the other one of said scaled and cropped left eye image and scaled and cropped right eye image as a reference image and ii) transform information providing scaling and shift information relating to the encoded image data which provides information on a shift operation performed prior to encoding to reduce a difference between the pair of images prior to encoding;

a first decoder configured to decode said encoded left eye image to produce a decoded left eye image;

a first scaling module configured to scale said decoded left eye image to generate a scaled decoded left eye image of a smaller size than said decoded left eye image by an amount based on the scaling information included in said received transform information; and

a first inverse transform module configured to perform an inverse left eye transform on said scaled decoded left eye image based on at least some of said received transform information to generate a left eye output image, wherein said first inverse transform module includes a first fill module configured to perform a first fill operation.