3D video encoding and decoding methods and apparatus

• US 9,407,902 B1
• Filed: 04/10/2012
• Issued: 08/02/2016
• Est. Priority Date: 04/10/2011
• Status: Active Grant

• Current Assignee: Nevermind Capital LLC
• Original Assignee: NextVR, Inc. (Apple Inc.)
• Inventors: Cole, David Michael, Moss, Alan McKay
• Primary Examiner(s): Rahaman, Mohammed
• Assistant Examiner(s): Lee, Jimmy S
• Application Number:
  US13/443,864
• Time in Patent Office: 1,575 Days
• Field of Search: 348/42, 348/43
• US Class Current: 1/1
• CPC Class Codes: H04N 13/106 Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/20 Image signal generators
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression

Abstract

Methods and apparatus relating to encoding and decoding stereoscopic (3D) image data, e.g., left and right eye images, are described. Various pre-encoding and post-decoding operations are described in conjunction with difference based encoding and decoding techniques. In some embodiments left and right eye image data is subject to scaling, transform operation(s) and cropping prior to encoding. In addition, in some embodiments decoded left and right eye image data is subject to scaling, transform operations(s) and cropping prior to being output to a display device. Transform information, scaling information and/or cropping information may be included in a bitstream communicating encoded left and right eye images. The amount of scaling can be the same for an entire scene and/or program.

Claims

1. A method of processing stereoscopic image data including a left eye image and a right eye image of a first frame pair, comprising:

comparing content of said left eye image and said right eye image of the first frame pair to determine a first image shift which can be used to reduce a difference between the left eye image and right eye image;

performing transform operations on said left eye image and said right eye image to reduce the difference between the left eye image and right eye image, said transform operations including scaling said left and right eye images by a first amount to generate scaled left and right eye images, cropping said scaled left eye image to generate a cropped scaled left eye image including scaled content shifted in a first direction relative to a center of the left eye image, and cropping said scaled right eye image to generate a cropped scaled right eye image including scaled content shifted in a second direction relative to a center of the right eye image;

encoding said cropped scaled left eye image and cropped scaled right eye image to generate a first encoded frame pair, wherein encoding said cropped scaled left eye image and cropped scaled right eye image includes using an encoder which performs difference encoding on at least one of said cropped scaled left and right eye images using the other one of said cropped scaled left and right eye images as a reference image; and

communicating information to a playback device, with said first encoded frame pair, indicating said first amount of scaling.

2. The method of claim 1, wherein said difference encoding includes motion compensated prediction encoding.

3. The method of claim 1, wherein said image shift which can be used to reduce the difference between the left eye image and right eye image is a maximum image shift in one of a vertical and horizontal direction required to minimize the difference between the left eye image and the right eye image.

4. The method of claim 1,

wherein the left eye image is of a first size;

and

wherein cropping said scaled left eye image to generate a cropped scaled left eye image includes;

cropping said scaled left eye image to the first size, said cropped scaled left eye image being the same size as said left eye image.

5. The method of claim 1,

wherein said first and second directions are opposite directions;

and

wherein said cropped scaled left eye image includes content which is shifted by an equal and opposite amount as compared to content of said cropped scaled right eye image.

6. The method of claim 1, further comprising:

generating a bitstream communicating said first encoded frame pair, scaling information corresponding to said first encoded frame pair, and shift information corresponding to said first encoded frame pair.

7. The method of claim 6, further comprising:

receiving a left eye image and a right eye image of a second frame pair, said second frame pair being a frame pair which follows said first frame pair in a sequence of frame pairs;

comparing content of said left eye image and said right eye image of the second frame pair to determine a second image shift which can be used to reduce a difference between the left eye image and right eye image of the second frame pair; and

performing transform operations on said left eye image and said right eye image of the second frame pair to reduce the difference between the left eye image and right eye image of the second frame pair, said transform operations including scaling said left and right eye images of the second frame pair by a second amount to generate scaled left and right eye images, said second amount being determined based on said first amount and said determined second image shift.

8. The method of claim 6, further comprising:

operating said playback device to;

decode said encoded first frame pair to produce decoded left and right eye images;

scale said decoded left and right eye images by an amount determined from scaling information communicated by the bitstream to generate a scaled decoded left eye image and a scaled decoded right eye image; and

crop said scaled decoded left eye image and scaled decoded right eye image.

9. The method of claim 8

wherein said cropping of said scaled decoded left and right eye images crops said scaled decoded left and right eye images to a size which is the size of said left eye image and right eye image of the first frame pair.

10. The method of claim 9,

wherein said scaling performed by said playback device on said decoded left and right eye images is by said first amount;

and

wherein said cropping of said scaled decoded left and right eye images produces a cropped decoded left eye image including scaled content shifted relative to a center of the decoded left eye image in a direction opposite the first direction and a cropped decoded right eye image including scaled content shifted relative to a center of the decoded right eye image in a direction opposite the second direction.

15. The method of claim 1, wherein said left and right eye transforms each include at least one of a null transform, shifting transform, anamorphic scaling transform, keystone transform or warping transform.

16. The method of claim 15, wherein said left eye transform is equal and opposite to said right eye transform when said left and right eye transforms are non-null transforms.

17. The method of claim 16

wherein said first amount of scaling is indicated by a global parameter used to control scaling of multiple frame pairs.

11. A method of processing encoded stereoscopic image data including an encoded left eye image and an encoded right eye image of a frame pair, comprising:

decoding said encoded left and right eye images to produce decoded left and right eye images, said encoded left and right eye images having been encoded using difference encoding on at least one of a left eye image and a right eye image using the other one of the left and right eye images as a reference image, said left and right eye images having been produced by transform operations including scaling said left and right eye images to generate scaled left and right eye images, cropping said scaled left eye image to generate a cropped scaled left eye image including scaled content shifted in a first direction and cropping said scaled right eye image to generate a cropped scaled right eye image including scaled content shifted in a second direction, to reduce a difference between the left and right eye images, said second direction being opposite to said first direction;

scaling said decoded left and right eye images by a first amount indicated to a playback device by information supplied to said playback device along with said encoded left eye image and said encoded right eye image or prior to said encoded left and right eye images being supplied to said playback device;

cropping said scaled decoded left and right eye images, said cropping of said scaled decoded left and right eye images producing a cropped scaled decoded left eye image including scaled content shifted relative to the center of the decoded left eye image and a cropped scaled decoded right eye image including scaled content shifted relative to the center of the decoded right eye image; and

outputting said cropped scaled decoded left eye image and right eye image.

12. The method of claim 11, further comprising:

performing an inverse left eye transform operation on said scaled decoded left eye image prior to cropping said scaled decoded left eye image;

performing an inverse right eye transform operation on said scaled decoded right eye image prior to cropping said scaled decoded right eye image; and

wherein said cropping of said scaled decoded left and right eye images crops said decoded left and right eye images to the size of said encoded left and right eye images.

13. The method of claim 12,

wherein said inverse left eye transform operation includes a shift in a direction and amount which is based on left eye transform information supplied with said encoded left eye image;

wherein said inverse right eye transform operation includes a shift in a direction and amount which is based on right eye transform information supplied with said encoded right eye image.

14. The method of claim 13, wherein outputting said cropped scaled decoded left and right eye images includes providing the images to a display device for display to a user.

18. An apparatus including a processor configured to process stereoscopic image data including a left eye image and a right eye image of a first frame pair, said processor being configured to:

compare content of said left eye image and said right eye image of the first frame pair to determine a first image shift which can be used to reduce a difference between the left eye image and right eye image;

perform transform operations on said left eye image and said right eye image to reduce the difference between the left eye image and right eye image, said transform operations including scaling said left and eye right images by a first amount to generate scaled left and right eye images, cropping said scaled left eye image to generate a cropped scaled left eye image including scaled content shifted in a first direction relative to a center of the left eye image, and cropping said scaled right eye image to generate a cropped scaled right eye image including scaled content shifted in a second direction relative to a center of the right eye image;

encode said cropped scaled left eye image and cropped scaled right eye image to generate a first encoded frame pair, wherein encoding said cropped scaled left eye image and cropped scaled right eye image includes using an encoder which performs difference encoding on at least one of said cropped scaled left and right eye images using the other one of said cropped scaled left and right eye images as a reference image; and

communicate information to a playback device, with said first encoded frame pair, indicating said first amount of scaling.