3D VIDEO ENCODING AND DECODING METHODS AND APPARATUS

• US 20170094247A1
• Filed: 08/01/2016
• Published: 03/30/2017
• Est. Priority Date: 04/10/2011
• Status: Active Grant

• Current Assignee: Nevermind Capital LLC
• Original Assignee: NextVR, Inc. (Apple Inc.)
• Inventors: Cole, David, Moss, Alan McKay
• Granted Patent: US 10,681,333 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: H04N 13/106 Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression

Abstract

Methods and apparatus relating to encoding and decoding stereoscopic (3D) image data, e.g., left and right eye images, are described. Various pre-encoding and post-decoding operations are described in conjunction with difference based encoding and decoding techniques. In some embodiments left and right eye image data is subject to scaling, transform operation(s) and cropping prior to encoding. In addition, in some embodiments decoded left and right eye image data is subject to scaling, transform operations(s) and filling operations prior to being output to a display device. Transform information and/or scaling information may be included in a bitstream communicating encoded left and right eye images. The amount of scaling can be the same for an entire scene and/or program.

Claims

1. A method of processing stereoscopic image data including a left eye image and a right eye image of a frame pair, comprising:

scaling said left eye image and said right eye image by a first amount to generate a scaled left eye image and a scaled right eye image;

performing a left eye transform on said scaled left eye image, said left eye transform including at least a cropping operation and generating a cropped left eye image; and

performing a right eye transform on said scaled right eye image, said right eye transform including at least a cropping operation and generating a cropped right eye image.

2. The method of claim 1, further comprising:

encoding said cropped left and right eye images using an encoder which performs difference encoding on at least one of said cropped left and right eye images using the other one of said cropped left and right eye images as a reference image, said encoding producing an encoded left eye image and an encoded right eye image.

3. The method of claim 2, wherein said difference encoding includes motion compensated prediction encoding.

4. The method of claim 2, further comprising:

determining a first transform to be used as said left eye transform and a second transform to be used as said right eye transform, said determining including;

comparing content of said scaled left eye image and said scaled right eye image to determine image transform operations which will reduce the difference between an area of said cropped left eye image and said cropped right eye image.

5. The method of claim 4, wherein the area of said cropped left eye image is equal in size to the area of the left eye image, and

wherein the area of the cropped right eye image is equal in size to the area of the right eye image.

6. The method of claim 4, wherein the area of the scaled left eye image is equal to the area of the scaled right eye image.

7. The method of claim 6, wherein the area of the scaled left eye image is larger than the area of the left eye image.

8. The method of claim 5, wherein the area of the left eye image is equal to the area of the right eye image.

9. The method of claim 1, wherein said left and right eye transforms each include a preliminary transform in addition to said cropping operation, said preliminary transform including at least one of a null transform, shifting transform, anamorphic scaling transform, keystone transform or warping transform.

10. An apparatus including a processor configured to process stereoscopic image data including a left eye image and a right eye image of a frame pair, said processor being configured to:

scale said left eye image and said right eye image by a first amount to generate a scaled left eye image and a scaled right eye image;

perform a left eye transform on said scaled left eye image, said left eye transform including at least a cropping operation and generating a cropped left eye image; and

perform a right eye transform on said scaled right eye image, said right eye transform including at least a cropping operation and generating a cropped right eye image.

11. A method of processing encoded stereoscopic image data including an encoded left eye image of a frame pair, comprising:

decoding said encoded left eye image to produce a decoded left eye image;

scaling said decoded left eye images to generate a scaled decoded left eye image; and

performing an inverse left eye transform on said scaled decoded left eye image to generate a left eye output image, wherein performing an inverse left eye transform includes performing a first fill operation.

12. The method of claim 11, wherein said frame pair includes an encoded right eye image, the method further comprising:

decoding said encoded right eye image to produce a decoded right eye image;

scaling said decoded right eye image to generate a scaled decoded right eye image; and

performing an inverse right eye transform on said scaled decoded right eye image to generate a right eye output image, wherein performing an inverse right eye transform includes performing a second fill operation.

13. The method of claim 12, further comprising:

transmitting said left and right eye output images to a display device.

14. The method of claim 12,

wherein said left eye output image is the same size as said encoded left eye image, and

wherein said right eye output image is the same size as said encoded right eye image.

15. The method of claim 14, wherein said encoded left eye image is the same size as said encoded right eye image.

16. The method of claim 12,

wherein performing an inverse left eye transform further includes:

performing a preliminary inverse left eye transform operation prior to performing said first fill operation; and

wherein performing an inverse right eye transform further includes;

performing a preliminary inverse right eye transform operation prior to performing said second fill operation.

17. The method of claim 15, wherein said preliminary inverse left eye transform operation includes one of:

a null transform, shifting transform, anamorphic scaling transform, keystone transform or warping transform.

18. The method of claim 14,

wherein said scaling of said left eye image and said right eye image is by a first amount, said first amount being an amount indicated by information supplied along with at least one of said encoded left eye image and said encoded right eye image or prior to said encoded left and right eye images being supplied;

wherein said inverse left eye transform includes a shift in a direction and amount which is based on left eye transform information supplied with said encoded left eye image; and

wherein said inverse right eye transform includes a shift in a direction and amount which is based on right eye transform information supplied with said encoded right eye image.

19. An apparatus configured to process encoded stereoscopic image data including an encoded left eye image of a frame pair, the apparatus comprising a processor configured to:

decode said encoded left eye image to produce a decoded left eye image;

scale said decoded left eye images to generate a scaled decoded left eye image; and

perform an inverse left eye transform on said scaled decoded left eye image to generate a left eye output image, wherein performing an inverse left eye transform includes performing a first fill operation.

20. An apparatus configured to process encoded stereoscopic image data including an encoded left eye image of a frame pair, the apparatus comprising:

a first decoder module decode configured to decoded said encoded left eye image to produce a decoded left eye image;

a first scaling module configured to scale said decoded left eye images to generate a scaled decoded left eye image; and

a first inverse transform module configured to perform an inverse left eye transform on said scaled decoded left eye image to generate a left eye output image, wherein said first inverse transform module includes a first fill module configured to perform a first fill operation.