Video encoding method and decoding method, apparatuses therefor, programs therefor, and storage media which store the programs

• US 8,204,118 B2
• Filed: 06/23/2008
• Issued: 06/19/2012
• Est. Priority Date: 06/25/2007
• Status: Active Grant

• Current Assignee: Nippon Telegraph and Telephone Corporation
• Original Assignee: Nippon Telegraph and Telephone Corporation
• Inventors: Shimizu, Shinya, Kimata, Hideaki, Kamikura, Kazuto, Yashima, Yoshiyuki
• Primary Examiner(s): Ustaris, Joseph
• Assistant Examiner(s): Gao, Hang
• Application Number:
  US12/665,890
• Publication Number:
  US 20100189177A1
• Time in Patent Office: 1,457 Days
• Field of Search: 375/240.01, 375/240.02, 375/240.12, 348/218.1, 396/310
• US Class Current: 375/240.12
• CPC Class Codes: H04N 19/105 Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
• CPC Class Codes: H04N 19/12 Selection from among a plurality of transforms or standards, e.g. selection between discrete cosine transform [DCT] and sub-band transform or selection between H.263 and H.264
• CPC Class Codes: H04N 19/134 characterised by the element, parameter or criterion affecting or controlling the adaptive coding
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding

Abstract

A video encoding apparatus used in encoding of a multi-viewpoint image. The apparatus generates a synthetic image for a camera used for obtaining an encoding target image, by using an already-encoded reference camera image having a viewpoint different from the viewpoint of the camera used for obtaining the encoding target image, and disparity information between the reference camera image and the encoding target image, thereby encoding the encoding target image. A predicted image for a differential image between an input image of an encoding target area to be encoded and the synthetic image generated therefor is generated, and a predicted image for the encoding target area, which is represented by the sum of the predicted differential image and the synthetic image for the encoding target area, is generated. A prediction residual represented by a difference between the predicted image for the encoding target area and the encoding target image of the encoding target area is encoded.

Claims

1. A video encoding apparatus used in encoding of a multi-viewpoint image obtained by cameras having different viewpoints, wherein the apparatus generates a synthetic image for a camera used for obtaining an encoding target image, by using at least one already-encoded reference camera image having a viewpoint different from the viewpoint of the camera used for obtaining the encoding target image, and disparity information between the reference camera image and the encoding target image, encodes the encoding target image by using the synthetic image, and comprises:

a differential image prediction device that generates a predicted image for a differential image between an input image of an encoding target area to be encoded and the synthetic image generated therefor;

a predicted image generation device that generates a predicted image for the encoding target area, where the predicted image is represented by the sum of the predicted differential image generated by the differential image prediction device and the synthetic image for the encoding target area; and

a prediction residual encoding device that encodes a prediction residual represented by a difference between the predicted image for the encoding target area generated by the predicted image generation device and the encoding target image of the encoding target area.

2. The video encoding apparatus in accordance with claim 1, wherein:

the differential image prediction device generates the predicted image for the differential image with respect to the encoding target area, by using a differential image at a time which is the same as the time when the encoding target image was obtained, where the used differential image is represented by a difference between a decoded image of an already-encoded area in the encoding target image and the synthetic image for the already-encoded area.

3. The video encoding apparatus in accordance with claim 1, wherein:

the differential image prediction device generates the predicted image for the differential image with respect to the encoding target area, by using a differential image at a time different from the time when the encoding target image was obtained, where the used differential image is represented by a difference between a decoded image of an already-encoded image which was obtained at the viewpoint used for obtaining the encoding target image and the synthetic image at the same time as the time of the decoded image.

4. The video encoding apparatus in accordance with claim 1, further comprising:

an image decoding device that generates a decoded image of the encoding target image by decoding encoded data of the encoding target image; and

a decoded differential image generation and storage device that generates a decoded differential image represented by a difference between the decoded image generated by the image decoding device and the synthetic image, and stores the generated decoded differential image into a storage memory,

wherein the differential image prediction device generates the predicted image for the differential image with respect to the encoding target area, by using the decoded differential image stored in the storage memory.

5. The video encoding apparatus in accordance with claim 1, further comprising:

a prediction residual decoding device that decodes the prediction residual encoded by the prediction residual encoding device; and

a decoded differential image generation and storage device that generates a decoded differential image represented by the sum of the decoded prediction residual generated by the prediction residual decoding device and the predicted differential image, and stores the generated decoded differential image into a storage memory,

wherein the differential image prediction device generates the predicted image for the differential image with respect to the encoding target area, by using the decoded differential image stored in the storage memory.

6. The video encoding apparatus in accordance with claim 5, wherein:

the decoded differential image generation and storage device performs clipping so that each pixel of the decoded differential image has a pixel value within a predetermined effective range.

7. The video encoding apparatus in accordance with claim 1, further comprising:

an original image prediction device that generates a predicted image for the encoding target image, by using any one of a decoded image of an already-encoded area in the encoding target image and a decoded image of an already-encoded image which was obtained at the same viewpoint as that used for obtaining the encoding target image and at a time different from the time of the encoding target image,

wherein the predicted image generation device selectively performs one of generation of the predicted image for the encoding target image based on the predicted differential image and the synthetic image and generation of the predicted image for the encoding target image by directly using the predicted image generated by the original image prediction device.

8. The video encoding apparatus in accordance with claim 1, wherein:

the predicted image generation device performs clipping so that each pixel of the predicted image has a pixel value within a predetermined effective range.

27. A non-transitory computer-readable storage medium which stores a video encoding program by which a computer executes a process for implementing the video encoding apparatus in accordance with claim 1.

9. A video decoding apparatus used in decoding of encoded data of a multi-viewpoint image obtained by cameras having different viewpoints, wherein the apparatus generates a synthetic image for a camera used for obtaining a decoding target image, by using at least one already-decoded reference camera image having a viewpoint different from the viewpoint of the camera used for obtaining a decoding target image, and disparity information between the reference camera image and the decoding target image, decodes encoded data of the decoding target image by using the synthetic image, and comprises:

a prediction residual decoding device that decodes a prediction residual of the decoding target image, where the prediction residual is included in the encoded data;

a differential image prediction device that generates a predicted image for a differential image between a decoding target image of a decoding target area to be decoded and the synthetic image generated therefor; and

an image decoding device that decodes the decoding target image by adding the predicted differential image generated by the differential image prediction device, the prediction residual decoded by the prediction residual decoding device, and the synthetic image.

10. The video decoding apparatus in accordance with claim 9, wherein:

the differential image prediction device generates the predicted image for the differential image with respect to the decoding target area, by using a differential image at a time which is the same as the time when the decoding target image was obtained, where the used differential image is represented by a difference between a decoded image of an already-decoded area in the decoding target image and the synthetic image for the already-decoded area.

11. The video decoding apparatus in accordance with claim 9, wherein:

the differential image prediction device generates the predicted image for the differential image with respect to the decoding target area, by using a differential image at a time different from the time when the decoding target image was obtained, where the used differential image is represented by a difference between an already-decoded reference image which was obtained at the viewpoint used for obtaining the decoding target image and the synthetic image at the same time as the time of the reference image.

12. The video decoding apparatus in accordance with claim 9, further comprising:

a decoded differential image generation and storage device that generates a decoded differential image represented by a difference between the decoded image generated by the image decoding device and the synthetic image, and stores the generated decoded differential image into a storage memory,

wherein the differential image prediction device generates the predicted image for the differential image with respect to the decoding target area, by using the decoded differential image stored in the storage memory.

13. The video decoding apparatus in accordance with claim 9, further comprising:

a decoded differential image generation and storage device that generates a decoded differential image represented by the sum of the decoded prediction residual generated by the prediction residual decoding device and the predicted differential image, and stores the generated decoded differential image into a storage memory,

wherein the differential image prediction device generates the predicted image for the differential image with respect to the decoding target area, by using the decoded differential image stored in the storage memory.

14. The video decoding apparatus in accordance with claim 13, wherein:

the decoded differential image generation and storage device performs clipping so that each pixel of the decoded differential image has a pixel value within a predetermined effective range.

15. The video decoding apparatus in accordance with claim 9, further comprising:

an original image prediction device that generates a predicted image for the decoding target image, by using any one of an already-decoded image of the decoding target image and an already-decoded image which was obtained at the same viewpoint as that used for obtaining the decoding target image and at a time different from the time of the decoding target image,

wherein for the decoding target area, the image decoding device selectively performs one of decoding of the decoding target image by adding the predicted differential image generated by the differential image prediction device, the prediction residual decoded by the prediction residual decoding device, and the synthetic image, and decoding of the decoding target image by adding the predicted image generated by the original image prediction device to the prediction residual decoded by the prediction residual decoding device.

28. A non-transitory computer-readable storage medium which stores a video decoding program by which a computer executes a process for implementing the video decoding apparatus in accordance with claim 9.

16. A video encoding method used in encoding of a multi-viewpoint image obtained by cameras having different viewpoints, wherein the method generates a synthetic image for a camera used for obtaining an encoding target image, by using at least one already-encoded reference camera image having a viewpoint different from the viewpoint of the camera used for obtaining the encoding target image, and disparity information between the reference camera image and the encoding target image, encodes the encoding target image by using the synthetic image, and comprises:

a differential image prediction step that generates a predicted image for a differential image between an input image of an encoding target area to be encoded and the synthetic image generated therefor;

a predicted image generation step that generates a predicted image for the encoding target area, where the predicted image is represented by the sum of the predicted differential image generated by the differential image prediction step and the synthetic image for the encoding target area; and

a prediction residual encoding step that encodes a prediction residual represented by a difference between the predicted image for the encoding target area generated by the predicted image generation step and the encoding target image of the encoding target area.

17. The video encoding method in accordance with claim 16, wherein:

the differential image prediction step generates the predicted image for the differential image with respect to the encoding target area, by using a differential image at a time which is the same as the time when the encoding target image was obtained, where the used differential image is represented by a difference between a decoded image of an already-encoded area in the encoding target image and the synthetic image for the already-encoded area.

18. The video encoding method in accordance with claim 16, wherein:

the differential image prediction step generates the predicted image for the differential image with respect to the encoding target area, by using a differential image at a time different from the time when the encoding target image was obtained, where the used differential image is represented by a difference between a decoded image of an already-encoded image which was obtained at the viewpoint used for obtaining the encoding target image and the synthetic image at the same time as the time of the decoded image.

19. The video encoding method in accordance with claim 16, further comprising:

a prediction residual decoding step that decodes the prediction residual encoded by the prediction residual encoding step; and

a decoded differential image generation and storage step that generates a decoded differential image represented by the sum of the decoded prediction residual generated by the prediction residual decoding step and the predicted differential image, and stores the generated decoded differential image into a storage memory,

wherein the differential image prediction step generates the predicted image for the differential image with respect to the encoding target area, by using the decoded differential image stored in the storage memory.

20. The video encoding method in accordance with claim 16, further comprising:

an original image prediction step that generates a predicted image for the encoding target image, by using any one of a decoded image of an already-encoded area in the encoding target image and a decoded image of an already-encoded image which was obtained at the same viewpoint as that used for obtaining the encoding target image and at a time different from the time of the encoding target image,

wherein the predicted image generation step selectively performs one of generation of the predicted image for the encoding target image based on the predicted differential image and the synthetic image and generation of the predicted image for the encoding target image by directly using the predicted image generated by the original image prediction step.

21. A video decoding method used in decoding of encoded data of a multi-viewpoint image obtained by cameras having different viewpoints, wherein the method generates a synthetic image for a camera used for obtaining a decoding target image, by using at least one already-decoded reference camera image having a viewpoint different from the viewpoint of the camera used for obtaining a decoding target image, and disparity information between the reference camera image and the decoding target image, decodes encoded data of the decoding target image by using the synthetic image, and comprises:

a prediction residual decoding step that decodes a prediction residual of the decoding target image, where the prediction residual is included in the encoded data;

a differential image prediction step that generates a predicted image for a differential image between a decoding target image of a decoding target area to be decoded and the synthetic image generated therefor; and

an image decoding step that decodes the decoding target image by adding the predicted differential image generated by the differential image prediction step, the prediction residual decoded by the prediction residual decoding step, and the synthetic image.

22. The video decoding method in accordance with claim 21, wherein:

the differential image prediction step generates the predicted image for the differential image with respect to the decoding target area, by using a differential image at a time which is the same as the time when the decoding target image was obtained, where the used differential image is represented by a difference between a decoded image of an already-decoded area in the decoding target image and the synthetic image for the already-decoded area.

23. The video decoding method in accordance with claim 21, wherein:

the differential image prediction step generates the predicted image for the differential image with respect to the decoding target area, by using a differential image at a time different from the time when the decoding target image was obtained, where the used differential image is represented by a difference between an already-decoded reference image which was obtained at the viewpoint used for obtaining the decoding target image and the synthetic image at the same time as the time of the reference image.

24. The video decoding method in accordance with claim 21, further comprising:

a decoded differential image generation and storage step that generates a decoded differential image represented by the sum of the decoded prediction residual generated by the prediction residual decoding step and the predicted differential image, and stores the generated decoded differential image into a storage memory,

wherein the differential image prediction step generates the predicted image for the differential image with respect to the decoding target area, by using the decoded differential image stored in the storage memory.

25. The video decoding method in accordance with claim 24, wherein:

the decoded differential image generation and storage step performs clipping so that each pixel of the decoded differential image has a pixel value within a predetermined effective range.

26. The video decoding method in accordance with claim 21, further comprising:

an original image prediction step that generates a predicted image for the decoding target image, by using any one of an already-decoded image of the decoding target image and an already-decoded image which was obtained at the same viewpoint as that used for obtaining the decoding target image and at a time different from the time of the decoding target image,

wherein for the decoding target area, the image decoding step selectively performs one of decoding of the decoding target image by adding the predicted differential image generated by the differential image prediction step, the prediction residual decoded by the prediction residual decoding step, and the synthetic image, and decoding of the decoding target image by adding the predicted image generated by the original image prediction step to the prediction residual decoded by the prediction residual decoding step.