MOVING PICTURE COMPRESSION APPARATUS, IMAGE PROCESSING APPARATUS, MOVING PICTURE COMPRESSION METHOD, IMAGE PROCESSING METHOD, AND DATA STRUCTURE OF MOVING PICTURE COMPRESSION FILE

• US 20150124883A1
• Filed: 10/24/2012
• Published: 05/07/2015
• Est. Priority Date: 12/27/2011
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Inada, Tetsugo, Ohba, Akio, Segawa, Hiroyuki
• Granted Patent: US 9,693,072 B2
• Time in Patent Office: Days
• US Class Current: 375/240.16
• CPC Class Codes: H04N 19/105 Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
• CPC Class Codes: H04N 19/137 Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/167 Position within a video image, e.g. region of interest [ROI]
• CPC Class Codes: H04N 19/174 the region being a slice, e.g. a line of blocks or a group of blocks
• CPC Class Codes: H04N 19/198 including smoothing of a sequence of encoding parameters, e.g. by averaging, by choice of the maximum, minimum or median value
• CPC Class Codes: H04N 19/428 Recompression, e.g. by spatial or temporal decimation
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/593 involving spatial prediction techniques

Abstract

A frame sequence of moving picture data is divided into a tile image sequence 250, and the color space of the tile image sequence 250 is converted to generate a YCbCr image sequence 252 (S10). Each frame is reduced to ½ time in the vertical and horizontal directions (S12), and a compression process is carried out to generate compression data 260 of a reference image (S14). The compression data 260 of the reference image is decoded and decompressed similarly as upon image display to restore a YCbCr image as the reference image, and a difference image sequence 262 is generated from the reference image and the original YCbCr image 252 (S16). Then, compression data 266 of a difference image is generated (S18), and compression data 268 obtained by connecting the compression data 260 of the reference image and the compression data 266 of the difference image is generated for every four frames of a tile image (S20).

Claims

1. A moving picture compression apparatus, comprising:

a tile image sequence generation section configured to generate a tile image sequence configured by dividing a frame sequence, which configures moving picture data, on an image plane;

a reference image compression section configured to compress the tile image sequence to generate compression data of a reference image;

a difference image generation section configured to generate a difference image sequence configured from difference images which represent a difference between tile images configuring the tile image sequence and images obtained by decoding the compression data of the reference image generated by the reference image compression section and corresponding to the tile images;

a difference image compression section configured to compress the difference image sequence in units of data blocks obtained by dividing the difference image sequence on the image plane and a time axis to generate compression data of the difference images; and

a compression data generation section configured to generate moving picture compression data including the compression data of the reference image and the compression data of the difference images in tile image units of a predetermined number of frames and output the moving picture compression data to a storage apparatus.

2. The moving picture compression apparatus according to claim 1, wherein:

the reference image compression section compresses the reference image in units of the data blocks obtained by dividing the reference image on the image plane and the time axis;

the reference image compression section and the difference image compression section reduce, when the sum total of differences of pixel values between images obtained by reducing once and then enlarging images included in the data blocks by a predetermined magnification and the original images is equal to or lower than a predetermined threshold value in the compression process, the images included in the data blocks by the pertaining magnification; and

the compression data generation section includes pointer information for associating a position of each data block in the tile images and the compression data of the reference image and the difference images corresponding to the position with each other into the moving picture compression data.

3. The moving picture compression apparatus according to claim 2, wherein the reference image compression section and the difference image compression section replace, when the sum total of differences of pixel values between an image obtained by averaging the image sequence included in the data blocks for each predetermined number and the original images is equal to or lower than the predetermined threshold value in the compression process, the original images with the averaged image to decrease the number of images.

4. The moving picture compression apparatus according to claim 1, wherein:

the difference image compression section determines, when absolute values of pixel values configuring each data block are equal to or lower than a predetermined threshold value, all of the pixel values of the data block so as to be zero; and

the compression data generation section associates the data block and the compression data with each other such that the compression data whose pixel values are zero is shared by a plurality of data blocks in the moving picture compression data.

5. The moving picture compression apparatus according to claim 1, wherein the reference image compression section generates, for each predetermined number of pixels of data blocks of the reference image obtained by dividing the reference image on the image plane and the time axis, a pallet which holds two values from among pixel values as representative values and an index which holds, in place of the pixel values, information which designates one of a plurality of intermediate values determined by linearly interpolating the representative values and the representative values to quantize the pixel values.

6. The moving picture compression apparatus according to claim 1, wherein the difference image compression section generates, for each predetermined number of pixels of data blocks of the difference image, a pallet which holds four values obtained by multiplying four values which are representative values from among pixel values of the pixels by 1/N (N is a natural number) and an index which holds, in place of the pixel values, information which designates one of the four values held by the pallet to quantize the pixel values.

7. The moving picture compression apparatus according to claim 6, wherein the difference image compression section represents an identification number representative of the magnification N to be multiplied by the values held by the pallet in order to restore the representative values by a magnitude relationship between and a storage order of the two values stored in a first half address from among the four values held by the pallet and a magnitude relationship between and a storage order of the two values stored in a latter half address.

8. The moving picture compression apparatus according to claim 5, wherein the reference image compression section and the difference image compression section represent an identification number for specifying a corresponding relationship between the index and a position of pixels of the original data block by a magnitude relationship between a value stored in the first half address and a value stored in the latter half address of the corresponding pallet.

9. The moving picture compression apparatus according to claim 5, wherein the difference image generation section performs, when the difference image generation section enlarges an image in the process for decoding the compression data of the reference image, interpolation and extrapolation in units equal to or smaller than an array of pixels in an image configured from the predetermined number of pixels which are the units when the pallet and the index are generated to generate new pixels.

10. An image processing apparatus, comprising:

an information processing section configured to successively calculate a frame to be displayed from within a frame sequence which configures moving picture data and a region to be displayed in the frame;

a loading section configured to load, from a storage apparatus in which moving picture compression data including compression data of a reference image obtained by compressing a tile image sequence obtained by dividing the frame sequence on an image plane and compression data of a difference image representative of a difference between an image obtained by decoding the compression data of the reference image and the corresponding tile image are stored, the moving picture compression data in units of tile images into a memory based on the information calculated by the information processing section;

a display image processing section configured to read out, from within the moving picture compression data loaded in the memory, moving picture compression data calculated by the information processing section and including a region to be displayed from within the frame to be displayed and decode and add the compression data of the reference image and the compression data of the difference image to sequentially render an image of the region to be displayed; and

a display section configured to sequentially display the rendered images.

11. The image processing apparatus according to claim 10, wherein:

the compression data of the reference image and the compression data of the difference image are data compressed in variable length in units of data blocks obtained by dividing the compression data on the image plane and a time axis; and

the display image processing section refers to pointer information included in the moving picture compression data and associating a position of each data block in the tile image and the compression data of the reference image and the compression data of the difference image corresponding to the position with each other to specify the compression data of the reference image and the compression data of the difference image of the data block corresponding to the region to be displayed.

12. The image processing apparatus according to claim 10, wherein:

the compression data of the difference image is configured, for each predetermined number of pixels, from a pallet which holds four values obtained by multiplying four values which are representative values from among pixel values of the pixels by 1/N (N is a natural number) and an index which holds, in place of a pixel value, information which designates one of the four values held by the pallet; and

the display image processing section specifies an identification number representative of the magnification N to be multiplied by the values held by the pallet in order to restore the representative values from a magnitude relationship between and a storage order of the two values stored in a first half address from among the four values held by the pallet and a magnitude relationship between and a storage order of the two values stored in a latter half address to restore the representative values.

13. The image processing apparatus according to claim 12, wherein the display image processing section specifies an identification number for specifying a corresponding relationship between the index and the position of the original pixel in the compression data of the difference image from a magnitude relationship of a value stored in the first half address and a value stored in the latter half address of the corresponding pallet and specifies the index of the pixels which configure the region to be displayed based on the specified identification number.

14. The image processing apparatus according to claim 10, wherein:

the compression data of the reference image and the compression data of the difference image are data obtained by quantizing pixel values for each predetermined number of pixels; and

when an image is to be enlarged in the process for decoding the compression data of the reference image and the compression data of the difference image, the display image processing section carries out interpolation and extrapolation in units equal to or smaller than an array of pixels in an image configured from the predetermined number of pixels which are the units when the quantization is carried out to generate new pixels.

15. A moving picture compression method, comprising:

by a moving picture compression apparatus,

a step of generating a tile image sequence configured by dividing a frame sequence, which is stored in a storage apparatus and configures moving picture data, on an image plane;

a step of compressing the tile image sequence to generate compression data of a reference image;

a step of generating a difference image sequence configured from difference images which represent a difference between tile images which configure the tile image sequence and images obtained by decoding the compression data of the generated reference image and corresponding to the tile images;

a step of compressing the difference image sequence in units of data blocks obtained by dividing the difference image sequence on the image plane and a time axis to generate compression data of the difference images; and

a step of generating moving picture compression data including the compression data of the reference image and the compression data of the difference images in tile image units of a predetermined number of frames and outputting the moving picture compression data to the storage apparatus.

16. An image processing method, comprising:

by an image processing apparatus,

a step of successively calculating a frame to be displayed from within a frame sequence which configures moving picture data and a region to be displayed in the frame;

a step of loading, from a storage apparatus in which moving picture compression data including compression data of a reference image obtained by compressing a tile image sequence obtained by dividing the frame sequence on an image plane and compression data of a difference image representative of a difference between an image obtained by decoding the compression data of the reference image and the corresponding tile image are stored, the moving picture compression data in units of tile images into a memory based on the information calculated by the calculating step;

a step of reading out, from within the moving picture compression data loaded in the memory, moving picture compression data calculated by the calculating step and including a region to be displayed from within the frame to be displayed and decoding and adding the compression data of the reference image and the compression data of the difference image to sequentially render an image of the region to be displayed; and

a step of sequentially displaying the rendered images.

17. A computer program for a computer, comprising:

generating a tile image sequence configured by dividing a frame sequence, which is stored in a storage apparatus and configures moving picture data, on an image plane;

compressing the tile image sequence to generate compression data of a reference image;

generating a difference image sequence configured from difference images which represent a difference between tile images which configure the tile image sequence and images obtained by decoding the compression data of the generated reference image and corresponding to the tile images;

compressing the difference image sequence in units of data blocks obtained by dividing the difference image sequence on the image plane and a time axis to generate compression data of the difference images; and

generating moving picture compression data including the compression data of the reference image and the compression data of the difference images in tile image units of a predetermined number of frames and outputting the moving picture compression data to the storage apparatus.

18. A computer program for a computer, comprising:

successively calculating a frame to be displayed from within a frame sequence which configures moving picture data and a region to be displayed in the frame;

loading, from a storage apparatus in which moving picture compression data including compression data of a reference image obtained by compressing a tile image sequence obtained by dividing the frame sequence on an image plane and compression data of a difference image representative of a difference between an image obtained by decoding the compression data of the reference image and the corresponding tile image are stored, the moving picture compression data in units of tile images into a memory based on the calculated information;

reading out, from within the moving picture compression data loaded in the memory, calculated moving picture compression data including a region to be displayed from within the frame to be displayed and decoding and adding the compression data of the reference image and the compression data of the difference image to sequentially render an image of the region to be displayed; and

sequentially displaying the rendered images.

19-20. -20. (canceled)

21. A data structure of a moving picture compression file

wherein compression data of a reference image obtained by compressing tile images, which configure a tile image sequence obtained by dividing a frame sequence which configures moving picture data on an image plane, in units of data blocks obtained by dividing the tile images on the image plane and a time axis and

compression data of a difference image obtained by compressing a difference image sequence, which is configured from difference images representative of a difference between an image obtained by decoding the compression data of the reference image and the corresponding tile image, in units of data blocks obtained by dividing the difference image sequence on the image plane and the time axis, are associated with each other in units of tile images of a predetermined number of frames;

and

in an image processing apparatus, the compression data of the reference image and the compression data of the difference image of the data block loaded in the tile image units based on information of the frame to be displayed and a region to be displayed in the frame and corresponding to the region to be displayed from within the frame to be displayed are decoded and added to be used to sequentially render an image of the region to be displayed.

22. (canceled)