Moving picture compression apparatus, image processing apparatus, moving picture compression method, image processing method, and data structure of moving picture compression file

• US 9,693,072 B2
• Filed: 10/24/2012
• Issued: 06/27/2017
• Est. Priority Date: 12/27/2011
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.), Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Ohba, Akio, Segawa, Hiroyuki, Inada, Tetsugo
• Primary Examiner(s): Czekaj, Dave
• Assistant Examiner(s): Abimbola, Kehinde O
• Application Number:
  US14/366,498
• Publication Number:
  US 20150124883A1
• Time in Patent Office: 1,707 Days
• CPC Class Codes: H04N 19/105 Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
• CPC Class Codes: H04N 19/137 Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/167 Position within a video image, e.g. region of interest [ROI]
• CPC Class Codes: H04N 19/174 the region being a slice, e.g. a line of blocks or a group of blocks
• CPC Class Codes: H04N 19/198 including smoothing of a sequence of encoding parameters, e.g. by averaging, by choice of the maximum, minimum or median value
• CPC Class Codes: H04N 19/428 Recompression, e.g. by spatial or temporal decimation
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/593 involving spatial prediction techniques

Abstract

A frame sequence of moving picture data is divided into a tile image sequence, and the color space of the tile image sequence is converted to generate a YCbCr image sequence. Each frame is reduced to ½ time in the vertical and horizontal directions, and a compression process is carried out to generate compression data of a reference image. The compression data of the reference image is decoded and decompressed similarly as upon image display to restore a YCbCr image as the reference image, and a difference image sequence is generated from the reference image and the original YCbCr image. Then, compression data of a difference image is generated, and compression data obtained by connecting the compression data of the reference image and the compression data of the difference image is generated for every four frames of a tile image.

Claims

1. A moving picture compression apparatus, comprising:

a tile image sequence generation section configured to generate a tile image sequence configured by dividing a frame sequence including a plurality of frame images sequentially positioned along a time axis, into a plurality of tile images, the frame sequence configuring moving picture data, the dividing of the frame sequence including dividing each frame image into a same number and spatial arrangement, on a respective image plane, of tile images, the tile image sequence being a subset of the tile images, the tile images in the subset being sequentially positioned along the time axis and each having a same spatial position on its respective image plane as the other tile images in the subset;

a reference image compression section configured to compress the tile image sequence to generate compression data of a reference image;

a difference image generation section configured to generate a difference image sequence configured from difference images which represent a difference between tile images configuring the tile image sequence and images obtained by decoding the compression data of the reference image generated by the reference image compression section and corresponding to the tile images;

a difference image compression section configured to compress the difference image sequence in units of data blocks obtained by dividing the difference image sequence on the image plane and a time axis to generate compression data of the difference images; and

a compression data generation section configured to generate moving picture compression data including the compression data of the reference image and the compression data of the difference images in tile image units of a predetermined number of frames and output the moving picture compression data to a storage apparatus,

wherein at least one of;

the reference image compression section generates, for each predetermined number of pixels of data blocks of the reference image obtained by dividing the reference image on the image plane and the time axis, a pallet which holds two values from among pixel values as representative values and an index which holds, in place of the pixel values, information which designates one of a plurality of intermediate values determined by linearly interpolating the representative values and the representative values to quantize the pixel values, and

the difference image compression section generates, for each predetermined number of pixels of data blocks of the difference image, a pallet which holds four values obtained by multiplying four values which are representative values from among pixel values of the pixels by 1/N (N is a natural number) and an index which holds, in place of the pixel values, information which designates one of the four values held by the pallet to quantize the pixel values.

2. The moving picture compression apparatus according to claim 1, wherein:

the reference image compression section compresses the reference image in units of the data blocks obtained by dividing the reference image on the image plane and the time axis;

the reference image compression section and the difference image compression section reduce, when the sum total of differences of pixel values between images obtained by reducing once and then enlarging images included in the data blocks by a predetermined magnification and the original images is equal to or lower than a predetermined threshold value in the compression process, the images included in the data blocks by the pertaining magnification; and

the compression data generation section includes pointer information for associating a position of each data block in the tile images and the compression data of the reference image and the difference images corresponding to the position with each other into the moving picture compression data.

3. The moving picture compression apparatus according to claim 2, wherein the reference image compression section and the difference image compression section replace, when the sum total of differences of pixel values between an image obtained by averaging the image sequence included in the data blocks for each predetermined number and the original images is equal to or lower than the predetermined threshold value in the compression process, the original images with the averaged image to decrease the number of images.

4. The moving picture compression apparatus according to claim 1, wherein:

the difference image compression section determines, when absolute values of pixel values configuring each data block are equal to or lower than a predetermined threshold value, all of the pixel values of the data block so as to be zero; and

the compression data generation section associates the data block and the compression data with each other such that the compression data whose pixel values are zero is shared by a plurality of data blocks in the moving picture compression data.

5. The moving picture compression apparatus according to claim 1, wherein the difference image compression section represents an identification number representative of the magnification N to be multiplied by the values held by the pallet in order to restore the representative values by a magnitude relationship between and a storage order of the two values stored in a first half address from among the four values held by the pallet and a magnitude relationship between and a storage order of the two values stored in a latter half address.

6. The moving picture compression apparatus according to claim 1, wherein the reference image compression section and the difference image compression section represent an identification number for specifying a corresponding relationship between the index and a position of pixels of the original data block by a magnitude relationship between a value stored in the first half address and a value stored in the latter half address of the corresponding pallet.

7. The moving picture compression apparatus according to claim 1, wherein the difference image generation section performs, when the difference image generation section enlarges an image in the process for decoding the compression data of the reference image, interpolation and extrapolation in units equal to or smaller than an array of pixels in an image configured from the predetermined number of pixels which are the units when the pallet and the index are generated to generate new pixels.

8. An image processing apparatus, comprising:

an information processing section configured to successively calculate a frame to be displayed from within a frame sequence and a region to be displayed in the frame;

a loading section configured to load, from a storage apparatus in which moving picture compression data including compression data of a reference image obtained by compressing a tile image sequence obtained by dividing the frame sequence and compression data of a difference image representative of a difference between an image obtained by decoding the compression data of the reference image and the corresponding tile image are stored, the moving picture compression data in units of tile images into a memory based on the information calculated by the information processing section;

a display image processing section configured to read out, from within the moving picture compression data loaded in the memory, moving picture compression data calculated by the information processing section and including a region to be displayed from within the frame to be displayed and decode and add the compression data of the reference image and the compression data of the difference image to sequentially render an image of the region to be displayed; and

a display section configured to sequentially display the rendered images,

wherein the tile image sequence is configured by dividing the frame sequence, which includes a plurality of frame images sequentially positioned along a time axis, into a plurality of tile images, the frame sequence configuring moving picture data, the dividing of the frame sequence including dividing each frame image into a same number and spatial arrangement, on a respective image plane, of tile images, the tile image sequence being a subset of the tile images, the tile images in the subset being sequentially positioned along the time axis and each having a same spatial position on its respective image plane as the other tile images in the subset, and

wherein at least one of;

(i) the compression data of the difference image is configured, for each predetermined number of pixels, from a pallet which holds four values obtained by multiplying four values which are representative values from among pixel values of the pixels by 1/N (N is a natural number) and an index which holds, in place of a pixel value, information which designates one of the four values held by the pallet, and the display image processing section specifies an identification number representative of the magnification N to be multiplied by the values held by the pallet in order to restore the representative values from a magnitude relationship between and a storage order of the two values stored in a first half address from among the four values held by the pallet and a magnitude relationship between and a storage order of the two values stored in a latter half address to restore the representative values; and

(ii) the compression data of the reference image and the compression data of the difference image are data obtained by quantizing pixel values for each predetermined number of pixels, and when an image is to be enlarged in the process for decoding the compression data of the reference image and the compression data of the difference image, the display image processing section carries out interpolation and extrapolation in units equal to or smaller than an array of pixels in an image configured from the predetermined number of pixels which are the units when the quantization is carried out to generate new pixels.

9. The image processing apparatus according to claim 8, wherein:

the compression data of the reference image and the compression data of the difference image are data compressed in variable length in units of data blocks obtained by dividing the compression data on the image plane and a time axis; and

the display image processing section refers to pointer information included in the moving picture compression data and associating a position of each data block in the tile image and the compression data of the reference image and the compression data of the difference image corresponding to the position with each other to specify the compression data of the reference image and the compression data of the difference image of the data block corresponding to the region to be displayed.

10. The image processing apparatus according to claim 8, wherein the display image processing section specifies an identification number for specifying a corresponding relationship between the index and the position of the original pixel in the compression data of the difference image from a magnitude relationship of a value stored in the first half address and a value stored in the latter half address of the corresponding pallet and specifies the index of the pixels which configure the region to be displayed based on the specified identification number.

11. A moving picture compression method, comprising:

by a moving picture compression apparatus,

generating a tile image sequence configured by dividing a frame sequence, which is stored in a storage apparatus and includes a plurality of frame images sequentially positioned along a time axis, into a plurality of tile images, the frame sequence configuring moving picture data, the dividing of the frame sequence including dividing each frame image into a same number and spatial arrangement, on a respective image plane, of tile images, the tile image sequence being a subset of the tile images, the tile images in the subset being sequentially positioned along the time axis and each having a same spatial position on its respective image plane as the other tile images in the subset;

compressing the tile image sequence to generate compression data of a reference image;

generating a difference image sequence configured from difference images which represent a difference between tile images which configure the tile image sequence and images obtained by decoding the compression data of the generated reference image and corresponding to the tile images;

compressing the difference image sequence in units of data blocks obtained by dividing the difference image sequence on the image plane and a time axis to generate compression data of the difference images; and

generating moving picture compression data including the compression data of the reference image and the compression data of the difference images in tile image units of a predetermined number of frames and outputting the moving picture compression data to the storage apparatus,

wherein at least one of;

compressing step generates, for each predetermined number of pixels of data blocks of the reference image obtained by dividing the reference image on the image plane and the time axis, a pallet which holds two values from among pixel values as representative values and an index which holds, in place of the pixel values, information which designates one of a plurality of intermediate values determined by linearly interpolating the representative values and the representative values to quantize the pixel values, and

the compressing the difference image sequence step includes generating, for each predetermined number of pixels of data blocks of the difference image, a pallet which holds four values obtained by multiplying four values which are representative values from among pixel values of the pixels by 1/N (N is a natural number) and an index which holds, in place of the pixel values, information which designates one of the four values held by the pallet to quantize the pixel values.

12. An image processing method, comprising:

by an image processing apparatus,

successively calculating a frame to be displayed from within a frame sequence and a region to be displayed in the frame;

loading, from a storage apparatus in which moving picture compression data including compression data of a reference image obtained by compressing a tile image sequence obtained by dividing the frame sequence and compression data of a difference image representative of a difference between an image obtained by decoding the compression data of the reference image and the corresponding tile image are stored, the moving picture compression data in units of tile images into a memory based on the information calculated by the calculating step;

reading out, from within the moving picture compression data loaded in the memory, moving picture compression data calculated by the calculating step and including a region to be displayed from within the frame to be displayed and decoding and adding the compression data of the reference image and the compression data of the difference image to sequentially render an image of the region to be displayed; and

sequentially displaying the rendered images,

wherein the tile image sequence is configured by dividing the frame sequence, which includes a plurality of frame images sequentially positioned along a time axis, into a plurality of tile images, the frame sequence configuring moving picture data, the dividing of the frame sequence including dividing each frame image into a same number and spatial arrangement, on a respective image plane, of tile images, the tile image sequence being a subset of the tile images, the tile images in the subset being sequentially positioned along the time axis and each having a same spatial position on its respective image plane as the other tile images in the subset, and

wherein at least one of;

(i) the compression data of the difference image is configured, for each predetermined number of pixels, from a pallet which holds four values obtained by multiplying four values which are representative values from among pixel values of the pixels by 1/N (N is a natural number) and an index which holds, in place of a pixel value, information which designates one of the four values held by the pallet, and the reading out step includes specifying an identification number representative of the magnification N to be multiplied by the values held by the pallet in order to restore the representative values from a magnitude relationship between and a storage order of the two values stored in a first half address from among the four values held by the pallet and a magnitude relationship between and a storage order of the two values stored in a latter half address to restore the representative values; and

(ii) the compression data of the reference image and the compression data of the difference image are data obtained by quantizing pixel values for each predetermined number of pixels, and when an image is to be enlarged in the process for decoding the compression data of the reference image and the compression data of the difference image, the reading out step includes carrying out interpolation and extrapolation in units equal to or smaller than an array of pixels in an image configured from the predetermined number of pixels which are the units when the quantization is carried out to generate new pixels.

13. A non-transitory, computer-readable storage medium containing a computer program for a computer, which when executed by the computer causes the computer to carry out actions, comprising:

generating a tile image sequence configured by dividing a frame sequence, which is stored in a storage apparatus and includes a plurality of frame images sequentially positioned along a time axis, into a plurality of tile images, the frame sequence configuring moving picture data, the dividing of the frame sequence including dividing each frame image into a same number and spatial arrangement, on a respective image plane, of tile images, the tile image sequence being a subset of the tile images, the tile images in the subset being sequentially positioned along the time axis and each having a same spatial position on its respective image plane as the other tile images in the subset;

compressing the tile image sequence to generate compression data of a reference image;

generating a difference image sequence configured from difference images which represent a difference between tile images which configure the tile image sequence and images obtained by decoding the compression data of the generated reference image and corresponding to the tile images;

compressing the difference image sequence in units of data blocks obtained by dividing the difference image sequence on the image plane and the time axis to generate compression data of the difference images; and

generating moving picture compression data including the compression data of the reference image and the compression data of the difference images in tile image units of a predetermined number of frames and outputting the moving picture compression data to the storage apparatus,

wherein at least one of;

compressing step generates, for each predetermined number of pixels of data blocks of the reference image obtained by dividing the reference image on the image plane and the time axis, a pallet which holds two values from among pixel values as representative values and an index which holds, in place of the pixel values, information which designates one of a plurality of intermediate values determined by linearly interpolating the representative values and the representative values to quantize the pixel values, and

the compressing the difference image sequence step includes generating, for each predetermined number of pixels of data blocks of the difference image, a pallet which holds four values obtained by multiplying four values which are representative values from among pixel values of the pixels by 1/N (N is a natural number) and an index which holds, in place of the pixel values, information which designates one of the four values held by the pallet to quantize the pixel values.

14. A non-transitory, computer-readable storage medium containing a computer program for a computer, which when executed by the computer causes the computer to carry out actions, comprising:

successively calculating a frame to be displayed from within a frame sequence and a region to be displayed in the frame;

loading, from a storage apparatus in which moving picture compression data including compression data of a reference image obtained by compressing a tile image sequence obtained by dividing the frame sequence and compression data of a difference image representative of a difference between an image obtained by decoding the compression data of the reference image and the corresponding tile image are stored, the moving picture compression data in units of tile images into a memory based on the calculated information;

reading out, from within the moving picture compression data loaded in the memory, calculated moving picture compression data including a region to be displayed from within the frame to be displayed and decoding and adding the compression data of the reference image and the compression data of the difference image to sequentially render an image of the region to be displayed; and

sequentially displaying the rendered images,

wherein the tile image sequence is configured by dividing the frame sequence, which includes a plurality of frame images sequentially positioned along a time axis, into a plurality of tile images, the frame sequence configuring moving picture data, the dividing of the frame sequence including dividing each frame image into a same number and spatial arrangement, on a respective image plane, of tile images, the tile image sequence being a subset of the tile images, the tile images in the subset being sequentially positioned along the time axis and each having a same spatial position on its respective image plane as the other tile images in the subset, and

wherein at least one of;

(i) the compression data of the difference image is configured, for each predetermined number of pixels, from a pallet which holds four values obtained by multiplying four values which are representative values from among pixel values of the pixels by 1/N (N is a natural number) and an index which holds, in place of a pixel value, information which designates one of the four values held by the pallet, and the reading out step includes specifying an identification number representative of the magnification N to be multiplied by the values held by the pallet in order to restore the representative values from a magnitude relationship between and a storage order of the two values stored in a first half address from among the four values held by the pallet and a magnitude relationship between and a storage order of the two values stored in a latter half address to restore the representative values; and

(ii) the compression data of the reference image and the compression data of the difference image are data obtained by quantizing pixel values for each predetermined number of pixels, and when an image is to be enlarged in the process for decoding the compression data of the reference image and the compression data of the difference image, the reading out step includes carrying out interpolation and extrapolation in units equal to or smaller than an array of pixels in an image configured from the predetermined number of pixels which are the units when the quantization is carried out to generate new pixels.

15. A non-transitory, computer-readable storage medium containing a data structure of a moving picture compression file,

wherein compression data of a reference image obtained by compressing tile images, which configure a tile image sequence obtained by dividing a frame sequence, in units of data blocks obtained by dividing the tile images on the image plane and a time axis and

compression data of a difference image obtained by compressing a difference image sequence, which is configured from difference images representative of a difference between an image obtained by decoding the compression data of the reference image and the corresponding tile image, in units of data blocks obtained by dividing the difference image sequence on the image plane and the time axis, are associated with each other in units of tile images of a predetermined number of frames;

and

in an image processing apparatus, the compression data of the reference image and the compression data of the difference image of the data block loaded in the tile image units based on information of the frame to be displayed and a region to be displayed in the frame and corresponding to the region to be displayed from within the frame to be displayed are decoded and added to be used to sequentially render an image of the region to be displayed,

wherein the tile image sequence is configured by dividing the frame sequence, which includes a plurality of frame images sequentially positioned along the time axis, into a plurality of tile images, the frame sequence configuring moving picture data, the dividing of the frame sequence including dividing each frame image into a same number and spatial arrangement, on a respective image plane, of tile images, the tile image sequence being a subset of the tile images, the tile images in the subset being sequentially positioned along the time axis and each having a same spatial position on its respective image plane as the other tile images in the subset, and

wherein at least one of;

(i) the compression data of the difference image is configured, for each predetermined number of pixels, from a pallet which holds four values obtained by multiplying four values which are representative values from among pixel values of the pixels by 1/N (N is a natural number) and an index which holds, in place of a pixel value, information which designates one of the four values held by the pallet, an identification number representative of the magnification N to be multiplied by the values held by the pallet is specified in order to restore the representative values from a magnitude relationship between and a storage order of the two values stored in a first half address from among the four values held by the pallet and a magnitude relationship between and a storage order of the two values stored in a latter half address to restore the representative values; and

(ii) the compression data of the reference image and the compression data of the difference image are data obtained by quantizing pixel values for each predetermined number of pixels, and when an image is to be enlarged in the process for decoding the compression data of the reference image and the compression data of the difference image, interpolation and extrapolation in units equal to or smaller than an array of pixels is carried out in an image configured from the predetermined number of pixels which are the units when the quantization is carried out to generate new pixels.