Method and apparatus for converting motion image data, method and apparatus for reproducing motion image data, computer program for converting motion image data, and computer program for reproducing motion image data

• US 20050259740A1
• Filed: 04/30/2005
• Published: 11/24/2005
• Est. Priority Date: 05/18/2004
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Kobayashi, Seiji, Hirasawa, Yasutaka
• Granted Patent: US 7,813,430 B2
• Time in Patent Office: Days
• US Class Current: 375/240.160
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/137 Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/182 the unit being a pixel

Abstract

In an apparatus that converts (compresses) motion image data, the amount of motion of a subject in a block of motion image data is detected, and spatial decimation is performed in a mode in which sampling point positions are fixed or in a mode in which sampling point positions are shifted, depending on the detected amount of motion. When predicted image quality corresponding to the moving speed of a subject detected by analyzing blocks is lower than a predetermined threshold level, spatial decimation is performed in the mode in sampling point positions are shifted such that the moving speed of the subject is virtually changed to a value at which a super resolution effect occurs, thereby achieving data conversion without causing significant degradation in image quality.

Claims

1. A motion image data conversion apparatus that performs data conversion of motion image data, comprising:

a block divider that divides each frame of the motion image data into blocks;

a motion detector that detects the amount of motion of a subject for each block generated by the block divider; and

a block processing unit that receives block data of blocks generated by the block divider and the data of motion detected by the motion detector and that decimates the block data,

the block processing unit including

a decimation mode determination unit that determines a decimation mode as to whether to perform spatial decimation in a fixed sampling point position mode or a sampling point position shifting mode, in accordance with the data of motion; and

a decimation execution unit that executes spatial decimation in the fixed sampling point position mode or the sampling point position shifting mode in accordance with a determination made by the decimation mode determination unit.

2. A motion image data conversion apparatus according to claim 1, wherein:

the decimation mode determination unit further determines a direction in which the sample point position is shifted with frame advance in the spatial decimation process in the sampling point position shifting mode; and

the decimation execution unit shifts the sample point position in the direction determined by the decimation mode determination unit when the spatial decimation process is performed in the sampling point position shifting mode.

3. A motion image data conversion apparatus according to claim 1, wherein the decimation mode determination unit determines the decimation mode in accordance with data indicating the correspondence between an evaluated image quality score and a moving speed of a subject, the evaluated image quality score being estimated for data obtained as a result of the decimation process in the fixed sampling point position mode, the determination being made such that

when a moving speed of a subject indicated by the data of motion detected by the motion detector corresponds to an evaluated image quality score equal to or higher than a predetermined threshold value, the decimation mode determination unit determines that the decimation process should be performed in the fixed process point position mode, but

when a moving speed of a subject indicated by the data of motion detected by the motion detector corresponds to an evaluated image quality score lower than the predetermined threshold value, the decimation mode determination unit determines that the decimation process should be performed in the process point position shifting mode.

4. A motion image data conversion apparatus according to claim 1, wherein:

the decimation mode determination unit further includes a memory adapted to store a decimation mode determined for a previous frame by the decimation mode determination unit; and

the decimation mode determination unit determines the decimation mode in accordance with data indicating the correspondence between an evaluated image quality score and a moving speed of a subject, the evaluated image quality score being estimated for data obtained as a result of the decimation process in the fixed sampling point position mode, and also in accordance with the decimation mode determined for the previous frame, the determination being made such that

when the decimation mode determined for the previous frame was the fixed sample point position mode and when a moving speed of a subject indicated by the data of motion detected by the motion detector corresponds to an evaluated image quality score equal to or higher than a predetermined threshold value T1, the decimation mode determination unit determines that the decimation process should be performed in the fixed process point position mode,

when the decimation mode determined for the previous frame was the fixed sample point position mode and when a moving speed of a subject indicated by the data of motion detected by the motion detector corresponds to an evaluated image quality score lower than the predetermined threshold value T1, the decimation mode determination unit determines that the decimation process should be performed in the process point position shifting mode,

when the decimation mode determined for the previous frame was the sample point position shifting mode and when a moving speed of a subject indicated by the data of motion detected by the motion detector corresponds to an evaluated image quality score lower than a predetermined threshold value T2, the decimation mode determination unit determines that the decimation process should be performed in the process point position shifting mode, and

when the decimation mode determined for the previous frame was the sample point position shifting mode and when a moving speed of a subject indicated by the data of motion detected by the motion detector corresponds to an evaluated image quality score equal to or higher than the predetermined threshold value T2, the decimation mode determination unit determines that the decimation process should be performed in the fixed process point position mode.

5. A motion image data conversion apparatus according to claim 1, wherein the block processing unit performs the spatial decimation process in the sample point position shifting mode such that a virtual moving speed of a subject resulting from execution of the decimation process in the sample point position shifting mode falls within a range of the moving speed in which an evaluated image quality score corresponding to the virtual moving speed is equal to or higher than a predetermined threshold value, the evaluated image quality score being estimated based on prepared data indicating the correspondence between the moving speed of a subject and the evaluated image quality score of data obtained as a result of the decimation process in the fixed sampling point position mode.

6. A motion image data conversion apparatus according to claim 1, wherein the block processing unit further includes a memory adapted to store data indicating a mode in which the sample point position is shifted or fixed in the spatial decimation process performed by the block processing unit for a previous frame;

and

the block processing unit performs a spatial decimation process on a current frame in a mode in which the sampling point position is shifted or fixed, the mode being determined such that

when the virtual moving speed of a subject obtained as a result of spatial decimation performed on the previous frame depending on the mode in which the sampling point position was shifted or fixed is evaluated based on data indicating the correspondence between the moving speed of a subject and the image quality score for data obtained as a result of decimation in the fixed sampling point position mode, if an image quality score corresponding to the virtual moving speed of the subject is equal to or higher than a predetermined threshold value T3, then the spatial decimation is performed in the same mode as the mode used in the spatial decimation performed on the previous frame, but

when the virtual moving speed of the subject obtained as the result of spatial decimation performed on the previous frame depending on the mode in which the sampling point position was shifted or fixed is evaluated based on data indicating the correspondence between the moving speed of a subject and the image quality score for data obtained as a result of decimation in the fixed sampling point position mode, if the image quality score corresponding to the virtual moving speed of the subject is lower than the predetermined threshold value T3, then the spatial decimation is performed in a mode selected such that the virtual speed of the subject obtained as a result of the decimation falls within a range of the moving speed in which the image quality score evaluated based on the data indicating the correspondence between the moving speed of a subject and the image quality score for data obtained as a result of decimation in the fixed sampling point position mode is equal to or higher than a predetermined threshold value T4.

7. A motion image data conversion apparatus according to claim 1, wherein the block processing unit performs the decimation process in one of following modes (a) to (c) selected based on the data of motion:

(a) mode in which only spatial decimation is performed;

(b) mode in which spatial decimation and temporal decimation are performed;

or

(c) mode in which only temporal decimation is performed.

8. A motion image data conversion apparatus according to claim 1, wherein:

the motion detector detects a motion vector based on blocks that correspond to each other and that belong to different frames; and

the block processing unit determines the decimation mode based on the motion vector and performs the decimation process in the determined mode.

9. A motion image reproduction apparatus that reproduces motion image data from converted motion image data, comprising:

a block expander that receives converted block data included in the converted motion image data and data indicating a conversion mode in which the converted block data was converted and that expands the converted block data based on the data indicating the conversion mode; and

a combiner that produces frame data by combining blocks reproduced via the block expansion performed by the block expander, wherein

the block expander reproduces the block by performing the block expansion process in a mode corresponding to at least one of a spatial decimation process in a fixed sampling point position mode, a spatial decimation process in a sampling point position shifting mode, and a temporal decimation process, performed in the production of the converted motion image data; and

the combiner produces the frame data by combining blocks reproduced via the block expansion process.

10. A motion image reproduction apparatus according to claim 9, wherein when the blocks reproduced by the block expander are blocks reproduced from blocks subjected to the spatial decimation process in the sampling point position shifting mode, the combiner shifts the positions at which the blocks are laid with frame advance.

11. A motion image reproduction apparatus according to claim 9, wherein when the blocks reproduced by the block expander are blocks reproduced from blocks subjected to the spatial decimation process in the sampling point position shifting mode, the combiner shifts the positions at which the blocks are laid with frame advance and performs a pixel value correction process to determine a pixel value in a pixel gap created when blocks were laid or determine a pixel value for an overlapping pixel.

12. A method of converting motion image data, comprising the steps of:

dividing each frame of the motion image data into blocks;

detecting the amount of motion of a subject for each block generated in the block dividing step;

processing block data by receiving block data of blocks generated in the block dividing step and the data of motion detected in the step of detecting the amount of motion, and decimating the block data,

the block data processing step including the steps of

determining a decimation mode as to whether to perform spatial decimation in a fixed sampling point position mode or a sampling point position shifting mode, in accordance with the data of motion; and

executing spatial decimation in the fixed sampling point position mode or the sampling point position shifting mode in accordance with a determination made in the decimation mode determination step.

13. A method of converting motion image data, according to claim 12, wherein:

the decimation mode determination step includes the step of determining a direction in which the sample point position is shifted with frame advance in the spatial decimation process in the sampling point position shifting mode; and

in the decimation execution step, spatial decimation is performed while shifting the sampling point position in the direction determined in the decimation mode determination step.

14. A method of converting motion image data, according to claim 12, wherein in the decimation mode determination step, the decimation mode is determined in accordance with data indicating the correspondence between an image quality score and a moving speed of a subject, the image quality score being estimated based on data obtained as a result of the decimation process in the fixed sampling point position mode, the determination being made such that

when a moving speed of a subject indicated by the data of motion detected in the step of detecting the amount of motion corresponds to an evaluated image quality score equal to or higher than a predetermined threshold value, it is determined that the decimation process should be performed in the fixed process point position mode, but

when a moving speed of a subject indicated by the data of motion detected in the step of detecting the amount of motion corresponds to an evaluated image quality score lower than the predetermined threshold value, it is determined that the decimation process should be performed in the process point position shifting mode.

15. A method of reproducing motion image data from converted motion image data, comprising the steps of:

expanding a block by receiving converted block data included in the converted motion image data and data indicating a conversion mode in which the converted block data was converted, and expanding the converted block data based on the data indicating the conversion mode; and

combining blocks reproduced in the block expansion step to produce frame data, wherein

the block expansion step reproduces the block by performing the block expansion process in a mode corresponding to at least one of a spatial decimation process in a fixed sampling point position mode, a spatial decimation process in a sampling point position shifting mode, and a temporal decimation process, performed in the production of the converted motion image data; and

the block combining step produces the frame data by combining blocks reproduced via the block expansion process.

16. A method of reproducing motion image data, according to Step 15, wherein in the block combining step, when the blocks reproduced by the block expander are blocks reproduced from blocks subjected to the spatial decimation process in the sampling point position shifting mode, the positions at which to lay the blocks are shifted with frame advance.

17. A method of reproducing motion image data, according to Step 15, wherein the block combining step includes the step of, when the blocks reproduced by the block expander are blocks reproduced from blocks subjected to the spatial decimation process in the sampling point position shifting mode, shifting the positions at which the blocks are laid with frame advance and performing a pixel value correction process to determine a pixel value in a pixel gap created when blocks were laid or determine a pixel value for an overlapping pixel.

18. A computer program for executing a motion image data conversion process, the process comprising the steps of:

dividing each frame of the motion image data into blocks;

detecting the amount of motion of a subject for each block generated in the block dividing step;

processing block data by receiving block data of blocks generated in the block dividing step and the data of motion detected in the step of detecting the amount of motion, and decimating the block data,

the block data processing step including the steps of

determining a decimation mode as to whether to perform spatial decimation in a fixed sampling point position mode or a sampling point position shifting mode, in accordance with the data of motion; and

executing spatial decimation in the fixed sampling point position mode or the sampling point position shifting mode in accordance with a determination made in the decimation mode determination step.

19. A computer program for executing a process of reproducing motion image data from converted motion image data, the process comprising the steps of:

expanding a block by receiving converted block data included in the converted motion image data and data indicating a conversion mode in which the converted block data was converted, and expanding the converted block data based on the data indicating the conversion mode; and

combining blocks reproduced in the block expansion step to produce frame data, wherein

the block expansion step reproduces the block by performing the block expansion process in a mode corresponding to at least one of a spatial decimation process in a fixed sampling point position mode, a spatial decimation process in a sampling point position shifting mode, and a temporal decimation process, performed in the production of the converted motion image data; and

the block combining step produces the frame data by combining blocks reproduced via the block expansion process.