Image processing apparatus, image processing method, and program for generating motion compensated image data

• US 10,587,892 B2
• Filed: 08/28/2014
• Issued: 03/10/2020
• Est. Priority Date: 06/18/2007
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Nakagami, Ohji, Tanaka, Junichi, Yagasaki, Yoichi
• Primary Examiner(s): Jebari, Mohammed
• Application Number:
  US14/471,108
• Publication Number:
  US 20150043646A1
• Time in Patent Office: 2,021 Days
• CPC Class Codes: H04N 19/105 Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
• CPC Class Codes: H04N 19/11 among a plurality of spatial predictive coding modes
• CPC Class Codes: H04N 19/139 Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/33 in the spatial domain
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/573 Motion compensation with multiple frame prediction using two or more reference frames in a given prediction direction
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure
• CPC Class Codes: H04N 5/91 Television signal processing therefor
• CPC Class Codes: H04N 7/0132 the field or frame frequency of the incoming video signal being multiplied by a positive integer, e.g. for flicker reduction
• CPC Class Codes: H04N 7/014 involving the use of motion vectors motion estimation and compensation in video coding H04N19/51
• CPC Class Codes: H04N 9/8045 using predictive coding

Abstract

There is provided an image processing apparatus including a an acquiring unit for acquiring moving image data containing a plurality of successive frames, and one or a plurality of image data corresponding to the frames and having a spatial resolution higher than the frames; a motion prediction unit for detecting a motion vector between the frames using the moving image data; a difference amount calculation unit for calculating a difference amount between a predetermined frame and the frame corresponding to the image data; and an image generation unit capable of generating motion compensated image data corresponding to the predetermined frame based on the frame corresponding to the image data and the motion vector.

Claims

1. An image processing apparatus, comprising:

a receiving unit configured to receive first video data containing a plurality of frames having a first resolution and second video data containing a plurality of frames having a second resolution higher than the first resolution;

a motion prediction unit configured to detect a motion vector between the frames contained in the first video data received by the receiving unit;

a difference amount calculation unit configured to calculate a difference amount between a reference block of a frame of the first video data having the first resolution and a reference block of a frame of the second video data having the second resolution higher than the first resolution, wherein the difference amount is a value representing a difference between the reference block of the frame of the first video data having the first resolution and the reference block of the frame of the second video data having the second resolution higher than the first resolution; and

an image generation unit configured to

perform a process associated with motion compensation by generating, based on the frame of the second video data having the second resolution and the motion vector detected by the motion prediction unit, motion compensated image data having a same resolution as the second video data having the second resolution, and to output the motion compensated image data as a result of the motion compensation when the difference amount calculated by the difference amount calculation unit is smaller than a predetermined value, and

suppress occurrence of noise due to motion vector prediction accuracy in information of a high frequency component contained in the second video data by preventing a shift in a position of the reference block of the frame of the second video data utilized in the process associated with motion compensation,

wherein the process associated with motion compensation is adaptively switched and outputting a picture, created by modifying the frame of the first video data, as the result of the motion compensation instead of outputting the motion compensated image data as the result of the motion compensation, when the difference amount is not smaller than the predetermined value and the process associated with motion compensation is adaptively switched, and

wherein the receiving unit, the motion prediction unit, the difference amount calculation unit, and the image generation unit are each implemented via at least one processor.

2. The image processing apparatus according to claim 1, wherein the image generation unit is further configured to generate a frame of video data in which the frame of the first video data is magnified to a same spatial resolution and having a same pixel density as the frame of the second video data when the difference amount calculated by the difference amount calculation unit is not smaller than the predetermined value.

3. The image processing apparatus according to claim 2, wherein the motion prediction unit detects the motion vector between magnified frames of the frames contained in the first video data received by the receiving unit, the magnified frames having a same resolution as the second resolution.

4. The image processing apparatus according to claim 2, wherein the difference amount calculation unit calculates a difference amount between a magnified frame of the first video data and the frame of the second video data, the magnified frame having a same resolution as the second resolution.

5. The image processing apparatus according to claim 2, wherein the image generation unit generates, based on the frame of the second video data and the motion vector detected by the motion prediction unit, the frame of video data in which the frame of the first video data is magnified to have a same number of pixels as the frame of the second video data when the difference amount calculated by the difference amount calculation unit is greater than the predetermined value.

6. The image processing apparatus according to claim 1, wherein the image generation unit is further configured to change a ratio of combining the frame of the first video data, which is a reference source of the motion vector, and the frame of the second video data, which is a reference destination of the motion vector, according to the difference amount calculated by the difference amount calculation unit when performing the motion compensation when the difference amount calculated by the difference amount calculation unit is greater than the predetermined value.

7. The image processing apparatus according to claim 1, wherein

the first video data has higher temporal resolution than the second video data, and

the second video data has lower temporal resolution than the first video data.

8. The image processing apparatus according to claim 1, wherein the difference amount calculation unit calculates a difference amount between a block of the frame of the first video data and a block of the frame of the second video data.

9. The image processing apparatus according to claim 1, wherein the difference amount is calculated by a Mean Square Error (MSE) between the frame of the first video data and the frame of the second video data.

10. The image processing apparatus according to claim 1, wherein the difference amount is calculated by a Sum of Absolute Difference (SAD) between the frame of the first video data and the frame of the second video data.

11. The image processing apparatus according to claim 1, wherein the difference amount is calculated by generating a differential picture between the frame of the first video data and the frame of the second video data.

14. The image processing apparatus according to claim 1, wherein the first resolution of the plurality of frames of the first video data is a higher temporal resolution than the second resolution of the plurality of frames of the second video data.

12. An image processing method, comprising:

receiving first video data containing a plurality of frames having a first resolution and second video data containing a plurality of frames having a second resolution higher than the first resolution;

detecting a motion vector between the frames contained in the received first video data;

calculating a difference amount between a reference block of a frame of the first video data having the first resolution and a reference block of a frame of the second video data having the second resolution higher than the first resolution, wherein the difference amount is a value representing a difference between the reference block of the frame of the first video data having the first resolution and the reference block of the frame of the second video data having the second resolution higher than the first resolution;

performing a process associated with motion compensation by generating, based on the frame of the second video data having the second resolution and the detected motion vector, motion compensated image data having a same resolution as the second video data having the second resolution, and outputting the motion compensated image data as a result of the motion compensation when the difference amount calculated in the difference amount calculation step is smaller than a predetermined value;

suppressing occurrence of noise due to motion vector prediction accuracy in information of a high frequency component contained in the second video data by preventing a shift in a position of the reference block of the frame of the second video data utilized in the process associated with motion compensation,

wherein the process associated with motion compensation is adaptively switched and outputting a picture, created by modifying the frame of the first video data, as the result of the motion compensation instead of outputting the motion compensated image data as the result of the motion compensation, when the difference amount is not smaller than the predetermined value and the process associated with motion compensation is adaptively switched.

15. The image processing method according to claim 12, wherein the first resolution of the plurality of frames of the first video data is a higher temporal resolution than the second resolution of the plurality of frames of the second video data.

13. A non-transitory computer-readable storage medium having embodied thereon a program, which when executed by a computer causes the computer to execute a method, the method comprising:

receiving first video data containing a plurality of frames having a first resolution and second video data containing a plurality of frames having a second resolution higher than the first resolution;

detecting a motion vector between the frames contained in the received first video data;

calculating a difference amount between a reference block of a frame of the first video data having the first resolution and a reference block of a frame of the second video data having the second resolution higher than the first resolution, wherein the difference amount is a value representing a difference between the reference block of the frame of the first video data having the first resolution and the reference block of the frame of the second video data having the second resolution higher than the first resolution;

performing a process associated with motion compensation by generating, based on the frame of the second video data having the second resolution and the detected motion vector, motion compensated image data having a same resolution as the second video data having the second resolution, and outputting the motion compensated image data as a result of the motion compensation when the difference amount calculated in the difference amount calculation step is smaller than a predetermined value;

suppressing occurrence of noise due to motion vector prediction accuracy in information of a high frequency component contained in the second video data by preventing a shift in a position of the reference block of the frame of the second video data utilized in the process associated with motion compensation,

wherein the process associated with motion compensation is adaptively switched and outputting a picture, created by modifying the frame of the first video data, as the result of the motion compensation instead of outputting the motion compensated image data as the result of the motion compensation, when the difference amount is not smaller than the predetermined value and the process associated with motion compensation is adaptively switched.

16. The non-transitory computer-readable storage medium according to claim 13, wherein the first resolution of the plurality of frames of the first video data is a higher temporal resolution than the second resolution of the plurality of frames of the second video data.