Image processing apparatus, image processing method, and program

• US 8,885,716 B2
• Filed: 06/10/2008
• Issued: 11/11/2014
• Est. Priority Date: 06/18/2007
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Nakagami, Ohji, Tanaka, Junichi, Yagasaki, Yoichi
• Primary Examiner(s): Senfi, Behrooz
• Assistant Examiner(s): Jebari, Mohammed
• Application Number:
  US12/664,208
• Publication Number:
  US 20100183072A1
• Time in Patent Office: 2,345 Days
• Field of Search: None
• US Class Current: 375/240.16
• CPC Class Codes: H04N 19/105 Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
• CPC Class Codes: H04N 19/11 among a plurality of spatial predictive coding modes
• CPC Class Codes: H04N 19/139 Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/33 in the spatial domain
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/573 Motion compensation with multiple frame prediction using two or more reference frames in a given prediction direction
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure
• CPC Class Codes: H04N 5/91 Television signal processing therefor
• CPC Class Codes: H04N 7/0132 the field or frame frequency of the incoming video signal being multiplied by a positive integer, e.g. for flicker reduction
• CPC Class Codes: H04N 7/014 involving the use of motion vectors motion estimation and compensation in video coding H04N19/51
• CPC Class Codes: H04N 9/8045 using predictive coding

Abstract

There is provided an image processing apparatus including a an acquiring unit for acquiring moving image data containing a plurality of successive frames, and one or a plurality of image data corresponding to the frames and having a spatial resolution higher than the frames; a motion prediction unit for detecting a motion vector between the frames using the moving image data; a difference amount calculation unit for calculating a difference amount between a predetermined frame and the frame corresponding to the image data; and an image generation unit capable of generating motion compensated image data corresponding to the predetermined frame based on the frame corresponding to the image data and the motion vector.

Claims

1. An image processing apparatus, comprising:

a receiving unit for receiving first video data containing a plurality of frames having a first resolution and second video data containing a plurality of frames having a second resolution higher than the first resolution;

a motion prediction unit for detecting a motion vector between the frames contained in the first video data received by the receiving unit;

a difference amount calculation unit for calculating a difference amount between a frame of the first video data and a frame of the second video data, wherein the difference amount is a value representing a difference between the frame of the first video data and the frame of the second video data; and

an image generation unit for, based on the frame of the second video data and the motion vector detected by the motion prediction unit, generating motion compensated image data having a same resolution as the second video data if the difference amount calculated by the difference amount calculation unit is smaller than a predetermined value, and generating a frame of video data in which the frame of the first video data is magnified to a same spatial resolution and having a same pixel density as the frame of the second video data if the difference amount calculated by the difference amount calculation unit is greater than the predetermined value.

2. The image processing apparatus according to claim 1, wherein the image generation unit changes a ratio of combining the frame of the first video data, which is a reference source of the motion vector, and the frame of the second video data, which is a reference destination of the motion vector, according to the difference amount calculated by the difference amount calculation unit when performing the motion compensation if the difference amount calculated by the difference amount calculation unit is greater than the predetermined value.

3. The image processing apparatus according to claim 1, wherein

the first video data has higher temporal resolution than the second video data, and

the second video data has lower temporal resolution than the first video data.

4. The image processing apparatus according to claim 1, wherein the motion prediction unit detects the motion vector between magnified frames of the frames contained in the first video data received by the receiving unit, the magnified frames having a same resolution as the second resolution.

5. The image processing apparatus according to claim 1, wherein the difference amount calculation unit calculates a difference amount between a magnified frame of the first video data and the frame of the second video data, the magnified frame having a same resolution as the second resolution.

6. The image processing apparatus according to claim 1, wherein the difference amount calculation unit calculates a difference amount between a block of the frame of the first video data and a block of the frame of the second video data.

7. The image processing apparatus according to claim 1, wherein the difference amount is calculated by a Mean Square Error (MSE) between the frame of the first video data and the frame of the second video data.

8. The image processing apparatus according to claim 1, wherein the difference amount is calculated by a Sum of Absolute Difference (SAD) between the frame of the first video data and the frame of the second video data.

9. The image processing apparatus according to claim 1, wherein the difference amount is calculated by generating a differential picture between frame of the first video data and the frame of the second video data.

10. The image processing apparatus according to claim 1, wherein the image generation unit, based on the frame of the second video data and the motion vector detected by the motion prediction unit, generates the frame of video data in which the frame of the first video data is magnified to have a same number of pixels as the frame of the second video data if the difference amount calculated by the difference amount calculation unit is greater than the predetermined value.

11. An image processing method, comprising the steps of:

receiving first video data containing a plurality of frames having a first resolution and second video data containing a plurality of frames having a second resolution higher than the first resolution;

detecting a motion vector between the frames contained in the first video data received in the receiving step;

calculating a difference amount between a frame of the first video data and a frame of the second video data, wherein the difference amount is a value representing a difference between the frame of the first video data and the frame of the second video data; and

generating, based on the frame of the second video data and the motion vector detected in the motion prediction step, motion compensated image data having a same resolution as the second video data if the difference amount calculated in the difference amount calculation step is smaller than a predetermined value, and a frame of video data in which the frame of the first video data is magnified to a same spatial resolution and having a same pixel density as the frame of the second video data if the difference amount calculated in the difference amount calculation step is greater than the predetermined value.

12. A non-transitory computer readable storage medium storing a program for causing a computer to execute the functions of:

receiving first video data containing a plurality of frames having a first resolution and second video data containing a plurality of frames having a second resolution higher than the first resolution;

detecting a motion vector between the frames contained in the first video data received by the receiving function;

calculating a difference amount between a frame of the first video data and a frame of the second video data, wherein the difference amount is a value representing a difference between the frame of the first video data and the frame of the second video data; and

generating, based on the frame of the second video data and the motion vector detected in the motion prediction function, motion compensated image data having a same resolution as the second video data if the difference amount calculated by the difference amount calculation function is smaller than a predetermined value, and a frame of video data in which the frame of the first video data is magnified to a same spatial resolution and having a same pixel density as the frame of the second video data if the difference amount calculated by the difference amount calculation function is greater than the predetermined value.

13. An image processing apparatus for generating a created picture, comprising:

a receiving unit for receiving first video data containing a plurality of frames having a first resolution and second video data containing a plurality of frames having a second resolution higher than the first resolution;

a motion prediction unit for detecting a motion vector between a magnified frame at time (t) contained in the first video data and a magnified frame at time (t+h) contained in the first video data received by the receiving unit, the magnified frames having a same spatial resolution and having a same pixel density as the plurality of frames having the second resolution;

an image generation unit for generating a tentatively created picture at time (t+h) having a same spatial resolution and having a same pixel density as the plurality of frames having the second resolution by executing motion compensation using the motion vector and a frame at time (t) contained in the second video data; and

a difference amount calculation unit for calculating a difference amount between the magnified frame at time (t+h) contained in the first video data and the tentatively created picture at time (t+h), wherein the difference amount is a value representing a difference between the magnified frame at time (t+h) contained in the first video data and the tentatively created picture at time (t+h);

wherein the image generation unit is configured to;

output the magnified frame at time (t+h) as a created picture when the difference amount is greater than a predetermined value, and

output the tentatively created picture at time (t+h) as the created picture when the difference amount is less than the predetermined value.