Image processing apparatus, image processing method, and program

• US 8,804,832 B2
• Filed: 05/21/2008
• Issued: 08/12/2014
• Est. Priority Date: 06/18/2007
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Nakagami, Ohji, Tanaka, Junichi, Yagasaki, Yoichi
• Primary Examiner(s): Senfi, Behrooz
• Assistant Examiner(s): Jebari, Mohammed
• Application Number:
  US12/597,792
• Publication Number:
  US 20100118963A1
• Time in Patent Office: 2,274 Days
• US Class Current: 375/240.16
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: H04N 1/217 Interfaces allowing access to a single user
• CPC Class Codes: H04N 19/53 Multi-resolution motion estimation; Hierarchical motion estimation
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 2101/00 Still video cameras
• CPC Class Codes: H04N 2201/3247 Data linking a set of images to one another, e.g. sequence, burst or continuous capture mode
• CPC Class Codes: H04N 2201/3267 of motion picture signals, e.g. video clip
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 25/42 by switching between different modes of operation using different resolutions or aspect ratios, e.g. switching between interlaced and non-interlaced mode
• CPC Class Codes: H04N 5/765 Interface circuits between an apparatus for recording and another apparatus
• CPC Class Codes: H04N 7/0125 one of the standards being a high definition standard
• CPC Class Codes: H04N 7/014 involving the use of motion vectors motion estimation and compensation in video coding H04N19/51
• CPC Class Codes: H04N 9/8042 involving data reduction

Abstract

There is provided an image processing apparatus including an acquiring unit for acquiring moving image data containing a plurality of successive frames, and one or a plurality of image data corresponding to the frames and having a spatial resolution higher than the frames; a motion prediction unit for detecting a motion vector between the frames using the moving image data; and an image generation unit for generating motion compensated image data corresponding to a predetermined frame based on the image data and the motion vector. The image generation unit generates the motion compensated image data being positioned between the predetermined frame and a frame corresponding to the image data and corresponding to the frame, and generates motion compensated image data corresponding to the predetermined frame based on the motion compensated image data and the motion vector.

Claims

1. An image processing apparatus, comprising:

a receiving unit for receiving first video data containing a plurality of frames and second video data having a spatial resolution higher than the frames contained in the first video data and a temporal resolution lower than the frames contained in the first video data;

an image conversion unit for magnifying the frames of the first video data to generate a plurality of magnified frames;

a motion prediction unit for detecting a motion vector between each sequential frame pair of the magnified frames; and

an image generation unit for, based on the second video data received by the receiving unit and the motion vector detected by the motion prediction unit, performing motion compensation in sequential order from one of the magnified frames associated with a time that corresponds to a time associated with the second video data, and sequentially generating motion compensated image data corresponding to the first video data based on the motion compensated frames and the motion vector,

wherein the image generation unit sequentially generates the motion compensated image data by combining, at a predetermined ratio, a reference block of a first frame of the second video data specified by a respective motion vector, and a corresponding reference block of a second frame of the plurality of magnified frames that is immediately subsequent in time to the first frame of the second video data and belonging to a corresponding respective frame pair of the magnified frames used to generate the respective motion vector, to thereby generate an intermediate created picture, and

wherein the predetermined ratio is based upon a difference amount between the reference block of the first frame of the second video data and the corresponding reference block of the second frame of the plurality of magnified frames.

2. The image processing apparatus according to claim 1, further comprising:

a storage unit for storing the first video data and the second video data, wherein the receiving unit receives the first video data and the second video data from the storage unit.

3. The image processing apparatus according to claim 2, wherein the storage unit stores image data obtained by down sampling an imaged image signal as the first video data.

4. The image processing apparatus according to claim 1, wherein if a difference amount between a predetermined frame contained in the first video data and a frame contained in the second video data is smaller than a predetermined value, the image generation unit outputs the image data corresponding to the predetermined frame without performing motion compensation on the second video data.

5. The image processing apparatus according to claim 1, wherein the image generation unit sequentially generates the motion compensated image data by repeating the combining by using the generated intermediate created picture as a next first frame of the second video data, to thereby generate an ultimate created picture.

6. The image processing apparatus according to claim 1, wherein the image generation unit sequentially generates the motion compensated image data by combining pixels of reference blocks of the second video data specified by respective motion vectors and pixels of corresponding reference blocks of the magnified frames at the predetermined ratio.

7. The image processing apparatus according to claim 1, wherein when the difference amount is less than a predetermined value, the image generation unit outputs the first frame of the second video data, without performing motion compensation, as the intermediate created picture.

8. The image processing apparatus according to claim 1, wherein the intermediate created picture is generated for the second video data to thereby increase the temporal resolution of the second video data.

9. An image processing method, comprising the steps of:

receiving first video data containing a plurality of frames and second video data having a spatial resolution higher than the frames contained in the first video data and a temporal resolution lower than the frames contained in the first video data;

magnifying the frames of the first video data to generate a plurality of magnified frames;

detecting a motion vector between each sequential frame pair of the magnified frames; and

performing, based on the second video data received in the receiving step and the motion vector detected in the motion prediction step, motion compensation in sequential order from one of the magnified frames associated with a time that corresponds to a time associated with the second video data, and sequentially generating motion compensated image data corresponding to the first video data based on the motion compensated frame and the motion vector,

wherein the motion compensated image data is sequentially generated by combining, at a predetermined ratio, a reference block of a first frame of the second video data specified by a respective motion vector, and a corresponding reference block of a second frame of the plurality of magnified frames that is immediately subsequent in time to the first frame of the second video data and belonging to a corresponding respective frame pair of the magnified frames used to generate the respective motion vector, to thereby generate an intermediate created picture,

wherein the predetermined ratio is based upon a difference amount between the reference block of the first frame of the second video data and the corresponding reference block of the second frame of the plurality of magnified frames.

10. The image processing method according to claim 9, wherein the motion compensated image data is sequentially generated by repeating the combining by using the generated intermediate created picture as a next first frame of the second video data, to thereby generate an ultimate created picture.

11. The image processing method according to claim 9, wherein the motion compensated image data is sequentially generated by combining pixels of reference blocks of the second video data specified by respective motion vectors and pixels of corresponding reference blocks of the magnified frames at the predetermined ratio.

12. The image processing method according to claim 9, wherein when the difference amount is less than a predetermined value, the first frame of the second video data is outputted, without performing motion compensation, as the intermediate created picture.

13. The image processing method according to claim 9, wherein the intermediate created picture is generated for the second video data to thereby increase the temporal resolution of the second video data.

14. A tangible, non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform a method, comprising:

receiving first video data containing a plurality of frames and second video data having a spatial resolution higher than the frames contained in the first video data and a temporal resolution lower than the frames contained in the first video data;

magnifying the frames of the first video data to generate a plurality of magnified frames;

detecting a motion vector between each sequential frame pair of the magnified frames; and

performing, based on the second video data received by the receiving unit and the motion vector detected by the motion prediction unit, motion compensation in sequential order from one of the magnified frames associated with a time that corresponds to a time associated with the second video data, and sequentially generating motion compensated image data corresponding to the first video data based on the motion compensated frame and the motion vector,

wherein the motion compensated image data is sequentially generated by combining, at a predetermined ratio, a reference block of a first frame of the second video data specified by a respective motion vector, and a corresponding reference block of a second frame of the plurality of magnified frames that is immediately subsequent in time to the first frame of the second video data and belonging to a corresponding respective frame pair of the magnified frames used to generate the respective motion vector, to thereby generate an intermediate created picture,

wherein the predetermined ratio is based upon a difference amount between the reference block of the first frame of the second video data and the corresponding reference block of the second frame of the plurality of magnified frames.

15. The computer-readable medium according to claim 14, wherein the motion compensated image data is sequentially generated by repeating the combining by using the generated intermediate created picture as a next first frame of the second video data, to thereby generate an ultimate created picture.

16. The computer-readable medium according to claim 14, wherein the motion compensated image data is sequentially generated by combining pixels of reference blocks of the second video data specified by respective motion vectors and pixels of corresponding reference blocks of the magnified frames at the predetermined ratio.

17. The computer-readable medium according to claim 14, wherein when the difference amount is less than a predetermined value, the first frame of the second video data is outputted, without performing motion compensation, as the intermediate created picture.

18. The computer-readable medium according to claim 14, wherein the intermediate created picture is generated for the second video data to thereby increase the temporal resolution of the second video data.