Image processing method and image processing device

• US 20070206678A1
• Filed: 03/02/2007
• Published: 09/06/2007
• Est. Priority Date: 03/03/2006
• Status: Active Grant

• Current Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Original Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Inventors: Kondo, Satoshi
• Granted Patent: US 8,116,576 B2
• Time in Patent Office: Days
• US Class Current: 375/240.170
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: H04N 19/523 with sub-pixel accuracy
• CPC Class Codes: H04N 19/527 Global motion vector estimation
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51

Abstract

The present invention provides an image processing device which is able to reconstruct a high-resolution (HR) picture from low-resolution (LR) pictures, by modifying instability in the pictures due to movement of hands in capturing the images, and also by preventing increase of a circuit size and processing amounts required for motion estimation between the pictures.

In the image processing device, a motion estimation unit reads the LR pictures from a memory, and estimates respective motion vectors between the pictures. A stabilization parameter calculation unit calculates a stabilized global motion parameter based on the motion vector. A motion vector compensation unit stabilizes the motion vectors calculated by the motion estimation unit, using the stabilized global motion parameter. A picture integration unit combines (integrates) a target picture and reference pictures with sub-pixel precision, using the stabilized motion vectors. An integrated picture processing unit generates equally-spaced pixels of a HR picture from unequally-spaced pixels of the integrated picture, and outputs the generated HR picture.

Claims

1. An image processing method of generating an integrated picture corresponding to a target picture, using the target picture and a plurality of reference pictures, the target picture being one of a plurality of pictures which are sequentially inputted every predetermined period, and the reference pictures being included in the sequentially inputted pictures and being temporally positioned prior or subsequent to the target picture, said image processing method comprising:

estimating a motion amount between a segment of the target picture and a segment of each of the reference pictures with sub-pixel precision;

calculating an in-stabilized variant component using at least a part of the estimated motion amounts;

integrating pixels of the target picture and pixels of the reference pictures onto a space for the integrated picture, so that resolution of the target picture is increased and instability of the target picture is modified, based on the estimated motion amounts and the calculated in-stabilized variant component; and

generating, from the pixels of the space, the integrated picture that has pixels whose number is determined using one of a magnification ratio and a predetermined number of pixels, and which are positioned equally spaced apart.

2. The image processing method according to claim 1, further comprising:

calculating a stabilized motion amount for the target picture, based on the in-stabilized variant component; and

calculating respective stabilized motion amounts for the respective reference pictures, based on the motion amounts and the in-stabilized variant component;

wherein said integrating includes;

mapping the pixels of the reference pictures onto the space, after shifting the pixels based on the stabilized motion amounts of the reference pictures; and

mapping the pixels of the target picture onto the space, after shifting the pixels based on the stabilized motion amount of the target picture.

3. The image processing method according to claim 2,

wherein said calculating of the in-stabilized variant component includes:

calculating a global motion parameter which represents dominant motion in the whole target picture; and

calculating the in-stabilized variant component, based on a history of the global motion parameter.

4. The image processing method according to claim 1,

wherein said integrating includes:

mapping (i) the pixels of the reference pictures onto the space after shifting the pixels based on the motion amounts, and (ii) the pixels of the target picture onto the space without shifting the pixels; and

modifying positions of the pixels mapped on the space, using the in-stabilized variant component.

5. The image processing method according to claim 4,

wherein said calculating of the in-stabilized variant component includes:

calculating a global motion parameter which represents dominant motion in the whole target picture; and

calculating the in-stabilized variant component, based on a history of the global motion parameter.

6. The image processing method according to claim 1,

wherein the magnification ratio or the predetermined number of pixels is designated from the outside.

7. The image processing method according to claim 1,

wherein the magnification ratio or the predetermined number is designated, so that the number of pixels of the integrated picture becomes equal to the number of pixels of the input pictures.

8. The image processing method according to claim 1, further comprising

designating a predetermined maximum value of an amount of the shifting,

wherein, in said calculating of the in-stabilized variant component, the in-stabilized variant component is calculated so that the amount of the shifting of the pixels do not exceed the maximum value, and

in said integrating, the magnification ratio is determined based on the number of pixels of the input pictures and the maximum value.

9. An image processing device which generates an integrated picture corresponding to a target picture, using the target picture and a plurality of reference pictures, the target picture being one of a plurality of pictures which are sequentially inputted every predetermined period, and the reference pictures being included in the sequentially inputted pictures and being temporally positioned prior or subsequent to the target picture, said image processing device comprising:

a motion estimation unit operable to estimate a motion amount between a segment of the target picture and a segment of each of the reference pictures with sub-pixel precision;

an in-stabilized variant component calculation unit operable to calculate an in-stabilized variant component using at least a part of the estimated motion amounts;

a picture integration unit operable to integrate pixels of the target picture and pixels of the reference pictures onto a space for the integrated picture, so that resolution of the target picture is increased and instability of the target picture is modified, based on the estimated motion amounts and the calculated in-stabilized variant component; and

an integrated picture processing unit operable to generate, from the pixels of the space, the integrated picture that has pixels whose number is determined using one of a magnification ratio and a predetermined number of pixels, and which are positioned equally spaced apart.

10. An integrated circuit which generates an integrated picture corresponding to a target picture, using the target picture and a plurality of reference pictures, the target picture being one of a plurality of pictures which are sequentially inputted every predetermined period, and the reference pictures being included in the sequentially inputted pictures and being temporally positioned prior or subsequent to the target picture, said integrated circuit comprising:

a motion estimation unit operable to estimate a motion amount between a segment of the target picture and a segment of each of the reference pictures with sub-pixel precision;

an in-stabilized variant component calculation unit operable to calculate an in-stabilized variant component using at least a part of the estimated motion amounts;

a picture integration unit operable to integrate pixels of the target picture and pixels of the reference pictures onto a space for the integrated picture, so that resolution of the target picture is increased and instability of the target picture is modified, based on the estimated motion amounts and the calculated in-stabilized variant component; and

an integrated picture processing unit operable to generate, from the pixels of the space, the integrated picture that has pixels whose number is determined using one of a magnification ratio and a predetermined number of pixels, and which are positioned equally spaced apart.

11. A camera comprising an image processing device which generates an integrated picture corresponding to a target picture, using the target picture and a plurality of reference pictures, the target picture being one of a plurality of pictures which are sequentially inputted every predetermined period, and the reference pictures being included in the sequentially inputted pictures and being temporally positioned prior or subsequent to the target picture,

wherein said image processing device includes:

a motion estimation unit operable to estimate a motion amount between a segment of the target picture and a segment of each of the reference pictures with sub-pixel precision;

an in-stabilized variant component calculation unit operable to calculate an in-stabilized variant component using at least a part of the estimated motion amounts;

a picture integration unit operable to integrate pixels of the target picture and pixels of the reference pictures onto a space for the integrated picture, so that resolution of the target picture is increased and instability of the target picture is modified, based on the estimated motion amounts and the calculated in-stabilized variant component; and

an integrated picture processing unit operable to generate, from the pixels of the space, the integrated picture that has pixels whose number is determined using one of a magnification ratio and a predetermined number of pixels, and which are positioned equally spaced apart.

12. A program which is stored in a computer-readable recording medium and is used to perform a method of generating an integrated picture corresponding to a target picture, using the target picture and a plurality of reference pictures, the target picture being one of a plurality of pictures which are sequentially inputted every predetermined period, and the reference pictures being included in the sequentially inputted pictures and being temporally positioned prior or subsequent to the target picture, said program causing a computer to execute:

estimating a motion amount between a segment of the target picture and a segment of each of the reference pictures with sub-pixel precision;

calculating an in-stabilized variant component using at least a part of the estimated motion amounts;

integrating pixels of the target picture and pixels of the reference pictures onto a space for the integrated picture, so that resolution of the target picture is increased and instability of the target picture is modified, based on the estimated motion amounts and the calculated in-stabilized variant component; and

generating, from the pixels of the space, the integrated picture that has pixels whose number is determined using one of a magnification ratio and a predetermined number of pixels, and which are positioned equally spaced apart.