Motion Vector Calculation Method and Hand-Movement Correction Device, Imaging Device and Moving Picture Generation Device

• US 20080180535A1
• Filed: 01/14/2005
• Published: 07/31/2008
• Est. Priority Date: 01/14/2005
• Status: Active Grant

• Current Assignee: Morpho Incorporated
• Original Assignee: Morpho Incorporated
• Inventors: Habuka, Kensuke, Hiraga, Masaki
• Granted Patent: US 7,847,823 B2
• Time in Patent Office: Days
• US Class Current: 348/208.4
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51

Abstract

[PROBLEM] To enable calculation of motion vectors with a high accuracy by a comparatively simple processing by using a small amount of data without applying a load onto a computer.

[MEANS FOR SOLVING PROBLEMS] A motion vector calculation method includes: processing for calculating multi-resolution data up to a predetermined resolution level L from a plurality of inputted image data; and processing for estimating motion vectors per resolution executing a matching step for temporarily calculating motion vectors minimizing an energy function within a predetermined range of the input motion vectors and a smoothing step for finally calculating the motion vectors by averaging the surrounding data. By using the motion vectors of the resolution level L finally calculated as input motion vectors of a further higher resolution level L-1, the processing for estimating motion vectors per resolution is successively repeated on image data of a high resolution, thereby calculating the motion vectors.

Claims

1. A motion vector calculation method in which multiple pieces of image data are input to calculate motion vectors between/among images, the method comprising:

multi-resolution image generation processing for calculating and storing multi-resolution data up to a predetermined resolution level L (where L is a natural number) of the input multiple pieces of image data;

input motion vector calculation processing for calculating input motion vectors for the resolution level L of a reference image; and

processing for estimating motion vectors per resolution by using the reference image, a target image and the input motion vectors at the resolution level L calculated by the processings above to perform;

a step (1) of defining an energy function on each starting point coordinate of the input motion vectors by handling the coordinate on the reference image and a coordinate on the target image as arguments, and temporarily calculating motion vectors, within predetermined range from the input motion vectors, based on the energy function;

a step (2) of temporarily calculating reliability of the motion vectors based on a feature amount of an image at each feature point of the reference image;

a step (3) of calculating correlation by using at least one of a difference in the feature amount of the image and a difference in the temporarily calculated motion vectors between feature points of the reference image; and

a step (4) of finally calculating motion vectors of the resolution level L by smoothing the temporarily calculated motion vectors and the reliability by using the motion vectors, the reliability and the correlation in predetermined ranges,

wherein the motion vectors between the reference image and the target image are calculated by using the finally calculated motion vectors of the resolution level L as input motion vectors of a resolution level L-1, which is a higher resolution, and by repeating the processing for estimating motion vectors per resolution on image data having higher resolutions sequentially.

2. The motion vector calculation method according to claim 1, further comprising processing for inputting specification information relating to the feature points of the reference image among the input images,

wherein the input motion vector calculating processing calculates the input motion vectors relating to the feature points.

3. The motion vector calculation method according to claim 2, further comprising image feature point group extraction processing for dispersedly extracting multiple feature points of the reference image among the input images,

wherein the input motion vector calculating processing calculates the input motion vectors relating to the feature points.

4. The motion vector calculation method according to claim 3, wherein the motion vectors are motion vector mesh data indicating respective movements of corresponding feature points, lines connecting the feature points and areas defined by the feature points or the lines between the reference image and the target image.

5. A motion vector calculation method in which multiple pieces of image data are input to calculate motion vectors between images, the method comprising:

processing for inputting specification information relating to feature points of a reference image;

processing for calculating motion vector mesh data representing correspondence relationships among points, lines and areas between the reference image and a target image based on input specification information;

processing for dispersedly extracting multiple feature points of the reference image from an area excluding the area range of the input specification information and calculating, as motion vector mesh data, correspondence relationships among points, lines and areas between the reference image and the target image based on the extracted feature points;

processing for dividing the area of the motion vector mesh data into one or more triangles, correcting the motion vector data to a predetermined resolution level L and calculating input motion vector mesh data;

processing for calculating and storing multi-resolution data up to the resolution level L of the input multiple pieces of image data; and

processing for estimating motion vectors per resolution by using the reference image, target image and input motion vector mesh data of the resolution level L, which are calculated by the processings above to perform;

a step (1) of defining an energy function on each starting point coordinate of the input motion vector mesh data of an area excluding the area range of the input specification information by handling the coordinates on the reference image and the coordinates on the target image as arguments, and temporarily calculating the motion vectors within predetermined range from the input motion vector mesh data based on the energy function;

a step (2) of temporarily calculating reliability of the motion vectors based on a feature amount of an image at each feature point of the reference image;

a step (3) of calculating correlation by using at least one of a difference in the feature amount of the image and a difference in the temporarily calculated motion vectors between feature points of the reference image; and

a step (4) of finally calculating motion vector mesh data of a resolution level L by smoothing the temporarily calculated motion vector mesh data and the reliability by using the motion vector mesh data, the reliability and the correlation in a predetermined ranges,

wherein the motion vector mesh data between the reference image and the target image is calculated by using the finally calculated motion vector mesh data of the resolution level L as input motion vector mesh data of a resolution level L-1, which is a higher resolution, and by repeating the processing for estimating motion vectors per resolution on image data having higher resolutions sequentially.

6. An imaging device that performs hand-movement correction, the device comprising:

a motion vector calculating unit for inputting multiple images, calculating motion vectors between a reference image and one or more target images by using the motion vector calculation method and outputting the calculation result;

synthesis image generation means for generating a synthesis image by performing pre-processing for superposing the target image over the reference image using the motion vectors calculated by the motion vector calculation unit and the target image;

image synthesis means for generating an image by superposing the one or more synthesis images generated by the synthesis image generation means over the reference image; and

image data output means for outputting the image data generated by the image synthesis means.

7. The imaging device according to claim 6, wherein the image synthesis means calculates the maximum range of a hand-movement and performs averaging processing based on the number of the synthesis images on the area from an image frame to the maximum range.

8. The imaging device having the hand-movement correction device according to claim 6, the imaging device comprising:

an imaging unit for shooting a subject in series, generating multiple images and inputting them to the hand-movement correction device; and

a storage unit for storing images from the hand-movement correction device.

9. The imaging device having the hand-movement correction device according to claim 7, the imaging device comprising:

an imaging unit for shooting a subject in series, generating multiple images and inputting them to the hand-movement correction device; and

a storage unit for storing images from the hand-movement correction device.

10. A moving picture generation device that generates a moving picture, the device comprising:

a motion vector calculation unit for inputting multiple images and calculating motion vectors between the reference image and the one or more target images;

motion vector editing means for calculating one or more intermediate motion vectors by dividing the motion vectors, which are calculated by the motion vector calculating unit, by a predetermined value; and

moving picture generation means for generating an intermediate image by applying the intermediate motion vectors to the reference image and enabling to display the initial image, intermediate images and final image in a time-series manner.

11. A computer-executable program that inputs multiple pieces of image data and calculating motion vectors between the images, the program comprising:

processing for calculating and storing multi-resolution data up to a predetermined resolution level L (where L is a natural number) of each of the input multiple pieces of image data;

processing for calculating the input motion vectors for the resolution level L of the reference image; and

processing for estimating motion vectors per resolution by using the reference image, a target image and the input motion vectors of the resolution level L, which are calculated by the processings above, to perform;

a step (1) of defining an energy function on each starting point coordinate of the input motion vectors by handling the coordinate the reference image and a coordinate on the target image as arguments, and temporarily calculating motion vectors, within predetermined range from the input motion vectors, based on the energy function;

a step (2) of temporarily calculating reliability of the motion vectors based on a feature amount of an image at each feature point of the reference image;

a step (3) of calculating correlation by using at least one of a difference in the feature amount of the image and a difference in the calculated motion vectors between feature points of the reference image; and

a step (4) of finally calculating motion vectors of a resolution level L by smoothing the temporarily calculated motion vectors and the reliability by using the motion vectors, the reliability and the correlation in predetermined ranges,

wherein the motion vectors between the reference image and the target image are calculated by using the finally calculated motion vectors of the resolution level L as input motion vectors of a resolution level L-1, which is a higher resolution, and by repeating the processing for estimating motion vectors per resolution on image data having higher resolutions sequentially.