Motion vector calculation method and hand-movement correction device, imaging device and moving picture generation device

• US 7,847,823 B2
• Filed: 01/14/2005
• Issued: 12/07/2010
• Est. Priority Date: 01/14/2005
• Status: Active Grant

• Current Assignee: Morpho Incorporated
• Original Assignee: Morpho Incorporated
• Inventors: Habuka, Kensuke, Hiraga, Masaki
• Primary Examiner(s): Ho; Tuan
• Assistant Examiner(s): Flohre; Jason A
• Application Number:
  US11/795,218
• Publication Number:
  US 20080180535A1
• Time in Patent Office: 2,153 Days
• Field of Search: None
• US Class Current: 348/208.4
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51

Abstract

A motion vector calculation method includes: processing for calculating multi-resolution data up to a predetermined resolution level L from a plurality of inputted image data; and processing for estimating motion vectors per resolution executing a matching step for temporarily calculating motion vectors minimizing an energy function within a predetermined range of the input motion vectors and a smoothing step for finally calculating the motion vectors by averaging the surrounding data. By using the motion vectors of the resolution level L finally calculated as input motion vectors of a further higher resolution level L-1, the processing for estimating motion vectors per resolution is successively repeated on image data of a high resolution, thereby calculating the motion vectors.

Claims

1. A motion vector calculation method in which multiple pieces of image data are input to calculate motion vectors between/among images, the method comprising:

multi-resolution image generation processing for calculating and storing multi-resolution data up to a predetermined resolution level L (where L is a natural number) of the input multiple pieces of image data;

input motion vector calculation processing for calculating input motion vectors for the resolution level L of a reference image; and

processing for estimating motion vectors per resolution by using the reference image, a target image and the input motion vectors at the resolution level L calculated by the processings above to perform;

a step (1) of defining an energy function on each starting point coordinate of the input motion vectors by handling a coordinate on the reference image and a coordinate on the target image as arguments, and temporarily calculating motion vectors, within predetermined range from the input motion vectors, based on the energy function;

a step (2) of temporarily calculating reliability of the motion vectors based on an amount of strength of an edge or a corner angle detected from a color space of the reference image;

a step (3) of calculating correlation by using a color difference between feature points of the reference image and a difference between the motion vectors temporarily calculated in the step (1); and

a step (4) of finally calculating motion vectors of the resolution level L by smoothing each of the motion vectors temporarily calculated in the step (1) by weighted averaging by the product of the reliability and the correlation where the smoothing utilizes the reliability temporarily calculated in the step (2) smoothed by weighted averaging by the correlation,

wherein the motion vectors between the reference image and the target image are calculated by using the finally calculated motion vectors of the resolution level L as input motion vectors of a resolution level L-1, which is a higher resolution, and by repeating the processing for estimating motion vectors per resolution on image data having higher resolutions sequentially.

2. The motion vector calculation method according to claim 1, further comprising processing for inputting specification information relating to the feature points of the reference image among the input images,

wherein the input motion vector calculating processing calculates the input motion vectors relating to the feature points.

3. The motion vector calculation method according to claim 2, further comprising image feature point group extraction processing for dispersedly extracting multiple feature points of the reference image among the input images,

wherein the input motion vector calculating processing calculates the input motion vectors relating to the feature points.

4. The motion vector calculation method according to claim 3, wherein the motion vectors are motion vector mesh data indicating respective movements of corresponding feature points, lines connecting the feature points and areas defined by the feature points or the lines between the reference image and the target image.

5. A motion vector calculation method in which multiple pieces of image data are input to calculate motion vectors between images, the method comprising:

processing for inputting specification information relating to feature points of a reference image;

processing for calculating motion vector mesh data representing correspondence relationships among points, lines and areas between the reference image and a target image based on input specification information;

processing for dispersedly extracting multiple feature points of the reference image from an area excluding the area range of the input specification information and calculating, as motion vector mesh data, correspondence relationships among points, lines and areas between the reference image and the target image based on the extracted feature points;

processing for dividing the area of the motion vector mesh data into one or more triangles, correcting the motion vector data to a predetermined resolution level L and calculating input motion vector mesh data;

processing for calculating and storing multi-resolution data up to the resolution level L of the input multiple pieces of image data; and

processing for estimating motion vectors per resolution by using the reference image, target image and input motion vector mesh data of the resolution level L, which are calculated by the processings above to perform;

a step (1) of defining an energy function on each starting point coordinate of the input motion vector mesh data of an area excluding the area range of the input specification information by handling coordinates on the reference image and coordinates on the target image as arguments, and temporarily calculating the motion vector mesh data within predetermined range from the input motion vector mesh data based on the energy function;

a step (2) of temporarily calculating reliability of the motion vector mesh data based on an amount of strength of an edge or a corner angle detected from a color space at each feature point of the reference image;

a step (3) of calculating correlation by using a color difference between feature points of the reference image and a difference between the motion vector mesh data temporarily calculated in the step (1); and

a step (4) of finally calculating motion vector mesh data of a the resolution level L by smoothing each of the motion vector mesh data temporarily calculated in the step (1) by weighted averaging by the product of the reliability and the correlation where the smoothing utilizes the reliability temporarily calculated in the step (2) smoothed by weighted averaging by the correlation,

wherein the motion vector mesh data between the reference image and the target image are calculated by using the finally calculated motion vector mesh data of the resolution level L as input motion vector mesh data of a resolution level L-1, which is a higher resolution, and by repeating the processing for estimating motion vectors per resolution on image data having higher resolutions sequentially.

6. A computer program product comprising a tangible non-transitory computer readable medium storing a computer-executable program that inputs multiple pieces of image data and calculating motion vectors between the images, the program comprising:

processing for calculating and storing multi-resolution data up to a predetermined resolution level L (where L is a natural number) of each of the input multiple pieces of image data;

processing for calculating the input motion vectors for the resolution level L of the reference image; and

processing for estimating motion vectors per resolution by using the reference image, a target image and the input motion vectors of the resolution level L, which are calculated by the processings above, to perform;

a step (1) of defining an energy function on each starting point coordinate of the input motion vectors by handling a coordinate on the reference image and a coordinate on the target image as arguments, and temporarily calculating motion vectors, within predetermined range from the input motion vectors, based on the energy function;

a step (2) of temporarily calculating reliability of the motion vectors based on an amount of strength of an edge or a corner angle detected from a color space of the reference image;

a step (3) of calculating correlation by using a color difference between feature points of the reference image and a difference between the motion vectors temporarily calculated in the step (1); and

a step (4) of finally calculating motion vectors of the resolution level L by smoothing each of the motion vectors temporarily calculated in the step (1) by weighted averaging by the product of the reliability and the correlation where the smoothing utilizes the reliability temporarily calculated in the step (2) smoothed by weighted averaging by the correlation,

wherein the motion vectors between the reference image and the target image are calculated by using the finally calculated motion vectors of the resolution level L as input motion vectors of a resolution level L-1, which is a higher resolution, and by repeating the processing for estimating motion vectors per resolution on image data having higher resolutions sequentially.