MOTION VECTOR GENERATION APPARATUS AND MOTION VECTOR GENERATION METHOD

• US 20110274175A1
• Filed: 12/21/2009
• Published: 11/10/2011
• Est. Priority Date: 01/09/2009
• Status: Active Grant

• Current Assignee: Konica Minolta Holdings Incorporated (Konica Minolta Inc.)
• Original Assignee: Konica Minolta Holdings Incorporated (Konica Minolta Inc.)
• Inventors: Sumitomo, Hironori
• Granted Patent: US 9,177,389 B2
• Time in Patent Office: Days
• US Class Current: 375/240.16
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/285 using a sequence of stereo image pairs
• CPC Class Codes: H04N 19/523 with sub-pixel accuracy
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51

Abstract

In a motion vector generation apparatus 1 according to the present invention, when motion vectors are sequentially generated for a plurality of images, a target point is determined at a sub-pixel level in any of the plurality of images, a corresponding point corresponding to the target point is searched at the sub-pixel level in another image different from this image, and a motion vector is calculated on the basis of the target point and the corresponding point. Then, the corresponding point is defined as a new target point, and the aforementioned process is repeated to sequentially generate the motion vectors. Thus, the motion vector generation apparatus 1 of the present invention can compute consecutive motion vectors by performing correspondence search at the sub-pixel level in the plurality of images.

Claims

1. A motion vector generation apparatus comprising:

an image acquisition unit for acquiring and holding a plurality of images;

a target point determination unit for determining a target point at a sub-pixel level in one of the plurality of images held in the image acquisition unit;

a corresponding point search unit for searching a corresponding point corresponding to the target point at the sub-pixel level in an image out of the plurality of images held in the image acquisition unit, the image being different from the image in which the target point is determined; and

a motion vector calculation unit for calculating a motion vector according to the target point and the corresponding point,

the target point determination unit determining, after the corresponding point search unit searches the corresponding point, the searched corresponding point as a target point in the searched image, and

the motion vector calculation unit sequentially calculating motion vectors based on the sequentially determined target points and the searched corresponding points corresponding to the target points.

2. The motion vector generation apparatus according to claim 1, wherein the corresponding point search unit defines a template in the image in which the target point is determined, the center of gravity of the template being located at the position of the target point, to search a corresponding point corresponding to the target point at the sub-pixel level, by using the template.

3. The motion vector generation apparatus according to claim 1, wherein the corresponding point search unit defines a template in the image in which the target point is determined, the center of gravity of the template being located at the position of a pixel that is nearest to the target point,

searches a point corresponding to the position of the center of gravity of the pixel using the template, and

displaces the position of the searched point by an amount of displacement between the target point and the position of the center of gravity of the pixel that is nearest to the target pixel to search the corresponding point at the sub-pixel level.

4. The motion vector generation apparatus according to claim 1, wherein the corresponding point search unit defines a template in the image in which the target point is determined, the center of gravity of the template being located at the position of a pixel that is nearest to the target point,

searches, using the template, a point in an image different from the image in which the target point is determined, the point being corresponding to the position of the center of gravity of the pixel, and

displaces the position of the searched point by an amount of displacement between the target point and the position of the center of gravity of the pixel that is nearest to the target pixel to search the corresponding point at the sub-pixel level.

5. The motion vector generation apparatus according to claim 1, wherein the plurality of images have two or more pairs of stereo images,

the target point determination unit determining the target point at the sub-pixel level in one image of the pair of stereo images, and

the corresponding point search unit further searching, at the sub-pixel level, a corresponding point corresponding to the target point in another image that pairs with the image in which the target point is determined.

6. The motion vector generation apparatus according to claim 5, wherein the motion vector calculation unit further computes distance information at the time when the pair of stereo images are taken, according to points corresponding to each other between the pair of stereo images,

the motion vector calculation unit being configured to use also the distance information in calculating the motion vector.

7. The motion vector generation apparatus according to claim 1, wherein the plurality of images have two or more pairs of stereo images in time series,

the target point determination unit determining the target point at the sub-pixel level in one image of the pair of stereo images,

the corresponding point search unit searching a corresponding point at the sub-pixel level in an image corresponding to the image in which the target point is determined, the corresponding point being corresponding to the target point, the image being one of the stereo images taken at a time instant that is different from the one at which the image in which the target point is determined is taken, and

the target point determination unit determining the corresponding point corresponding to the target point as a target point, sequentially searching a corresponding point at the sub-pixel level in an image corresponding to the image in which the target point is determined, the corresponding point being corresponding to the target point, the image being one of the stereo images taken at a time instant that is different from the one at which the image in which the target point is determined is taken, and searching, at the sub-pixel level, a corresponding point corresponding to the target point in another image that pairs with the image in which the target point is determined.

8. The motion vector generation apparatus according to claim 1, wherein the plurality of images have two or more pairs of stereo images in time series,

the target point determination unit determining the target point at the sub-pixel level in one image of the pair of stereo images,

the corresponding point search unit further searching, at the sub-pixel level, a corresponding point corresponding to the target point in another image that pairs with the image in which the target point is determined,

the target point determination unit determining the corresponding point corresponding to the target point in the one image as well as the target point in the other image, as target points, and

sequentially searching corresponding points corresponding to the respective target points at the sub-pixel level in an image corresponding to the images having the respective target points, out of a pair of stereo images taken at a time instant that is different from the one at which the images having the respective target points are taken.

9. The motion vector generation apparatus according to claim 2, wherein the corresponding point search unit generates the template by means of interpolating the image.

10. The motion vector generation apparatus according to claim 2, wherein the corresponding point search unit generates the template using a window function.

11. The motion vector generation apparatus according to claim 7, wherein when searching two or more corresponding points that correspond to one target point, the corresponding point search unit uses the same correspondence search method for the search of any corresponding point.

12. The motion vector generation apparatus according to claim 5, wherein the corresponding point search unit searches one corresponding point according to two or more target points.

13. The motion vector generation apparatus according to claim 5, wherein when searching a corresponding point in one image of a pair of stereo images taken at the same time instant based on a point in another image of the pair, the corresponding point search unit searches, under a predetermined condition, the corresponding point based on a point in an image corresponding in time series to the other image, taken at a time instant that is different from the one at which the other image is taken.

14. The motion vector generation apparatus according to claim 5, wherein when searching a corresponding point based on a point in an image taken at a different time instant and corresponding in time series, the corresponding point search unit searches, under a predetermined condition, the corresponding point based on a point in one image of a pair of stereo images taken at the same time instant as the image in which the corresponding point is searched.

15. The motion vector generation apparatus according to claim 13, wherein the predetermined condition is a predetermined number of frames.

16. The motion vector generation apparatus according to claim 13, wherein the predetermined condition is determined according to the similarity in the correspondence search.

17. The motion vector generation apparatus according to claim 13, wherein the predetermined condition is determined according to the motion vector.

18. The motion vector generation apparatus according to claim 13, wherein the predetermined condition is determined according to a contrast in the template.

19. The motion vector generation apparatus according to claim 2, wherein the corresponding point search unit searches a point which corresponds using an image pattern of the template of which frequency is resolved and amplitude component is restricted.

20. The motion vector generation apparatus according to claim 19, wherein when searching two or more corresponding points that correspond to one target point, the corresponding point search unit uses the same image pattern of the template of which frequency is resolved and amplitude component is restricted, for the search of any corresponding point.

21. The motion vector generation apparatus according to claim 19, wherein the corresponding point search unit generates the template by rotating a phase component of the image pattern in a frequency domain.

22. The motion vector generation apparatus according to claim 19, wherein the frequency resolution is either one of FFT, DFT, DCT, DST, wavelet transform, and Hadamard transform.

23. The motion vector generation apparatus according to claim 2, wherein the corresponding point search unit searches a corresponding point using phase-only correlation.

24. A motion vector generation method comprising:

a target point determination step of determining a target point at a sub-pixel level in one image of a plurality of images;

a corresponding point search step of searching a corresponding point corresponding to the target point at the sub-pixel level in an image that is different from the image in which the target point is determined, out of the plurality of images;

a step of determining the corresponding point searched in the corresponding point search step as a target point in the image in which the corresponding point is searched, and repeating the corresponding point search step; and

a motion vector calculation step of calculation a motion vector according to the target point and the corresponding point.