Movement vector detecting apparatus

• US 5,731,849 A
• Filed: 03/11/1993
• Issued: 03/24/1998
• Est. Priority Date: 03/13/1992
• Status: Expired due to Term

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Sekine, Masayoshi, Kondo, Toshiaki
• Primary Examiner(s): Britton, Howard W.
• Assistant Examiner(s): Le, Vu
• Application Number:
  US08/029,550
• Time in Patent Office: 1,839 Days
• Field of Search: 348/416, 348/699, 348/208, 348/407, 348/413, 348/402, 348/218, 348/219, 348/220, 348/390
• US Class Current: 348/699
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/223 using block-matching
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/686 with a variable apex prism

Abstract

A movement vector detecting apparatus comprising: a movement vector detecting circuit for dividing an input image into a plurality of blocks and for detecting a movement vector every block; and a movement vector calculating circuit for extracting areas which perform the same movement from the movement vectors from the movement vector detecting circuit and for calculating movement vector values in such areas by executing a weight averaging process according to position information which occupies in the area of each of the blocks. A video camera having such a movement vector detecting apparatus is provided.

Claims

1. A movement vector detecting apparatus comprising:

a) movement vector detecting means for dividing an input image into a plurality of blocks and for detecting movement vectors in every one of the blocks; and

b) movement vector calculating means for detecting an effective area including effective blocks on the basis of a distribution of the movement vectors of the plurality of blocks and for calculating a representative movement vector in the effective area by

(i) calculating and assigning different weighting values for the movement vectors of the effective blocks according to the position of each effective block within the effective area, and

(ii) averaging the weighted movement vectors.

2. An apparatus according to claim 1, further having image pickup means for photoelectrically converting an object image formed on an image pickup surface and for outputting an image pickup signal.

3. An apparatus according to claim 2, wherein said input image is an image formed on the image pickup surface of said image pickup means, and said movement vector detecting means detects the movement vectors from the image pickup signal.

4. An apparatus according to claim 3, wherein said movement vector detecting means obtains the movement vectors from a luminance signal which is obtained by eliminating a color signal component from the image pickup signal.

5. An apparatus according to claim 1, wherein said movement vector detecting means generates a distribution (optical flow) of the movement vectors on a picture plane.

6. An apparatus according to claim 1, wherein said movement vector calculating means detects the blocks which output the movement vectors due to a vibration as a target for correction from said optical flow and applies weights according to a degree of surrounding of said blocks to output signals from said blocks and calculates a movement amount of the image at that moment in a real-time manner.

7. An apparatus according to claim 1, wherein said movement vector calculating means detects the blocks which output the movement vectors due to a vibration as the target for correction from said optical flow and executes a convolution mask arithmetic operation to the output signals from said blocks, thereby applying the weights to said signals.

8. An apparatus according to claim 7, wherein the convolution mask arithmetic operation is executed in a manner such that in the case where blocks which are adjacent to the target blocks in vertical and lateral directions exist, a point 1 is given to the target blocks, and in the case where blocks which are adjacent in oblique directions exist, a point 0.5 is given so that a higher point total is provided to target blocks surrounded by more adjacent blocks, and the weight of the movement vector is increased for target blocks of a high point total.

9. An apparatus according to claim 8, wherein said movement vector calculating means is constructed by a microcomputer.

10. An apparatus according to claim 1, further comprising correcting means for correcting a vibration on the basis of an output of the movement vector calculating means.

11. An apparatus according to claim 10, wherein said correcting means variably changes a read-out position when image information is read out from an image memory on the basis of the output of said movement vector calculating means, thereby setting off the vibration.

12. A movement vector detecting apparatus comprising:

a) movement vector detecting means for dividing an input image into a plurality of blocks and for detecting movement vectors in every one of the blocks; and

b) movement vector calculating means for detecting an effective area including effective blocks on the basis of a distribution of the movement vectors of the plurality of blocks and for calculating a representative movement vector in the input image in the effective area by

(i) calculating and assigning different weighting values for the movement vectors of the effective blocks according to the position of each effective block within the effective area, and

(ii) averaging the weighted movement vectors.

13. An apparatus according to claim 12, wherein said movement vector detecting means generates a distribution (optical flow) of the movement vectors on a picture plane and extracts the blocks which output the movement vectors due to a vibration as a target for correction from said optical flow and performs a weighing process according to a degree of surrounding of said blocks to output signals from said blocks, thereby calculating a movement amount of the image at that moment in a real-time manner.

14. An apparatus according to claim 13, wherein said movement vector calculating means detects the blocks which output the movement vectors due to the vibration serving as the target for correction from said optical flow and executes a convolution mask arithmetic operation to the output signals from said blocks, thereby applying a weight.

15. An apparatus according to claim 12, further having correcting means for correcting a vibration on the basis of an output of said movement vector calculating means.

16. An apparatus according to claim 15, wherein said correcting means includes an image memory and variably changes a read-out position when image information is read out from said image memory on the basis of an output of said movement vector calculating means, thereby setting off the vibration.

17. A movement vector detecting apparatus comprising:

a) movement vector detecting means for dividing an input image into a plurality of blocks and for detecting movement vectors in every one of the blocks;

b) movement vector calculating means for extracting effective areas including effective blocks on the basis of a distribution of the movement vectors from said movement vector detecting means and for calculating representative movement vector values for each of the effective areas by

(i) calculating and assigning different weighting values for the movement vectors of the effective blocks in each effective area according to the position of each effective block within the respective effective area, and

(ii) averaging the weighted movement vectors of each effective area; and

c) evaluating means for evaluating the representative movement vectors for the effective areas calculated by said movement vector calculating means on the basis of the total number of weight coefficients due to the weight averaging process.

18. An apparatus according to claim 17, wherein said movement vector detecting means generates a distribution (optical flow) of the movement vectors on a picture plane and extracts the blocks which output the movement vectors due to a vibration serving as a target for correction from said optical flow and executes a convolution mask arithmetic operation to output signals from said blocks, thereby applying a weight.

19. An apparatus according to claim 18, wherein said convolution mask arithmetic operation is executed in a manner such that in the case where blocks adjacent to the target blocks in vertical and lateral directions exist, a point 1 is given to said target blocks, and in the case where blocks adjacent in oblique directions exist, a point 0.5 is given, thereby raising a point total as the number of adjacent blocks which surround the target blocks is large, and the weight of the movement vector is increased as the point total of the blocks is increased.

20. An apparatus according to claim 19, wherein said evaluating means has a predetermined evaluation function, and when the point total is small, there is a possibility of an erroneous detection, so that the weight is multiplied by a small coefficient less than 1, thereby reducing an influence.

21. An apparatus according to claim 20, wherein assuming that the number of blocks in the picture plane is set to M×

N, where M is a number of blocks in a row direction and N is a number of blocks in a column direction, said evaluating means expresses a point distribution by 0≦

total number of points≦

7 Mn-4M-4N+2.

22. An apparatus according to claim 17, wherein said evaluating means evaluates a reliability of each block by calculating a correlation among picture planes which differ with respect to a time for a weight coefficient distribution in the effective blocks.

23. A vibrationproof apparatus comprising:

a) movement vector detecting means for detecting movement vectors of an image in every area obtained by dividing the image into a plurality of areas;

b) first detecting means for detecting an effective vibration detecting area according to vibration amounts and a distribution of the movement vectors on the basis of movement vector information in each of the plurality of areas from said movement vector detecting means;

c) boundary detecting means for detecting a boundary of the image;

d) second detecting means for modifying an effective vibration detecting area detected by said first detecting means on the basis of an output of said boundary detecting means and an output of said movement vector detecting means; and

e) correcting means for correcting a vibration in accordance with the information from said second detecting means.

24. An apparatus according to claim 23, further comprising image pickup means for photoelectrically converting an object image formed on an image pickup surface and for outputting an image pickup signal.

25. An apparatus according to claim 24, wherein said input image is an image formed on the image pickup surface of said image pickup means, and said movement vector detecting means detects the movement vectors from a luminance signal in the image pickup signal.

26. An apparatus according to claim 23, wherein said movement vector detecting means generates a distribution (optical flow) of the movement vectors on a picture plane.

27. An apparatus according to claim 26, wherein said second detecting means discriminates an area in which a main object to be photographed exists by adding said optical flow for a predetermined time.

28. An apparatus according to claim 27, wherein said boundary detecting means detects a boundary of the image on the basis of the result of a Laplacian arithmetic operation of the movement vector distribution.

29. An apparatus according to claim 28, wherein said boundary detecting means specifies the block in which the movement vectors cannot correctly be detected from the magnitude of the result of the Laplacian arithmetic operation of the movement vector distribution.

30. An apparatus according to claim 29, wherein said second detecting means decides a coarse vibration detecting area from a distribution of the result in which the optical flow has been added for a predetermined time, and further an area in which accurate movement vectors can be detected is selected from the result of the Laplacian arithmetic operation.

31. An apparatus according to claim 30, further having correcting means for correcting a vibration on the basis of an output of said vibration amount detecting means.

32. An apparatus according to claim 31, wherein said correcting means is a variable vertical angle prism.

33. An apparatus according to claim 23, wherein said boundary detecting means detects a boundary of the image from the result of the Laplacian arithmetic operation of the input image.

34. An apparatus according to claim 33, wherein said boundary detecting means detects a zero crossing point indicative of an edge portion of the image as a result of the Laplacian arithmetic operation of the input image, and said second detecting means sets the block including said zero crossing point into a non-vibration area which is not used for vibration detection and sets the other blocks into the vibration detecting areas.

35. A vibrationproof apparatus comprising:

a) movement vector detecting means for detecting movement vectors in every area which is obtained by dividing an image into a plurality of areas;

b) vibration detecting area deciding means (i) for assigning different weights to the movement vectors from said movement vector detecting means according to a plurality of weighting functions, (ii) for integrating the weighted movement vectors in each of the plurality of areas for a predetermined time, and (iii) for deciding the vibration detecting area on the basis of the integrated values;

c) vibration amount detecting means for detecting a vibration amount of the basis of the movement vector information in said vibration detecting area; and

d) correcting means for correcting a vibration in accordance with information from said vibration amount detecting means.

36. An apparatus according to claim 35, further having image pickup means for photoelectrically converting an object image formed on an image pickup surface and for outputting an image pickup signal.

37. An apparatus according to claim 36, wherein said input image is an image formed on the image pickup surface of said image pickup means, and said movement vector detecting means detects the movement vectors from said image pickup signal.

38. An apparatus according to claim 35, wherein said movement vector detecting means generates a distribution (optical flow) of the movement vectors on a picture plane.

39. An apparatus according to claim 38, wherein said vibration detecting area deciding means decides the vibration detecting area on the basis of a movement vector distribution as a result in which said optical flow has been added for a predetermined time.

40. An apparatus according to claim 38, further comprising a plurality of memory means for applying different weights to the information from said movement vector detecting means and for accumulating the weighted information.

41. An apparatus according to claim 40, wherein the movement vector weight addition information which is accumulated in said memory means is movement vector information corresponding to different photographing states.

42. An apparatus according to claim 41, wherein said different photographing states include a panning operation, an object tracking photographing operation, a still photographing operation (camera is fixed), and the like.

43. An apparatus according to claim 41, further comprising comparing and integrating means for comparing and integrating the different weight addition values of the movement vectors stored in said memory means, for detecting the present photographing state, for giving significance to the weight addition value in the memory means corresponding to said present photographing state, and for outputting the weight addition value.

44. An apparatus according to claim 43, wherein said vibration detecting area deciding means decides the vibration detecting area from the magnitudes of the movement vectors outputted from said comparing and integrating means.

45. A video camera apparatus comprising:

a) image pickup means for photoelectrically converting an object image formed on an image pickup surface and for outputting an image pickup signal;

b) movement vector detecting means for detecting movement vectors from said image pickup signal for every block obtained by dividing said image pickup surface into a plurality of blocks;

c) operating means for assigning different weights to the movement vectors output from said movement vector detecting means according to a plurality of weighting functions and for integrating the weighted movement vectors in each of the plurality of blocks for a predetermined time;

d) memory means for storing the integrated information;

e) vibration detecting area deciding means for deciding a vibration detecting area by evaluating integrated weight values stored in said memory means;

f) vibration amount detecting means for detecting a vibration amount on the basis of movement vector information in the vibration detecting area from said movement vector detecting means; and

g) correcting means for correcting a vibration in accordance with the information from said vibration amount detecting means.