Multi-eye image pickup apparatus, and method and apparatus for measuring or recognizing three-dimensional shape

• US 6,118,475 A
• Filed: 06/01/1995
• Issued: 09/12/2000
• Est. Priority Date: 06/02/1994
• Status: Expired due to Term

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Matsugu, Masakazu, Mori, Katsuhiko, Iijima, Katsumi, Yano, Kotaro, Kondo, Toshiaki
• Primary Examiner(s): Le, Vu
• Application Number:
  US08/457,397
• Time in Patent Office: 1,930 Days
• Field of Search: 348/42-43, 348/47-48, 348/208, 348/214, 348/219, 348/262, 348/416, 348/699, 382/154, 382/285, 359/466
• US Class Current: 348/42
• CPC Class Codes: G01C 11/06 by comparison of two or more pictures of the same area
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/221 using the relative movement between cameras and objects
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0085 Motion estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0092 Image segmentation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0096 Synchronisation or controlling aspects

Abstract

An apparatus is arranged to have two cameras set at a predetermined angle of convergence, image motion vector extracting portions each for extracting motion vectors between predetermined frames from images obtained in time series from the respective cameras, a stereoscopic parallax extracting portion for extracting a stereoscopic parallax between the predetermined frames, and a portion for obtaining final motion vectors or distance information from the motion vectors extracted by the respective motion vector extracting portions and the stereoscopic parallax extracted by the stereoscopic parallax extracting portion.

Claims

1. A multi-eye image pickup apparatus comprising:

a plurality of image pickup means set at a predetermined angle of convergence, each for picking up an image of a desired object;

a plurality of motion vector extracting means provided corresponding to said image pickup means, each for extracting a motion vector of said object between predetermined frames from images obtained in time series by the corresponding image pickup means;

stereoscopic parallax extracting means for extracting a stereoscopic parallax of said object between said predetermined frames from images obtained in time series from said respective image pickup means; and

means for obtaining a final motion vector of said object between said predetermined frames from motion vectors extracted by said respective motion vector extracting means and the stereoscopic parallax extracted by said stereoscopic parallax extracting means.

2. The apparatus according to claim 1, wherein the motion vectors extracted by said respective motion vector extracting means are defined as first motion vectors;

said apparatus comprising;

stereoscopic parallax motion vector extracting means for extracting a stereoscopic parallax motion vector of said object between said predetermined frames from the stereoscopic parallax extracted by said stereoscopic parallax extracting means;

converting means for converting the stereoscopic parallax motion vector extracted by said stereoscopic parallax motion vector extracting means into second motion vectors corresponding to said first motion vectors, based on said predetermined angle of convergence; and

motion vector determining means for determining the final motion vector of said object between said predetermined frames from the first motion vectors extracted by said respective motion vector extracting means and the second motion vectors converted into by said converting means.

3. The apparatus according to claim 2, wherein said motion vector determining means comprises comparing and investigating means provided corresponding to said respective motion vector extracting means, each for investigating if the first motion vectors each extracted by the motion vector extracting means are coincident with the second motion vectors converted by the converting means, and integrating means for integrating said first motion vectors with said second motion vectors, based on results of investigation by said respective comparing and investigating means;

and

wherein said integrating means is so arranged that if a first motion vector coincides with a second motion vector said integrating means takes a mean value or weighted mean of the first and second motion vectors, that if either one of the first motion vectors coincides with a second motion vector the integrating means takes a mean value or weighted mean of said coincident first motion vector and said second motion vector, and that if a second motion vector does not coincide with any of the first motion vectors the integrating means employs either one of said first motion vectors.

4. The apparatus according to claim 2, wherein extraction of the motion vectors, the stereoscopic parallax, and the stereoscopic parallax motion vector by said motion vector extracting means, said stereoscopic parallax extracting means, and said stereoscopic parallax motion vector extracting means is carried out by extracting common regions from images obtained in time series from the image pickup means or from images obtained from the respective image pickup means and obtaining a relative deviation amount between said common regions extracted.

5. A three-dimensional shape measuring method comprising:

a step of performing a stereo distance-measuring method for measuring an object distance, using a plurality of images obtained when an object is taken from a plurality of different points of view;

a step of performing a zoom distance-measuring method for measuring an object distance, using a plurality of time-serial images obtained when said object is taken at a plurality of different angles of view; and

a step of measuring a three-dimensional shape of said object by combining a measurement result obtained by said stereo distance-measuring method with a measurement result obtained by said zoom distance-measuring method.

6. The method according to claim 5, wherein differences of zoom magnifications are corrected between the plurality of images used in said zoom distance-measuring method, using the object distance measured by said stereo distance-measuring method.

7. The method according to claim 5, wherein a plurality of image pickup apparatus are used and wherein said each image pickup apparatus is directed so that said object is included in an angle of view of said apparatus, depending upon a zoom magnification.

8. The method according to claim 5, wherein measurement of the object distance is performed adaptively utilizing a past measurement result in at least one of said stereo distance-measuring method and zoom distance-measuring method.

9. A three-dimensional shape measuring apparatus comprising:

image pickup means for picking up images of an object from a plurality of different points of view and at a plurality of different angles of view;

stereo distance-measuring means for measuring an object distance, using a plurality of images obtained when said image pickup means takes said object from the plurality of points of view;

zoom distance-measuring means for measuring an object distance, using a plurality of images obtained when said image pickup means takes said object at the plurality of angles of view; and

integration measuring means for measuring a three-dimensional shape of said object, using a measurement result obtained by said stereo distance-measuring means and a measurement result obtained by said zoom distance-measuring means.

10. The apparatus according to claim 9, further comprising image correcting means for correcting differences of zoom magnifications between the plurality of images used in said zoom distance-measuring means, using the object distance measured by said stereo distance-measuring means.

11. The apparatus according to claim 9, wherein said image pickup means comprises a plurality of image pickup apparatus and wherein said each image pickup apparatus is directed so that said object is included in an angle of view of said apparatus, depending upon a zoom magnification.

12. The apparatus according to claim 9, wherein measurement of the object distance is performed adaptively utilizing a past measurement result in at least one of said stereo distance-measuring means and zoom distance-measuring means.

13. A three-dimensional object shape recognizing method comprising:

a step of recognizing an object by a multi-viewpoint image pickup method of a horizontal/vertical plane parallax type using a plurality of images input from different points of view at a same object distance;

a step of recognizing an object by a multi-viewpoint image pickup method of a front-to-back parallax type using a plurality of images input from a same point of view at different object distances; and

a step of taking a weighted mean integrally using recognition results of the three-dimensional object each obtained in said above steps.

14. The method according to claim 13, comprising a step of performing magnification correction between images of different magnifications used in said multi-viewpoint image pickup method of the front-to-back parallax type.

15. The method according to claim 13, comprising a step of performing magnification correction depending upon an object distance obtained by said multi-viewpoint image pickup method of the horizontal/vertical plane parallax type.

16. The method according to claim 13, wherein a Fourier-Mellin transformation method is used for correspondence between images of different magnifications used in said multi-viewpoint image pickup method of the front-to-back parallax type.

17. The method according to claim 13, wherein a filter is used to separate oblique spatial gradient components of said images from horizontal/vertical spatial gradient components of said images, said horizontal/vertical spatial gradient components of said images are supplied with priority to said multi-viewpoint image pickup method of the horizontal/vertical plane parallax type, and said oblique spatial gradient components of said images are supplied with priority to said multi-viewpoint image pickup method of the front-to-back parallax type.

18. The method according to claim 13, wherein a past three-dimensional object shape recognition result is utilized.

19. The method according to claim 13, wherein a plurality of image pickup apparatus are arranged to be directed so that a same object is included in fields of the apparatus, depending upon speeds of forward or backward motions of said image pickup apparatus.

20. A three-dimensional object shape recognizing apparatus comprising:

recognizing means for recognizing an object by a multi-viewpoint image pickup method of a horizontal/vertical plane parallax type using a plurality of images input from different points of view at a same object distance;

recognizing means for recognizing an object by a multi-viewpoint image pickup method of a front-to-back parallax type using a plurality of images input from a same point of view at different object distances; and

integration processing means for taking a weighted mean of recognition results of the three-dimensional object obtained by said respective recognizing means.

21. The apparatus according to claim 20, comprising correcting means for performing magnification correction between images of different magnifications used in said multi-viewpoint image pickup method of the front-to-back parallax type.

22. The apparatus according to claim 20, comprising correcting means for performing magnification correction depending upon an object distance obtained by said multi-viewpoint image pickup method of the horizontal/vertical plane parallax type.

23. The apparatus according to claim 20, wherein a Fourier-Mellin transformation method is used for correspondence between images of different magnifications used in said multi-viewpoint image pickup method of the front-to-back parallax type.

24. The apparatus according to claim 20, which comprises a filter for separating oblique spatial gradient components of the images from horizontal/vertical spatial gradient components of said images, wherein said horizontal/vertical spatial gradient components of the images are supplied with priority to said recognizing means by said multi-viewpoint image pickup method of the horizontal/vertical plane parallax type and wherein said oblique spatial gradient components of the images are supplied with priority to said recognizing means by said multi-viewpoint image pickup method of the front-to-back parallax type.

25. The apparatus according to claim 20, which utilizes a past three-dimensional object shape recognizing result.

26. The apparatus according to claim 20, wherein a plurality of image pickup apparatus are arranged to be directed so that a same object is included in fields of the apparatus, depending upon speeds of forward or backward motions of said image pickup apparatus.