Dual-input image processor for recognizing, isolating, and displaying specific objects from the input images

• US 5,995,649 A
• Filed: 09/22/1997
• Issued: 11/30/1999
• Est. Priority Date: 09/20/1996
• Status: Expired due to Fees

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Marugame, Atsushi
• Primary Examiner(s): Bella, Matthew C.
• Application Number:
  US08/935,141
• Time in Patent Office: 799 Days
• Field of Search: 382/190, 382/192, 382/193, 382/195, 382/199, 382/201, 382/203, 382/206, 382/151, 382/154, 382/291, 382/191, 348/94, 348/95, 345/425, 345/419, 364/559
• US Class Current: 382/154
• CPC Class Codes: G06V 10/751 Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching

Abstract

An image processor comprising an image pickup portion to take in an image of an object, a reference point extracting portion that extracts a reference point to be used in object extraction from images taken by the image pickup portion,a corresponding point detecting portion that detects a corresponding point in other images, a three dimensional coordinates value calculating portion that calculates a three dimensional coordinates value of an object by using the absolute position and posture parameters calculated by using image of a reference solid and coordinates values of reference point and corresponding point, a contour point extracting portion that extracts a contour point to show contour of an object on the basis of coordinate values of reference point and three dimensional coordinate values, and an object display control portion that extracts an image of the object from images photographed by the image pickup portion based on coordinates value of contour point and displays it on a display device.

Claims

1. An image processor that extracts only an image of a specific object from input images, comprising:

an image pickup means that is equipped with plural image pickup devices for picking up said object;

a parameter calculating means for extracting feature points on a reference solid from the images obtained by picking up said reference solid where a known pattern is picked up by said image pickup means, and calculating the relative position and posture parameters of each said image pickup device arranged on said image pickup means;

a reference point extracting means for extracting a reference point as a feature point having features on an image, wherein the image is one image of plural images picked up by said image pickup means, and calculating coordinate values of said image of the reference point;

a corresponding point detecting means for detecting corresponding points to be feature points having features on an image which correspond with said reference points, wherein said corresponding points are detected for all the images except the images that extract said reference points, among said plural images, by coordinate values of said reference points extracted by said reference point extracting means, and calculating coordinate values in each image at the corresponding points;

a three dimensional coordinates calculating means for calculating coordinate values in the three dimensional space of said reference points, by using the relative position and posture parameters of each said image pickup device calculated by said parameter calculating means, coordinate values of said reference points extracted by said reference point extracting means, and coordinate values of said corresponding points detected by said corresponding point detecting means;

a contour point extracting means for extracting contour points showing a contour of said object among images picked up by said image pickup means, on the basis of coordinate values of said reference points extracted by said reference point extracting means and said three dimensional coordinates values calculated by said three dimensional coordinates value calculating means, and calculating their coordinate values; and

an object display means for extracting an image of said object from images picked up by said image pickup means based on coordinate values of said contour points extracted by said contour point extracting means, and displaying them on a display device.

2. An image processor as set forth in claim 1, wherein

said parameter calculating means,

extracts six feature points or more having features on image from plural planes of said reference solid, and calculates their coordinate values, and

and calculates said parameters by using the least square method in equations including said calculated coordinate values.

3. An image processor as set forth in claim 1, wherein,

said corresponding point detecting means,

leads a straight line including said corresponding points corresponding to said reference points, in other images than images that extract said reference points, by using parameters calculated by said parameter calculating means,

searches for candidates of point corresponding to be feature points having features on image, on said straight line, and

carries out a block matching processing on the searched the above corresponding point candidate, and makes said corresponding point candidates as corresponding points where an obtained block error is below a threshold value.

4. An image processor as set forth in claim 1, wherein

said parameter calculating means,

extracts six feature points or more having features on image from plural planes of said reference solid, and calculates their coordinate values,

and calculates said parameters by using the least square method in equations including said calculated coordinate values,

said corresponding point detecting means,

leads a straight line including said corresponding points corresponding to said reference points, in other images than images that extract said reference points, by using parameters calculated by said parameter calculating means,

searches for candidates of point corresponding to be feature points having features on image, on said straight line, and

carries out a block matching processing on the searched the above corresponding point candidates, and makes said corresponding point candidates as corresponding points at which an obtained block error is below a threshold value.

5. An image processor as set forth in claim 1, further comprising:

a noise correcting means for judging whether each said extracted contour point is a true contour point or a noise based on the presence or absence of continuity of the coordinate values in each said contour point extracted by said contour point extracting means, and to a portion that is judged as a noise, extracting other candidates of contour points on the basis of coordinate values of a true contour point near the portion, and outputting them to said corresponding point detecting means and said three dimensional coordinates calculating means, and outputting the coordinate values of the contour points where there is no noise portion or noise portion has been corrected to said object display means,

wherein said corresponding point detecting means detects corresponding points based on the contour point candidate, when contour point candidates are input from said noise correcting means,

said three dimensional coordinates calculating means calculates three dimensional coordinates values of the contour point on the basis of corresponding points of the contour point candidates detected by contour point candidate and said corresponding point detecting means, when contour point candidates are input from said noise correction means, and

said object display means extracts an image of said object from images photographed by said image pickup means based on coordinate values of the contour point input from said noise correction means.

6. An image processor as set forth in claim 1, further comprising:

a noise correcting means for judging whether each said extracted contour point is a true contour point or a noise based on the presence or absence of continuity of the coordinate values in each said contour point extracted by said contour point extracting means, and to a portion that is judged as a noise, extracting other contour candidate points based on coordinate values of a true contour point near the portion, and outputting them to said corresponding point detecting means and said three dimensional coordinates calculating means, and outputting the coordinate values of the contour points where there is no noise portion or noise portion has been corrected to said object display means,

wherein said corresponding point detecting means,

leads a straight line including said corresponding points corresponding to said reference points, in other images than images that extract said reference points, by using parameters calculated by said parameter calculating means,

searches for corresponding point candidates to be feature points having features on image, on said straight line,

carries out a block matching processing on the searched above corresponding point candidate, and makes said corresponding point candidates as corresponding points at which an obtained block error is below a threshold value,

and when contour point candidates are input from said noise correcting means, detects corresponding points based on the contour point candidates,

said three dimensional coordinates calculating means calculates three dimensional coordinates values of the contour point based on corresponding points of the contour point candidates detected by contour point candidate and said corresponding point detecting means, when contour point candidates are input from said noise correction means, and

said object display means extracts an image of said object from images photographed by said image pickup means based on coordinate values of the contour point input from said noise correction means.

7. An image processor as set forth in claim 1, further comprising:

a noise judging means for judging whether each said contour point extracted based on the presence or absence of continuity of coordinate values in each said contour point extracted by said contour point extracting means is a true contour point or a noise,

a contour smoothing processing means for correcting the contour point that said noise judging means judges as a noise in consideration of the continuity with other contour points that are judged as true, and outputting it to said object display means, wherein

said object display means extracts an image of said object from images photographed by said image pickup means based on the coordinate values of the corrected contour point input from said contour smoothing processing means, and displays it on a display device.

8. An image processor set forth in claim 1, further comprising:

a noise judging means for judging whether each said contour point extracted based on the presence or absence of continuity of coordinate values in each said contour point extracted by said contour point extracting means is a true contour point or a noise,

a contour smoothing processing means for correcting the contour point that said noise judging means judges as a noise in consideration of the continuity with other contour points that are judged as true, and outputting it to said object display means, wherein

said object display means,

extracts an image of said object from images photographed by said image pickup means based on the coordinate values of the corrected contour point input from said contour smoothing processing means, and displays it on a display device,

said corresponding point detecting means,

leads a straight line including said corresponding points corresponding to said reference points, in other images than images that extract said reference points, by using parameters calculated by said parameter calculating means,

searches for candidates of points corresponding to be feature points having features on image, on said straight line, and

carries out a block matching processing on the searched above corresponding point candidate, and makes said corresponding point candidates as corresponding points at which an obtained block error is below a threshold value.

9. An image processor as set forth in claim 1, wherein said reference solid is not the same as said object.

10. An image processor that extracts only an image of a specific object from input images, comprising:

an image pickup means that is equipped with plural image pickup devices for picking up said object;

a parameter calculating means for extracting feature points on a reference solid from the images obtained by picking up said reference solid where a known pattern is picked up by said image pickup means, and calculating the relative position and posture parameters of each said image pickup device arranged on said image pickup means;

a reference point extracting means for extracting a reference point as a feature point having features on an image, wherein the image is one image of plural images picked up by said image pickup means, and calculating coordinate values of said image of the reference point;

a corresponding point detecting means for detecting corresponding points to be feature points having features on an image which correspond with said reference points, wherein said corresponding points are detected for all the images except the images that extract said reference points, among said plural images, by coordinate values of said reference points extracted by said reference point extracting means, and carrying out a block matching processing by using plural size blocks to the corresponding point candidate, and setting all said corresponding point candidates where obtained block error is below a threshold value as corresponding points, and calculating the unit of coordinate value in each image of all the corresponding points;

a block size designating means for designating the block size to be used in said block matching;

a three dimensional coordinates calculating means for calculating coordinate values in the three dimensional space of said reference point per coordinate value of said corresponding point, by using the relative position and posture parameters of each said image pickup device calculated by said parameter calculating means, coordinate values of said reference point extracted by said reference point extracting means, and unit of coordinate values of said corresponding points detected by said corresponding point detecting means;

an object extracting means fore extracting a contour point to show a contour of said object from images picked up by said image pickup means per said three dimensional coordinate value, based on unit of coordinate value of said reference point extracted by said reference point extracting means and said three dimensional coordinates value calculated by said three dimensional coordinates value calculating means, and calculating its coordinate value, and based on each coordinate value of the contour point, extracting an image of said object from images photographed by said image pickup means and creating unit of object image; and

an image synthesizing means for synthesizing unit of object image created by said object extracting means and displaying it on a display device.

11. An image processor as set forth in claim 10, wherein

said parameter calculating means,

extracts six feature points or more having features on image from plural planes of said reference solid, and calculates their coordinate values, and

and calculates said parameters by using the least square method in equations including said calculated coordinate values.

12. An image processor as set forth in claim 10, wherein,

said corresponding point detecting means,

leads a straight line including said corresponding points corresponding to said reference points, in other images than images that extract said reference points, by using parameters calculated by said parameter calculating means,

searches for corresponding point candidates to be feature points having features on image, on said straight line, and

carries out a block matching processing on the searched above corresponding point candidate by using the block of size designated by said block size designating device.

13. An image processor as set forth in claim 10, wherein

said parameter calculating means,

extracts six feature points or more having features on image from plural planes of said reference solid, and calculates their coordinate values,

and calculates said parameters by using the least square method in equations including said calculated coordinate values,

said corresponding point detecting means,

leads a straight line including said corresponding points corresponding to said reference points, in other images than images that extract said reference points, by using parameters calculated by said parameter calculating means,

searches for corresponding point candidates to be feature points having features on image, on said straight line, and

carries out a block matching processing on the searched above corresponding point candidate by using the block of size designated by said block size designating device.

14. An image processor set forth in claim 10, further comprising:

a noise correcting means for judging whether said extracted contour point is a true contour point or a noise based on the presence or absence of continuity of the coordinate values in said contour point extracted by said contour point extracting means, and to a portion that is judged as a noise, extracting other contour candidate points based on coordinate values of a true contour point near the portion, and outputting them to said corresponding point detecting means and said three dimensional coordinates calculating means, and outputting the coordinate values of the contour points where there is no noise portion or noise portion has been corrected to said object display means,

wherein said corresponding point detecting means detects corresponding points based on the contour point candidate, when contour point candidates are input from said noise correcting means,

said three dimensional coordinates calculating means calculates three dimensional coordinates values of the contour point based on corresponding points of the contour point candidates detected by contour point candidate and said corresponding point detecting means, when contour point candidates are input from said noise correction means, and

said object display means extracts an image of said object from images photographed by said image pickup means based on coordinate values of the contour point input from said noise correction means.

15. An image processor as set forth in claim 10, further comprising:

a noise correcting means for judging whether said extracted contour point is a true contour point or a noise based on the presence or absence of continuity of the coordinate values in said contour point extracted by said contour point extracting means, and to a portion that is judged as a noise, extracting other contour candidate points based on coordinate values of a true contour point near the portion, and outputting them to said corresponding point detecting means and said three dimensional coordinates calculating means, and outputting the coordinate values of the contour points where there is no noise portion or noise portion has been corrected to said object display means,

wherein said corresponding point detecting means,

leads a straight line including said corresponding points corresponding to said reference points, in other images than images that extract said reference points, by using parameters calculated by said parameter calculating means,

searches for candidate of points corresponding to be feature points having features on image, on said straight line,

carries out a block matching processing on the searched above corresponding point candidate, and makes said corresponding point candidates as corresponding points at which an obtained block error is below a threshold value,

and when contour point candidates are input from said noise correcting means, detects corresponding points based on the contour point candidates,

said three dimensional coordinates calculating means calculates three dimensional coordinates values of the contour point based on corresponding points of the contour point candidates detected by contour point candidate and said corresponding point detecting means, when contour point candidates are input from said noise correction means, and

said object display means extracts an image of said object from images photographed by said image pickup means based on coordinate values of the contour point input from said noise correction means.

16. An image processor set forth in claim 10, further comprising:

a noise judging means for judging whether said contour point extracted based on the presence or absence of continuity of coordinate values in said contour point extracted by said contour point extracting means is a true contour point or a noise,

a contour smoothing processing means for correcting the contour point that said noise judging means judges as a noise in consideration of the continuity with other contour points that are judged as true, and outputting it to said object display means,

wherein said object display means extracts an image of said object from images photographed by said image pickup means based on the coordinate values of the corrected contour point input from said contour smoothing processing means, and displays it on a display device.

17. An image processor as set forth in claim 10, further comprising:

a noise judging means for judging whether said contour point extracted based on the presence or absence of continuity of coordinate values in said contour point extracted by said contour point extracting means is a true contour point or a noise,

a contour smoothing processing means for correcting the contour point that said noise judging means judges as a noise in consideration of the continuity with other contour points that are judged as true, and outputting it to said object display means,

wherein said object display means,

extracts an image of said object from images photographed by said image pickup means based on the coordinate values of the corrected contour point input from said contour smoothing processing means, and displays it on a display device,

said corresponding point detecting means,

leads a straight line including said corresponding points corresponding to said reference points, in other images than images that extract said reference points, by using parameters calculated by said parameter calculating means,

searches for corresponding point candidates to be feature points having features on image, on said straight line, and

carries out a block matching processing on the searched above corresponding point candidate, and makes said corresponding point candidates as corresponding points at which an obtained block error is below a threshold value.

18. An image processor as set forth in claim 10, wherein said reference solid is not the same as said object.

19. A method of extracting an image, the method comprising:

taking two images of a reference object at approximately the same time, a first image with a first camera and a second image with a second camera, wherein the first and second cameras are in different positions;

determining position and orientation of each of the cameras by analyzing the two images of the reference object;

taking two images of a second object at approximately the same time, a third image with the first camera and a fourth image with the second camera;

selecting a feature point in the third image;

locating a corresponding point in the fourth image, which corresponds to the feature point, by forming an Epipolar line; and

determining the three-dimensional coordinates of the feature point.