Object extraction method and system

• US 6,226,396 B1
• Filed: 07/31/1998
• Issued: 05/01/2001
• Est. Priority Date: 07/31/1997
• Status: Expired due to Term

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Marugame, Atsushi
• Primary Examiner(s): Au, Amelia
• Assistant Examiner(s): AHMED, SAMIR ANWAR
• Application Number:
  US09/127,446
• Time in Patent Office: 1,005 Days
• Field of Search: 382/151, 382/154, 382/190, 382/192, 382/193, 382/195, 382/199, 382/201, 382/203, 382/206, 382/291, 345/139, 345/418, 345/419, 345/425, 356/12, 364/559
• US Class Current: 382/154
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

In an image processing system having a primary image pickup device and a plurality of reference image pickup devices, a feature region consisting of a plurality of feature segments each defined by a pair of feature points is extracted from a primary image. Subsequently, a correspondence point candidate of each of the feature points is extracted from each of the reference images. Based on the feature points and the correspondence point candidate of each of the feature points for each of the reference images, three-dimensional coordinates corresponding to the feature points are calculated, and the feature region is extracted as a target object from the primary image based on the feature points and the three-dimensional coordinates corresponding to the feature points for each of the reference images.

Claims

1. An object extraction method comprising the steps of:

a) generating a first image and a plurality of second images through a first image pickup device and a plurality of second image pickup devices, respectively;

b) extracting a feature region from the first image, the feature region consisting of a plurality of feature segments each defined by a pair of feature points;

c) extracting a correspondence point candidate of each of the feature points from each of the second images;

d) calculating three-dimensional coordinates corresponding to the feature points based on the feature points based on the feature points and the correspondence point candidate of each of the feature points for each of the second images; and

e) extracting the feature region as a target object from the first image based on the feature points and the three-dimensional coordinates corresponding to the feature points for each of the second images, wherein step e) comprises;

e-1) determining feature point correspondences between the first image and each of the second images using the three-dimensional coordinates for each of the second images;

e-2) calculating a segment length difference between the first image and each of the second images by using segment correspondence;

e-3) calculating a disparity of feature points between the first image and each of the second images; and

e-4) extracting the feature region as the target object from the first image when both the segment length difference and the disparity fall into a predetermined length difference range and a predetermined disparity range, respectively.

2. The object extraction method according to claim 1, wherein the step e-1) comprises the steps of:

calculating possible distances between counterparts of three-dimensional coordinates over the second images;

selecting a minimum distance from the possible distances; and

determining a pair of three-dimensional coordinates providing the minimum distance among point correspondence between the first image and each of the second images.

3. The object extraction method according to claim 1, wherein the step e-4) comprises the steps of:

generating the predetermined length difference range depending on a first distribution of the number of points included in the feature region with respect to the segment length difference;

generating the predetermined disparity range depending on a second distribution of the number of points included in the feature region with respect to the disparity; and

extracting the feature region as the target object from the first image when both the length difference and the disparity fall into the predetermined segment length difference range and the predetermined disparity range, respectively.

4. The object extraction method according to claim 3, wherein the predetermined length difference range is a range where the first distribution is larger than a first predetermined threshold level.

5. The object extraction method according to claim 3, wherein the predetermined disparity range is a range where the second distribution is larger than a second predetermined threshold level.

6. The object extraction method according to claim 5, wherein, when a plurality of ranges where the second distribution is larger than a second predetermined threshold level, a range corresponding to a maximum disparity is determined to be the predetermined disparity range.

7. An object extraction system comprising:

a first image pickup device for generating a first image of a space;

a plurality of second image pickup devices for generating second images of the space, respectively;

a first extractor for extracting a feature region from the first image, the feature region consisting of a plurality of a feature segments each defined by a pair of adjoining feature points;

a second extractor for extracting a correspondence point candidate of each of the feature points from each of the second images;

a three-dimensional coordinate calculator for calculating three-dimensional coordinates corresponding to the feature points based on the feature points and the correspondence point candidate of each of the feature points for each of the second images; and

a third extractor for extracting the feature region as a target object from the first image based on the feature points and the three-dimensional coordinates corresponding to the feature points for each of the second images, wherein the third extractor comprises;

a correspondence point determiner for determining first and second correspondence points of the feature points using feature point correspondences between the first image and each of the second images;

a segment length difference calculator for calculating a segment length difference between a first distance calculated between the feature points and a second distance calculated between the first image and each of the second images by using segment correspondence;

a disparity calculator for calculating a disparity of feature points between the first image and each of the second images; and

a selector for selecting the feature region as the target object from the first image when both the segment length difference and the disparity fall into a predetermined length difference range and a predetermined disparity range respectively.

8. The object extraction system according to claim 7, wherein the correspondence point determiner calculates possible distance between counterparts of three-dimensional coordinates over the second images, selects a minimum distance from the possible distances, and determines a pair of three-dimensional coordinates providing the minimum distance to be the first and second correspondence points, respectively.

9. An object extraction system comprising:

a first image pickup device for generating a first image of a space;

a plurality of second image pickup devices for generating second images of the space, respectively;

a first extractor for extracting a feature region from the first image, the feature region consisting of a plurality of feature segments each defined by a pair of feature points;

a second extractor for extracting a correspondence point candidate of each of the feature points from each of the second images;

a three-dimensional coordinate calculator for calculating three-dimensional coordinates corresponding to the feature points based on the feature points and the correspondence point candidate of each of the feature points for each of the second images;

a correspondence point determiner for determining first and second correspondence points of the feature points using the three-dimensional coordinates for each of the second images;

a segment length difference calculator for calculating a segment length difference between the first image and each of the second images by using segment correspondence;

e-3) a disparity calculator for calculating a disparity of feature points between the first image and each of the second images;

a first range generator for generating a segment length difference range depending on a first distribution of the number of points included in the feature region with respect to the segment length difference;

a second range generator for generating a disparity range depending on a second distribution of the number of points included in the feature region with respect to the disparity; and

a third extractor for selecting the feature region as a target object from the first image when both the segment length difference and the disparity fall into the segment length difference range and a disparity range, respectively.

10. The object extraction system according to claim 9, wherein the first range generator comprises:

a first histogram generator for generating the first distribution; and

a first determiner for determining the segment length difference range to be a range where the first distribution is larger than a first predetermined threshold level.

11. The object extraction system according to claim 10, wherein the second range generator comprises:

a second histogram generator for generating the second distribution; and

a second determiner for determining the disparity range to be a range where the second distribution is larger than a second predetermined threshold level.

12. The object extraction system according to claim 11, wherein, when a plurality of ranges where the second the second distribution is larger than a second predetermined threshold level, the second range generator determines a range corresponding to a maximum disparity to be the disparity range.

13. A program memory storing a program for performing an object extraction, the program comprising the steps of;

a) storing a first image and a plurality of second images onto image memories, wherein the first image is received from a first image pickup device and the second images is received from a plurality of second image pickup devices, respectively;

b) extracting a feature region from the first image, the feature region consisting of a plurality of feature segments each defined by a pair of adjoining feature points;

c) extracting a correspondence point candidate of each of the feature points from each of the second images;

d) calculating three-dimensional coordinates corresponding to the feature points based on the feature points and the correspondence point candidate of each of the feature points of each of the second images; and

e) extracting the feature region as a target object from the first image based on the feature points and the three-dimensional coordinates corresponding to the feature points for each of the second images, wherein step e) comprises;

e-1) determining feature point correspondences between the first image and each of the second images using the three-dimensional coordinates for each of the second images;

e-2) calculating a segment length difference between the first image and each of the second images by using segment correspondence;

e-3) calculating a disparity of feature points between the first image and each of the second images; and

e-4) extracting the feature region as the target object from the first image when both the segment length difference and the disparity fall into a segment length difference range and the disparity range, respectively.

14. The program memory according to claim 13, wherein the step e-1) comprises the steps of:

calculating possible distances between counterparts of three-dimensional coordinates over the second images;

selecting a minimum distance from the possible distances; and

determining a pair of three-dimensional coordinates providing the minimum distance to be the first and second correspondence points, respectively.

15. The program memory according to claim 13, wherein the step e-4) comprises the steps of:

generating the segment length difference range depending on a first distribution of the number of points included in the feature region with respect to the segment length difference;

generating the disparity range depending on a second distribution of the number of points included in the feature region with respect to the disparity; and

extracting the feature region as the target object from the first image when both the segment length difference and the disparity fall into the segment length difference range and the disparity range, respectively.

16. The program memory according to claim 15, wherein the segment length difference range is a range where the first distribution is larger than a first predetermined threshold level, and the disparity range is an range where the second distribution is larger than a second predetermined threshold level.

17. The program memory according to claim 16, wherein, when a plurality of ranges where the second the second distribution is larger than a second predetermined threshold level, a range corresponding to a maximum disparity is determined to be the disparity range.

18. An object extraction method comprising the steps of:

a) generating a first image and a plurality of second images through a first image pickup device and a plurality of second image pickup devices, respectively;

b) extracting a feature region from the first image, the feature region consisting of a plurality of feature segments each defined by a pair of feature points;

c) extracting a correspondence point candidate of each of the feature points from each of the second images;

d) calculating three-dimensional coordinates corresponding to the feature points based on the feature points and the correspondence point candidate of each of the feature points for each of the second images; and

e) extracting the feature region as a target object from the first image based on the feature points and the three-dimensional coordinates corresponding to the feature points for each of the second images, wherein step e) comprises;

e-1) determining feature point correspondences between the first image and each of the second images using the three-dimensional coordinates for each of the second images;

e-2) calculating a segment length difference between the first image and each of the second images by using segment correspondence; and

e-3) extracting the feature region as the target object from the first image when the segment length difference falls into a predetermined length difference range.