Image recognition device using feature points, method for recognizing images using feature points, and robot device which recognizes images using feature points

• US 7,627,178 B2
• Filed: 04/22/2004
• Issued: 12/01/2009
• Est. Priority Date: 04/28/2003
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Sabe, Kohtaro, Suzuki, Hirotaka, Fujita, Masahiro
• Primary Examiner(s): Park; Edward
• Application Number:
  US10/517,615
• Publication Number:
  US 20050213818A1
• Time in Patent Office: 2,049 Days
• Field of Search: 382/190, 382/181, 382/201, 382/209, 382/216, 382/170
• US Class Current: 382/190
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/443 by matching or filtering
• CPC Class Codes: G06V 10/757 Matching configurations of points or features
• CPC Class Codes: G06V 10/758 Involving statistics of pixels or of feature values, e.g. histogram matching

Abstract

In an image recognition apparatus, feature point extraction sections and extract feature points from a model image and an object image. Feature quantity retention sections extract a feature quantity for each of the feature points and retain them along with positional information of the feature points. A feature quantity comparison section compares the feature quantities with each other to calculate the similarity or the dissimilarity and generates a candidate-associated feature point pair having a high possibility of correspondence. A model attitude estimation section repeats an operation of projecting an affine transformation parameter determined by three pairs randomly selected from the candidate-associated feature point pair group onto a parameter space. The model attitude estimation section assumes each member in a cluster having the largest number of members formed in the parameter space to be an inlier. The model attitude estimation section finds the affine transformation parameter according to the least squares estimation using the inlier and outputs a model attitude determined by this affine transformation parameter.

Claims

1. An image recognition apparatus which compares an object image containing a plurality of objects with a model image containing a model to be detected and extracts the model from the object image, the apparatus comprising:

feature point extracting means for extracting a feature point from each of the object image and the model image;

feature quantity retention means for extracting and retaining, as a feature quantity, a density gradient direction histogram at least acquired from density gradient information in a neighboring region at the feature point in each of the object image and the model image, the density gradient direction histogram storing a number of points near the feature point having each of a plurality of gradient directions;

feature quantity comparison means for comparing the feature quantity of each feature point of the object image with the feature quantity of each feature point of the model image and generating a candidate-associated feature point pair having similar feature quantities; and

model attitude estimation means for detecting the presence or absence of the model on the object image using the candidate-associated feature point pair and estimating a position and an attitude of the model, if any,

wherein the feature quantity comparison means itinerantly shifts one of the density gradient direction histograms of feature points to be compared in density gradient direction to find distances between the density gradient direction histograms by sequentially shifting all of the feature points in the one of the density gradient direction histograms one by one to generate a plurality of shifted histograms, and generates the candidate-associated feature point pair by determining a shortest distance between (1) an other of the density gradient direction histograms and (2) the one of the density gradient direction histograms and the shifted histograms.

2. The image recognition apparatus according to claim 1, wherein the feature quantity retention means extracts and retains, as the feature quantity, an average density gradient vector for each of plurality of partial regions into which the neighboring region is further divided, and the feature quantity comparison means generates the candidate-associated feature point pair based on a distance between density gradient direction histograms for the feature points to be compared and on similarity between feature vectors which are collected in the neighboring region as average density gradient vectors in each of the partial regions.

3. The image recognition apparatus according to claim 2, wherein the feature quantity comparison means generates a provisional candidate-associated feature point pair based on a distance between the density gradient direction histograms for the feature points to be compared and, based on the similarity between feature vectors, selects the candidate-associated feature point pair from the provisional candidate-associated feature point pair.

4. The image recognition apparatus according to claim 3, wherein the feature quantity comparison means uses a rotation angle equivalent to a shift amount giving the shortest distance to correct a density gradient direction of a density gradient vector in the neighboring region and selects the candidate-associated feature point pair from the provisional candidate-associated feature point pair based on similarity between the feature vectors in a corrected neighboring region.

5. The image recognition apparatus according to claim 1, further comprising:

candidate-associated feature point pair selection means for creating a rotation angle histogram concerning a rotation angle equivalent to a shift amount giving the shortest distance and selects a candidate-associated feature point pair giving a rotation angle for a peak in the rotation angle histogram from the candidate-associated feature point pair generated by the feature quantity comparison means, wherein the model attitude estimation means detects the presence or absence of the model on the object image using a candidate-associated feature point pair selected by the candidate-associated feature point pair selection means and estimates a position and an attitude of the model, if any.

6. An image recognition apparatus which compares an object image containing a plurality of objects with a model image containing a model to be detected and extracts the model from the object image, the apparatus comprising:

feature point extracting means for extracting a feature point from each of the object image and the model image;

feature quantity retention means for extracting and retaining, as a feature quantity, a density gradient direction histogram at least acquired from density gradient information in a neighboring region at the feature point in each of the object image and the model image, the density gradient direction histogram storing a number of points near the feature point having each of a plurality of gradient directions;

feature quantity comparison means for comparing the feature quantity of each feature point of the object image with the feature quantity of each feature point of the model image and generating a candidate-associated feature point pair having similar feature quantities; and

model attitude estimation means for detecting the presence or absence of the model on the object image using the candidate-associated feature point pair and estimating a position and an attitude of the model, if any,

wherein the feature quantity comparison means itinerantly shifts one of the density gradient direction histograms of feature points to be compared in density gradient direction to find distances between the density gradient direction histograms and generates the candidate-associated feature point pair by assuming a shortest distance to be a distance between the density gradient direction histograms, and

wherein the model attitude estimation means repeatedly projects an affine transformation parameter determined from three randomly selected candidate-associated feature point pairs onto a parameter space and finds an affine transformation parameter to determine a position and an attitude of the model based on an amine transformation parameter belonging to a cluster having the largest number of members out of clusters formed on a parameter space.

7. The image recognition apparatus according to claim 6, wherein the model attitude estimation means assumes a centroid for the cluster having the largest number of members to be an affine transformation parameter to determine a position and an attitude of the model.

8. The image recognition apparatus according to claim 6, wherein the model attitude estimation means assumes a candidate-associated feature point pair giving the affine transformation parameter belonging to a cluster having the largest number of members to be a true candidate-associated feature point pair and uses the true candidate-associated feature point pair for least squares estimation to find an affine transformation parameter for determining a position and an attitude of the model.

9. An image recognition apparatus which compares an object image containing a plurality of objects with a model image containing a model to be detected and extracts the model from the object image, the apparatus comprising:

feature point extracting means for extracting a feature point from each of the object image and the model image;

feature quantity retention means for extracting and retaining, as a feature quantity, a density gradient direction histogram at least acquired from density gradient information in a neighboring region at the feature point in each of the object image and the model image, the density gradient direction histogram storing a number of points near the feature point having each of a plurality of gradient directions;

feature quantity comparison means for comparing the feature quantity of each feature point of the object image with the feature quantity of each feature point of the model image and generating a candidate-associated feature point pair having similar feature quantities;

model attitude estimation means for detecting the presence or absence of the model on the object image using the candidate-associated feature point pair and estimating a position and an attitude of the model, if any; and

candidate-associated feature point pair selection means for performing generalized Hough transform for a candidate-associated feature point pair generated by the feature quantity comparison means, assuming a rotation angle, enlargement and reduction ratios, and horizontal and vertical linear displacements to be a parameter space, and selecting a candidate-associated feature point pair having voted for the most voted parameter from candidate-associated feature point pairs generated by the feature quantity comparison means, wherein the model attitude estimation means detects the presence or absence of the model on the object image using a candidate-associated feature point pair selected by the candidate-associated feature point pair selection means and estimates a position and an attitude of the model, if any,

wherein the feature quantity comparison means itinerantly shifts one of the density gradient direction histograms of feature points to be compared in density gradient direction to find distances between the density gradient direction histograms and generates the candidate-associated feature point pair by assuming a shortest distance to be a distance between the density gradient direction histograms.

10. An image recognition apparatus which compares an object image containing a plurality of objects with a model image containing a model to be detected and extracts the model from the object image, the apparatus comprising:

feature point extracting means for extracting a feature point from each of the object image and the model image;

feature quantity retention means for extracting and retaining, as a feature quantity, a density gradient direction histogram at least acquired from density gradient information in a neighboring region at the feature point in each of the object image and the model image, the density gradient direction histogram storing a number of points near the feature point having each of a plurality of gradient directions;

feature quantity comparison means for comparing the feature quantity of each feature point of the object image with the feature quantity of each feature point of the model image and generating a candidate-associated feature point pair having similar feature quantities; and

model attitude estimation means for detecting the presence or absence of the model on the object image using the candidate-associated feature point pair and estimating a position and an attitude of the model, if any,

wherein the feature quantity comparison means itinerantly shifts one of the density gradient direction histograms of feature points to be compared in density gradient direction to find distances between the density gradient direction histograms and generates the candidate-associated feature point pair by assuming a shortest distance to be a distance between the density gradient direction histograms, and

wherein the feature point extraction means extracts a local maximum point or a local minimum point in second-order differential filter output images with respective resolutions as the feature point, i.e., a point free from positional changes due to resolution changes within a specified range in a multi-resolution pyramid structure acquired by repeatedly applying smoothing filtering and reduction resampling to the object image or the model image.

11. An image recognition apparatus which compares an object image containing a plurality of objects with a model image containing a model to be detected and extracts the model from the object image, the apparatus comprising:

feature point extracting means for extracting a feature point from each of the object image and the model image;

feature quantity retention means for extracting and retaining a feature quantity in a neighboring region at the feature point in each of the object image and the model image, the feature quantity being a density gradient direction histogram storing a number of points near the feature point having each of a plurality of gradient directions;

feature quantity comparison means for comparing the feature quantity of each feature point of the object image with the feature quantity of each feature point of the model image and generating a candidate-associated feature point pair having similar feature quantities, each candidate-associated feature point pair including one feature point of the object image and one feature point of the model image; and

model attitude estimation means for detecting the presence or absence of the model on the object image using the candidate-associated feature point pair and estimating a position and an attitude of the model, if any,

wherein the model attitude estimation means repeatedly projects an affine transformation parameter determined from three randomly selected candidate-associated feature point pairs onto a parameter space and finds an affine transformation parameter to determine a position and an attitude of the model based on an affine transformation parameter belonging to a cluster having the largest number of members out of clusters formed on a parameter space.

12. The image recognition apparatus according to claim 11, wherein the model attitude estimation means assumes a centroid for the cluster having the largest number of members to be an affine transformation parameter to determine a position and an attitude of the model.

13. The image recognition apparatus according to claim 11, wherein the model attitude estimation means assumes a candidate-associated feature point pair giving the affine transformation parameter belonging to a cluster having the largest number of members to be a true candidate-associated feature point pair and uses the true candidate-associated feature point pair for least squares estimation to find an affine transformation parameter for determining a position and an attitude of the model.

14. An image recognition apparatus which compares an object image containing a plurality of objects with a model image containing a model to be detected and extracts the model from the object image, the apparatus comprising:

feature point extracting means for extracting a feature point from each of the object image and the model image;

feature quantity retention means for extracting and retaining a feature quantity in a neighboring region at the feature point in each of the object image and the model image, the feature quantity being a density gradient direction histogram storing a number of points near the feature point having each of a plurality of gradient directions;

feature quantity comparison means for comparing the feature quantity of each feature point of the object image with the feature quantity of each feature point of the model image and generating a candidate-associated feature point pair having similar feature quantities;

model attitude estimation means for detecting the presence or absence of the model on the object image using the candidate-associated feature point pair and estimating a position and an attitude of the model, if any; and

candidate-associated feature point pair selection means for performing generalized Hough transform for a candidate-associated feature point pair generated by the feature quantity comparison means, assuming a rotation angle, enlargement and reduction ratios, and horizontal and vertical linear displacements to be a parameter space, and selecting a candidate-associated feature point pair having voted for the most voted parameter from candidate-associated feature point pairs generated by the feature quantity comparison means,

wherein the model attitude estimation means repeatedly projects an affine transformation parameter determined from three randomly selected candidate-associated feature point pairs onto a parameter space and finds an affine transformation parameter to determine a position and an attitude of the model based on an affine transformation parameter belonging to a cluster having the largest number of members out of clusters formed on a parameter space, and

wherein the model attitude estimation means detects the presence or absence of the model on the object image using a candidate-associated feature point pair selected by the candidate-associated feature point pair selection means and estimates a position and an attitude of the model, if any.

15. An image recognition apparatus which compares an object image containing a plurality of objects with a model image containing a model to be detected and extracts the model from the object image, the apparatus comprising:

feature point extracting means for extracting a feature point from each of the object image and the model image;

feature quantity retention means for extracting and retaining a feature quantity in a neighboring region at the feature point in each of the object image and the model image, the feature quantity being a density gradient direction histogram storing a number of points near the feature point having each of a plurality of gradient directions;

feature quantity comparison means for comparing the feature quantity of each feature point of the object image with the feature quantity of each feature point of the model image and generating a candidate-associated feature point pair having similar feature quantities;

model attitude estimation means for detecting the presence or absence of the model on the object image using the candidate-associated feature point pair and estimating a position and an attitude of the model, if any,

wherein the model attitude estimation means repeatedly projects an affine transformation parameter determined from three randomly selected candidate-associated feature point pairs onto a parameter space and finds an affine transformation parameter to determine a position and an attitude of the model based on an affine transformation parameter belonging to a cluster having the largest number of members out of clusters formed on a parameter space, and

wherein the feature point extraction means extracts a local maximum point or a local minimum point in second-order differential filter output images with respective resolutions as the feature point, i.e., a point free from positional changes due to resolution changes within a specified range in a multi-resolution pyramid structure acquired by repeatedly applying smoothing filtering and reduction resampling to the object image or the model image.

16. An image recognition method which compares an object image containing a plurality of objects with a model image containing a model to be detected and extracts the model from the object image, the method comprising:

at least one processor performing the steps of,

extracting a feature point from each of the object image and the model image;

extracting and retaining, as a feature quantity, a density gradient direction histogram at least acquired from density gradient information in a neighboring region at the feature point in each of the object image and the model image, the density gradient direction histogram storing a number of points near the feature point having each of a plurality of gradient directions;

comparing the feature quantity of each feature point of the object image with the feature quantity of each feature point of the model image and generating a candidate-associated feature point pair having similar feature quantities; and

detecting the presence or absence of the model on the object image using the candidate-associated feature point pair and estimating a position and an attitude of the model, if any,

wherein the comparing itinerantly shifts one of the density gradient direction histograms of feature points to be compared in density gradient direction to find distances between the density gradient direction histograms by sequentially shifting all of the feature points in the one of the density gradient direction histograms one by one to generate a plurality of shifted histograms, and generates the candidate-associated feature point pair by determining a shortest distance between (1) an other of the density gradient direction histograms and (2) the one of the density gradient direction histograms and the shifted histograms.

17. An image recognition method which compares an object image containing a plurality of objects with a model image containing a model to be detected and extracts the model from the object image, the method comprising:

at least one processor performing the steps of,

extracting a feature point from each of the object image and the model image;

extracting and retaining a feature quantity in a neighboring region at the feature point in each of the object image and the model image, the feature quantity being a density gradient direction histogram storing a number of points near the feature point having each of a plurality of gradient directions;

comparing the feature quantity of each feature point of the object image with the feature quantity of each feature point of the model image and generating a candidate-associated feature point pair having similar feature quantities, each candidate-associated feature point pair including one feature point of the object image and one feature point of the model image; and

detecting the presence or absence of the model on the object image using the candidate-associated feature point pair and estimating a position and an attitude of the model, if any,

wherein the detecting repeatedly projects an affine transformation parameter determined from three randomly selected candidate-associated feature point pairs onto a parameter space and finds an affine transformation parameter to determine a position and an attitude of the model based on an affine transformation parameter belonging to a cluster having the largest number of members out of clusters formed on a parameter space.

18. An autonomous robot apparatus capable of comparing an input image with a model image containing a model to be detected and extracting the model from the input image, the apparatus comprising:

image input means for imaging an outside environment to generate the input image;

feature point extracting means for extracting a feature point from each of the input image and the model image;

feature quantity retention means for extracting and retaining, as a feature quantity, a density gradient direction histogram at least acquired from density gradient information in a neighboring region at the feature point in each of the input image and the model image, the density gradient direction histogram storing a number of points near the feature point having each of a plurality of gradient directions;

feature quantity comparison means for comparing the feature quantity of each feature point of the input image with the feature quantity of each feature point of the model image and generating a candidate-associated feature point pair having similar feature quantities; and

model attitude estimation means for detecting the presence or absence of the model on the input image using the candidate-associated feature point pair and estimating a position and an attitude of the model, if any,

wherein the feature quantity comparison means itinerantly shifts one of the density gradient direction histograms of feature points to be compared in density gradient direction to find distances between the density gradient direction histograms by sequentially shifting all of the feature points in the one of the density gradient direction histograms one by one to generate a plurality of shifted histograms, and generates the candidate-associated feature point pair by determining a shortest distance between (1) an other of the density gradient direction histograms and (2) the one of the density gradient direction histograms and the shifted histograms.

19. An autonomous robot apparatus capable of comparing an input image with a model image containing a model to be detected and extracting the model from the input image, the apparatus comprising:

image input means for imaging an outside environment to generate the input image;

feature point extracting means for extracting a feature point from each of the input image and the model image;

feature quantity retention means for extracting and retaining a feature quantity in a neighboring region at the feature point in each of the input image and the model image, the feature quantity being a density gradient direction histogram storing a number of points near the feature point having each of a plurality of gradient directions;

feature quantity comparison means for comparing the feature quantity of each feature point of the input image with the feature quantity of each feature point of the model image and generating a candidate-associated feature point pair having similar feature quantities, each candidate-associated feature point pair including one feature point of the object image and one feature point of the model image; and

a model attitude estimation means for detecting the presence or absence of the model on the input image using the candidate-associated feature point pair and estimating a position and an attitude of the model, if any, wherein the model attitude estimation means repeatedly projects an affine transformation parameter determined from three randomly selected candidate-associated feature point pairs onto a parameter space and finds an affine transformation parameter to determine a position and an attitude of the model based on an affine transformation parameter belonging to a cluster having the largest number of members out of clusters formed on a parameter space.

20. An image recognition apparatus which compares an object image containing a plurality of objects with a model image containing a model to be detected and extracts the model from the object image, the apparatus comprising:

a feature point extracting unit configured to extract a feature point from each of the object image and the model image;

a feature quantity retention unit configured to extract and retain, as a feature quantity, a density gradient direction histogram at least acquired from density gradient information in a neighboring region at the feature point in each of the object image and the model image, the density gradient direction histogram storing a number of points near the feature point having each of a plurality of gradient directions;

a feature quantity comparison unit configured to compare the feature quantity of each feature point of the object image with the feature quantity of each feature point of the model image and generating a candidate-associated feature point pair having similar feature quantities; and

a model attitude estimation unit configured to detect the presence or absence of the model on the object image using the candidate-associated feature point pair and estimating a position and an attitude of the model, if any,

wherein the feature quantity comparison unit itinerantly shifts one of the density gradient direction histograms of feature points to be compared in density gradient direction to find distances between the density gradient direction histograms by sequentially shifting all of the feature points in the one of the density gradient direction histograms one by one to generate a plurality of shifted histograms, and generates the candidate-associated feature point pair by determining a shortest distance between (1) an other of the density gradient direction histograms and (2) the one of the density gradient direction histograms and the shifted histograms.

21. An image recognition apparatus which compares an object image containing a plurality of objects with a model image containing a model to be detected and extracts the model from the object image, the apparatus comprising:

a feature point extracting unit configured to extract a feature point from each of the object image and the model image;

a feature quantity retention unit configured to extract and retain, as a feature quantity, a density gradient direction histogram at least acquired from density gradient information in a neighboring region at the feature point in each of the object image and the model image, the density gradient direction histogram storing a number of points near the feature point having each of a plurality of gradient directions;

a feature quantity comparison unit configured to compare the feature quantity of each feature point of the object image with the feature quantity of each feature point of the model image and to generate a candidate-associated feature point pair having similar feature quantities, each candidate-associated feature point pair including one feature point of the object image and one feature point of the model image, each feature quantity not including gradient magnitude information; and

a model attitude estimation unit configured to detect the presence or absence of the model on the object image using the candidate-associated feature point pair and estimating a position and an attitude of the model, if any,

wherein the model attitude estimation unit is configured to repeatedly project an affine transformation parameter determined from three randomly selected candidate-associated feature point pairs onto a parameter space and to find an affine transformation parameter to determine a position and an attitude of the model based on an affine transformation parameter belonging to a cluster having the largest number of members out of clusters formed on a parameter space.

22. The image recognition apparatus according to claim 21, wherein the feature quantity comparison unit is configured to generate each candidate-associated feature point pair to include one feature point of the object image and one feature point of the model image with a dissimilarity less than a threshold.

23. An image recognition apparatus which compares an object image containing a plurality of objects with a model image containing a model to be detected and extracts the model from the object image, the apparatus comprising:

a feature point extracting unit configured to extract a feature point from each of the object image and the model image;

a feature quantity retention unit configured to extract and retain, as a feature quantity, a density gradient direction histogram at least acquired from density gradient information in a neighboring region at the feature point in each of the object image and the model image, the density gradient direction histogram storing a number of points near the feature point having each of a plurality of gradient directions;

a feature quantity comparison unit configured to compare the feature quantity of each feature point of the object image with the feature quantity of each feature point of the model image and to generate a candidate-associated feature point pair having similar feature quantities, each feature quantity not including gradient magnitude information; and

a model attitude estimation unit configured to detect the presence or absence of the model on the object image using the candidate-associated feature point pair and estimating a position and an attitude of the model, if any,

wherein the model attitude estimation unit is configured to repeatedly project an affine transformation parameter determined from three randomly selected candidate-associated feature point pairs onto a parameter space and to find an affine transformation parameter to determine a position and an attitude of the model based on an affine transformation parameter belonging to a cluster having the largest number of members out of clusters formed on a parameter space, and

wherein the feature quantity comparison unit is configured to generate the dissimilarity for each respective candidate-associated feature point pair by itinerantly shifting by one step the plurality of gradient directions for one of the object image and the model image to compute a number of similarities to a number of the plurality of gradient directions, and to take a minimum dissimilarity to be the dissimilarity.