Image processing apparatus and image processing method

• US 6,072,903 A
• Filed: 12/31/1997
• Issued: 06/06/2000
• Est. Priority Date: 01/07/1997
• Status: Expired due to Fees

• Current Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Original Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Inventors: Maki, Atsuto, Matsuda, Natsuko, Wiles, Charles, Watanabe, Mutsumi
• Primary Examiner(s): Boudreau, Leo H.
• Assistant Examiner(s): Mariam, Daniel G.
• Application Number:
  US09/001,465
• Time in Patent Office: 888 Days
• Field of Search: 382/153, 382/154, 382/190, 382/201, 382/203, 382/103, 382/169, 348/48, 348/50, 348/77
• US Class Current: 382/190
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/251 involving models

Abstract

A moving picture processing apparatus comprises an image capture unit for taking in time-series images (moving picture series) of a target object, a feature point extraction unit for extracting feature points constituting the surface of the object whose model is to be made, a three-dimensional position information extraction unit for finding three-dimensional position information on the feature points by correlating the feature points contained in the image at a point in time with those at another point in time in the time-series images and analyzing positional coordinate information on the correlated feature points, a three-dimensional patch group generating unit for generating three-dimensional patch groups constituting the surface of the target object on the basis of the feature points for which the three-dimensional position information has been found and supplying the three-dimensional patch groups as information on a model of the target object to an output section.

Claims

1. An image processing apparatus for acquiring a shape of a target object contained in a series of moving pictures, comprising:

an image capture unit configured to take in time-series images obtained in time series by shooting a target object;

a feature point extraction unit configured to extract feature points of said target object from said time-series images taken in by said image capture unit;

a motion information extraction unit configured to determine a change of a position and posture of said target object in each point of time in said time-series images by correlating feature points contained in the image at a point of time with those at another point of time in said time-series images extracted by said feature point extraction unit to obtain correlated feature points, and configured to analyze positional coordinate information on the correlated feature points;

an inter-image linear combination coefficient calculating unit configured to calculate a linear combination coefficient of image luminance between said time-series images by analyzing information on luminance at the feature points correlated by said motion information extraction unit and luminance around the correlated feature points; and

a distance information detecting unit configured to estimate information on a distance to each point of said target from the position and posture of said target object determined by said motion information extraction unit at each point of time in said time-series images and the linear combination coefficient of image luminance between said time-series images, the linear combination coefficient being calculated by said inter-image linear combination coefficient calculating unit.

2. An image processing apparatus according to claim 1, wherein said distance information detecting unit includes:

a section configured to choose as a reference image an image at a specific time A from said time-series images, configured to set a distance Z of each pixel of the reference image to said target object, and configured to determine pixels corresponding to said each pixel in images at three or more points of time different from said specific time A from said distance Z and the position and posture of said target object determined by said motion information extraction unit at each point of time;

a matching degree calculating section configured to calculate a degree of matching of the luminance at the corresponding pixels with luminance at the pixel in said reference image using the linear combination coefficient calculated by said inter-image linear combination coefficient calculating unit; and

a section configured to evaluate the distance Z estimated under a geometric condition and an optical condition according to the degree of matching found by said matching degree calculating section and configured to estimate the distance information on the basis of the evaluation.

5. The image processing apparatus according to claim 1, wherein said inter-image linear combination coefficient calculating unit comprises:

a feature point correlation unit configured to correlate isolated feature points and connected feature points between the time-series images, the isolated feature points and the connected feature points being extracted by said feature point extraction unit;

an evaluation function calculator configured to calculate a specific evaluation function on the basis of a square error of the luminance synthesized from a basic image and the linear combination coefficient and the actual luminance; and

a distance decision section configured to calculate an evaluation function while changing the distance and configured to determine a distance which makes the evaluation function the smallest to be the distance at that point.

6. The image processing apparatus according to claim 1, wherein said inter-image linear combination coefficient calculating unit includes a section configured to determine a linear combination coefficient between images by modifying a relation including as a coefficient matrix, a matrix generated from information on the luminance of each of the correlated feature points.

7. The image processing apparatus according to claim 1, wherein said feature point extraction unit comprises:

an isolated feature point extracting section configured to extract isolated feature points;

a connected feature point extracting section configured to extract connected feature points;

a filtering processor configured to perform a filtering process on time-series image data representing the time-series images;

a two-dimensional spatial derivative processor configured to emphasize brightness of the isolated feature points by performing a two-dimensional spatial derivative process on the time-series image data filtered by said filtering processor; and

an isolated feature point pair extraction section configured to extract the isolated feature points in the images by extracting the portions for which the result of emphasizing the brightness is larger than a suitably set threshold value.

8. The image processing apparatus according to claim 7, wherein said isolated feature point extracting section includes:

a local mask setting section configured to set a local mask area used to extract consecutive feature points to a filtering result obtained by said filtering processor; and

a direction-based variance calculator configured to compute a variance value by direction in each of the local masks set by said local mask setting section and a variance value in each direction using brightness values of the consecutive pixels in each direction in the local masks to output the variance value in each direction to said connected feature point extracting section.

9. The image processing apparatus according to claim 8, wherein said connected feature point extracted section extracts, as a connected feature point, a central point of the local masks where the direction-based variance value obtained at the direction-based variance calculator is larger than a specific value in two or more directions.

10. The image processing apparatus according to claim 8, wherein said direction-based variance calculator computes the variance value in each of a vertical direction, a horizontal direction, a direction at 45 degrees to a right, and a direction at 45 degrees to a left, using brightness values of the consecutive pixels in each direction in the local masks.

11. The image processing apparatus according to claim 1, wherein said motion information extraction unit includes:

a feature point correction section configured to correlate isolated feature points and connected feature points extracted at the feature point extraction unit with each other between the consecutive time-series images.

12. The image processing apparatus according to claim 11, wherein said feature point correlation section includes:

a feature point pair select section configured to select a set of feature points to be correlated between the consecutive time-series images;

a local mask setting section configured to set a local mask area including feature points in each image;

a correlation coefficient calculator configured to find a correlation coefficient between the local mask areas;

a correspondence decision section configured to perform a process of determining a set of feature points whose correlation coefficient is larger than a specific threshold value and the largest to be the one already correlated with each other and configured to store the set in a storage section; and

a correspondence decision section configured to perform the process of determining for each of the extracted feature points.

3. An image processing method for acquiring a shape of a target object contained in a series of moving pictures, comprising the steps of:

taking in images obtained in time-series by shooting a target object;

extracting feature points of said target object from said time-series images taken in at the image taking-in step;

determining a change of position and posture of said target object in each point in time in said time-series images by correlating the feature points contained in the image at a point of time with those at another point of time in said time-series images extracted in said feature point extracting step to obtain correlated feature points and analyzing positional coordinate information on the correlated feature points;

calculating a linear combination coefficient of image luminance between said time-series images by analyzing information on luminance at the correlated feature points and luminance around the correlated feature points; and

estimating information on a distance to each point of said determined target object from the position and posture of said target object at each point of time in said time-series images and the linear combination coefficient of image luminance between said calculated time-series images.

4. An image processing method according to claim 3, wherein said step of estimating distance information on said target object includes:

choosing as a reference image the image at a specific time A from said time-series images;

setting a distance Z of each pixel of the reference image to said target object in the reference image, evaluating the distance Z under a geometric condition and an optical condition; and

determining pixels corresponding to the each pixel in the images at three or more points in time different from said specific time A from said distance Z and the position and posture at each point in time of said determined target object;

calculating a degree of matching of the luminance at the corresponding pixels with the luminance at the each pixel in said reference image using the linear combination coefficient between said calculated time-series images; and

evaluating the distance Z estimated under a geometric condition and an optical condition according to the degree of matching and estimating the distance information on the basis of evaluation.

13. The image processing method according to claim 3, wherein said step of calculating a linear combination coefficient includes:

correlating isolated feature points and connected feature points between the time-series images, the isolated feature points and the connected feature points being extracted by said feature point extracting step;

calculating a specific evaluation function on the basis of a square error of the luminance synthesized from a basic image and the linear combination coefficient and actual luminance; and

calculating an evaluation function while changing the distance and configured to determine a distance which makes the evaluation function the smallest to be the distance at that point.

14. The image processing method according to claim 3, wherein said step of calculating a linear combination coefficient includes:

determining a linear combination coefficient between image by modifying a relation including as a coefficient matrix a matrix generated from information on the luminance of each of the correlated feature points.

15. The image processing method according to claim 3, wherein said step of extracting feature points comprises:

extracting isolated feature points;

extracting connected feature points;

performing a filtering process on time-series image data representing the time-series images;

emphasizing brightness of the isolated feature points by performing a two-dimensional spatial derivative process on the time-series image data filtered by said filtering step; and

extracting the isolated feature points in the images by extracting the portions for which the result of emphasizing the brightness is larger than a suitably set threshold value.

16. The image processing method according to claim 15, wherein said step of extracting isolated feature points includes:

setting a local mask area used to extract consecutive feature points to a filtering result obtained by said filtering step; and

computing a variance value by direction in each of the local masks and a variance value in each direction using brightness values of the consecutive pixels in each direction in the local masks to output the variance value in each direction.

17. The image processing method according to claim 16, wherein said step of extracting connected feature points includes:

extracting, as a connected feature point, a central point of the local masks where the direction-based variance value obtained at the direction-based variance calculator is larger than a specific value in two or more directions.

18. The image processing method according to claim 16, wherein said step of computing a variance value includes:

computing the variance value in each of a vertical direction, a horizontal direction, a direction at 45 degrees to a right, and a direction at 45 degrees to a left, using brightness values of the consecutive pixels in each direction in the local masks.

19. The image processing method according to claim 3, wherein said step of determining the change of position and posture includes:

correlating the isolated feature points and the connected feature points extracted at the feature point extraction unit with each other between the consecutive time-series images.

20. The image processing method according to claim 19, wherein said step of correlating the isolated feature points and the connected feature points includes:

selecting a set of feature points to be correlated between the consecutive time-series images;

setting a local mask area including feature points in each image;

finding a correlation coefficient between the local mask areas;

performing a process of determining a set of feature points whose correlation coefficient is larger than a specific threshold value and the largest to be the one already correlated with each other and storing the set in a storage section; and

performing the process of determining for each of the extracted feature points.