Image recognizing apparatus and method

• US 7,221,797 B2
• Filed: 04/26/2002
• Issued: 05/22/2007
• Est. Priority Date: 05/02/2001
• Status: Expired due to Fees

• Current Assignee: Honda Giken Kogyo Kabushiki Kaisha (Honda Motor Company)
• Original Assignee: Honda Giken Kogyo Kabushiki Kaisha (Honda Motor Company)
• Inventors: Tsujino, Hiroshi, Koshizen, Takamasa, Akatsuka, Koji
• Primary Examiner(s): Mancuso; Joseph
• Assistant Examiner(s): LIEW, ALEX KOK SOON
• Application Number:
  US10/133,114
• Publication Number:
  US 20030007682A1
• Time in Patent Office: 1,852 Days
• Field of Search: 382/190, 382/153, 382/159, 382/195, 382/181, 382/155, 382/156, 382/157, 382/158, 382/160, 382/161
• US Class Current: 382/190
• CPC Class Codes: G05D 1/0253 extracting relative motion information from a plurality of images taken successively, e.g. visual odometry, optical flow determining position or orientation from images G06T7/70
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

An image recognizing apparatus and method is provided for recognizing behavior of a mobile unit accurately with an image of external environment acquired during the mobile unit is moving.

Behavior command output block 12 outputs behavior commands to cause the mobile unit 32 move. Local feature extraction block 16 extracts features of local areas of the image from the image of external environment acquired on the mobile unit 32 when the behavior command is output. Global feature extraction block 18 extracts feature of global area of the image using the features of local areas. Learning block 20 calculates probability models for recognizing behavior given to the mobile unit 32 based on the feature of global area of the image. After learning is finished, behavior of the mobile unit 32 may be recognized rapidly and accurately by applying the probability models to an image of external environment acquired in mobile unit 32 afresh.

Claims

1. An apparatus for recognizing images, comprising:

a) behavior command output means for outputting behavior commands to cause a mobile unit to move;

b) local feature extraction means for extracting local features from image of external environment captured by said mobile unit when said mobile unit moves according to said behavior command, said local feature extraction means;

b1) capturing two temporally consecutive images,

b2) applying Gabor filters to the two images in plural directions with respect to each of a plurality of local areas produced by segmenting a total area of the image, thereby determining a magnitude of optical flow of said two images in each direction of the Gabor filter, and

b3) extracting local feature data for each of said local areas by determining probability density distribution for the direction having the largest optical flow in each local area;

c) global feature extraction means for extracting feature of global area of said image using said extracted features of said local areas; and

d) learning means for calculating probability models to recognize behavior of the mobile unit based on said extracted feature of said global area.

2. The apparatus according to claim 1, wherein said local feature extraction means extracts said local features from the image by utilizing said optical flow obtained by applying both positive and negative component of Gabor filters on two images.

3. The apparatus according to claim 2, wherein each of said Gabor filters is applied on one of eight different directions respectively.

4. The apparatus according to claim 1, wherein said learning means calculates said probability models utilizing expectation maximization algorithm and supervised learning with the use of neural network.

5. The apparatus according to claim 4, further comprising behavioral recognition means for applying Bayes'"'"' rule with use of said probability model on an image acquired afresh, said behavioral recognition means calculating confidence for each of said behavior commands to recognize behavior of said mobile unit.

6. An apparatus for recognizing images, comprising:

behavior command output means for outputting behavior commands to cause a mobile unit to move;

local feature extraction means for extracting features of local areas within an image of external environment acquired on said mobile unit when said behavior command is output;

global feature extraction means for combining said extracted features of said local areas into global feature of global area of said image with the use of a predetermined function;

learning means for calculating probability models by utilizing expectation maximization algorithm and supervised learning with the use of neural network based on said global feature;

behavioral recognition means for applying Bayes'"'"' rule with use of said probability model on an image acquired afresh and calculating confidence for each of said behavior commands to recognize behavior of said mobile unit;

wherein said behavioral recognition means selects one confidence having largest value among the plurality of confidence;

behavioral assessment means for comparing said selected confidence with a predetermined value to assess said recognized behavior;

wherein said behavioral recognition means recognizing the behavior corresponding to the selected confidence as the behavior of said mobile unit if said confidence is resulted to be larger than the predetermined value;

attention generation means for generating attentional demanding which demands to cause said probability model to be updated if said confidence is resulted to be equal or smaller than the predetermined value; and

attentional modulation means for changing the number of Gaussian mixture in an equation of said learning means for calculating said probability models in response to said attentional demanding;

wherein said learning means recalculates said probability model after said parameter has been changed.

7. A method for recognizing images in computing system, comprising:

a) outputting behavior commands to cause a mobile unit to move;

b) extracting local features from images of external environment captured by said mobile unit when said mobile unit moves according to said behavior command, comprising;

b1) capturing two temporally consecutive images;

b2) applying Gabor filters to the two images in plural directions with respect to each of a plurality of local areas produced by segmenting a total area of the image, thereby determining a magnitude of optical flow of said two images in each direction of the Gabor filter, and

b3) extracting local feature data for each of said local areas by determining probability density distribution for the direction having the largest optical flow in each local area;

c) extracting feature of global area of said image using said extracted features of said local areas; and

d) calculating probability models to recognize behavior of the mobile unit based on said extracted feature of said global area.

8. The method according to claim 7, wherein said step of extracting local features from images uses said optical flow obtained by applying both positive and negative components of the Gabor filter to the two images.

9. The method according to claim 8, wherein each of said Gabor filters is applied on one of eight different directions respectively.

10. The method according to claim 7, wherein said step of calculating the probability models includes utilizing expectation maximization algorithm and supervised learning with the use of neural network.

11. The method according to claim 10, further comprising:

applying Bayes'"'"' rule with use of said probability models on an image acquired afresh; and

calculating confidence for each of said behavior commands to recognize behavior of said mobile unit.

12. A method for recognizing images in computing system, comprising:

outputting behavior commands to cause a mobile unit to move;

extracting features of local areas within an image of external environment acquired on said mobile unit when said behavior command is output;

combining said extracted features of said local areas into global feature of global area of said image with the use of a predetermined function;

calculating probability models by utilizing expectation maximization algorithm and supervised learning with the use of neural network based on said global feature;

applying Bayes'"'"' rule with use of said probability model on an image acquired afresh and calculating confidence for each of said behavior commands to recognize behavior of said mobile unit;

selecting one confidence having largest value among the plurality of confidence for said comparing step;

comparing said selected confidence with a predetermined value to assess said recognized behavior;

recognizing the behavior corresponding to the selected confidence as the behavior of said mobile unit if said confidence is resulted to be larger than the predetermined value;

generating attentional demanding which demands to cause said probability model to be updated if said confidence is resulted to be equal or smaller than the predetermined value; and

changing the number of Gaussian mixture in an equation of said calculating step for calculating said probability models in response to said attentional demanding;

wherein said probability model is recalculated after said parameter has been changed.

13. An apparatus for recognizing images, comprising:

a) behavior command output means for outputting behavior commands to cause a mobile unit to move;

b) local feature extraction means for extracting local features from image of external environment captured by said mobile unit when said mobile unit moves according to said behavior command, said local feature extraction means;

b1) capturing two temporally consecutive images,

b2) applying Gabor filters to the two images in plural directions with respect to each of a plurality of local areas produced by segmenting a total area of the image, thereby determining image intensity in each direction of the Gabor filter, and

b3) extracting local feature data for each of said local areas by determining probability density distribution for the direction having the largest image intensity in each local area;

c) global feature extraction means for extracting feature of global area of said image using said extracted features of said local areas; and

d) learning means for calculating probability models utilizing expectation maximization algorithm and supervised learning with the use of neural network to recognize behavior of the mobile unit based on said extracted feature of said global area;

e) behavioral recognition means for applying Bayes'"'"' rule with use of said probability model on an image acquired afresh, said behavioral recognition means calculating confidence for each of said behavior commands to recognize behavior of said mobile unit;

f) behavioral assessment means for comparing said confidence with a predetermined value to assess said recognized behavior;

g) attention generation means for generating attentional demanding which demands to cause said probability models to be updated based on the result of said assessment; and

h) attentional modulation means for changing specified parameter of said learning means in response to said attentional demanding;

wherein said learning means recalculates said probability model after said parameter has been changed.

14. The apparatus according to claim 13, wherein said behavioral recognition means selects one confidence having largest value among the plurality of confidences and supplies it to said behavioral assessment means, said behavioral recognition means recognizing the behavior corresponding to the selected confidence as the behavior of said mobile unit if said confidence is resulted to be larger than the predetermined value by said comparing by said behavioral assessment means.

15. The apparatus according to claim 14, wherein said parameter is the number of Gaussian mixture in an equation of said learning means for calculating said probability models.

16. A method for recognizing images in computing system, comprising:

a)outputting behavior commands to cause a mobile unit to move;

b)extracting local features from images of external environment captured by said mobile unit when said mobile unit moves according to said behavior command, comprising;

b1) capturing two temporally consecutive images;

b2) applying Gabor filters to the two images in plural directions with respect to each of a plurality of local areas produced by segmenting a total area of the image, thereby determining image intensity in each direction of the Gabor filter, and

b3) extracting local feature data for each of said local areas by determining probability density distribution for the direction having the largest image intensity in each local area;

c) extracting feature of global area of said image using said extracted features of said local areas;

d) calculating probability models utilizing expectation maximization algorithm and supervised learning with the use of neural network to recognize behavior of the mobile unit based on said extracted feature of said global area;

applying Bayes'"'"' rule with use of said probability models on an image acquired afresh;

calculating confidence for each of said behavior commands to recognize behavior of said mobile unit;

comparing said confidence with a predetermined value to assess said recognized behavior;

generating attentional demanding which demands to cause said probability models to be updated based on the result of said assessment; and

changing specified parameter of said learning step in response to said attentional demanding;

wherein said probability model is recalculated after said parameter has been changed.

17. The apparatus according to claim 16, further comprising:

subsequent to said step of calculating confidence, selecting one confidence having largest value among the plurality of confidences for said comparing step,

recognizing the behavior corresponding to the selected confidence as the behavior of said mobile unit if said confidence is resulted to be larger than the predetermined value in said comparing step.

18. The method according to claim 17, wherein said parameter is the number of Gaussian mixture in an equation of said calculating step for calculating said probability models.