ROBOT AND METHOD FOR RECOGNIZING HUMAN FACES AND GESTURES THEREOF

• US 20110158476A1
• Filed: 07/01/2010
• Published: 06/30/2011
• Est. Priority Date: 12/24/2009
• Status: Abandoned Application

• Current Assignee: National Taiwan University of Science and Technology
• Original Assignee: National Taiwan University of Science and Technology
• Inventors: Wang, Chih-Hsin, Chu, Keng-Yu, Fahn, Chin-Shyurng
• Application Number:
  US12/829,370
• Publication Number:
  US 20110158476A1
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: B25J 11/0005 Manipulators having means for high-level communication with users, e.g. speech generator, face recognition means
• CPC Class Codes: B25J 13/08 by means of sensing devices, e.g. viewing or touching devices
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A robot and a method for recognizing human faces and gestures are provided, and the method is applicable to a robot. In the method, a plurality of face regions within an image sequence captured by the robot are processed by a first classifier, so as to locate a current position of a specific user from the face regions. Changes of the current position of the specific user are tracked to move the robot accordingly. While the current position of the specific user is tracked, a gesture feature of the specific user is extracted by analyzing the image sequence. An operating instruction corresponding to the gesture feature is recognized by processing the gesture feature through a second classifier, and the robot is controlled to execute a relevant action according to the operating instruction.

Claims

1. A method for recognizing human faces and gestures that are suitable for recognizing movement of a specific user to operate a robot, the method comprising:

processing a plurality of face regions within an image sequence captured by the robot through a first classifier, so as to locate a current position of the specific user according to the face regions, the image sequence comprising a plurality of images;

tracking the changes of the current position of the specific user and moving the robot based on the current position of the specific user, such that the specific user can be constantly appear in the image sequence continuously captured by the robot;

analyzing the image sequence to extract a gesture feature of the specific user;

processing the gesture feature through a second classifier to recognize an operating instruction corresponding to the gesture feature; and

controlling the robot to execute an action based on the operating instruction.

2. The method as claimed in claim 1, the step of processing the face regions through the first classifier to locate the current position of the specific user comprising:

detecting the face regions in each of the images of the image sequence through the first classifier;

recognizing each of the face regions to authenticate a corresponding user identity;

extracting a specific face region from all of the face regions, wherein the corresponding user identity of the specific face region is consistent with the specific user; and

indicating the current position of the specific user based on the positions of the specific face region in an image containing the specific face region.

3. The method as claimed in claim 2, wherein the first classifier is a hierarchical classifier constructed based on a plurality of Haar-like features of individual training samples, and the step of detecting the face regions in each of the images of the image sequence comprises:

dividing each of the images into a plurality of blocks based on an image pyramid rule;

detecting each of the blocks by a detection window to extract a plurality of block features of each of the blocks; and

processing the block features through the first classifier to detect the face regions from the blocks.

4. The method as claimed in claim 3, wherein each of the training samples is corresponding to a sample feature parameter calculated based on the Haar-like features of the individual training samples, and the step of recognizing each of the face regions to authenticate the corresponding user identity comprises:

extracting the Haar-like features of each of the face regions to calculate a region feature parameter respectively corresponding to each of the face regions; and

calculating a Euclidean distance between the region feature parameter and the sample feature parameter of each of the training samples, so as to recognize each of the face regions and authenticate the corresponding user identity based on the Euclidean distance.

5. The method as claimed in claim 1, the step of tracking the change of the current position of the specific user comprising:

a. defining a plurality of sampling points adjacent to the current position;

b. respectively calculating a probability of the specific user moving from the current position to each of the sampling points;

c. acquiring one of the sampling points with a highest probability as a local current position;

d. defining a plurality of second-stage sampling points, wherein a distance between each of the second-stage sampling points and the local current position does not exceed a predetermined value;

e. respectively calculating a probability of the specific user moving from the current position to each of the second-stage sampling points;

f. if the probability corresponding to one of the second-stage sampling points is greater than the probability corresponding to the local current position, setting one of the second-stage sampling points as the local current position and repeating the step d to the step f; and

g. if the probability corresponding to the local current position is greater than the probability of each of the second-stage sampling points, then determining that the specific user is going to move to the local current position, and the local current position is regarded as a latest current position and repeating step a to step g to continuously track the changes of the current position of the specific user.

6. The method as claimed in claim 1, wherein before the step of analyzing the image sequence to extract the gesture feature of the specific user, further comprising:

detecting a plurality of skin tone regions in addition to the face regions;

determining a plurality of local maximum circles exactly covering the skin tone regions, respectively; and

determining one of the skin tone regions as a hand region based on a radius of each of the local maximum circles corresponding to the skin tone regions.

7. The method as claimed in claim 6, the step of analyzing the image sequence to extract the gesture feature of the specific user comprising:

calculating a moving distance and a moving angle of the hand region in the images and determining the moving distance and the moving angle as the gesture feature based on a position of the hand region in each of the images of the image sequence.

8. The method as claimed in claim 1, wherein the second classifier is a hidden Markov model (HMM) classifier constructed based on a plurality of training track samples.

9. A robot comprising:

an image extraction apparatus;

a marching apparatus; and

a processing module coupled to the image extraction apparatus and the marching apparatus,

wherein the processing module processes a plurality of face regions within an image sequence captured by the image extraction apparatus through a first classifier, locates a current position of a specific user from the face regions, tracks changes of the current position of the specific user, and controls the marching apparatus to move the robot based on the current position of the specific user so as to ensure that the specific user constantly appears in the image sequence continuously captured by the image extraction apparatus, the image sequence comprising a plurality of images,

the processing module analyses the image sequence to extract a gesture feature of the specific user and processing the gesture feature through a second classifier to recognize an operating instruction corresponding to the gesture feature, and controls the robot to execute an action according to the operating instruction.

10. The robot as claimed in claim 9, wherein the processing module detects the face regions in each of the images of the image sequence through the first classifier, recognizes each of the face regions to authenticate a corresponding user identity, extracts a specific face region from all of the face regions, in which the corresponding user identity of the specific face region is consistent with the specific user, and indicates the current position of the specific user based on positions of the specific face region in an image containing the specific face region.

11. The robot as claimed in claim 10, wherein the first classifier is a hierarchical classifier constructed based on a plurality of Haar-like features of individual training samples, and the processing module divides each of the images into a plurality of blocks based on an image pyramid rule, detects each of the blocks by a detection window to extract a plurality of block features of each of the blocks, and processes the block features of each of the blocks through the first classifier to detect the face regions from the blocks.

12. The robot as claimed in claim 10, wherein each of the training samples is corresponding to a sample feature parameter calculated based on the Haar-like features of the individual training samples, and the processing module extracts the Haar-like features of the face regions to calculate a region feature parameter corresponding to each of the face regions respectively and calculates a Euclidean distance between the region feature parameter and the sample feature parameter of each of the training samples so as to recognize each of the face regions and authenticate the corresponding user identity based on the Euclidean distance.

13. The robot as claimed in claim 9, the processing module defining a plurality of sampling points adjacent to the current position, respectively calculating a probability of the specific user moving from the current position to each of the sampling points, and acquiring one of the sampling points with a highest probability as a local current position,

the processing module defining a plurality of second-stage sampling points that are not more far away from the local current position than a predetermined value and respectively calculating a probability of the specific user moving from the current position to each of the second-stage sampling points,

if the probability corresponding to one of the second-stage sampling points is greater than the probability corresponding to the local current position, the processing module regards one of the second-stage sampling points as the local current position and repeatedly defines the second-stage sampling points and calculates the probability of the specific user moving from the current position to each of the second-stage sampling points,

if the probability corresponding to the local current position is greater than the robability of each of the second-stage sampling points, the specific user is determined to move to the local current position by the processing module, and the local current position is regarded as a latest current position,

the processing module repeating above procedure to continuously track the change of the current position of the specific user.

14. The robot as claimed in claim 9, wherein the processing module detects a plurality of skin tone regions in addition to the face regions, respectively determines a plurality of local maximum circles exactly covering each of the skin tone regions, and determines one of the skin tone regions as a hand region based on a radius of each of the local maximum circles corresponding to the skin tone regions.

15. The robot as claimed in claim 14, wherein the processing module calculates a moving distance and a moving angle of the hand region in the images and regards the moving distance and the moving angle as the gesture feature based on a position of the hand region in each of the images of the image sequence.

16. The robot as claimed in claim 9, wherein the second classifier is a hidden Markov model classifier constructed based on a plurality of training track samples.