Sign based human-machine interaction

• US 20050271279A1
• Filed: 05/12/2005
• Published: 12/08/2005
• Est. Priority Date: 05/14/2004
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company), The Ohio State University Research Foundation
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Fujimura, Kikuo, Liu, Xia
• Granted Patent: US 7,308,112 B2
• Time in Patent Office: Days
• US Class Current: 382/203
• CPC Class Codes: E05F 15/00 Power-operated mechanisms for wings motor-operated accessories in locks for completing closing or initiating opening of a wing E05B17/00
• CPC Class Codes: E05Y 2400/852 Sensors
• CPC Class Codes: E05Y 2400/86 by hand
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Communication is an important issue in man-to-robot interaction. Signs can be used to interact with machines by providing user instructions or commands. Embodiment of the present invention include human detection, human body parts detection, hand shape analysis, trajectory analysis, orientation determination, gesture matching, and the like. Many types of shapes and gestures are recognized in a non-intrusive manner based on computer vision. A number of applications become feasible by this sign-understanding technology, including remote control of home devices, mouse-less (and touch-less) operation of computer consoles, gaming, and man-robot communication to give instructions among others. Active sensing hardware is used to capture a stream of depth images at a video rate, which is consequently analyzed for information extraction.

Claims

1. A computer based method for man-machine interaction comprising the steps of:

receiving a digital image of a target, the digital image comprising depth data associated with image pixels, the depth data corresponding to the distance to the camera;

determining a location of a hand of the target within the pixels of the image based on the depth data associated with the pixels; and

matching a candidate shape of the hand of the target captured in the digital image with a stored shape image from a plurality of shape images by comparing one or more of the plurality of stored shape images with the candidate shape.

2. The method of claim 1, wherein matching the candidate shape of the hand further comprises:

generating a skeleton representation of the hand from the image pixel data of the location of the hand; and

determining one or more hand attributes from the skeleton representation, including at least a number of fingers,

wherein the one or more of the plurality of stored shape images for comparing with the candidate shape are filtered based at least in part on the number of fingers.

3. The method of claim 1, wherein matching the candidate shape of the hand further comprises:

determining an edge of the hand in the determined location of the image;

applying a pixel distance transform to the image with respect to the edge of the hand to compose a candidate edge image,

wherein the one or more of the plurality of stored shape images include edge pixel distance transforms of the images for comparing with the candidate shape based on a projection of the candidate shape image over the stored shape images to determine a similarity value based on the correlation between the candidate edge image and the distance transforms of the stored images.

4. The method of claim 1, wherein determining a location of a hand of the target further comprises:

dividing the digital image in a plurality of predetermined regions;

associating a predetermined region with a location of the hand; and

determining one or more regions associated with at least one of a location of the target'"'"'s head or a location of the target'"'"'s torso,

wherein the one or more of the plurality of stored shape images for comparing with the candidate shape include region information defining hand locations associated with the shape.

5. A computer based method for man-machine interaction comprising the steps of:

receiving a digital image of a target;

pre-processing the digital image to determine a location of at least a hand and at least one of a head or a torso of the target;

matching a shape of the hand with one of a plurality of stored hand shape images to provide a matched shape associated with the digital image;

creating a candidate image data object including information associated with the location of a hand with respect to one of a head or a torso of the target and information indicating the matched shape associated with the digital image; and

matching a gesture captured by one or more digital images with a template gesture by comparing the candidate image data object with one ore more stored gesture profiles for template gestures, the gesture profiles including hand to body part location information and hand shape information for comparing with the candidate image data object.

6. The method of claim 5, wherein pre-processing includes a training process comprising:

performing a principal component analysis of a training set of images with known poses of the target to derive a derive a projection matrix; and

determining a pose of the target in the digital image by projecting a vector corresponding to a candidate pose of the target based on the projection matrix and finding a nearest neighbor in the training set with a known pose.

7. The method of claim 5, wherein the candidate image data object further comprises a palm information section and a finger information section, the palm information section comprising shape information, orientation information, and trajectory information, the finger information section including number of fingers and direction of the fingers.

8. The method of claim 5 wherein the digital image of the target comprises hybrid depth-image information.

9. The method of claim 8, wherein pre-processing includes analyzing the hybrid depth-image information to determine location based on distance of the target to an image capturing system.

10. The method of claim 5, wherein matching the gesture captured by one or more digital images with a template gesture further comprises comparing values associated with one or more attributes from the group consisting of number of hands used, number of fingers used, distance between fingers, hand location, hand location variation, shape of hand information, and finger information.

11. The method of 10, wherein the shape of hand information further comprises one or more attributes from the group consisting of number of fingers, finger identifiers, finger shapes, and finger orientations.

12. A computer based method for human-machine interaction comprising:

matching a hand shape captured in one or more digital images with a plurality of hand shape patterns to determine a candidate gesture hand shape;

determining a trajectory curve of the hand from the one ore more digital images;

matching the trajectory curve of the hand with a plurality of trajectory curve templates to determine a candidate gesture motion; and

determining a gesture corresponding to the candidate gesture hand shape and the candidate gesture motion by comparing the candidate gesture hand shape and the candidate gesture motion with a plurality of gesture profiles with associated gesture hand shapes and gesture motions.

13. The method of claim 12, further comprising:

determining whether the hand shape captured in the one or more digital images is still; and

in response to determining the hand shape to be still matching the hand shape with a plurality of hand shape patterns corresponding to still gestures to determine the gesture.

14. A system for human-machine interaction comprising:

an image data pre-processing module for receiving frames of image data comprising human image information, the image pre-processing module configured to determine locations of one or more body parts of the human in the frames;

a shape matching module coupled to the image data pre-processing module for receiving the determined locations and configured to match a shape of at least one of the body parts by comparing information associated with the image data with stored shape profiles in a shape database; and

a gesture matching module coupled to the image data-preprocessing module and the shape matching module for receiving shape and location information, the gesture matching module configured to match a gesture of the at least one body part by comparing gesture attributes of the at least one body part with gesture profiles stored in a gesture database, the gesture attributes comprising values associated with body part locations, and shape information.

15. The computer based system of claim 14, further comprising an image data capturing system, the image data capturing system configured to capture hybrid image-depth information of a scene within a range of depths from the image data capturing system.

16. The computer based system of claim 15, wherein the image data capturing system comprises a time-of-flight camera configured to capture the hybrid image-depth information along a single optical axis.

17. The computer based system of claim 14, further comprising a data storage system coupled to the shape matching module and the gesture matching module, the data storage system comprising the shape database and the gesture database.

18. The computer based system of claim 14, wherein the coupling between the image data pre-processing module, the shape matching module, and the gesture matching module comprises a random access memory device.

19. A computer readable media for human-machine interaction comprising:

program instructions for receiving a digital image of a target;

program instructions for pre-processing the digital image to determine a location of at least a hand and at least one of a head or a torso of the target;

program instructions for matching a shape of the hand with one of a plurality of stored hand shape images to provide a matched shape associated with the digital image;

program instructions for creating a candidate image data object including information associated with the location of a hand with respect to one of a head or a torso of the target and information indicating the matched shape associated with the digital image; and

program instructions for matching a gesture captured by one or more digital images with a template gesture by comparing the candidate image data object with one ore more stored gesture profiles for template gestures, the gesture profiles including hand to body part location information and hand shape information for comparing with the candidate image data object.

20. A computer based system for human-machine interaction comprising:

means for receiving a digital image of a target;

means for pre-processing the digital image to determine a location of at least a hand and at least one of a head or a torso of the target;

means for matching a shape of the hand with one of a plurality of stored hand shape images to provide a matched shape associated with the digital image;

means for creating a candidate image data object including information associated with the location of a hand with respect to one of a head or a torso of the target and information indicating the matched shape associated with the digital image; and

means for matching a gesture captured by one or more digital images with a template gesture by comparing the candidate image data object with one ore more stored gesture profiles for the template gestures, the gesture profiles including hand to body part location information and hand shape information for comparing with the candidate image data object.