Apparatus and method for detecting hands of subject in real time

• US 20100329511A1
• Filed: 06/25/2010
• Published: 12/30/2010
• Est. Priority Date: 06/25/2009
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd., Sungkyunkwan University Foundation For Corporate Collaboration (SungKyunKwan University)
• Original Assignee: Samsung Electronics Co. Ltd., Sungkyunkwan University Foundation For Corporate Collaboration (SungKyunKwan University)
• Inventors: Jin, Seung-Hun, Song, Tae-Houn, Pham, Cong-Thien, Jeon, Jae-Wook, Nguyen, Duc-Dung, Park, Young-Hee, Shim, Hyun-Sik, Yoon, Je-Han, Song, Hee-Jun
• Granted Patent: US 8,588,467 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06V 40/11 Hand-related biometrics; Hand pose recognition

Abstract

An apparatus and method can effectively detect both hands and hand shape of a user from images input through cameras. A skin image detecting skin regions from one of the input images and a stereoscopic distance image are used. For hand detection, background and noise are eliminated from a combined image of the skin image and the distance image and regions corresponding to actual both hands are detected from effective images having a high probability of hands. For hand shape detection, a non-skin region is eliminated from the skin image based on the stereoscopic distance information, hand shape candidate regions are detected from the remaining region after elimination, and finally a hand shape is determined.

Claims

1. An apparatus capable of detecting hands of a subject in real time, the apparatus comprising:

a first image input part configured to generate a first image photographed at a first side with respect to one subject;

a second image input part configured to generate a second image photographed at second side with respect to the subject;

a stereoscopic processor configured to generate a distance image indicating 3-dimensional distance information by comparison information of the first image and the second image;

a skin region detector configured to determine whether each pixel within any one of the first and second images is a skin pixel and generate a skin image indicating whether each pixel within any one of the first and second images is a skin pixel;

a skin/distance pixel combiner configured to generate a combined image by combining the distance image with the skin image in units of pixels;

a background and noise eliminator configured to generate an effective image including two or more hand candidate regions by eliminating background and noise from the combined image; and

a hand correlation determiner configured to determine calculating effective values with respect to respective pairs of the hand candidate regions which can be combined within the effective image and determine a pair of hand candidate regions having the largest effective values as hand regions.

2. The apparatus of claim 1, wherein the skin/distance pixel combiner includes a skin pixel buffer configured to compensate for a difference in processing time between the skin region detector and the stereoscopic processor.

3. The apparatus of claim 2, wherein the skin/distance pixel combiner is configured to combine a skin image pixel and a distance image pixel which have the same location on an image of an original pixel using the skin pixel buffer.

4. The apparatus of claim 1, wherein the background and noise eliminator is configured to eliminate background from the combined image based on the distance image and eliminate a skin region having a size less than a predetermined size as noise.

5. The apparatus of claim 1, wherein the hand correlation determiner is configured to calculate and then adds size priorities, size differences, distance priorities and location differences with respect to the respective pairs of the hand candidate regions which can be combined and determines a pair of hand candidate regions having the largest effective values among the added effective values as hand regions.

6. The apparatus of claim 1, further comprising:

a hand candidate region detector configured to generate a hand candidate image from which hand candidate regions are detected;

an erosion image generator configured to generate an erosion image by repeatedly performing erosion operations considering an expected thickness of a finger with respect to the hand candidate image;

a hand center operator configured to detect the center of a hand using a phase relationship from the erosion image;

a dilation image generator configured to generate a dilation image by performing a dilation operation as many times as the repeated number of the erosion operations;

a subtraction image generator configured to generate a subtraction image by subtracting the dilation image from the hand candidate image; and

a hand shape determiner configured to determine a hand shape using a size of a region remaining in the subtraction image.

7. The apparatus of claim 6, wherein the hand candidate image is an image in which background and noise are eliminated from the skin image based on the distance image.

8. The apparatus of claim 6, wherein the erosion image generator is configured to determine the number of erosion operations which is to be repeatedly performed to eliminate pixels corresponding to an expected thickness of a finger within the hand candidate image, with reference to a table which defines the number of erosion operations according to a distance value of a hand.

9. The apparatus of claim 6, wherein the hand shape determiner is configured to compare the size of the region remaining in the subtraction image with a predefined size of a finger region and determines a comparison result as one of a hand shape in which a finger is spread and a hand shape in which a finger is bent.

10. A method for detecting hands of a subject in real time, comprising:

determining whether pixels within any of a first image and a second image correspond to left and right images of a subject in a photograph are skin pixels;

generating a skin image indicating whether each pixel within any of the first and second images is a skin pixel;

generating a distance image indicating 3-dimensional distance information by comparison information of the first and second images;

generating a combined image by combining the distance image with the skin image in units of pixels;

generating an effective image including at least two hand candidate regions by eliminating background and noise from the combined image; and

calculating effective values with respect to respective pairs of the hand candidate regions which can be combined within the effective image and determining a pair of hand candidate regions having the largest effective values as hand regions.

11. The method of claim 10, wherein the combination combines a skin image pixel and a distance image pixel which have the same location on an image of an original pixel using a skin pixel buffer for compensating for a difference in processing time between the distance image and the skin image.

12. The method of claim 10, wherein the generation of the effective image eliminates background from the combined image based on the distance image and eliminates a skin region having a size less than a predetermined size as noise.

13. The method of claim 10, wherein the determination of the hand regions includes:

calculating and then adds size priorities, size differences, distance priorities and location differences with respect to respective pairs of the hand candidate regions which can be combined; and

determining a pair of hand candidate region having the largest effective values among the added effective values as hand regions.

14. The method of claim 10, further comprising:

generating a hand candidate image from which the hand candidate regions are detected;

generating an erosion image by repeatedly performing erosion operations considering an expected thickness of a finger with respect to the hand candidate image;

detecting the center of a hand using a phase relationship from the erosion image;

generating a dilation image by performing dilation operations with respect to the erosion image as many times as the repeated number of the erosion operations;

generating a subtraction image by subtracting the dilation image from the hand candidate image; and

determining a hand shape using a size of a region remaining in the subtraction image.

15. The method of claim 14, wherein the hand candidate image is an image in which background and noise are eliminated from the skin image based on the distance image.

16. The method of claim 14, wherein the generation of the erosion image determines the number of erosion operations which is to be repeatedly performed to eliminate pixels corresponding to an expected thickness of a finger within the hand candidate image, with reference to a table which defines the number of erosion operations according to a distance value of a hand.

17. The method of claim 14, wherein the determination of the hand shape compares the size of the region remaining in the subtraction image with a predefined size of a finger region and determines a comparison result as one of a hand shape in which a finger is spread and a hand shape in which a finger is bent.

18. An image processing system comprising:

a hand shape detection unit; and

a hand detection unit, wherein the image processing system is configured to detect hands of a subject in real time.

19. The system of claim 18, wherein the hand detection unit comprises:

a camera interface that includes a first image input part configured to generate a first image photographed at a first side with respect to one subject and a second image input part configured to generate a second image photographed at second side with respect to the subject;

a stereoscopic processor configured to generate a distance image indicating 3-dimensional distance information by comparison information of the first image and the second image;

a skin region detector configured to determine whether each pixel within any one of the first and second images is a skin pixel and generate a skin image indicating whether each pixel within any one of the first and second images is a skin pixel;

a skin/distance pixel combiner configured to generate a combined image by combining the distance image with the skin image in units of pixels;

a background and noise eliminator configured to generate an effective image including two or more hand candidate regions by eliminating background and noise from the combined image; and

a hand correlation determiner configured to determine calculating effective values with respect to respective pairs of the hand candidate regions which can be combined within the effective image and determine a pair of hand candidate regions having the largest effective values as hand regions.

20. The system of claim 18, wherein the hand shape detection unit comprises:

a hand candidate region detector configured to generate a hand candidate image from which hand candidate regions are detected;

an erosion image generator configured to generate an erosion image by repeatedly performing erosion operations considering an expected thickness of a finger with respect to the hand candidate image;

a hand center operator configured to detect the center of a hand using a phase relationship from the erosion image;

a dilation image generator configured to generate a dilation image by performing a dilation operation as many times as the repeated number of the erosion operations;

a subtraction image generator configured to generate a subtraction image by subtracting the dilation image from the hand candidate image; and

a hand shape determiner configured to determine a hand shape using a size of a region remaining in the subtraction image.