METHOD AND APPARATUS FOR SHOULDER-LINE DETECTION AND GESTURE SPOTTING DETECTION

• US 20080037875A1
• Filed: 02/12/2007
• Published: 02/14/2008
• Est. Priority Date: 08/14/2006
• Status: Active Grant

• Current Assignee: Electronics and Telecommunications Research Institute
• Original Assignee: Electronics and Telecommunications Research Institute
• Inventors: YOON, Ho Sub, KIM, Do Hyung, LEE, Jae Yeon, KIM, Hye Jin
• Granted Patent: US 8,204,311 B2
• Time in Patent Office: Days
• US Class Current: 382/199
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A gesture spotting detection method and apparatus employ a shoulder-line algorithm. The shoulder-line detecting method recognizes a GSD calling gesture that occurs in a shoulder-line, head or higher part in a remote distance or a short distance, although a user does not have a fixed posture. In the method, an image of people is received, and skin information of a person in the image is detected to detect a face area. Then, the cloth color information of the person is modeled from the inputted image to detect a cloth area. An external space is defined from the image based on the body space area, and an edge is extracted from the image based on the body space and the external space. Then, shoulder-line information is acquired based on an energy function obtained based on the body space, the external space, and the edge.

Claims

1. A shoulder-line detecting method, comprising:

receiving a image of the object people;

defining a body space area from the inputted image;

defining an external space area from the inputted image based on the body space area;

extracting an edge area from the inputted image based on the body space area and the external space area; and

acquiring shoulder-line information based on an energy function obtained based on the body space area, the external space area, and the edge area.

2. The shoulder-line detecting method according to claim 1, wherein the step of defining the external space area includes:

detecting skin information of a person in the inputted image to detect a face area; and

modeling cloth color information of the person whose face position from the inputted image determine the relative color region of the people'"'"'s clothes.

3. The shoulder-line detecting method according to claim 2, wherein a Face Detection algorithm provided by OpenCV is used in the step of extracting the skin information.

4. The shoulder-line detecting method according to claim 2, wherein an area including skin color information is extracted from the inputted image and determined as a face area in the step of detecting the skin information.

5. The shoulder-line detecting method according to claim 4, wherein the skin color information is based on Peer'"'"'s color model.

6. The shoulder-line detecting method according to claim 2, wherein the cloth color information is extracted from the inputted image and a cloth area is detected by performing a color histogram backprojection onto the image based on the above-detected face area in the step of modeling the cloth color information.

7. The shoulder-line detecting method according to claim 2, wherein cloth color information of an upper half body is modeled and traced along with the face information of the people, and an HSV histogram is obtained in the upper half body area of the image to thereby represent first, second and third-order statistic values thereof in a 9-dimensional arrangement in the step of modeling the cloth color information.

8. The shoulder-line detecting method according to claim 1, wherein the external space area is defined as an area having the same length and width as the body space area, which is defined in the defining of a body space, and in the step of defining the external space.

9. The shoulder-line detecting method according to claim 1, wherein edge area information is acquired from the inputted image based on spatial gradient information of the inputted image in the step of extracting the edge area.

10. The shoulder-line detecting method according to claim 1, wherein a boundary area between the body space area and the external space area is defined as an edge area in the step of extracting the edge area.

11. The shoulder-line detecting method according to claim 1, wherein in the step of acquiring the shoulder-line information, when the energy function for detecting a shoulder-line is defined as the following Equation 1:

$\begin{array}{cc}S=\left(\underset{\forall <br><br>x\in <br><br>R<br><br><br><br>2}{\sum <br><br>}<br><br>f<br><br>\left(x\right)-\underset{\forall <br><br>x\in <br><br>R<br><br><br><br>1}{\sum <br><br>}<br><br>f<br><br>\left(x\right)\right)/r+\underset{\mathrm{vx}\in <br><br>L}{\sum <br><br>}<br><br>f<br><br>\left(x\right)& \mathrm{Eq}.<br><br>1\end{array}$

where R1 denotes a body space area including a face area and a cloth area and having a predetermined length (l) and a predetermined width (r);

R2 denotes an external space area which is an area other than the body space area and having the same length (l) and width (r) of the body space area R1;

an edge area (L) denotes an area between the body space area R1 and the external space area R2 and defining a boundary between the body and the outside of the body; and

α

denotes an angle which an edge branched out from an edge space (L) forms against the edge space when the edge rotates based on its contact point to the edge space (L),

the angle (Θ

_max) that maximizes an energy S is acquired by varying the angle α and

based on the following Equation 2;

$\begin{array}{cc}{\mathrm{\theta <br><br>}}_{\max }={\text{arg}}_{\mathrm{\theta <br><br>}}<br><br>\left\{\max <br><br>\left\{\underset{i=1}{\overset{l}{\sum <br><br>}}<br><br>I_s<br><br>\left(x_k+i<br><br><br><br>\cos <br><br><br><br>\mathrm{\theta <br><br>},y_k+i<br><br><br><br>\sin <br><br><br><br>\mathrm{\theta <br><br>}\right)\right\}\right\}& \mathrm{Eq}.<br><br>2\end{array}$

where I(x,y) signifies an intensity at a position (x,y).

12. A gesture spotting detection (GSD) method, comprising:

receiving a image of the object people, extracting a body space area, an external space area, and an edge area from the inputted image, forming an energy function based on the body space area, the external space area, and the edge area, and acquiring shoulder-line information based on the energy function;

tracing calling information among gestures of the people from the inputted image;

when a calling gesture is made, checking whether the calling gesture is repeatedly made by the same person based on the shoulder-line information; and

when the calling gesture is made by the same person repeatedly and a repetition frequency is larger than a predetermined threshold value, confirming the person as a calling user.

13. The gesture spotting detection method according to claim 12, wherein the step of detecting the shoulder-line includes:

receiving a image of the object people;

defining a body space area from the inputted image;

defining an external space area from the inputted image based on the body space area;

detecting an edge area from the inputted image based on the body space area and the external space area; and

acquiring shoulder-line information based on an energy function acquired from the body space area, the external space area, and the edge area.

14. The gesture spotting detection method according to claim 13, wherein the step of defining the body space area includes:

detecting a face area by extracting skin information of a person from the inputted image; and

detecting a cloth area by modeling cloth color information of the person whose face area is detected from the inputted image.

15. The gesture spotting detection method according to claim 12, wherein the step of tracing the calling information includes:

detecting a hand gesture; and

checking whose hand is making the hand gesture.

16. The gesture spotting detection method according to claim 15, wherein in the hand gesture detecting step b1), a Mean Shift algorithm of OpenCV is used to find out a value Y_c that makes a Mean Shift vector m_h,G(y_j)y_j−

y_j‘

0’

(m_h,G(y_j)y_j−

y_j=0), when m_h,G(x) is expressed as the following Equation 3;

$\begin{array}{cc}{m}_{h,G}<br><br>\left(x\right)=\frac{\underset{i=1}{\overset{n}{\sum <br><br>}}<br><br>x_i<br><br>g<br><br>\left({<br><br>\frac{x-x_i}{h}<br><br>}^2\right)}{\underset{i=1}{\overset{n}{\sum <br><br>}}<br><br>g<br><br>\left({<br><br>\frac{x-x_i}{h}<br><br>}^2\right)}-x& \mathrm{Eq}.<br><br>3\end{array}$

where g denotes a Gaussian kernel.

17. The gesture spotting detection method according to claim 12, wherein the step of calling the user includes:

when the calling gesture is made by the same person repeatedly and a repetition frequency is larger than a predetermined threshold value, confirming the person as a calling user;

waking up a gesture recognition system; and

informing an initial hand position to the gesture recognition system.

18. A computer-readable recording medium for storing a computer program that implements a method for detecting a shoulder-line, comprising:

receiving a image of the object people;

defining a body space area from the inputted image;

defining an external space area from the inputted image based on the body space area;

detecting an edge area from the inputted image based on the body space area and the external space area; and

acquiring shoulder-line information based on an energy function acquired from the body space area, the external space area, and the edge area.

19. A computer-readable recording medium for storing a computer program that implements a gesture spotting detection method, comprising:

receiving a photographed image of the object people, extracting a body space area, an external space area, and an edge area from the inputted photographed image, forming an energy function based on the body space area, the external space area, and the edge area, and acquiring shoulder-line information based on the energy function;

tracing calling information among gestures of the people from the inputted image;

when a calling gesture is made, checking whether the calling gesture is repeatedly made by the same person based on the shoulder-line information; and

when the calling gesture is made by the same person repeatedly and a repetition frequency is larger than a predetermined threshold value, confirming the person as a calling user.

20. A shoulder-line detecting apparatus, comprising:

an image input module for receiving a image of the object people through an external vision sensor;

a body space area defining module for detecting a body space area from the inputted image provided by the image input module;

an external space area defining module for defining an external space area based on the inputted image provided by the image input module and the body space area provided by the body space area defining module;

an edge area defining module for defining an edge area based on the body space area information and the external space area information; and

a shoulder-line detecting module for detecting shoulder-line information based on a value that maximizes a ratio of the body space area to the external space area of the above defined body.

21. The shoulder-line detecting apparatus according to claim 20, wherein the body space area defining module extracts a face area and a cloth area from the inputted image obtained in the image input module, and defines a body space area based on the face area and a cloth area.

22. The shoulder-line detecting apparatus according to claim 20, wherein the edge area defining module defines an edge space area, which is a boundary area between the body space area and the external space area.

23. The shoulder-line detecting apparatus according to claim 20, wherein when the energy function for detecting a shoulder-line is defined as the following Equation 1:

$\begin{array}{cc}S=\left(\underset{\forall <br><br>x\in <br><br>R<br><br><br><br>2}{\sum <br><br>}<br><br>f<br><br>\left(x\right)-\underset{\forall <br><br>x\in <br><br>R<br><br><br><br>1}{\sum <br><br>}<br><br>f<br><br>\left(x\right)\right)/r+\underset{\mathrm{vx}\in <br><br>L}{\sum <br><br>}<br><br>f<br><br>\left(x\right)& \mathrm{Eq}.<br><br>1\end{array}$

where R1 denotes a body space area including a face area and a cloth area and having a predetermined length (l) and a predetermined width (r);

R2 denotes an external space area which is an area other than the body space area and having the same length (l) and width (r) of the body space area R1;

an edge area (L) denotes an area between the body space area R1 and the external space area R2 and defining a boundary between the body and the outside of the body; and

α

denotes an angle which an edge branched out from an edge space (L) forms against the edge space when the edge rotates based on its contact point to the edge space (L),

the shoulder-line detecting module acquires the shoulder-line information by varying the angle α and

obtaining the angle (Θ

_max) that maximizes an energy S based on the following Equation 2;

$\begin{array}{cc}{\mathrm{\theta <br><br>}}_{\max }={\text{arg}}_{\mathrm{\theta <br><br>}}<br><br>\left\{\max <br><br>\left\{\underset{i=1}{\overset{l}{\sum <br><br>}}<br><br>I_s<br><br>\left(x_k+i<br><br><br><br>\cos <br><br><br><br>\mathrm{\theta <br><br>},y_k+i<br><br><br><br>\sin <br><br><br><br>\mathrm{\theta <br><br>}\right)\right\}\right\}& \mathrm{Eq}.<br><br>2\end{array}$

where I(x, y) signifies an intensity at a position (x, y).

24. A gesture spotting detection apparatus, comprising:

an image input block for receiving a image of the object people through an external vision sensor;

a shoulder-line detecting block which includes a body space area defining module for detecting a body space area from the inputted image provided by the image input block, an external space area defining module for defining an external space area based on the inputted image provided by the image input block and the body space area provided by the body space area defining module, an edge area defining module for defining an edge area based on the body space area and the external space area, and a shoulder-line detecting module for detecting shoulder-line information based on a value that maximizes a ratio of the body space area to the external space area of the above-defined body;

a calling gesture determining block for determining whether there is a calling gesture of a calling user based on the inputted image provided by the image input block and the shoulder-line information provided by the shoulder line detecting block; and

a system calling block for, when a calling gesture is confirmed, confirming the calling user, waking up a gesture recognition system, and transmitting initial hand position information of the calling user to the gesture recognition system.

25. The gesture spotting detection apparatus according to claim 24, wherein the calling gesture determining block includes:

a hand gesture extracting module for checking whether a repeated calling gesture of the calling user is a hand gesture; and

a calling gesture detecting module for checking whose hand is making the calling gesture and whether the galling gesture is repeated and a repetition frequency is larger than a predetermined threshold value.