METHOD AND TERMINAL DEVICE FOR CONTROLLING CONTENT BY SENSING HEAD GESTURE AND HAND GESTURE, AND COMPUTER-READABLE RECORDING MEDIUM

• US 20140168074A1
• Filed: 07/09/2012
• Published: 06/19/2014
• Est. Priority Date: 07/08/2011
• Status: Active Grant

• Current Assignee: Media Interactive Inc., The DNA Company Limited
• Original Assignee: Media Interactive Inc., The DNA Company Limited
• Inventors: Lim, Chang Joo, Jeong, Yun Guen, Min, Young Sam, Park, Seung Goo
• Granted Patent: US 9,298,267 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

The present invention relates to a method, a terminal, and a computer-readable medium for controlling content by detecting head and hand gestures. The method for controlling a content by detecting hand gestures and head gestures, includes steps of: (a) detecting a head region and a hand region of a user by analyzing an input image with an object detection technology; (b) tracking the gestures in the head region and the hand region of the user by using a computer vision technology; and (c) allowing the content to be controlled by referring to gestures-combining information including data on the tracked gestures in the head region and the hand region.

Claims

1. A method for controlling a content by detecting hand gestures and head gestures, comprising steps of:

(a) detecting a head region and a hand region of a user by analyzing an input image with an object detection technology;

(b) tracking the gestures in the head region and the hand region of the user by using a computer vision technology; and

(c) allowing the content to be controlled by referring to gestures-combining information including data on the tracked gestures in the head region and the hand region.

2. The method of claim 1, wherein step (c) includes the step of:

(c1) combining information on locations and directions included in the data on the gestures in the head region and the hand region.

3. The method of claim 2, wherein the step (c) includes the steps of:

(c2) recognizing the combined data as a specific command by comparing the combined data with prefixed command signal data in a motion dictionary database; and

(c3) allowing the content to be controlled according to the recognized specific command.

4. The method of claim 3, wherein, at the step (c2), the command signal data included in the motion dictionary database and the combined data are compared by referring to information on location values and rotation values in the combined data themselves.

5. The method of claim 1, wherein, at the step (a), a recognition range of the hand region is set in order to detect the hand region by referring to pixel values in a concentric circle in center of the central point of a palm of the user while the radius of the concentric circle is increased such that the pixel values in the concentric circle to be matched with the previously-analyzed shape of the hand of the user.

6. The method of claim 5, wherein the radius of the concentric circle is increased until information on the endpoint of each finger is acquired.

7. The method of claim 6, wherein, at the step (b), the gestures in the hand region of the user are tracked by referring to a moving trajectory of at least one straight line that connects the central point of the palm of the user with the endpoint of each finger of the user.

8. The method of claim 1, wherein, at the step (a), the head region and the hand region are detected by analyzing the input image based on a skin color model.

9. The method of claim 1, wherein, at the step (b), the gestures in the head region of the user are tracked by using a trajectory of a sight line of the user, which corresponds to a normal vector to a plane of a geometric model formed based on at least multiple morphological feature points extracted from the user'"'"'s head region.

10. The method of claim 9, wherein the multiple morphological feature points include feature points of eyes, nostrils, and endpoints of lips on a facial part of the user.

11. The method of claim 9, wherein data on the tracked gestures in the head region include information on an absolute coordinate value of an intersecting point when the normal vector is intersected on a display and the time of staying at the intersecting point.

12. The method of claim 1, wherein, at the step (b), data on the gestures are acquired by storing data acquired by tracking a face, a central point of a palm of a hand, and fingers of the user for a predetermined time.

13. The method of claim 12, wherein the data on the gestures include information on 3D coordinate values and average movement speed.

14. The method of claim 12, wherein the step (c) includes the steps of:

(c4) expressing a movement trajectory in an autoregressive model with reference of the data on the gestures acquired and stored for the predetermined time;

(c5) applying a hidden Markov model (HMM) to the movement trajectory data expressed in the autoregressive model; and

(c6) recognizing the data where the HMM is applied as a special command for controlling the content by comparing the data where the HMM is applied with prefixed command signal data in a motion dictionary database.

27. A medium recording a computer readable program to execute the method of claim 1.

15. A terminal for controlling a content by detecting hand gestures and head gestures, comprising:

an effective region detecting part for detecting a head region and a hand region of a user by analyzing an input image in use of an object detection technology;

a detected region tracking part for tracking gestures in the head region and the hand region of the user by using a computer vision technology; and

a control command interpreting part for allowing the content to be controlled by referring to gestures-combining information including data on the tracked gestures in the head region and the hand region.

16. The terminal of claim 15, where the control command interpreting part combines information on locations and directions in the data on the gestures in the head region and the hand region, recognizes the combined data as a specific command by comparing the combined data with prefixed command signal data in a motion dictionary database, and then allows the content to be controlled according to the recognized specific command.

17. The terminal of claim 16, wherein command signal data included in the motion dictionary database and the combined data are compared by referring to information on location values and rotation values in the combined data themselves.

18. The terminal of claim 15, wherein a recognition range of the hand region is set in order to detect the hand region by referring to pixel values in a concentric circle in center of the central point of a palm of the user while the radius of the concentric circle is increased such that the pixel values in the concentric circle to be matched with the previously-analyzed shape of the hand of the user.

19. The terminal of claim 18, wherein the radius of the concentric circle is increased until information on the endpoint of each finger is acquired.

20. The terminal of claim 19, wherein the detected region tracking part tracks gestures in the hand region of the user by referring to a moving trajectory of at least one straight line that connects the central point of the palm of the hand of the user with the endpoint of each finger of the user.

21. The terminal of claim 15, wherein the detected region tracking part tracks the gestures in the head region of the user by using a trajectory of a sight line of the user, which corresponds to a normal vector to a plane of a geometric model formed based on at least multiple morphological feature points extracted from the head region of the user.

22. The terminal of claim 21, wherein the multiple morphological feature points include feature points of eyes, nostrils, and endpoints of lips on a facial part of the user.

23. The terminal of claim 21, wherein data on the tracked gestures in the head region include information on an absolute coordinate value of an intersecting point when the normal vector is intersected on a display and the time of staying at the intersecting point.

24. The terminal of claim 15, wherein the detected region tracking part acquires data on the gestures by storing data acquired by tracking the face, a central point of a palm of a hand, and the fingers of the user for a predetermined time.

25. The terminal of claim 24, wherein the data on the gestures include information on 3D coordinate values and average movement speed.

26. The terminal of claim 24, wherein the control command interpreting part expresses a movement trajectory in an autoregressive model with reference of the data on the gestures acquired and stored for the predetermined time, applies a hidden Markov model (HMM) to the movement trajectory data expressed in the autoregressive model, compares the data where the HMM is applied with prefixed command signal data in a motion dictionary database, and then recognizes the data where the HMM is applied as a special command for controlling the content.