Command input by hand gestures captured from camera

• US 9,317,124 B2
• Filed: 09/28/2006
• Issued: 04/19/2016
• Est. Priority Date: 09/28/2006
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Technologies Oy (Nokia Corporation)
• Inventors: Kongqiao, Wang, Takala, Roope, Fang, Yikai, Lu, Hanqing, Liu, Qingshan
• Primary Examiner(s): ENGLAND, SARA M
• Application Number:
  US11/536,373
• Publication Number:
  US 20080244465A1
• Time in Patent Office: 3,491 Days
• Field of Search: 715/863
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 40/107 Static hand or arm

Abstract

A method and system for invoking an operation of a communication terminal in response to registering and interpreting a predetermined motion or pattern of an object. An input is received, the image data of the object is captured and the object in the image data is identified.

Claims

1. A method for invoking an operation of a communication terminal in response to registering and interpreting a predetermined motion or pattern of an object, the method comprising:

receiving an indication of a predetermined motion or pattern of a single occurrence of a hand input gesture received from a digital camera of the communication terminal;

identifying at least a first and second centroid in the object;

identifying at least one convex hull in the object;

matching the hand input gesture received from the digital camera with a reference gesture object selected from a set of reference gesture objects based on a positional relationship of the at least first and second centroids in the object and a change in size of the convex hull; and

executing a command related to an item of a display of the communication terminal in response to and upon matching the single occurrence of the hand input gesture received from the digital camera with the reference gesture object.

2. The method according to claim 1, wherein said interpretation of a motion or pattern is performed by a software of said terminal.

3. The method according to claim 1, wherein said operation involves provision of a command input to the communication terminal based on the single occurrence of the hand input gesture, and that said method comprises:

capturing image data of said hand input gesture using the digital camera of the communication terminal;

identifying an object in said image data;

recognizing the object as a hand;

recognizing and associating characteristics of said object as a gesture of said hand with a single, first reference gesture from a set of predetermined reference gestures;

providing a command input associated with said single reference gesture.

4. The method according to claim 3, wherein said identification involves classifying skin color.

5. The method according to claim 4, wherein said skin color classification comprises performing Gaussian mixture modeling.

6. The method according to claim 4, wherein said skin color classification involves color space analysis and/or probability analysis.

7. The method according to claim 6, wherein said color space analysis involves converting image data to chrominance plane (CbCr) color space image data.

8. The method according to claim 3, wherein said object recognition involves eliminating visual noise using connected component extraction.

9. The method according to claim 8, wherein said connected component extraction comprises any of the following:

determining aspect ratio of the object;

determining size of object compared to image size;

determining regions connecting to the borders of the input image; and

wherein said noise is eliminated in case the following requirements are fulfilled;

said aspect ratio is within 10;

said object size is greater than a predetermined value set in relation to the input image size; and

there is either only one region connecting to the borders of the input image, or a plurality of regions which do not meet the other requirements.

10. The method according to claim 3 further comprising determining orientation of said hand and involving:

determining a Karhunen-Loeve (KL) axis orientation of said object;

determining a first geometric centerpoint of said object; and

determining a second geometric centerpoint of a convex polygon of said object, and

wherein the orientation of said KL axis is used to determine a positional relationship of said first and second centerpoints.

11. The method according to claim 10, wherein said orientation determining arrives at one of the following:

performing a first operation if said KL-axis extends along a first direction, and the first and second centerpoints are displaced in a first displacement direction in relation to each other, essentially along said first direction;

performing a second operation if said KL-axis extends along said first direction, and the first and second centerpoints are displaced reversely in said displacement direction in relation to each other, essentially along said first direction;

performing a third operation if said KL-axis extends along a second direction, essentially perpendicular to said first direction, and the first and second centerpoints are displaced in a second displacement direction in relation to each other, essentially along said second direction;

performing a fourth operation if said KL-axis extends along said second direction, and the first and second centerpoints are displaced reversely in said displacement direction in relation each other, essentially along said second direction;

performing a fifth operation if said centerpoints are essentially superpositioned, and a first area of said object of said gesture is less than at least half a second area of a previously determined object of a previously recognized gesture;

performing a sixth operation if said centerpoints are essentially superpositioned, a first area of said object of said gesture is greater than at least twice a second area of a previously determined object of a previously recognized gesture, and said gesture corresponds to said previously recognized gesture;

performing a seventh operation if said centerpoints are essentially superpositioned, a first area of said object of said gesture is greater than at least twice a second area of a previously determined object of a previously recognized gesture, and said gesture does not correspond to said previous gesture.

12. The method according to claim 11, wherein said first, second, third, and fourth operations correspond to moving focus up, down, left, and right respectively, and said fifth, sixth, and seventh operations correspond to open an item, close an item, and stop the focus motion respectively.

13. The method according to claim 1, wherein said registering is performed using a camera of said communication terminal.

14. The method according to claim 1, further comprising a step of activation by proximity detection.

15. A computer-readable storage medium having computer-executable components embodied there, said computer-readable medium being configured to execute the method according to claim 1.

16. The computer-readable storage medium according to claim 15, further being configured to:

receive an input;

capture image data of said object;

identify said object in said image data;

recognize said object as a hand;

recognize and associate characteristics of said object as a gesture of said hand with a first reference gesture from a set of predetermined reference gestures;

provide a command input associated with said reference object.

17. The method of claim 1 wherein the item is a document, a folder, a web page, a call, a message, a contact item, a multimedia content item, a reminder, or a textual entry.

18. The method of claim 17 wherein the hand gesture corresponds to a command to maneuver a menu on the display or toggle between items on the display.

19. An apparatus comprising:

a processor; and

a memory including computer program code, wherein the memory and the computer program code are configured to, along with the processor, cause the apparatus at least to perform;

invoking an operation of the apparatus in response to registering and interpreting a predetermined motion or pattern of an object;

receiving an indication of a predetermined motion or pattern of a single occurrence of a hand input gesture received from an imaging unit;

identifying at least a first and second centroid in the object;

identifying at least one convex hull in the object;

matching the hand input gesture received from the imaging unit with a reference gesture object selected from a set of reference gesture objects based on a positional relationship of the at least first and second centroids in the object and a change in size of the convex hull; and

executing a command in the apparatus in response to and upon matching the single occurrence of the hand input gesture received from the imaging unit with the reference gesture object.

20. The apparatus of claim 19, wherein detecting a predetermined motion or pattern of a single occurrence of a hand input gesture comprises classifying a skin color of the hand.

21. The apparatus of claim 19, wherein detecting a predetermined motion or pattern of a single occurrence of a hand input gesture comprises determining orientation of a hand, comprising:

determining a Karhunen-Loeve (KL) axis orientation of said object;

determining a first geometric centerpoint of said object; and

determining a second geometric centerpoint of a convex polygon of said object, and wherein the orientation of said KL axis is used to determine a positional relationship of said first and second centerpoints.

22. The apparatus of claim 19, wherein the apparatus is a communication terminal.

23. The apparatus of claim 19, wherein the command operates on a document, a folder, a web page, a call, a message, a contact item, a multimedia content item, a reminder, or a textual entry.