Hand-Gesture Recognition Method

• US 20090103780A1
• Filed: 12/17/2008
• Published: 04/23/2009
• Est. Priority Date: 07/13/2006
• Status: Active Grant

• Current Assignee: Northrop Grumman Systems Corporation (Northrop Grumman Corporation)
• Original Assignee: Northrop Grumman Systems Corporation (Northrop Grumman Corporation)
• Inventors: Kaehler, Adrian, Hsu, Shi-Ping, Jangaard, Lars, Nishihara, H. Keith
• Granted Patent: US 9,696,808 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

One embodiment of the invention includes a method of providing device inputs. The method includes illuminating hand gestures performed via a bare hand of a user in a foreground of a background surface with at least one infrared (IR) light source. The method also includes generating a first plurality of silhouette images associated with the bare hand based on an IR light contrast between the bare hand and the background surface and generating a second plurality of silhouette images associated with the bare hand based on an IR light contrast between the bare hand and the background surface. The method also includes determining a plurality of three-dimensional features of the bare hand relative to the background surface based on a parallax separation of the bare hand in the first plurality of silhouette images relative to the second plurality of silhouette images. The method also includes determining a provided input gesture based on the plurality of three-dimensional features of the bare hand and comparing the provided input gesture with a plurality of predefined gesture inputs in a gesture library. The method further includes providing at least one device input corresponding to interaction with displayed visual content based on the provided input gesture corresponding to one of the plurality of predefined gesture inputs.

Claims

1. A method of providing device inputs, the method comprising:

illuminating hand gestures performed via a bare hand of a user in a foreground of a background surface with at least one infrared (IR) light source;

generating a first plurality of silhouette images associated with the bare hand based on an IR light contrast between the bare hand and the background surface;

generating a second plurality of silhouette images associated with the bare hand based on an IR light contrast between the bare hand and the background surface;

determining a plurality of three-dimensional features of the bare hand relative to the background surface based on a parallax separation of the bare hand in the first plurality of silhouette images relative to the second plurality of silhouette images;

determining a provided input gesture based on the plurality of three-dimensional features of the bare hand;

comparing the provided input gesture with a plurality of predefined gesture inputs in a gesture library; and

providing at least one device input corresponding to interaction with displayed visual content based on the provided input gesture corresponding to one of the plurality of predefined gesture inputs.

2. The method of claim 1, wherein determining the provided input gesture comprises determining that the bare hand is pointing at the background surface with at least one extended finger.

3. The method of claim 2, wherein providing the at least one device input comprises providing a visual reference on a portion of the displayed visual content.

4. The method of claim 1, wherein determining the provided input gesture comprises determining that the bare hand is one of contacting a portion of the visual content with a first digit while extending at least one additional digit from the bare hand and touching the background surface with at least two digits of the bare hand.

5. The method of claim 4, wherein providing the at least one device input comprises providing a simulated mouse-button input associated with the contacted portion of the displayed visual content.

6. The method of claim 1, wherein determining the provided input gesture comprises determining that the bare hand is positioned as a flat, open hand with the fingers and thumb position in one of together and spread, and the bare hand is at least one of moving in six-degrees of freedom and moving one of upwards and downwards.

7. The method of claim 6, wherein providing the at least one device input comprises at least one of moving a selected portion of the displayed visual content in a corresponding six-degrees of freedom, and a changing a point-of-view of the displayed visual content in at least one of the corresponding six-degrees of freedom and one of upwards and downwards, respectively.

8. The method of claim 1, wherein determining the provided input gesture comprises determining that the bare hand is pointing to a portion of the visual content with an extended index finger and is changing an angle between an extended thumb and the extended index finger.

9. The method of claim 8, wherein providing the at least one device input comprises controlling zoom associated with the portion of the visual content based on the changing of the angle between the extended thumb and the extended index finger.

10. The method of claim 1, wherein determining the provided input gesture comprises determining that the bare hand is closing from a predefined hand position to a grip hand gesture comprising one of a closed-fist, contact between a thumb and a distal end of at least one finger, and a bending of each finger and the thumb at a distal knuckle with respect to the hand.

11. The method of claim 10, wherein determining the provided input gesture comprises determining that the bare hand is selecting a portion of the displayed visual content for further interaction in response to additional hand gestures.

12. The method of claim 11, wherein determining the provided input gesture comprises determining that the bare hand is maintaining the grip hand gesture and is one of rotating at the wrist and moving in at least one of the X, Y, and Z direction.

13. The method of claim 12, wherein providing the at least one device input further comprises a respective one of rotating the selected portion of the visual content and moving the selected portion of the visual content in the at least one of the X, Y, and Z direction relative to a remaining portion of the visual content.

14. The method of claim 1, wherein determining the provided input gesture comprises determining that the bare hand is positioned in a flat, open hand gesture and is rotating at the wrist approximately 180 degrees.

15. The method of claim 14, wherein determining the provided input gesture further comprises determining that the bare hand is moving forward concurrently with rotating the flat, open hand gesture.

16. The method of claim 14, wherein providing the at least one device input comprises providing one of a page-up input and a page-down input associated with a document.

17. The method of claim 16, wherein determining the provided input gesture further comprises determining that the bare hand is maintaining the flat, open hand gesture and moving one of upwards and downwards, and wherein providing the at least one device input further comprises scrolling a page of the document upwards and downwards based on the movement of the bare hand.

18. The method of claim 16, wherein determining the provided input gesture further comprises:

determining that the bare hand is maintaining the flat, open hand gesture; and

determining at least one additional gesture input from one of the bare hand and the other bare hand of the user;

wherein providing the at least one device input further comprises one of repeating the one of the page-up and the page-down input associated with the document in response to the at least one additional gesture input and auto-turning the page at one of an acceleration and deceleration that is based on a duration at which the at least one additional gesture input is maintained.

19. The method of claim 1, wherein determining the provided input gesture comprises:

determining that each bare hand of the user is extending an index finger to point to two respective portions of the displayed visual content; and

determining that each of the bare hands are maintaining the extension of the index finger and are moving one of closer and further with respect to each other.

20. The method of claim 19, wherein providing the at least one device input comprises one of zooming in and zooming out the displayed visual content in response to the bare hands moving closer and further, respectively.

21. The method of claim 1, wherein determining the provided input gesture comprises determining that the bare hand is alternating between a flat, open hand and a closed-fist at least once.

22. The method of claim 21, wherein providing the at least one device input comprises toggling between a list of menu items on an application in the displayed visual content.

23. The method of claim 1, wherein determining the provided input gesture comprises determining that the bare hand is positioned in a first gesture comprising an extended thumb, an extended index finger, and an extended little finger and is moved one of upward and downward.

24. The method of claim 23, wherein providing the at least one device input comprises scrolling selected visual data one of upward and downward at a first speed in response to the movement of the bare hand the one of upward and downward, respectively.

25. The method of claim 24, wherein determining the provided input gesture comprises determining that the bare hand switches from the first gesture to a second gesture comprising a retracted thumb, the extended index finger, and the extended little finger and is moved one of upward and downward.

26. The method of claim 25, wherein providing the at least one device input comprises scrolling selected visual data one of upward and downward at a second speed in response to the movement of the bare hand the one of upward and downward, respectively, the second speed being faster than the first speed.

27. The method of claim 1, wherein determining the provided input gesture comprises determining that the bare hand switches from a first gesture comprising an extended thumb, an extended index finger, and an extended middle finger to a second gesture comprising a retracted thumb, an extended index finger, and an extended middle finger and is moved one of up, down, left, and right while maintaining the second gesture.

28. The method of claim 27, wherein providing the at least one device input comprises:

tilting a view one of up and down in response to the movement of the maintained second gesture up and down, respectively; and

panning the view one of left and right in response to the movement of the maintained second gesture left and right, respectively.

29. The method of claim 1, wherein determining the provided input gesture comprises determining that the bare hand is pointing to a portion of the displayed visual content with an extended index finger and is moving toward the portion of the visual content.

30. The method of claim 29, wherein providing the at least one device input comprises one of opening a menu list and expanding menu options associated with a menu of the portion of the displayed visual content.

31. The method of claim 1, wherein determining the provided input gesture comprises determining that the bare hand begins in a first gesture comprising an extended index finger that points to a portion of the displayed visual content, initiates a trigger motion with the extended index finger, and resumes the first gesture.

33. The method of claim 1, wherein determining the provided input gesture comprises determining that the bare hand, while maintaining an extended index finger that points to a portion of the displayed visual content, rapidly moves the bare hand in a downward direction, then rapidly moves the bare hand back upward.

35. The method of claim 1, wherein determining the provided input gesture comprises determining that the bare hands of the user, while maintaining a gesture that comprises extended index fingers being in longitudinal contact with each other and both pointing at a portion of the displayed visual content, rapidly move in a forward direction, then rapidly move in a backward direction.

36. The method of claim 35, wherein providing the at least one device input comprises providing a simulated mouse-button input associated with the portion of the displayed visual content.

37. The method of claim 1, wherein determining the provided input gesture comprises determining that the bare hand of the user switches from a first gesture comprising an extended thumb and an extended index finger that points to a portion of the displayed visual content to a second gesture comprising a retracted thumb and the extended index finger that remains pointing to the portion of the displayed visual content.

38. The method of claim 37, wherein providing the at least one device input comprises scrolling visual data associated with the portion of the displayed visual content while the thumb is retracted.

39. The method of claim 1, wherein determining the provided input gesture comprises determining that the bare hand of the user is pointing at a portion of the displayed visual content with an extended index finger and comprises an extended thumb that is subsequently retracted followed by a rotation of the wrist.

40. The method of claim 39, wherein providing the at least one device input comprises tilting a view associated with the displayed visual content in response to the retraction of the thumb, the tilting being based on the rotation of the wrist.

32. The method of claim 32, wherein providing the at least one device input comprises returning to an immediately preceding web-page in a selected Internet browser associated with the portion of the displayed visual content.

34. The method of claim 32, wherein providing the at least one device input comprises providing a simulated mouse-button input associated with the portion of the displayed visual content.