Hand gesture API using finite state machine and gesture language discrete values

• US 10,599,324 B2
• Filed: 12/31/2015
• Issued: 03/24/2020
• Est. Priority Date: 12/31/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Karmon, Kfir
• Primary Examiner(s): Bhargava, Anil K
• Application Number:
  US14/985,691
• Publication Number:
  US 20170192665A1
• Time in Patent Office: 1,545 Days
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06V 10/772 Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

System for associating a computerized hand gestures model with application functions, comprising:

• • (a) A storage storing a plurality of hand pose features records and hand motion features records. Each of the hand pose features records and hand motion features records is defined by a set of discrete pose values and discrete motion values respectively.
  • (b) An interface receiving programmer instructions.
  • (c) A memory storing code.
  • (d) One or more processors coupled to the interface, storage and memory for executing the code which comprises:
    • 1) Code instructions to define hand gestures by constructing a unique logical sequence of the hand pose features records and hand motion features records.
    • 2) Code instructions to associate the unique logical sequence with application functions per the instructions for initiating execution of the functions during the application runtime in response to detection of the unique logical sequence from analysis of images depicting movement of user'"'"'s hand(s).

Claims

1. A system for associating between a computerized model of hand gestures and application functions, comprising:

a storage storing a plurality of hand pose features records and a plurality of hand motion features records, each one of said plurality of hand pose features records include values for one or more of a palm pose feature, a finger flexion feature, finger tangency feature, and a finger relative location feature and each one of said plurality of hand motion features records include one or more values for a motion property feature that relate to a property of a motion;

an interface for receiving instructions from a programmer;

a memory storing a code;

at least one processor coupled to said interface, said storage and said memory for executing said stored code, said code comprising;

code instructions to define at least one hand gesture by constructing a unique logical sequence of at least one of said plurality of hand pose features records and said plurality of hand motion features records based on the values of the hand pose features and the hand motion features; and

code instructions to associate said unique logical sequence with at least one function of an application per said instructions for initiating an execution of said at least one function during runtime of said application and in response to a detection of said unique logical sequence from an analysis of a sequence of images which depict a movement of a hand of a user of said application during said runtime.

2. The system of claim 1, wherein each of said discrete pose values is indicative of a corresponding one of a plurality of hand pose features and each of said discrete motion values is indicative of a corresponding one of a plurality of hand motion features.

3. The system of claim 1, further comprising said programmer defines at least one of:

a plurality of hand pose features and a plurality of hand motion features, each of said plurality of hand pose features is associated with at least one of said discrete pose values and each of said plurality of hand motion features is associated with at least one of said discrete motion values.

4. The system of claim 1, wherein each of the one of said plurality of hand motion features records further includes one or more values for a motion script feature which defines an actual movement.

5. The system of claim 1, wherein said hand pose feature is a member selected from a group consisting of:

active hand, palm direction, palm rotation, flexion of at least one finger, direction of at least one finger, relative location between at least two fingers and tangency between at least two fingers.

6. The system of claim 1, wherein said hand motion feature is a member selected from a group consisting of:

motion properties and motion script, said motion script defines at least one of;

hand motion and motion of at least one finger.

7. The system of claim 1, wherein each finger of a hand is assigned one of the unique set of discrete motion values.

8. The system of claim 1, wherein said unique sequence which represents said at least one hand gesture is defined by a unique finite state machine (FSM) model.

9. A computer implemented method for associating between a computerized model of hand gestures and application functions, comprising:

using an application code executed on at least one processor for;

accessing a memory storing a plurality of hand pose features records and a plurality of hand motion features records, each one of said plurality of hand pose features records include values for one or more of a palm pose feature, a finger flexion feature, finger tangency feature, and a finger relative location feature and each one of said plurality of hand motion features records include one or more values for a motion property feature that relate to a property of a motion;

presenting a user interface for receiving instructions from a programmer;

defining at least one hand gesture by constructing a unique logical sequence of at least one of said plurality of hand pose features records and said plurality of hand motion features records based on the values of the hand pose features and the hand motion features; and

associating said unique logical sequence with at least one function of an application per said instructions for initiating an execution of said at least one function during runtime of said application and in response to a detection of said unique logical sequence from an analysis of a sequence of images which depict a movement of a hand of a user of said application during said runtime.

10. The computer implemented method of claim 9, wherein each of said discrete pose values is indicative of a corresponding one of a plurality of hand pose features and each of said discrete motion values is indicative of a corresponding one of a plurality of hand motion features.

11. The computer implemented method of claim 9, further comprising said programmer defines at least one of:

a plurality of hand pose features and a plurality of hand motion features, each of said plurality of hand pose features is associated with at least one of said discrete pose values and each of said plurality of hand motion features is associated with at least one of said discrete motion values.

12. The computer implemented method of claim 9, wherein said hand pose feature is a member selected from a group consisting of:

active hand, palm direction, palm rotation, flexion of at least one finger, direction of at least one finger, relative location between at least two fingers and tangency between at least two fingers.

13. The computer implemented method of claim 9, wherein said hand motion feature is a member selected from a group consisting of:

motion properties and motion script, said motion script defines at least one of;

hand motion and motion of at least one finger.

14. The computer implemented method of claim 9, wherein said unique sequence which represents said at least one hand gesture is defined by a unique finite state machine (FSM) model.

15. A software program product for associating between a computerized model of hand gestures and application functions, comprising:

a non-transitory computer readable storage medium;

first program instructions to access a memory storing a plurality of hand pose features records and a plurality of hand motion features records, each one of said plurality of hand pose features records include values for one or more of a palm pose feature, a finger flexion feature, finger tangency feature, and a finger relative location feature and each one of said plurality of hand motion features records include one or more values for a motion property feature that relate to a property of a motion;

second program instructions to present a user interface for receiving instructions from a programmer;

third program instructions for defining at least one hand gesture by constructing a unique logical sequence of at least one of said plurality of hand pose features records and said plurality of hand motion features records based on the values of the hand pose features and the hand motion features; and

fourth program instructions for associating said unique logical sequence with at least one function of an application per said instructions for initiating an execution of said at least one function during runtime of said application and in response to a detection of said unique logical sequence from an analysis of a sequence of images which depict a movement of a hand of a user of said application during said runtime;

wherein said first, second, third and fourth program instructions are executed by at least one computerized processor from said non-transitory computer readable storage medium.

16. The software program product of claim 15, wherein each of said discrete pose values is indicative of a corresponding one of a plurality of hand pose features and each of said discrete motion values is indicative of a corresponding one of a plurality of hand motion features.

17. The software program product of claim 15, further comprising fifth program instructions for enabling said programmer to define at least one of:

a plurality of hand pose features and a plurality of hand motion features, each of said plurality of hand pose features is associated with at least one of said discrete pose values and each of said plurality of hand motion features is associated with at least one of said discrete motion values, wherein said fifth program instructions are executed by said at least one computerized processor from said non-transitory'"'"' computer readable storage medium.

18. The software program product of claim 15, wherein said hand pose feature is a member selected from a group consisting of:

active hand, palm direction, palm rotation, flexion of at least one finger, direction of at least one finger, relative location between at least two fingers and tangency between at least two fingers.

19. The software program product of claim 15, wherein said hand motion feature is a member selected from a group consisting of:

motion properties and motion script, said motion script defines at least one of;

hand motion and motion of at least one finger.

20. The software program product of claim 15, wherein said unique sequence which represents said at least one hand gesture is defined by a unique finite state machine (FSM) model.