Method and system for human-to-computer gesture based simultaneous interactions using singular points of interest on a hand

• US 9,864,433 B2
• Filed: 07/12/2013
• Issued: 01/09/2018
• Est. Priority Date: 07/13/2012
• Status: Active Grant

• Current Assignee: SoftKinetic Software
• Original Assignee: SoftKinetic Software
• Inventors: Pinault, Gilles, Salihoglu, Utku, Souza, Carlos, Thollot, Julien, Guigues, Laurent
• Primary Examiner(s): Tzeng, Fred
• Application Number:
  US14/413,411
• Publication Number:
  US 20150153833A1
• Time in Patent Office: 1,642 Days
• Field of Search: 345156
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Described herein is a method for enabling human-to-computer three-dimensional hand gesture-based natural interactions from depth images provided by a range finding imaging system. The method enables recognition of simultaneous gestures from detection, tracking and analysis of singular points of interests on a single hand of a user and provides contextual feedback information to the user. The singular points of interest of the hand: include hand tip(s), fingertip(s), palm center and center of mass of the hand, and are used for defining at least one representation of a pointer. The point(s) of interest is/are tracked over time and are analyzed to enable the determination of sequential and/or simultaneous “pointing” and “activation” gestures performed by a single hand.

Claims

1. A method for providing natural human-to-computer interaction based on a three-dimensional hand gesture recognition system, the method comprising:

a) imaging a scene including at least one hand of at least one user;

b) processing the imaged scene to determine at least three points of interest associated with said at least one hand;

c) tracking said at least three points of interest to provide a tracked movement of each point of interest with respect to time;

d) analysing said tracked movement of each point of interest;

e) determining, from the analysis of said tracked movement of said at least three points of interest, a simultaneous performance of an activation gesture based on two points of interest of the at least three points of interest, and a pointing gesture based on a single point of interest of the at least three points of interest; and

f) using said determined performance of said activation gesture and said pointing gesture for human-to-computer interaction.

2. A method according to claim 1, wherein any of said at least three points of interest comprise one of:

a finger tip;

a hand tip;

a palm centre;

a centre of mass of the hand; and

a derivative of a combination of at least two of;

the finger tip, the hand tip, the palm centre and the centre of mass of the hand.

3. A method according to claim 2, further comprising determining the position of the palm centre using one of:

a distance map and a Principal Component Analysis.

4. A method according to claim 2, further comprising determining a position of the hand tip by determining the position in space corresponding to a predetermined range in a histogram of a vector onto which all orthonormal pixels of the hand are projected, the vector being characterised in that its direction is correlated to the main direction of a Principal Component Analysis performed on a hand three dimensional point cloud, and its origin is located at a position in correlation with the palm centre.

5. A method according to claim 2, further comprising determining hand tips or extremities by computing a response map, each of the hand tips corresponding to a set of pixels the values of which are determined as being a third smallest value of a set of neighbouring pixels, each neighbouring pixel being at a predetermined distance in a set of predetermined directions and having a depth value different from a considered pixel which is above a predetermined value.

6. A method according to claim 1, wherein said single point of interest used for the pointing gesture is interpolated from two points of interest of the at least three points of interest.

7. A method according to claim 1, wherein said two points of interest used for the activation gesture comprise two finger tips, one finger tip and a hand tip, or one finger tip and a palm centre.

8. A method according to claim 1, wherein two said at least three points of interest comprise a palm centre providing the pointing gesture and a finger tip for a snap gesture.

9. A method according to claim 1, wherein said single point of interest corresponds to a palm centre.

10. A method according to claim 1, wherein said single point of interest comprises an interpolated point of interest between two finger tips, said two finger tips providing said activation gesture.

11. A method according to claim 1, wherein said pointing gesture comprises a pointing gesture which is invariant to rotation.

12. A method according to claim 1, wherein said pointing gesture comprises a pointing gesture which is determined using two finger tips, a palm centre and an interpolated point of interest.

13. A method according to claim 1, further comprising providing contextual feedback to a user in response to the performance of a gesture.

14. A method according to claim 13, wherein providing contextual feedback to a user comprises providing the contextual feedback via a graphical user interface comprising virtual objects which are able to render at least one pointer with which the virtual objects interact.

15. A method according to claim 14, wherein an interaction between the virtual objects and the at least three points of interest represented by at least one pointer is a natural interaction which comprises manipulation of the virtual objects including at least one of touching, squeezing, pinching, grabbing, rotating, pushing, dropping, clicking and their derivative gestures.

16. A method according to claim 14, wherein an interaction of the at least one pointer and the virtual objects comprises manipulation of the virtual objects using at least two points of interest represented by the at least one pointer.

17. A method according to claim 14, wherein an interaction of the at least one pointer and the virtual objects comprises manipulation of the virtual objects using at least two control points represented by two pointers or grabbers.

18. A method according to claim 14, wherein an interaction of the at least one pointer and the virtual objects comprises manipulation of the virtual objects using at least two control points represented by two pointers or grabbers and another pointer the position of which is determined as being between the two pointers.

19. A method according to claim 14, further comprising modifying a representation of the virtual objects and the at least one pointer in the graphical user interface according to the gesture performed by the user and their respective position in space.

21. A non-transitory computer readable medium storing a program for causing a computer to execute the steps of the method according to claim 1.

20. A three-dimensional hand gesture recognition system, the system comprising an imaging system capable of imaging a scene including at least one hand of at least one user and an interactive computer system associated with the imaging system, the interactive computer system being capable of performing the following steps:

i) processing the imaged scene to determine at least three points of interest associated with said at least one hand;

ii) tracking said at least three points of interest to provide a tracked movement of each point of interest with respect to time;

iii) analysing said tracked movement of each of said at least three points of interest;

iv) determining, from the analysis of said tracked movement of said at least three points of interest, a simultaneous performance of an activation gesture based on two points of interest of said at least three points of interest, and a pointing gesture based on a single point of interest of said at least three points of interest; and

v) using said determined performance of said activation gesture and said pointing gesture for human-to-computer interaction.