METHOD AND SYSTEM FOR HUMAN-TO-COMPUTER GESTURE BASED SIMULTANEOUS INTERACTIONS USING SINGULAR POINTS OF INTEREST ON A HAND

• US 20150153833A1
• Filed: 07/12/2013
• Published: 06/04/2015
• Est. Priority Date: 07/13/2012
• Status: Active Grant

• Current Assignee: SoftKinetic Software
• Original Assignee: SoftKinetic Software
• Inventors: Pinault, Gilles, Guigues, Laurent, Salihoglu, Utku, Souza, Carlos, Thollot, Julien
• Granted Patent: US 9,864,433 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Described herein is a method for enabling human-to-computer three-dimensional hand gesture-based natural interactions from depth images provided by a range finding imaging system. The method enables recognition of simultaneous gestures from detection, tracking and analysis of singular points of interests on a single hand of a user and provides contextual feedback information to the user. The singular points of interest of the hand: include hand tip(s), fingertip(s), palm centre and centre of mass of the hand, and are used for defining at least one representation of a pointer. The point(s) of interest is/are tracked over time and are analysed to enable the determination of sequential and/or simultaneous “pointing” and “activation” gestures performed by a single hand.

Claims

1-40. -40. (canceled)

41. A method for providing natural human-to-computer interaction based on a three-dimensional hand gesture recognition system, the method comprising the steps of:

a) imaging a scene including at least one hand of at least one user;

b) processing the imaged scene to determine at least two points of interest associated with said at least one hand;

c) tracking said at least two points of interest to provide a tracked movement of each point of interest with respect to time;

d) analysing said tracked movement of each point of interest;

e) determining, from the analysis of said tracked movement, the simultaneous performance of an activation gesture based on two points of interest, and a pointing gesture based on a single point of interest; and

f) using said determined performance of said activation gesture and said pointing gesture for human-to-computer interaction.

42. A method according to claim 41, wherein any of said points of interest comprise one of:

a finger tip;

a hand tip;

a palm centre;

a centre of mass of the hand; and

a derivative of a combination of at least two of;

the finger tip, the hand tip, the palm centre and the centre of mass of the hand.

43. A method according to claim 42, further comprising determining the position of the palm centre using one of:

a distance map and a PCA analysis.

44. A method according to claim 42, further comprising determining a position of the hand tip by determining the position in space corresponding to a predetermined range in the histogram of a vector onto which all orthonormal pixels of the hand are projected, the vector being characterised in that its direction is correlated to the main direction of a PCA analysis performed on the hand three dimensional point cloud, and its origin is located at a position in correlation with the hand palm centre.

45. A method according to claim 42, further comprising determining hand tips or extremities by computing a response map, each hand tip corresponding to a set of pixels the values of which are determined as being the third smallest value of a set of neighbouring pixels, each neighbouring pixel being at a predetermined distance in a set of predetermined directions and having a depth value different from the considered pixel which is above a predetermined value.

47. A method according to claim 41, wherein said two points of interest comprise any one of:

two finger tips;

one finger tip and a hand tip; and

one finger tip and a palm centre.

48. A method according to claim 41, wherein two points of interest comprise a palm centre providing the pointing gesture and a finger tip for a snap gesture.

49. A method according to claim 41, wherein step b) determines at least three points of interest and step c) comprises using the analysis of said at least three points of interest for the determination of the performance of said activation and pointing gestures.

50. A method according to claim 49, further comprising the step of determining simultaneous performance of two gestures by said three points of interest. (New) A method according to claim 50, wherein said simultaneously performed two gestures comprise a pointing gesture and an activation gesture.

58. A method according to claim 41, further comprising the step of providing contextual feedback to a user in response to the performance of a gesture.

59. A method according to claim 58, wherein the system enabling the providing of feedback information is a graphical user interface comprising virtual elements which are able to render at least one pointer with which the virtual elements interact.

60. A method according to claim 59, wherein interaction between the system and the points of interest of the user'"'"'s hand represented by at least one pointer is a natural interaction which comprises manipulation of the virtual objects including at least one of touching, squeezing, pinching, grabbing, rotating, pushing, dropping, clicking and their derivative gestures.

61. A method according to claim 59, wherein interaction comprises manipulation of the virtual objects using at least two points of interest represented by at least one pointer.

62. A method according to claim 59, wherein interaction comprises manipulation of the virtual objects using at least two control points represented by two pointers or grabbers.

63. A method according to claim 59, wherein interaction comprises in particular manipulation of the virtual objects using at least two control points represented by two pointers or grabbers and another pointer the position of which is determined as being between the two other pointers.

64. A method according to claim 59, further comprising modifying the representation of the objects and pointers in the graphical user interface according to the gesture performed by the user and their respective position in space.

66. A non-transitory computer readable medium storing a program for causing a computer to execute the steps of the method according to claim 41.

52. A method according to claim 51, wherein said pointing gesture is performed by a single point of interest and said activation gesture is performed by two points of interest.

53. A method according to claim 52, wherein said single point of interest corresponds to a palm centre.

54. A method according to claim 52, wherein said activation gesture is performed by one of:

two finger tips, and a finger tip and a hand tip.

55. A method according to claim 52, wherein said single point of interest comprises an interpolated point of interest between two finger tips, said two finger tips providing said activation gesture.

56. A method according to claim 52, wherein said pointing gesture comprises a pointing gesture which is invariant to rotation.

57. A method according to claim 52, wherein said pointing gesture comprises a pointing gesture which is determined using two finger tips, a palm centre and an interpolated point of interest.

65. A three-dimensional hand gesture recognition system, the system comprising an imaging system capable of imaging a scene including at least one hand of at least one user and an interactive computer system associated with the imaging system, the interactive computer system being capable of performing the following steps:

i) processing the imaged scene to determine at least two points of interest associated with said at least one hand;

ii) tracking said at least two points of interest to provide a tracked movement of each point of interest with respect to time;

iii) analysing said tracked movement of each point of interest;

iv) determining, from the analysis of said tracked movement, the simultaneous performance of an activation gesture based on two points of interest, and a pointing gesture based on a single point of interest; and

v) using said determined performance of said activation gesture and said pointing gesture for human-to-computer interaction.