Natural gesture based user interface methods and systems

• US 9,081,419 B2
• Filed: 06/28/2013
• Issued: 07/14/2015
• Est. Priority Date: 01/05/2011
• Status: Active Grant

• Current Assignee: SoftKinetic Software
• Original Assignee: SoftKinetic Software
• Inventors: Simons, Kevin, Pinault, Gilles, Baele, Xavier, Dal Zot, David, Tocino Diaz, Juan Carlos, Thollot, Julien
• Primary Examiner(s): Chu, Randolph I
• Application Number:
  US13/931,332
• Publication Number:
  US 20140007022A1
• Time in Patent Office: 746 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Described herein is a user interface that provides contextual feedback, controls and interface elements on a display screen of an interactive three-dimensional imaging system. A user interacts with the interface to provide control signals in accordance with those recognized by the system to a makes use of at least one POI in a three-dimensional scene that is imaged by the imaging system to provide control signals for the user interface. Control signals are provided by means of gestures which are analyzed in real-time by gesture recognition processes that analyze statistical and geometrical properties of POI motion and trajectories.

Claims

1. A method for interacting with a user interface system using gesture recognition, the method comprising the steps of:

a) forming at least one multi-dimensional representation of a scene within a field of view of at least one multi-dimensional imaging system;

b) performing a multi-dimensional constrained clustering operation on said multi-dimensional representation of the scene to provide a clustered representation of said scene; and

c) identifying point of interest candidates from said clustered representation which can be used for gesture recognition, a point of interest candidate being identified as a cluster that is connected to only one other cluster and which exhibits coherent motion; and

d) controlling said user interface system in accordance with recognised gestures performed by said at least one point of interest candidate.

2. A method according to claim 1, further comprising the step of recognizing the performance of a gesture of a point of interest candidate using at least continuous trajectory analysis including the steps of:

identifying trajectory points along a trajectory of a point of interest candidate;

identifying at least one multi-dimensional change in trajectory direction of a point of interest candidate, said at least one multi-dimensional change forming a reference seed point having a reference seed identification order;

identifying successive changes in trajectory direction, each of said successive changes forming successive reference seed points; and

using said reference seed points and said trajectory points to recognise a gesture.

3. A method according to claim 2, wherein said trajectory analysis further comprises the step of:

determining, for each seed point, at least one of;

distance, velocity vector and acceleration vector relative to one of;

previous points or previous seed points.

4. A method according to claim 3, wherein said trajectory analysis further comprises the step of using geometric and statistical computation for each point to recognise the gesture.

5. A method according to claim 1, further comprising the step of activating a point of interest candidate as an activated point of interest if it has first performed a predetermined gesture, said activated point of interest still having point of interest candidate status.

6. A method according to claim 5, further comprising the step of determining a region of interest associated with said activated point of interest.

7. A method according to claim 5, further comprising activating at least one further point of interest candidate as at least one further activated point of interest when it has performed a predetermined gesture, said at least one further activated point of interest still having point of interest candidate status.

8. A method according to claim 7, further comprising determining at least one further region of interest associated to said at least one further activated point of interest.

9. A method according to claim 6, wherein each region of interest has its position and dimensions set relative to a position of a point of interest candidate.

10. A method according to claim 6, further comprising the step of recognising a predetermined gesture performed by at least one predetermined point of interest candidate within said at least one determined region of interest.

11. A method according to claim 1, wherein said user interface system includes a gesture based virtual keyboard having a graphical user interface, said graphical user interface having a predetermined arrangement of elements for selection using only minimal natural gestures performed by at least one point of interest candidate.

12. A method according to claim 1, further comprising the step of:

e) controlling said user interface system in accordance with coherent motion of at least one point of interest candidate.

13. A method according claim 1, wherein step e) further comprises the step of providing contextual feedback information in accordance with current status of said user interface system.

14. A method according to claim 1, wherein step b) comprises using multi-resolution image processing.

15. A method according to claim 1, wherein each multi-dimensional representation comprises a three-dimensional representation.

16. A system for interacting with a user interface system using gesture recognition, the system comprising:

means for forming at least one multi-dimensional representation of a scene within a field of view of at least one multi-dimensional imaging system;

means for performing a multi-dimensional constrained clustering operation on said multi-dimensional representation to provide a clustered representation of said scene; and

means for identifying point of interest candidates from said clustered representation which can be used for gesture recognition; and

means for controlling said user interface system in accordance with recognised gestures;

wherein the means for identifying points of interest candidates comprises means for identifying at least one cluster that is connected to only one other cluster and which exhibits coherent motion as a point of interest candidate.

17. A system according to claim 16, further comprising means for recognizing the performance of a gesture of a point of interest candidate which uses at least continuous trajectory analysis including:

means for identifying trajectory points along a trajectory of a point of interest candidate,

means for identifying at least one multi-dimensional change in trajectory direction of a point of interest candidate, said at least one multi-dimensional change forming a reference seed point having a reference seed identification order,

means for identifying successive changes in trajectory direction, each of said successive changes forming successive reference seed points, and

means for using said reference seed points and said trajectory points to recognise a gesture.

18. A system according to claim 17, wherein the means for using trajectory analysis further comprises means for determining, for each seed point, at least one of:

distance, velocity vector and acceleration vector relative to previous seed points.

19. A system according to claim 18, wherein the means for using trajectory analysis further comprises means for using geometric and statistical computation for each point to recognise the gesture.

20. A system according to claim 16, further including a gesture based virtual keyboard having a graphical user interface, said graphical user interface having a predetermined arrangement of elements for selection using only minimal natural gestures.