Method and system for vision based interfacing with a computer

• US 9,305,229 B2
• Filed: 11/29/2012
• Issued: 04/05/2016
• Est. Priority Date: 07/30/2012
• Status: Active Grant

• Current Assignee: Bruno Delean, Guy Le Henaff
• Original Assignee: Bruno Delean
• Inventors: DeLean, Bruno, Le Henaff, Guy
• Primary Examiner(s): Fitzpatrick, Atiba O
• Application Number:
  US13/688,463
• Publication Number:
  US 20140029789A1
• Time in Patent Office: 1,223 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 10/10 Image acquisition document image scanning and transmission H04N1/00; control of digital cameras H04N23/60
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 40/113 Recognition of static hand signs

Abstract

System and method which allow a user to interface with a machine/computer using an image capturing device (e.g. camera) instead of conventional physical interfaces e.g. keyboard, mouse. The system allows the user to interface from any physical and non-physical location within the POV of the camera at a distance that is determined by the resolution of the camera. Using images of the hand, the system may detect a change of hand states. If the new state is a known state that represents a hit state, the system would map the change of state to a key hit in a row and column of the keyboard and sends the function associated with that key for execution. In an embodiment, the system determines the row based on the rotation of the wrist and/or position of the hand.

Claims

1. A method for interfacing with a machine using a virtual input device, said method comprising:

receiving from an image feed device an image in which a hand is present;

detecting fingertips of the hand in said image;

classifying said fingertips in a vector signature based on a position of each fingertip with respect to a reference axis defined by the remaining fingertips;

comparing said vector signature to pre-stored vector signatures representing known hand states;

if the vector signature represents a new state defining a hit state, associating a change of state to a virtual key in a first row of the virtual input device;

Sending a function represented by said virtual key to said machine for execution.

2. The method of claim 1, wherein classifying further comprises establishing the reference axis based on a point of interest in the fingertips of the index and annular fingers.

3. The method of claim 2, further comprising establishing the point of interest of each fingertip as the center of gravity (COG) of that fingertip.

4. The method of claim 3, wherein the vector signature comprises at least four vectors, the method further comprising:

establishing a first vector perpendicular to the reference axis, a magnitude of said first vector being defined by the distance between the COG of the ringer finger and the reference axis;

establishing a second vector collinear with the reference axis, a magnitude of said second vector being defined by the distance between the COG of the little finger and the point of contact between the first vector and the reference axis;

establishing a third vector perpendicular to the reference axis, a magnitude of said third vector being defined by the distance between the COG of the major and the reference axis;

establishing a fourth vector perpendicular to the reference axis, a magnitude of said fourth vector being defined by the distance between the COG of the index and the point of contact between the third vector and the reference axis.

5. The method of claim 1, wherein when fingertips are detected by group of two contiguous fingertips classification further comprises establishing the reference axis based on a point of interest of this in a group of two fingertips and the remaining fingers, and establishing a vector signature including two vectors.

6. The method of claim 5, further comprising:

establishing a fifth vector perpendicular to the reference axis, a magnitude of said fifth vector being defined by the distance between the COG of the group of fingers and the reference axis;

establishing a sixth vector collinear with the reference axis, a magnitude of said second vector being defined by the distance between the COG of one of little finger or index and the point of contact between the fifth vector and the reference axis;

comparing said vector signature to pre-stored vector signatures specific for the group of fingers representing known hand states

establishing from this comparison a probability that this signature represent a specific state

comparing this probability established from 5^th and 6^th vector with probabilities of signatures established from analysis of 4^th first vectors to a pre-stored vector signatures representing known hand states to establish a final probability of the asserted gesture

if the final probability establish that a new state defining a hit state, associating a change of state to a virtual key in a first row of the virtual input device then send a function represented by said virtual key to said machine for execution.

7. The method of claim 1, wherein comparing comprises applying a hysteresis based compensation of probabilities to a set of successive images including:

determining a probability for each potential state that the vector signature may represent;

comparing the probability associated with each potential state with a sum of the probabilities associated with all other potential states;

Setting the potential state as the new state when the probability associated with the potential state increases over the sum of probabilities.

8. The method of claim 7, wherein determining comprises:

calculating a coordinate point of the vector signature in a multi-dimensional environment including at least four dimensions defined by at least four fingers of the hand;

for each known state, transforming a hand movement leading to the known state into a multi-dimensional trajectory in the multi-dimensional environment;

calculating a distance between the coordinate point and each trajectory of each potential state;

setting the probability of each potential state based on the distance between the coordinate point of the vector signature and the multi-dimensional trajectory of that potential state, wherein the higher the distance the lower is the probability and vice versa.

9. The method of claim 8, further comprising:

scanning distances between the coordinate point of the vector signature and each trajectory of each state within a galaxy of the multidimensional environment;

setting the probability of each state based on the distance between the coordinate point of the vector signature and the multi-dimensional trajectory of that state, wherein the higher the distance the lower is the probability and vice versa.

10. The method of claim 8, further comprising updating a knowledge base including the trajectories as an interfacing session is running, said updating including providing one or more alternative trajectories for one or more states based on typing movements of the hand during the interfacing cession.

11. The method of claim 1, further comprising:

searching for two adjacent wrists of a user'"'"'s hands in said image;

establishing a virtual input plane for each hand based on a virtual wrist line of the hand;

associating a location hit by a certain fingertip in each virtual input plane to a function.

12. The method of claim 1, further comprising:

performing an assisted learning for said machine including asking a user to type specific keys;

storing vector signatures representing the typing movements in memory; and

associating each vectors signature with a hand state.

13. The method of claim 1, further comprising:

performing a dynamic training of said machine as an interfacing session is running;

storing vector signatures representing the typing movements in memory; and

associating each vector signature with a hand state.

14. The method of claim 1, further comprising:

upon detecting a specific gesture, associating the change of state to a key provided in a second row of the virtual input device, said second row being different than the first row.

15. The method of claim 1, further comprising:

Selecting the first row of the virtual input device based on a rotation angle of a wrist of the hand.

16. The method of claim 1, further comprising determining an allowable distance between the fingertips and the image feed device based upon the image resolution of the image feed device, wherein at least 16 pixels of the image are assigned to the fingertips.

17. The method of claim 1, wherein said known hand states include one:

relaxed state with all fingers up, hit state with major down, hit state with index down, hit state with ring finger down, hit state with little finger down.

18. The method of claim 1, wherein the known hand states represented in the pre-stored vector signatures are the states of the user'"'"'s hands.

19. The method of claim 1, wherein the known hand states represented in the pre-stored vector signatures are the states of hands of someone other than the user.

20. The method of claim 1, wherein receiving comprises receiving an image from one or more of a two dimensional (2D) image feed device, a depth sensor, and a three-dimensional image feed device.

21. The method of claim 1, wherein receiving comprises receiving different images from multiple image feed devices provided in a non-stereoscopic configuration and having different point of views for aggregating multiple take-ups and reducing detection errors.

22. The method of claim 1, further comprising aggregating ancillary intelligence prior to associating the change of state to the virtual key, said ancillary intelligence including one or more of a dictionary data, vocabulary data, and grammar rules.

23. The method of claim 1, further comprising implementing frequently used qualifier keys using thumbs.

24. A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions which when executed by a computing device, cause the computing device to implement the method of claim 1.