Hand and finger registration for control applications

• US 9,013,430 B2
• Filed: 06/22/2011
• Issued: 04/21/2015
• Est. Priority Date: 08/20/2010
• Status: Active Grant

• Current Assignee: University Of Massachusetts
• Original Assignee: University Of Massachusetts
• Inventors: McCann, Eric, Micire, Mark J., Yanco, Holly A., Norton, Adam D.
• Primary Examiner(s): HALEY, JOSEPH R
• Application Number:
  US13/818,007
• Publication Number:
  US 20130207920A1
• Time in Patent Office: 1,399 Days
• Field of Search: None
• US Class Current: 345/173
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Robot control typically requires many physical joysticks, buttons and switches. We have created a dynamically resizing, ergonomic, and multi-touch (DREAM) controller that enables a software-based operator control unit (SoftOCU) using a multi-touch screen. The DREAM controller can be formed wherever a person places his or her hand on a touch screen. We have also developed an algorithm for accurate hand and finger registration that can be used to create the DREAM controller.

Claims

1. A method of identifying one or more fingers of a hand, the method comprising:

calculating a center position of the hand based at least on points corresponding to a plurality of the fingers;

calculating angles between the points based at least on positions of the points and the center position of the hand; and

identifying, based on the angles, which one or more fingers of the hand correspond to one or more of the points,

wherein the center position is calculated by determining a center of a bounding box around the points.

2. The method of claim 1, further comprising:

receiving the points, wherein the points represent locations of contact between the fingers and a surface.

3. The method of claim 1, further comprising:

calculating a slope corresponding to non-thumb points of the hand, wherein the bounding box is determined based on the slope and the positions of the points.

11. The method of claim 1, wherein the method is initiated on a repeated basis with a timing independent of times at which the fingers are detected.

4. A method of identifying one or more fingers of a hand, the method comprising:

calculating a center position of the hand based at least on points corresponding to a plurality of the fingers;

calculating angles between the points based at least on positions of the points and the center position of the hand;

identifying, based on the angles, which one or more fingers of the hand correspond to one or more of the points;

determining a largest angle between any two nearest points; and

determining a second largest angle between any two nearest points,

wherein the step of identifying comprises identifying a first point at an intersection of the largest and second largest angles as corresponding to a thumb.

5. The method of claim 4, wherein the center position is calculated by determining a center of a bounding box around the points.

6. The method of claim 4, wherein the center position is calculated by determining an average or weighted average of the points.

7. The method of claim 4, wherein the step of identifying further comprises identifying a second point separated from the first point by the largest angle as corresponding to a little finger.

8. The method of claim 7, wherein the step of identifying further comprises identifying a third point separated from the first point by the second largest angle as corresponding to an index finger.

9. The method of claim 8, wherein the step of identifying further comprises identifying a fourth point nearest the third point as corresponding to a middle finger.

10. The method of claim 8, wherein the step of identifying further comprises identifying a fifth point nearest the second point as corresponding to a ring finger.

14. The method of claim 6, further comprising:

displaying a graphical user interface based on positions of the points and information identifying which fingers of the hand correspond to the points.

15. The method of claim 14, wherein displaying the graphical user interface comprises displaying a virtual joystick at a position corresponding to the thumb.

16. The method of claim 14, wherein displaying the graphical user interface comprises displaying a button at a position corresponding to an index finger and/or at a position corresponding to a middle finger.

17. The method of claim 14, further comprising moving a position at which the graphical user interface is displayed to track movement of the hand.

18. The method of claim 14, wherein displaying the graphical user interface comprises displaying at least a portion of a virtual keyboard.

19. The method of claim 14, further comprising:

determining whether the hand is a right hand or a left hand; and

if the hand is a right hand, displaying one or more controls to the left of a position between the thumb and an index finger of the right hand;

if the hand is a left hand, displaying one or more controls to the right of a position between the thumb and an index finger of the left hand.

20. The method of claim 14, further comprising:

determining whether the hand is a right hand or a left hand; and

if the hand is a right hand, displaying a video display to the left of a position between the thumb and an index finger of the right hand;

if the hand is a left hand, displaying a video display to the right of a position between the thumb and an index finger of the left hand.

21. The method of claim 14, further comprising:

displaying a circle around the hand.

12. A method of determining whether fingers correspond to a right hand or a left hand, the method comprising:

determining a first vector along a first direction between a first point, corresponding to a first finger, and a second point corresponding to a center of a hand;

determining a second vector along a second direction between a third point, corresponding to a second finger, and the second point;

calculating a cross product between the first and second vectors; and

determining whether the fingers correspond to a right hand or a left hand based on the cross product.

13. The method of claim 12, wherein the first point corresponds to detection of the first finger being in contact with a surface and the second point corresponds to detection of the second finger being in contact with the surface, and the method further comprises:

determining a sign of the cross product, wherein the sign of the cross product indicates whether the hand is a right hand or a left hand.