Biometric aware object detection and tracking

• US 10,733,429 B2
• Filed: 06/02/2017
• Issued: 08/04/2020
• Est. Priority Date: 03/13/2014
• Status: Active Grant

• Current Assignee: Lmi Liquidating Co., LLC
• Original Assignee: Ultrahaptics Ip Two Limited (Ultraleap Holdings Ltd.)
• Inventors: Sills, Maxwell, Smith, Aaron, Holz, David S., He, Hongyuan (Jimmy)
• Primary Examiner(s): Roz, Mark
• Application Number:
  US15/611,784
• Publication Number:
  US 20170270356A1
• Time in Patent Office: 1,159 Days
• Field of Search: None
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 40/11 Hand-related biometrics; Hand pose recognition
• CPC Class Codes: G06V 40/117 Biometrics derived from hands
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

The technology disclosed can provide methods and systems for identifying users while capturing motion and/or determining the path of a portion of the user with one or more optical, acoustic or vibrational sensors. Implementations can enable use of security aware devices, e.g., automated teller machines (ATMs), cash registers and banking machines, other secure vending or service machines, security screening apparatus, secure terminals, airplanes, automobiles and so forth that comprise sensors and processors employing optical, audio or vibrational detection mechanisms suitable for providing gesture detection, personal identification, user recognition, authorization of control inputs, and other machine control and/or machine communications applications. A virtual experience can be provided to the user in some implementations by the addition of haptic, audio and/or other sensory information projectors.

Claims

1. A method of finding an authorized command issued by gesture of a hand moving freely in 3D sensory space, the method including:

capturing a plurality of images of a hand while the hand makes a hand gesture in a three-dimensional (3D) sensory space proximate to, but not in contact with, a surface of a security aware device;

processing the images of the hand to extract one or more biometric features of the hand including at least one of;

one or more scars of the hand;

a vein pattern of the hand; and

a palm print of the hand;

authenticating the hand based on the extracted biometric features of the hand;

processing a sequence of images of the plurality of images to interpret the hand gesture;

determining a command indicated by the hand gesture;

determining whether the authenticated hand is authorized to issue the command; and

issuing an authorized command to the security aware device proximate to the three-dimensional (3D) sensory space in which the hand gesture was made, wherein the authorized command initiates a financial transaction involving an account of the authenticated user.

2. The method of claim 1, further including rejecting the command whenever the authenticated hand is determined unauthorized to issue the command.

3. The method of claim 1, further including triggering a response to gestures performed by an authenticated hand authorized to issue a command without triggering a response to gestures performed by non-authenticated hands.

4. The method of claim 1, further including authenticating a second hand captured in the plurality of images when the second hand belongs to a user authorized to issue commands.

5. The method of claim 1, further including extracting at least one of:

shape of the hand;

size of the hand;

measurements across a palm of the hand;

finger width at a first knuckle of the hand; and

complexion of the hand.

6. The method of claim 1, further including:

using stereo-imaging to capture the plurality of images of the hand.

7. The method of claim 1, further including:

registering multiple images of different poses of the hand during the hand gesture.

8. The method of claim 7, wherein the multiple images of different poses of the hand include at least one of:

palm view of the hand;

backhand view of the hand; and

closed first view of the hand.

9. The method of claim 1, further including:

responsive to the authenticating of the hand based on the extracted biometric features of the hand, interpreting subsequent hand gestures based on specific interpretation criteria.

10. The method of claim 1, further including:

issuing the authorized command to a security aware device.

11. The method of claim 1, wherein the processing of the images to extract biometric features of the hand further includes:

using at least one image characteristic of the images, determining an image property that specifies changes to the image characteristic across various image points;

determining variations to the image property across the various image points; and

based on the variations determined, identifying points of interest in the images that represent biometric features.

12. The method of claim 11, further including:

using a Hessian Matrix for determining changes to the image property across the various image points.

13. The method of claim 11, further including:

using eigenvalues and eigenvectors for identifying the points of interest.

14. The method of claim 11, wherein the image characteristic is pixel intensity.

15. The method of claim 11, wherein the image property is gradient of pixel intensity.

16. The method of claim 11, further including:

determining normalized biometric features by at least one of rotating, translating, and scaling biometric features.

17. The method of claim 11, wherein the determining of whether the authenticated hand is authorized to issue the command further includes:

comparing the biometric features to a stored biometric profile.

18. The method of claim 17, further including:

constructing a biometric profile based on the biometric features, including;

prompting a user to perform hand gestures in the 3D sensory space and capturing a plurality of images of the hand during the hand gestures;

processing one or more of the images to extract biometric features of the hand;

determining normalized biometric features by at least one of rotating, translating, and scaling biometric features;

registering multiple images of different poses of the hand during the hand gestures; and

storing the normalized biometric features and corresponding poses for recognizing gestures.

19. The method of claim 1, wherein the determining of whether the authorized hand is authorized to issue the command further includes:

determining whether a user who performs the hand gesture has a role authorized to issue the command.

20. The method of claim 1, wherein a biometric profile is constructed at a first device and an authorized command is issued to a second device.

23. The method of claim 1, wherein the security aware device includes one of an automated teller machine, a cash register, a banking machine, a vending machine, a security screening apparatus and secure terminals, airplanes and automobiles.

21. A non-transitory, computer-readable medium having computer executable instructions for performing a method comprising:

capturing a plurality of images of a hand while the hand makes a hand gesture in a three-dimensional (3D) sensory space proximate to, but not in contact with, a surface of a security aware device;

processing the images of the hand to extract one or more biometric features of the hand including at least one of;

one or more scars of the hand;

a vein pattern of the hand; and

a palm print of the hand;

authenticating the hand based on the extracted biometric features of the hand;

processing a sequence of images of the plurality of images to interpret the hand gesture;

determining a command indicated by the hand gesture;

determining whether the authenticated hand is authorized to issue the command; and

issuing an authorized command to the security aware device proximate to the three-dimensional (3D) sensory space in which the hand gesture was made, wherein the authorized command initiates a financial transaction involving an account of the authenticated user.

22. A computer system with one or more processors adapted to perform a method comprising:

capturing a plurality of images of a hand while the hand makes a hand gesture in a three-dimensional (3D) sensory space proximate to, but not in contact with, a surface of a security aware device;

processing the images of the hand to extract one or more biometric features of the hand including at least one of;

one or more scars of the hand;

a vein pattern of the hand; and

a palm print of the hand;

authenticating the hand based on the extracted biometric features of the hand;

processing a sequence of images of the plurality of images to interpret the hand gesture;

determining a command indicated by the hand gesture;

determining whether the authenticated hand is authorized to issue the command; and

issuing an authorized command to the security aware device proximate to the three-dimensional (3D) sensory space in which the hand gesture was made, wherein the authorized command initiates a financial transaction involving an account of the authenticated user.