Gesture recognition devices and methods with user authentication

• US 9,880,629 B2
• Filed: 02/25/2013
• Issued: 01/30/2018
• Est. Priority Date: 02/24/2012
• Status: Active Grant

• Current Assignee: Thomas J. Moscarillo
• Original Assignee: Thomas J. Moscarillo
• Inventors: Moscarillo, Thomas J.
• Primary Examiner(s): Lee, Gene W
• Application Number:
  US13/776,439
• Publication Number:
  US 20130265218A1
• Time in Patent Office: 1,800 Days
• CPC Class Codes: G06F 21/32 using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06F 21/83 input devices, e.g. keyboards, mice or controllers thereof
• CPC Class Codes: G06F 2203/04104 Multi-touch detection in digitiser, i.e. details about the simultaneous detection of a plurality of touching locations, e.g. multiple fingers or pen and finger
• CPC Class Codes: G06F 2221/2101 Auditing as a secondary aspect
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0416 Control or interface arrangements specially adapted for digitisers
• CPC Class Codes: G06F 3/0426 tracking fingers with respect to a virtual keyboard projected or printed on the surface virtual keyboards on touch screens G06F3/04886
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus

Abstract

Devices and related methods are disclosed herein that generally involve detecting and interpreting gestures made by a user to generate user input information for use by a digital data processing system. In one embodiment, a device includes first and second sensors that observe a workspace in which user gestures are performed. The device can be set to a keyboard input mode, a number pad input mode, or a mouse input mode based on the positioning of the user'"'"'s hands. Subsequent gestures made by the user can be interpreted as keyboard inputs, mouse inputs, etc., using observed characteristics of the user'"'"'s hands and various motion properties of the user'"'"'s hands. These observed characteristics can also be used to implement a security protocol, for example by identifying authorized users by the anatomical properties of their hands or the behavioral properties exhibited by the user while gesturing.

Claims

1. A method of authenticating a user of a digital data processing system while entering data, comprising:

using at least one non-contact sensor to generate data indicative of one or more anatomical parameters of a user'"'"'s hands during movement of the user'"'"'s hands comprising a typing movement in a workspace;

determining typing gesture information indicative of one or more data manipulation gestures corresponding to keyboard input made by the user'"'"'s hands within the workspace based on said data indicative of one or more anatomical parameters of a user'"'"'s hands;

comparing the typing gesture information to predetermined gesture information to determine whether the user is an authorized user of the digital data processing system;

generating user keyboard input information based on the determined typing gesture information; and

manipulating data secured by the digital data processing system according to the user keyboard input information if the user is an authorized user.

2. The method of claim 1, wherein the gesture information comprises changes in configuration of the user'"'"'s hand.

3. The method of claim 1, wherein the gesture information comprises a speed, a cadence, or a style of the user'"'"'s hand movement.

4. The method of claim 1, further comprising repeating the determining and comparing steps each time a hand enters the workspace to determine whether the hand belongs to an authorized user of the digital data processing system.

5. The method of claim 1, further comprising repeating the determining and comparing steps to determine if the user is an authorized user during manipulation of data secured by the digital data processing system.

6. The system of claim 1, wherein the user input information generated by the processor further comprises at least one of pointing device input information and number pad input information.

7. The method of claim 1, wherein the gesture information comprises at least one of keyboard input information, pointing device input information, and number pad input information.

8. The method of claim 1, further comprising calculating changes in at least one of the one or more parameters of the user'"'"'s hands.

9. The method of claim 1, wherein the workspace comprises a three-dimensional space within the field of view of the at least one sensor.

10. The method of claim 1, further comprising setting a current operating mode from a plurality of operating modes based at least in part on at least one of a location of the user'"'"'s one or more hands, a gesture made by the user'"'"'s one or more hands, and a configuration of the user'"'"'s one or more hands.

11. The method of claim 1, further comprising:

determining at least one anatomical parameter of the user'"'"'s hands based on the data from the at least one sensor; and

comparing the determined anatomical parameter to a predetermined value to determine whether the user is an authorized user of the digital data processing system.

12. The method of claim 11, wherein the parameter comprises at least one of an anatomical geometry of a portion of the user, a color of the portion of the user, and a surface texture of the portion of the user.

13. The method of claim 1, further comprising detecting landmarks of the user'"'"'s hands.

14. The method of claim 13, wherein the landmarks detected by the processor comprise at least one of a finger, a finger segment, a finger shape, a finger joint, a finger nail, a skin surface contour, and a hand surface.

15. The method of claim 1, wherein the at least one sensor comprises a plurality of sensors positioned around the perimeter of the workspace, the workspace comprising a region framed by the plurality of sensors.

16. The method of claim 15, wherein the plurality of operating modes comprises a keyboard input mode, a pointing device input mode, a number pad input mode, a template-based input mode, and a custom pad input mode.

17. The method of claim 1, further comprising associating gestures made by the user with one or more input candidates.

18. The method of claim 17, wherein the input candidates comprise alphanumeric characters, punctuation marks, symbols, or functional elements.

19. The method of claim 1, further comprising:

identifying a plurality of anatomical landmarks of the user'"'"'s hand; and

determining locations of said landmarks within the workspace based on data generated by the at least one sensor;

comparing the data generated by the at least one sensor over time to generate a set of values indicative of changes in said landmarks; and

associating changes in the landmarks with user input information.

20. The method of claim 19, wherein comparing the data generated by the at least one sensor over time comprises measuring at least one of changes in distance of the landmarks relative to a starting position, changes in velocity of the landmarks, changes in acceleration of the landmarks, changes in an angular relationship of at least two landmarks, an angular displacement of an angle defined by a vertex and at least two landmarks, an angular velocity of an angle defined by a vertex and at least two landmarks, and an angular acceleration of an angle defined by a vertex and at least two landmarks.

21. The method of claim 19, wherein comparing the data generated by the at least one sensor over time comprises measuring changes in velocity of said landmarks.

22. The method of claim 19, wherein comparing the data generated by the at least one sensor over time comprises measuring changes in acceleration of said landmarks.

23. The method of claim 19, wherein comparing the data generated by the at least one sensor over time comprises measuring changes in an angular relationship of at least two landmarks.

24. The method of claim 19, wherein comparing the data generated by the at least one sensor over time comprises measuring an angular displacement of an angle defined by a vertex and at least two landmarks.

25. The method of claim 19, wherein comparing the data generated by the at least one sensor over time comprises measuring an angular velocity of an angle defined by a vertex and at least two landmarks, and the classification module associates such measurements with user input information.

26. The method of claim 19, wherein comparing the data generated by the at least one sensor over time comprises measuring an angular acceleration of an angle defined by a vertex and at least two landmarks.

27. The method of claim 19, wherein comparing the data generated by the at least one sensor over time comprises comparing at least one of a distance traveled, a velocity, and an acceleration of a particular landmark to at least one of a distance traveled, a velocity, and an acceleration of at least one other landmark.

28. A system for determining whether a user is an authorized user of a digital data processing system, comprising:

at least one non-contact sensor that detects one or more data manipulation gestures comprising typing movement corresponding to keyboard input made by the user'"'"'s hands during movement of the user'"'"'s hands within a workspace; and

a processor that compares the generated typing gesture information to predetermined gesture information stored in a storage medium, the processor determining whether the user is an authorized user based on a degree to which the generated typing gesture information matches the predetermined gesture information, the processor further configured to generate user keyboard input information based on the detected data manipulation typing gestures and to manipulate said data secured by the digital data processing system according to the user keyboard input information if the user is an authorized user.

29. The system of claim 28, wherein the processor detects landmarks of the user'"'"'s hands.

30. The system of claim 29, wherein the landmarks detected by the processor comprise at least one of a finger, a finger segment, a finger shape, a finger joint, a finger nail, a skin surface contour, and a hand surface.

31. The system of claim 28, wherein the processor further determines a physical characteristic of the user'"'"'s hands and compares the detected physical characteristic to predetermined characteristic information stored in the storage medium.

32. The system of claim 28, wherein the processor further determines whether the user is an authorized user prior to providing access to data secured by the digital data system.

33. The system of claim 32, wherein the processor is further configured to intermittently compare generated gesture information indicative of one or more data manipulation gestures to predetermined gesture information to determine whether the user is an authorized user during manipulation of data secured by the digital data processing system.

34. The system of claim 28, wherein the processor further determines a physical characteristic of the user and compares the detected physical characteristic to predetermined characteristic information stored in the storage medium, wherein the one or more parameters of the user comprise a size of a portion of the user, a color of the portion of the user, and a surface texture of the portion of the user.

35. The system of claim 28, wherein the at least one sensor generates data indicative of one or more parameters of a user in a workspace and wherein the processor calculates changes in at least one of the one or more parameters of the user.

36. The system of claim 28, wherein the workspace comprises a three-dimensional space within the field of view of the at least one sensor.

37. The system of claim 28, wherein the at least one sensor comprises a plurality of sensors positioned around the perimeter of the workspace, the workspace comprising a region framed by the plurality of sensors.

38. The system of claim 28, wherein the processor associates gestures made by the agent with one or more input candidates.

39. The system of claim 38, wherein the input candidates comprise alphanumeric characters, punctuation marks, symbols, or functional elements.

40. The system of claim 28, wherein the processor is configured to set a current operating mode based at least in part on at least one of a location of the user'"'"'s one or more hands, a gesture made by the user'"'"'s one or more hands, and a configuration of the user'"'"'s one or more hands.

41. The system of claim 40, wherein the plurality of operating modes comprises a keyboard input mode, a pointing device input mode, a number pad input mode, a template-based input mode, and a custom pad input mode.

42. The system of claim 28, wherein the processor comprises:

a user profile module that identifies a plurality of anatomical landmarks of a user'"'"'s hand and determines locations of said landmarks within the workspace based on data generated by the at least one sensor;

a motion detection module that compares the data generated by the at least one sensor over time to generate a set of values indicative of changes in said landmarks; and

a classification module that associates changes in the landmarks with user input information.

43. The system of claim 42, wherein the motion detection module measures at least one of changes in distance of the landmarks relative to a starting position, changes in velocity of the landmarks, changes in acceleration of the landmarks, changes in an angular relationship of at least two landmarks, an angular displacement of an angle defined by a vertex and at least two landmarks, an angular velocity of an angle defined by a vertex and at least two landmarks, and an angular acceleration of an angle defined by a vertex and at least two landmarks, and wherein the classification module associates such measurements with user input information.

44. The system of claim 42, wherein the motion detection module compares at least one of a distance traveled, a velocity, and an acceleration of a particular landmark to at least one of a distance traveled, a velocity, and an acceleration of at least one other landmark, and wherein the classification module generates user input information based on said comparison.