Gesture Recognition Devices And Methods

• US 20180181208A1
• Filed: 01/12/2018
• Published: 06/28/2018
• Est. Priority Date: 02/24/2012
• Status: Active Grant

• Current Assignee: Thomas J. Moscarillo
• Original Assignee: Thomas J. Moscarillo
• Inventors: Moscarillo, Thomas J.
• Granted Patent: US 11,009,961 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 21/32 using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06F 21/83 input devices, e.g. keyboards, mice or controllers thereof
• CPC Class Codes: G06F 2203/04104 Multi-touch detection in digitiser, i.e. details about the simultaneous detection of a plurality of touching locations, e.g. multiple fingers or pen and finger
• CPC Class Codes: G06F 2221/2101 Auditing as a secondary aspect
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0416 Control or interface arrangements specially adapted for digitisers
• CPC Class Codes: G06F 3/0426 tracking fingers with respect to a virtual keyboard projected or printed on the surface virtual keyboards on touch screens G06F3/04886
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus

Abstract

Devices and related methods are disclosed herein that generally involve detecting and interpreting gestures made by a user to generate user input information for use by a digital data processing system. In one embodiment, a device includes first and second sensors that observe a workspace in which user gestures are performed. The device can be set to a keyboard input mode, a number pad input mode, or a mouse input mode based on the positioning of the user'"'"'s hands. Subsequent gestures made by the user can be interpreted as keyboard inputs, mouse inputs, etc., using observed characteristics of the user'"'"'s hands and various motion properties of the user'"'"'s hands. These observed characteristics can also be used to implement a security protocol, for example by identifying authorized users by the anatomical properties of their hands or the behavioral properties exhibited by the user while gesturing.

Claims

1. An input device for a digital data processing system, comprising:

at least one sensor that observes a workspace and generates data indicative of one or more parameters of an input agent within the workspace; and

a processor that identifies gestures made by the agent from the data generated by the at least one sensor and that generates user input information based on the identified gestures.

2. The device of claim 1, wherein the user input information generated by the processor comprises at least one of keyboard input information, pointing device input information, and number pad input information.

3. The device of claim 1, wherein the at least one sensor comprises a single sensor configured to generate data indicative of the three-dimensional position and orientation of the agent.

4. The device of claim 3, wherein the sensor comprises a structured light sensor.

5. The device of claim 4, wherein the structured light sensor comprises a structured infrared light sensor.

6. The device of claim 3, wherein the sensor utilizes time-of-flight detection.

7. The device of claim 1, wherein the at least one sensor comprises a first sensor that observes the workspace from a first perspective and a second sensor, spaced a distance apart from the first sensor, that observes the workspace from a second perspective different from the first perspective.

8. The device of claim 7, wherein the first sensor generates data indicative of the three-dimensional position and orientation of the agent, and the second sensor comprises an optical imager.

9. The device of claim 1, wherein the processor detects landmarks of the agent.

10. The device of claim 9, wherein the agent comprises a hand and the landmarks detected by the processor comprise at least one of a finger, a finger segment, a finger shape, a finger joint, a finger nail, a skin surface contour, and a hand surface.

11. The device of claim 1, wherein the one or more parameters of the agent comprise a size of the agent, a color of the agent, a surface texture of the agent, a position of the agent, and an orientation of the agent.

12. The device of claim 1, wherein the processor calculates changes in at least one of the one or more parameters of the agent.

13. The device of claim 1, wherein the workspace comprises a three-dimensional space within the field of view of the at least one sensor.

14. The device of claim 1, wherein the workspace comprises a surface on which the input device is positioned.

15. The device of claim 1, wherein the at least one sensor comprises a plurality of sensors positioned around the perimeter of the workspace, the workspace comprising a region framed by the plurality of sensors.

16. The device of claim 1, wherein the processor associates gestures made by the agent with one or more input candidates.

17. The device of claim 16, wherein the input candidates comprise alphanumeric characters, punctuation marks, symbols, or functional elements.

18. The device of claim 1, wherein the agent comprises one or more hands, each of the one or more hands comprising a plurality of fingers and wherein the input device is operable in a plurality of operating modes, the processor being configured to set a current operating mode based at least in part on at least one of a location of the one or more hands, a gesture made by the one or more hands, and a configuration of the one or more hands.

19. The device of claim 18, wherein the plurality of operating modes comprises a keyboard input mode, a pointing device input mode, a number pad input mode, a template-based input mode, and a custom pad input mode.

20. The device of claim 19, wherein the processor sets the current operating mode to the pointing device input mode when only one of the plurality of fingers is extended.

21. The device of claim 19, wherein the processor sets the current operating mode to the keyboard input mode when the processor detects a threshold number of digits within the workspace.

22. The device of claim 19, wherein the processor sets the current operating mode to the number pad input mode when the processor detects movement of one of the one or more hands to a position laterally offset from a home position or when the processor detects that only one of the one or more hands is presented to the workspace.

23. The device of claim 1, further comprising a configuration module that assigns particular user input information to a particular gesture such that the processor generates the particular user input information when the particular gesture is detected.

24. The device of claim 23, wherein the configuration module displays a gesture strength indicator based on a degree to which the particular gesture can be reliably detected.

25. An input device for a digital data processing system, comprising:

at least one sensor for generating data indicative of a workspace; and

a processor comprising;

a user profile module that identifies a plurality of anatomical landmarks of a user'"'"'s hand and determines locations of said landmarks within the workspace based on data generated by the at least one sensor;

a motion detection module that compares the data generated by the at least one sensor over time to generate a set of values indicative of changes in said landmarks; and

a classification module that associates changes in the landmarks with user input information.

26. The input device of claim 25, wherein the motion detection module measures at least one of changes in distance of the landmarks relative to a starting position, changes in velocity of the landmarks, changes in acceleration of the landmarks, changes in an angular relationship of at least two landmarks, an angular displacement of an angle defined by a vertex and at least two landmarks, an angular velocity of an angle defined by a vertex and at least two landmarks, and an angular acceleration of an angle defined by a vertex and at least two landmarks, and wherein the classification module associates such measurements with user input information.

27. The input device of claim 25, wherein the motion detection module measures at least changes in velocity of landmarks and the classification module associates such measurements with user input information.

28. The input device of claim 25, wherein the motion detection module measures at least changes in acceleration of landmarks and the classification module associates such measurements with user input information.

29. The input device of claim 25, wherein the motion detection module measures at least changes in an angular relationship of at least two landmarks and the classification module associates such measurements with user input information.

30. The input device of claim 25, wherein the motion detection module measures at least an angular displacement of an angle defined by a vertex and at least two landmarks, and the classification module associates such measurements with user input information.

31. The input device of claim 25, wherein the motion detection module measures at least an angular velocity of an angle defined by a vertex and at least two landmarks, and the classification module associates such measurements with user input information.

32. The input device of claim 25, wherein the motion detection module measures at least an angular acceleration of an angle defined by a vertex and at least two landmarks, and the classification module associates such measurements with user input information.

33. The input device of claim 25, wherein the motion detection module compares at least one of a distance traveled, a velocity, and an acceleration of a particular landmark to at least one of a distance traveled, a velocity, and an acceleration of at least one other landmark, and wherein the classification module generates user input information based on said comparison.

34. The input device of claim 25, further comprising a template identification module that determines the presence of a template within the workspace.

35. The input device of claim 34, wherein the template is identified based on at least one marking on the template.

36. The input device of claim 34, wherein the template comprises an object that can be manipulated by the user, and wherein the object is identified based on at least one characteristic of the object.

37. The input device of claim 34, wherein the classification module associates changes in landmarks relative to the template with user input information associated with the template.

38-51. -51. (canceled)