Handheld computer systems and techniques for character and command recognition related to human movements

• US 9,174,123 B2
• Filed: 07/08/2010
• Issued: 11/03/2015
• Est. Priority Date: 11/09/2009
• Status: Active Grant

• Current Assignee: Invensense Incorporated (TDK Corporation)
• Original Assignee: Invensense Incorporated (TDK Corporation)
• Inventors: Nasiri, Steve, Sachs, David, Castro, Alex, Li, Richard, Gu, Anjia, Lin, Shang-Hung, Nakayama, Takashi
• Primary Examiner(s): BROWN, VERNAL U
• Application Number:
  US12/832,707
• Publication Number:
  US 20120007713A1
• Time in Patent Office: 1,944 Days
• Field of Search: 340/5.81, 340/12.22, 340/5.61, 341/20, 341/176, 382/185, 702/150
• US Class Current: 1/1
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/23 for interfacing with the game device, e.g. specific interfaces between game controller and console
• CPC Class Codes: A63F 13/24 Constructional details thereof, e.g. game controllers with detachable joystick handles
• CPC Class Codes: A63F 13/30 Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2009/2447 Motion detector
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/204 the platform being a handheld device
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 9/24 Electric games; Games using electronic circuits not otherwise provided for video games A63F13/00; computerized gaming systems G07F17/32
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 1/3215 Monitoring of peripheral devices
• CPC Class Codes: G06F 1/3287 by switching off individual functional units in the computer system
• CPC Class Codes: G06F 21/32 using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06F 2221/2105 Dual mode as a secondary aspect
• CPC Class Codes: G06F 3/017 : Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0233 : Character input methods
• CPC Class Codes: G06F 3/0346 : with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 : Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/167 : Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06V 30/347 : Sampling; Contour coding; Stroke extraction
• CPC Class Codes: G06V 40/28 : Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G08C 2201/32 : Remote control based on movements, attitude of remote control device
• CPC Class Codes: H04M 1/67 : by electronic means
• CPC Class Codes: H04M 1/724 : User interfaces specially adapted for cordless or mobile telephones
• CPC Class Codes: H04M 1/72403 : with means for local support of applications that increase the functionality
• CPC Class Codes: H04M 2250/12 : including a sensor for measuring a physical value, e.g. temperature or motion telephones including GPS signal receivers H04M2250/10
• CPC Class Codes: H04M 2250/22 : including a touch pad, a touch sensor or a touch detector digitizers for touch screens G06F3/041
• CPC Class Codes: H04W 12/06 : Authentication
• CPC Class Codes: H04W 12/68 : Gesture-dependent or behaviour-dependent
• CPC Class Codes: H04W 4/027 : using movement velocity, acceleration information
• CPC Class Codes: H04W 52/0254 : detecting a user operation or a tactile contact or a motion of the device
• CPC Class Codes: H04W 52/0264 : by selectively disabling software applications
• CPC Class Codes: H04W 52/027 : by controlling a display operation or backlight unit
• CPC Class Codes: H04W 52/0274 : by switching on or off the equipment or parts thereof
• CPC Class Codes: H04W 88/02 : Terminal devices
• CPC Class Codes: Y02D 10/00 : Energy efficient computing, e.g. low power processors, power management or thermal management
• CPC Class Codes: Y02D 30/70 : in wireless communication networks

Abstract

Systems and methods for human hand gesture recognition through a training mode and a recognition mode are disclosed. In the training mode, a user can move a handheld device with a hand gesture intended to represent a command. Sensors within the handheld device can record raw data, which can be processed to obtain a set of values corresponding to a set of discrete features, which is stored in a database and associated with the intended command. The process is repeated for various hand gestures representing different commands. In the recognition mode, the user can move the handheld device with a hand gesture. A computer system can compare a set of values corresponding to a set of discrete features derived from the hand gesture with the sets of values stored in the database, select a command with the closest match and displays and/or executes the command.

Claims

1. A handheld device, comprising:

inertial sensors comprising a gyroscope that generates gyroscope data representing a trajectory of a movement of the handheld device and an accelerometer that generates accelerometer data representing a tilt of the handheld device relative to earth'"'"'s gravity; and

a processing component that;

determines the trajectory of the movement of the handheld device by combining the gyroscope data representing the trajectory of the movement of the handheld device and the accelerometer data representing the tilt of the handheld device relative to earth'"'"'s gravity;

converts, via a training mode, a first portion of the data into a first set of values corresponding to a first plurality of discrete features,

associates the first set of values with a character, stores the first set of values and information representing the character in a training database,

converts, via a recognition mode, a second portion of the data into a second set of values corresponding to a second plurality of discrete features, wherein a discrete feature of the first plurality of discrete features or the second plurality of discrete features includes at least one of an angle of the trajectory, an inverse tangent of a change in a pitch associated with the movement divided by a change in a yaw associated with the movement, a location of a crossing point associated with the character, or a number of times in which the trajectory changes direction; and

in response to a determination that the second set of values matches the first set of values with respect to a defined condition, displays the character.

2. The handheld device of claim 1, wherein the inertial sensors further comprise a compass.

3. The handheld device of claim 1, wherein the processing component stores at least a portion of the second set of values in a data store.

4. The handheld device of claim 1, further comprising:

a display for displaying the trajectory of the movement.

5. The handheld device of claim 1, wherein the character is associated with at least one of a letter from an alphabet, a roman numeral, a signature associated with an identity of a user, or an authentication signal.

6. The handheld device of claim 1, wherein the processing component executes a command associated with an application program associated with the handheld device or an operating system associated with the handheld device.

7. The handheld device of claim 6, wherein the command triggers an event in a video game.

8. The handheld device of claim 6, wherein the command opens or closes the application program or controls a set of applications running within the operating system.

9. The handheld device of claim 1, wherein the processing component matches, via the recognition mode, the first set of values with the second set of values using at least one of a look-up table, a Hidden Markov Model (HMM), a neural network, or a support vector machine.

10. The handheld device of claim 1, wherein the processing component matches, via the recognition mode, the first set of values with the second set of values according to one or more segments.

11. The handheld device of claim 10, wherein the one or more segments represent a letter and the first set of values or the second set of values represents a word.

12. The handheld device of claim 1, wherein the processing component compares, via the determination, the first set of values with the second set of values.

13. The handheld device of claim 1, wherein the processing component executes a command in response to the determination that the second set of values matches the first set of values.

14. A method for gesture recognition, comprising:

obtaining first training data related to a first trajectory of a first movement of a handheld device using a gyroscope and an accelerometer;

determining the first trajectory in response to integrating a portion of the first training data associated with the gyroscope with another portion of the first training data associated with the accelerometer representing a first tilt of the handheld device relative to earth'"'"'s gravity;

converting the first training data into first values corresponding to a first set of a plurality of discrete features;

wherein a discrete feature of the plurality of discrete features includes at least one of an angle of the trajectory, an inverse tangent of a change in a pitch associated with the movement divided by a change in a yaw associated with the movement, a location of a crossing point associated with the character, or a number of times in which the trajectory changes direction;

associating the first values with a first character;

storing the first values and information representing the first character in a data store; and

displaying the first character based on the first values.

15. The method of claim 14, wherein the obtaining the first training data comprises:

obtaining the first training data using the gyroscope, the accelerometer, and a sensor comprising a compass.

16. The method of claim 14, wherein the plurality of discrete features represents a number of direction changes.

17. The method of claim 14, wherein the plurality of discrete features relate to an angular movement of the handheld device.

18. The method of claim 14, further comprising:

learning a character with a plurality of strokes according to a plurality of vectors.

19. The method of claim 14, further comprising:

executing a first command associated with the first values.

20. The method of claim 14, further comprising:

obtaining second training data related to a second trajectory of a second movement of the handheld device using the gyroscope and the accelerometer;

determining the second trajectory in response to integrating a portion of the second training data associated with the gyroscope with another portion of the second training data associated with the accelerometer representing a second tilt of the handheld device relative to earth'"'"'s gravity;

converting the second training data into second values corresponding to a second set of the plurality of discrete features;

associating the second values with a second character;

storing the second values and information representing the second character in the data store;

obtaining trajectory data related to a third trajectory of a third movement of the handheld device using the gyroscope and the accelerometer;

converting the trajectory data into third values corresponding to a third set of the plurality of discrete features;

matching the third values with the first values or the second values; and

displaying the first character or the second character in response to the matching.

21. A handheld device, comprising:

a gyroscope;

an accelerometer;

a processing component that receives gyroscope data from the gyroscope representing a trajectory of a movement of the handheld device and receives accelerometer data from the accelerometer representing a tilt of the handheld device relative to earth'"'"'s gravity;

a programmable module coupled to the processing component that includes a look up table that associates a plurality of discrete features comprising the trajectory of the movement of the handheld device and the tilt of the handheld device with a plurality of characters; and

a data store for storing a plurality of sets of values corresponding to the plurality of discrete features, wherein the processing component converts a first portion of the data into a first set of values corresponding to first features of the plurality of discrete features, associates, utilizing the look up table, the first set of values with a character of the plurality of characters based on the first features, stores the first set of values and information representing the character in the data store, converts a second portion of the data into a second set of values corresponding to second features of the plurality of discrete features, matches, using the data store, the second set of values to one of the plurality of sets of values, and displays the character using the information, wherein a discrete feature of the plurality of discrete features includes at least one of an angle of the trajectory, an inverse tangent of a change in a pitch associated with the movement divided by a change in a yaw associated with the movement, a location of a crossing point associated with the character, or a number of times in which the trajectory changes direction.

22. The handheld device of claim 21, further comprising:

a display for displaying the trajectory in real time and displaying the character based on a third set of values corresponding to the trajectory.

23. The handheld device of claim 21, wherein the processing component executes commands, and wherein the look up table associates the plurality of discrete features with the commands.

24. The handheld device of claim 21, wherein the look up table associates the plurality of discrete features with the plurality of characters utilizing sets of values corresponding to rows of the look up table.

25. A method, comprising:

obtaining first information from a gyroscope based on a first trajectory of a first movement of a handheld device;

storing the first information and authentication information representing an identity of a user in a training data store;

obtaining second information from the gyroscope based on a second trajectory of a second movement of the handheld device;

in response to determining an orientation of the handheld device utilizing a magnetometer, modifying the first information and the second information according to the orientation;

wherein the first trajectory and the second trajectory are connected to a set of values corresponding to a set of selected discrete features wherein a discrete feature of the plurality of discrete features includes at least one of an angle of the trajectory, an inverse tangent of a change in a pitch associated with the movement divided by a change in a yaw associated with the movement, a location of a crossing point associated with the character, or a number of times in which the trajectory changes direction;

in response to the modifying, comparing the second information with the first information; and

authenticating the identity of the user in response to the comparing of the second information with the first information.

26. The method of claim 25, wherein the first movement or the second movement represents a signature of the user.

27. The method of claim 25, further comprising:

generating a sound within a predetermined time period before or after the second movement;

detecting the sound using a microphone embedded in the handheld device; and

authenticating the user in response to the comparing and the detecting.

28. A handheld device, comprising:

a first processor;

a programmable module comprising a look up table that associates a plurality of discrete features representing trajectories of movements of the handheld device with a plurality of commands;

wherein a discrete feature of the plurality of discrete features includes at least one of an angle of the trajectory, an inverse tangent of a change in a pitch associated with the movement divided by a change in a yaw associated with the movement, a location of a crossing point associated with the character, or a number of times in which the trajectory changes direction; and

a gyroscope, wherein the first processor receives at least a portion of first information related to a first movement of the handheld device from the gyroscope, determines the first movement based on the portion, consults the look up table and matches the first movement to a command from the plurality of commands, and wherein a first component coupled to the first processor enters a first active mode of operation from a sleep mode of operation based on the command.

29. The handheld device of claim 28, further comprising:

a second processor coupled to the gyroscope for processing the first information.

30. The handheld device of claim 29, wherein the second processor receives the first information from the gyroscope, and wherein

the first processor enters a second active mode of operation in response to receiving another portion of the first information from the second processor.

31. The handheld device of claim 28, further comprising:

an accelerometer for providing second information related to the first movement of the handheld device, wherein

the first component enters the first active mode of operation based on the second information.

32. The handheld device of claim 28, further comprising a smartphone.

33. The handheld device of claim 28, further comprising:

a camera; and

a display, wherein the camera switches from the sleep mode to the first active mode based on the first movement, wherein the camera uses more electrical power during the first active mode than during the sleep mode, and wherein the display is not switched from the sleep mode to the first active mode based on the first movement.

34. The handheld device of claim 28, further comprising a second processor coupled to the gyroscope for causing the first processor to enter into the first active mode of operation based on the first movement.

35. The handheld device of claim 28, further comprising:

a second processor that facilitates switching the microprocessor from the sleep mode to the first active mode based on the first movement,

wherein the first processor comprises a microprocessor,

wherein the first component comprises a camera,

wherein the microprocessor facilitates switching the camera from the sleep mode to the first active mode based on the first movement,

wherein the microprocessor facilitates switching the display from the sleep mode to the first active mode based on a second movement of the handheld device and not based on the first movement, and

wherein the first active mode of operation is associated with a first consumption of electrical power that is greater than a second consumption of electrical power associated with the sleep mode of operation.

36. The handheld device of claim 28, wherein the look up table associates the plurality of movements with the plurality of commands using rows corresponding to sets of values.

37. A method, comprising:

receiving, by a system comprising a processor, information from a gyroscope regarding a movement of the system;

consulting a look up table that associates discrete features representing trajectories of movements of the system with a plurality of commands;

wherein a discrete feature of the first plurality of discrete features or the second plurality of discrete features includes at least one of an angle of the trajectory, an inverse tangent of a change in a pitch associated with the movement divided by a change in a yaw associated with the movement, a location of a crossing point associated with the character, or a number of times in which the trajectory changes direction;

matching the movement to a command from the plurality of commands; and

switching, based on the command, a component of the system from a sleep mode of operation to an active mode of operation.

38. The method of claim 37, wherein the information further comprises information received from an accelerometer.

39. The method of claim 37, wherein the component comprises a camera.

40. The method of claim 37, wherein the component comprises a display.

41. The method of claim 37, wherein the component comprises a microphone.

42. The method of claim 37, wherein the movement corresponds to a touch of the system.

43. The method of claim 37, further comprising:

preventing another component of the system from switching from the sleep mode to the active mode based on the movement.

44. The method of claim 37, wherein the plurality of movements are associated with the plurality of commands based on sets of values corresponding to rows of the look up table.

45. A method, comprising:

detecting a first position of a mobile phone using a gyroscope;

consulting a look up table and matching the first position with a first command corresponding to a first column of the look up table representing a first trajectory of the mobile phone;

causing, based on the first command, a component of the mobile phone to enter into a sleep mode of operation;

detecting a second position of the mobile phone by using the gyroscope;

consulting the look up table and matching the second position with a second command corresponding to a second column of the look up table representing a second trajectory of the mobile phone;

causing, based on the second command, the component of the mobile phone to enter into an active mode of operation.

46. The method of claim 45, wherein the detecting the first position further comprises detecting the first position using an accelerometer.