HANDHELD COMPUTER SYSTEMS AND TECHNIQUES FOR CHARACTER AND COMMAND RECOGNITION RELATED TO HUMAN MOVEMENTS

• US 20120007713A1
• Filed: 07/08/2010
• Published: 01/12/2012
• Est. Priority Date: 11/09/2009
• Status: Active Grant

• Current Assignee: Invensense Incorporated (TDK Corporation)
• Original Assignee: Invensense Incorporated (TDK Corporation)
• Inventors: Sachs, David, Lin, Shang-Hung, Nasiri, Steve, Castro, Alex, Li, Richard, Gu, Anjia, Nakayami, Takashi
• Granted Patent: US 9,174,123 B2
• Time in Patent Office: Days
• US Class Current: 340/5.81
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/23 for interfacing with the game device, e.g. specific interfaces between game controller and console
• CPC Class Codes: A63F 13/24 Constructional details thereof, e.g. game controllers with detachable joystick handles
• CPC Class Codes: A63F 13/30 Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2009/2447 Motion detector
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/204 the platform being a handheld device
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 9/24 Electric games; Games using electronic circuits not otherwise provided for video games A63F13/00; computerized gaming systems G07F17/32
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 1/3215 Monitoring of peripheral devices
• CPC Class Codes: G06F 1/3287 by switching off individual functional units in the computer system
• CPC Class Codes: G06F 21/32 using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06F 2221/2105 Dual mode as a secondary aspect
• CPC Class Codes: G06F 3/017 : Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0233 : Character input methods
• CPC Class Codes: G06F 3/0346 : with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 : Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/167 : Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06V 30/347 : Sampling; Contour coding; Stroke extraction
• CPC Class Codes: G06V 40/28 : Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G08C 2201/32 : Remote control based on movements, attitude of remote control device
• CPC Class Codes: H04M 1/67 : by electronic means
• CPC Class Codes: H04M 1/724 : User interfaces specially adapted for cordless or mobile telephones
• CPC Class Codes: H04M 1/72403 : with means for local support of applications that increase the functionality
• CPC Class Codes: H04M 2250/12 : including a sensor for measuring a physical value, e.g. temperature or motion telephones including GPS signal receivers H04M2250/10
• CPC Class Codes: H04M 2250/22 : including a touch pad, a touch sensor or a touch detector digitizers for touch screens G06F3/041
• CPC Class Codes: H04W 12/06 : Authentication
• CPC Class Codes: H04W 12/68 : Gesture-dependent or behaviour-dependent
• CPC Class Codes: H04W 4/027 : using movement velocity, acceleration information
• CPC Class Codes: H04W 52/0254 : detecting a user operation or a tactile contact or a motion of the device
• CPC Class Codes: H04W 52/0264 : by selectively disabling software applications
• CPC Class Codes: H04W 52/027 : by controlling a display operation or backlight unit
• CPC Class Codes: H04W 52/0274 : by switching on or off the equipment or parts thereof
• CPC Class Codes: H04W 88/02 : Terminal devices
• CPC Class Codes: Y02D 10/00 : Energy efficient computing, e.g. low power processors, power management or thermal management
• CPC Class Codes: Y02D 30/70 : in wireless communication networks

Abstract

Systems and methods for human hand gesture recognition through a training mode and a recognition mode are disclosed. In the training mode, a user can move a handheld device with a hand gesture intended to represent a command. Sensors within the handheld device can record raw data, which can be processed to obtain a set of values corresponding to a set of discrete features, which is stored in a database and associated with the intended command. The process is repeated for various hand gestures representing different commands. In the recognition mode, the user can move the handheld device with a hand gesture. A computer system can compare a set of values corresponding to a set of discrete features derived from the hand gesture with the sets of values stored in the database, select a command with the closest match and displays and/or executes the command.

Claims

1. A handheld device, comprising:

inertial sensors including a gyroscope and an accelerometer for providing data related to a trajectory of a movement of the handheld device; and

a processor coupled to the handheld device and operational in a training mode and a recognition mode,

wherein in the training mode, the processor receives the data from the inertial sensors, converts the data into a first set of values corresponding to a first plurality of discreet features, associate the first set of values with a command or a character, and stores the first set of values and the command or character in a training database, and

wherein in the recognition mode, the processor receives the data from the inertial sensors, converts the data into a second set of values corresponding to a plurality of second discreet features, compares the second set of values to a plurality of training sets of values in the training device including the second set of values, matches the first set of values to the second set of values, and executes the command or displaying the character.

2. The handheld device of claim 1, wherein the inertial sensors further comprise a compass.

3. The handheld device of claim 1, wherein in the training mode, the processor is configured to populate the database with the plurality of training sets of values.

4. The handheld device of claim 1, further comprising:

a display for displaying the trajectory of the movement in real time.

5. The handheld device system of claim 1, wherein the processor is located remotely from the handheld device, and the processor for communicating with the handheld device over a network by using a cloud computing technique.

6. The handheld device of claim 1, wherein a discrete feature of the first plurality of discrete features or the second plurality if discrete features is selected from the group consisting of:

(i) the angle of the trajectory at a point in time, (ii) the inverse tangent of a change in pitch divided by a change in yaw, (iii) the pitch component, (iv) the yaw component, (v) the rate of change of the trajectory, (vi) the location of a crossing point, and (vii) the number of times within a trajectory in which the trajectory changes direction.

8. The handheld device of claim 1, wherein the character is selected from the group consisting of a letter from the alphabet, a roman numeral, a user'"'"'s signature and an authentication signal.

9. The handheld device of claim 1, wherein the command triggers an event in an application program or in an operating system of the handheld device.

10. The handheld device of claim 1, wherein the command triggers an event in a video game.

11. The handheld device of claim 1, wherein the command opens or closes an application program or controls a set of applications running within an operating system.

12. The handheld device of claim 1, wherein, in the recognition mode, the processor is configured to match the first set of values with the second set of values using an algorithmic device selected from the group consisting of look-up tables, Hidden Markov Models (HMM), neural networks, and support vector machines.

13. The handheld device of claim 1, wherein, in the recognition mode, the processor is configured to match the first set of values with the second set of values according to one or more segments.

14. The handheld device of claim 13, wherein the one or more segments represent a letter and the first set of values or the second set of values represents a word.

7. (canceled)

15. A method for gesture recognition comprising:

entering a training mode, wherein the training mode comprises;

obtaining first training data related to a first trajectory of a first movement of a handheld device by using a gyroscope;

converting the first training data into first values corresponding to a first set of a plurality of discrete features;

associating the first values with a first command or a first character;

saving the first values and the first command or the first character in a database;

obtaining second training data related to a second trajectory of a second movement of the handheld device by using the gyroscope;

converting the second training data into second values corresponding to a second set of the plurality of discrete features;

associating the second values with a second command or a second character; and

saving the second values and the second command or second character in the database; and

entering a recognition mode, comprising;

obtaining trajectory data related to a third trajectory of a third movement of the handheld device by using the gyroscope;

converting the third trajectory data into third values corresponding to a third set of the plurality of discrete features;

comparing the third values with the first values and the second values stored in the database;

matching the third values with the first values or the second values; and

displaying the first character or the second character or executing the first command or the second command based on the matching, wherein

the first data, the second data, and the trajectory data are related to rotational and linear movements of the handheld device.

16. The method of claim 15, wherein the obtaining the first data, obtaining the second data, or obtaining the trajectory data further comprises:

obtaining the first data, obtaining the second data, or obtaining the trajectory data by using the gyroscope and a sensor selected from the group consisting of a accelerometer and a compass.

17. The method of claim 15, wherein the plurality of discrete features include a number of direction changes.

18. The method of claim 15, wherein the plurality of discrete features relate to an angular movement of the handheld device.

19. The method of claim 15, wherein the training mode further comprises:

learning a character with a plurality of strokes according to a plurality of vectors.

20. A handheld device, comprising:

a gyroscope;

an accelerometer;

a compass, wherein the gyroscope, the accelerometer and the compass provide data related to a trajectory of a movement of the handheld device;

a processor coupled to the gyroscope, the accelerometer and the compass configured to receive the data from the gyroscope, the accelerometer and the compass;

a programmable module coupled to the processor that includes a look up table that associates a plurality of discreet features with a plurality of characters or commands; and

a memory coupled to the processor including a database that stores a plurality of sets of values corresponding to the plurality of discrete features associated with the plurality of characters or commands;

wherein

the processor is operable in a training mode to populate the database by receiving a first data from the gyroscope, the accelerometer, and the compass, converting the first data into a first set of values corresponding to a first plurality of discreet features, consulting the look up table and associating the first set of values with a first command or a character based on the first plurality of discreet features, and storing the first set of values and the command or character in the database, and

the processor is operable in a recognition mode to determine a tilt of the handheld device, receive a second data from the gyroscope, the accelerometer, and the compass, convert the second data into a second set of values corresponding to a second plurality of discreet features based on the tilt of the handheld device, matching the second set of values to one of the plurality of sets of values in the database, and displaying the associated character or executing the associated command.

21. The handheld device of claim 20, further comprising:

a display for displaying the trajectory in real time and displaying the character associated with a set of values corresponding to the trajectory.

22. A method for authentication a user comprising:

obtaining a first information from a gyroscope related to a first trajectory of a first movement of a handheld device;

populating a training database with the first information from the first trajectory and authentication information for a user;

obtaining a second information from the gyroscope related to a second trajectory of a second movement of the handheld device;

comparing the second information with the database; and

authenticating the user based on the comparison.

23. The method of claim 22, wherein the first movement or the second movement represents the user'"'"'s signature.

24. The method of claim 22, further comprising:

causing a sound to occur within a predetermined time period before or after the second movement;

detecting the sound by using a microphone embedded in the handheld device; and

authenticating the user based on the comparison and the sound.

25. A handheld device, comprising:

a first processor;

a first component coupled to the first processor, wherein the first component is in a sleep mode;

a first programmable module that stores a look up table that associates a plurality of movements with a plurality of commands; and

a gyroscope that provides first information related to a first movement of the handheld device to the first processor, wherein

the first processor receives the first information from the gyroscope, determines the first movement, consults the loop up table and matches the first movement to a first command from the plurality of commands, and causes the first component to enter into an active mode of operation from the sleep mode in response to the first movement.

27. The handheld device of claim 25, wherein:

the first processor includes a microprocessor; and

a second processor coupled to the gyroscope for processing the information provided by the gyroscope.

28. The handheld device of claim 27, wherein the second processor

receives the first information from the gyroscope;

causes the first processor to enter into a second active mode of operation in response to receiving the first information; and

provides the first information to the first processor.

29. The handheld device of claim 25, further comprising:

an accelerometer for providing second information related to the first movement of the handheld device, wherein

the first processor causes the first component to enter into the active mode of operation based on the first information and the second information.

30. The handheld device of claim 25, wherein the handheld device includes a smart phone.

31. The handheld device of claim 25, further comprising:

the first component including a camera;

a second component including a display; and

the first processor for causing the camera to switch from a sleep mode to the active mode in response to the first movement;

wherein,

when camera uses more electrical power during the active mode than the sleep mode; and

the display is not switched from a sleep mode to the an active mode in response to the first movement.

32. The handheld device of claim 25, further comprising a second processor coupled to the gyroscope for causing the first processor to enter into an active mode of operation in response to the first movement.

33. The handheld device of claim 25, further comprising:

the first processor including a microprocessor;

the first component including a camera;

the microprocessor coupled to a plurality of components including the camera, a display, a microphone, a network browser application program and a keypad;

the microprocessor and the plurality of components being set to the sleep mode of operation;

a second processor coupled to the gyroscope for processing the first information provided by the gyroscope;

the second processor for causing the microprocessor to switch from the sleep mode to the active mode of operation in response to the first movement; and

the microprocessor for causing the camera to switch from the sleep mode to the active mode of operation in response to the first movement;

wherein,

the microprocessor for causing the display to switch from the sleep mode to the active mode of operation in response to a second movement and not in response to the first movement; and

the active mode of operation requires more electrical power than the sleep mode of operation.

26. (canceled)

34. A method for managing power for a handheld device, comprising:

receiving first information from a gyroscope regarding a first movement of a handheld device at a microprocessor;

switching a first processor associated with the microprocessor from a sleep mode of operation to an active mode of operation;

consulting a look up table comprising a plurality of movements associated with a plurality of commands;

matching the first movement to a first command from the plurality of commands;

switching a first component to an active mode of operations in response to the command, wherein

the command is a trigger to switch from the sleep mode to the active mode.

35. The method of claim 34, wherein the first information further comprises information from an accelerometer.

37. The method of claim 35, wherein the first component is a camera included in the handheld device.

38. The method of claim 35, wherein the first component is a.

39. The method of claim 35, wherein the first component is a microphone.

40. The method of claim 39, wherein the movement is caused by a user tapping the handheld device.

41. The method of claim 35, further comprising:

not causing a second component to switch to an active mode of operation in response to the movement.

36. (canceled)

42. A method for managing power for a mobile phone, comprising:

detecting a first position of a mobile phone by using a gyroscope;

consulting a look up table and matching the first position with a first command;

causing a component of the mobile phone to enter into a sleep mode of operation in response to the first command;

detecting a second position of the mobile phone by using the gyroscope;

consulting the look up table and matching the second position with a second command;

causing the component of the mobile phone to enter into an active mode of operation in response to the second command.

43. The method of claim 42, further comprising:

detecting the first position of the mobile phone by using an accelerometer.

44. (canceled)