METHOD AND APPARATUS FOR PERFORMING MOTION RECOGNITION USING MOTION SENSOR FUSION, AND ASSOCIATED COMPUTER PROGRAM PRODUCT

• US 20130162525A1
• Filed: 02/08/2013
• Published: 06/27/2013
• Est. Priority Date: 07/14/2009
• Status: Active Grant

• Current Assignee: China Mobile Hong Kong Company Limited (China Mobile Limited)
• Original Assignee: CyWee Group Ltd.
• Inventors: Ye, Zhou, Liou, Shun-Nan, Lu, Ying-Ko, Lee, Chin-Lung
• Granted Patent: US 9,690,386 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

A method and apparatus for performing motion recognition using motion sensor fusion and an associated computer program product are provided, where the method is applied to an electronic device. The method includes the steps of: utilizing a plurality of motion sensors of the electronic device to obtain sensor data respectively corresponding to the plurality of motion sensors, the sensor data measured at a device coordinate system of the electronic device, wherein the plurality of motion sensors includes inertial motion sensors; and performing sensor fusion according to the sensor data by converting at least one portion of the sensor data and derivatives of the sensor data into converted data based on a global coordinate system of a user of the electronic device, to perform motion recognition based on the global coordinate system, in order to recognize the user'"'"'s motion.

Claims

1. A method, applied to an electronic device, for performing motion recognition using motion sensor fusion, the method comprising:

utilizing a plurality of motion sensors of the electronic device to obtain sensor data respectively corresponding to the plurality of motion sensors, the sensor data measured at a device coordinate system of the electronic device, wherein the plurality of motion sensors comprises inertial motion sensors; and

performing sensor fusion according to the sensor data by converting at least one portion of the sensor data and derivatives of the sensor data into converted data based on a global coordinate system of a user of the electronic device, to perform motion recognition based on the global coordinate system, in order to recognize the user'"'"'s motion.

2. The method of claim 1, wherein the plurality of motion sensors comprises at least one accelerometer and at least one rotation sensor;

and the sensor fusion comprises six-axis sensor fusion that is accomplished by utilizing the at least one accelerometer and the at least one rotation sensor.

3. The method of claim 1, wherein the plurality of motion sensors comprises at least one accelerometer, at least one rotation sensor, and at least one magnetometer;

and the sensor fusion comprises nine-axis sensor fusion that is accomplished by utilizing the at least one accelerometer, the at least one rotation sensor, and the at least one magnetometer.

4. The method of claim 1, wherein the step of performing sensor fusion according to the sensor data further comprises:

performing sensor fusion according to the sensor data to obtain motion data based on the device coordinate system and to obtain orientation of the electronic device, the orientation based on the global coordinate system, wherein the motion data based on the device coordinate system comprise linear acceleration based on the device coordinate system; and

converting the motion data based on the device coordinate system into at least one portion of the converted data according to the orientation based on the global coordinate system, wherein the at least one portion of the converted data comprise linear acceleration based on the global coordinate system;

wherein the linear acceleration based on the global coordinate system and the orientation based on the global coordinate system are utilized for sensing linear translation and rotation motion of the electronic device respectively in three-dimensional (3D) space.

5. The method of claim 4, further comprising:

according to the converted data based on the global coordinate system, performing motion recognition to recognize the user'"'"'s motion in the 3D space and at least one character drawn by the user in the 3D space.

6. The method of claim 5, further comprising:

performing character recognition by mapping the converted data onto at least one predetermined plane of the 3D space to obtain trajectories on the at least one predetermined plane, wherein the at least one predetermined plane is parallel to two of three axes of the global coordinate system, and the three axes of the global coordinate system are orthogonal to each other.

7. The method of claim 6, further comprising:

providing a pre-built database in the electronic device, for use of character recognition, allowing the user to follow drawing rules corresponding to meaningful characters to be recognized, without need for the user to train the database in advance.

8. The method of claim 6, further comprising:

loading the at least one character drawn by the user in the 3D space into a character recognition software module, for performing character recognition.

9. The method of claim 6, wherein the orientation based on the global coordinate system comprises a roll angle, a pitch angle, and a yaw angle of the electronic device;

and the method further comprises;

when starting performing character recognition, aligning the yaw angle with the user by resetting the yaw angle of the electronic device to be a predetermined value.

10. The method of claim 9, further comprising:

when it is detected that a button of the electronic device is pressed, triggering resetting the yaw angle of the electronic device to be the predetermined value.

11. The method of claim 9, further comprising:

when it is detected that force corresponding to beginning of the motion exists, triggering resetting the yaw angle of the electronic device to be the predetermined value.

12. The method of claim 6, wherein the orientation based on the global coordinate system comprises a roll angle, a pitch angle, and a yaw angle of the electronic device;

and the method further comprises;

performing Spherical-Cartesian transformation according to changes of one or more of the pitch angle and the yaw angle to generate data of 3D points in the 3D space;

performing Spherical-Cartesian transformation according to changes of one or more of the pitch angle and the roll angle to generate data of 3D points in the 3D space;

or

performing Spherical-Cartesian transformation according to changes of one or more of the roll angle and the yaw angle to generate data of 3D points in the 3D space.

13. The method of claim 4, further comprising:

according to the converted data based on the global coordinate system, performing motion recognition to obtain at least one motion recognition result representing the user'"'"'s motion in the 3D space; and

when obtaining the at least one motion recognition result, triggering at least one user-defined macro function corresponding to the at least one motion recognition result.

14. The method of claim 1, further comprising:

providing a user-defined gesture database in the electronic device, allowing the user to train the user-defined gesture database with different gestures of moving the electronic device by the user;

wherein the user-defined gesture database is utilized for performing motion recognition and triggering an operation corresponding to a motion recognition result obtained from performing motion recognition.

15. A computer program product, having program instructions for instructing a processor of an electronic device to perform a method comprising the steps of:

utilizing a plurality of motion sensors of the electronic device to obtain sensor data respectively corresponding to the plurality of motion sensors, the sensor data measured at a device coordinate system of the electronic device, wherein the plurality of motion sensors comprises inertial motion sensors; and

performing sensor fusion according to the sensor data by converting at least one portion of the sensor data and derivatives of the sensor data into converted data based on a global coordinate system of a user of the electronic device, to perform motion recognition based on the global coordinate system, in order to recognize the user'"'"'s motion.

16. An apparatus for performing motion recognition using motion sensor fusion, the apparatus comprising at least one portion of an electronic device, the apparatus comprising:

a storage module arranged to store information of at least one database for use of performing motion recognition;

a motion sensing module, the motion sensing module comprising a plurality of motion sensors positioned within the electronic device, wherein the plurality of motion sensors comprises inertial motion sensors; and

at least one processor, coupled to the storage module and the motion sensing module, arranged to execute program instructions to perform a method comprising the steps of;

utilizing the plurality of motion sensors of the electronic device to obtain sensor data respectively corresponding to the plurality of motion sensors, the sensor data measured at a device coordinate system of the electronic device; and

performing sensor fusion according to the sensor data by converting at least one portion of the sensor data and derivatives of the sensor data into converted data based on a global coordinate system of a user of the electronic device, to perform motion recognition based on the global coordinate system, with aid of the at least one database, in order to recognize the user'"'"'s motion.

17. The apparatus of claim 16, wherein the plurality of motion sensors comprises at least one accelerometer and at least one rotation sensor;

and the sensor fusion comprises six-axis sensor fusion that is accomplished by utilizing the at least one accelerometer and the at least one rotation sensor.

18. The apparatus of claim 16, wherein the plurality of motion sensors comprises at least one accelerometer, at least one rotation sensor, and at least one magnetometer;

and the sensor fusion comprises nine-axis sensor fusion that is accomplished by utilizing the at least one accelerometer, the at least one rotation sensor, and the at least one magnetometer.

19. The apparatus of claim 16, wherein the step of performing sensor fusion according to the sensor data further comprises:

performing sensor fusion according to the sensor data to obtain motion data based on the device coordinate system and to obtain orientation of the electronic device, the orientation based on the global coordinate system, wherein the motion data based on the device coordinate system comprise linear acceleration based on the device coordinate system; and

converting the motion data based on the device coordinate system into at least one portion of the converted data according to the orientation based on the global coordinate system, wherein the at least one portion of the converted data comprise linear acceleration based on the global coordinate system;

wherein the linear acceleration based on the global coordinate system and the orientation based on the global coordinate system are utilized for sensing linear translation and rotation motion of the electronic device respectively in three-dimensional (3D) space.

20. The apparatus of claim 16, wherein the at least one database comprises a user-defined gesture database, allowing the user to train the user-defined gesture database with different gestures of moving the electronic device by the user;

and the user-defined gesture database is utilized for performing motion recognition and triggering an operation corresponding to a motion recognition result obtained from performing motion recognition.