Method and apparatus for performing motion recognition using motion sensor fusion, and associated computer program product

• US 10,275,038 B2
• Filed: 06/05/2017
• Issued: 04/30/2019
• Est. Priority Date: 07/14/2009
• Status: Active Grant

• Current Assignee: China Mobile Hong Kong Company Limited (China Mobile Communications Group Co., Ltd.)
• Original Assignee: China Mobile Hong Kong Company Limited (China Mobile Communications Group Co., Ltd.)
• Inventors: Ye, Zhou, Liou, Shun-Nan, Lu, Ying-Ko, Lee, Chin-Lung
• Primary Examiner(s): Hong, Richard J
• Application Number:
  US15/613,426
• Publication Number:
  US 20170269701A1
• Time in Patent Office: 694 Days
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

A method and apparatus for performing motion recognition using motion sensor fusion and an associated computer program product are provided, where the method is applied to an electronic device. The method includes the steps of: utilizing a plurality of motion sensors of the electronic device to obtain sensor data respectively corresponding to the plurality of motion sensors, the sensor data measured at a device coordinate system of the electronic device, wherein the plurality of motion sensors includes inertial motion sensors; and performing sensor fusion according to the sensor data by converting at least one portion of the sensor data and derivatives of the sensor data into converted data based on a global coordinate system of a user of the electronic device, to perform motion recognition based on the global coordinate system, in order to recognize the user'"'"'s motion.

Claims

1. A method performed by a processor of an electronic device configured to be held by a user, the method comprising:

obtaining sensor data corresponding to a plurality of motion sensors of the electronic device, the sensor data being measured with respect to a device coordinate system of the electronic device, wherein the plurality of motion sensors comprises inertial motion sensors;

performing sensor fusion according to the sensor data to obtain motion data based on the device coordinate system and orientation of the electronic device based on a global coordinate system;

selecting one of at least one predetermined plane based on the orientation;

mapping the motion data onto the selected one of the at least one predetermined plane to obtain a trajectory on the selected one of the at least one predetermined plane; and

performing motion recognition based on the trajectory on the selected one of the at least one predetermined plane at the global coordinate system in order to recognize the user'"'"'s motion in 3D space and at least one character drawn by the user in the 3D space,

wherein, in mapping the motion data, the least one predetermined plane is parallel to two of three axes of the global coordinate system, and the three axes of the global coordinate system are orthogonal to each other.

2. The method of claim 1, wherein:

the plurality of motion sensors comprises at least one rotation sensor; and

performing the sensor fusion comprises utilizing the at least one rotation sensor.

3. The method of claim 2, wherein:

the plurality of motion sensors comprises at least one magnetometer; and

performing the sensor fusion comprises utilizing the at least one magnetometer.

4. The method of claim 1, further comprising:

providing a pre-built character recognition database in the electronic device corresponding to drawing rules associated with characters to be recognized.

5. The method of claim 1, wherein performing motion recognition further comprises:

loading the at least one character drawn by the user in the 3D space into a character recognition software module configured to perform character recognition; and

performing character recognition.

6. The method of claim 5, wherein the orientation based on the global coordinate system comprises a roll angle, a pitch angle, and a yaw angle of the electronic device.

7. The method of claim 6, wherein performing character recognition comprises:

resetting the yaw angle of the electronic device to a predetermined value to align the yaw angle with the user.

8. The method of claim 7, further comprising:

resetting the yaw angle of the electronic device to the predetermined value in response to a button of the electronic device being pressed.

9. The method of claim 7, further comprising:

resetting the yaw angle of the electronic device to the predetermined value in response to detecting onset of motion.

10. The method of claim 5, further comprising:

performing Spherical-Cartesian transformation according to changes of one or more of the pitch angle, roll angle, and the yaw angle to generate data of 3D points in the 3D space.

11. The method of claim 1, wherein:

performing motion recognition comprises obtaining at least one motion recognition result representing the user'"'"'s motion in the 3D space; and

the method further comprises triggering at least one user-defined macro function corresponding to the at least one motion recognition result.

12. The method of claim 11, wherein:

the method further comprises providing a user-defined gesture database in the electronic device; and

obtaining the at least one motion recognition result comprises using the user-defined gesture database.

13. An electronic device configured to be held by a user, comprising:

a plurality of motion sensors configured to generate sensor data measured with respect to a device coordinate system, wherein the plurality of motion sensors comprises inertial motion sensors;

a processor having processor circuitry configured to;

perform sensor fusion according to the sensor data to obtain motion data at the device coordinate system and orientation of the electronic device at a global coordinate system;

select one of at least one predetermined plane based on the orientation;

mapping the motion data onto the selected one of the at least one predetermined plane to obtain a trajectory on the selected one of the at least one predetermined plane; and

performing motion recognition based on the trajectory on the selected one of the at least one predetermined plane at the global coordinate system in order to recognize the user'"'"'s motion in 3D space and at least one character drawn by the user in the 3D space;

wherein, in mapping the motion data, the least one predetermined plane is parallel to two of three axes of the global coordinate system, and the three axes of the global coordinate system are orthogonal to each other.

14. The device of claim 13, wherein:

the plurality of motion sensors comprises at least one rotation sensor; and

the processor is configured to perform the sensor fusion utilizing the at least one rotation sensor.

15. The device of claim 14, wherein:

the plurality of motion sensors comprises at least one magnetometer; and

the processor is configured to perform the sensor fusion utilizing the at least one magnetometer.

16. The device of claim 13, further comprising:

a pre-built character recognition database corresponding to drawing rules associated with characters to be recognized such that the processor accesses the pre-built character recognition database during motion recognition.

17. The device of claim 13, further comprising:

a character recognition software module configured to perform character recognition during the performing of motion recognition.

18. The device of claim 17, wherein:

the orientation based on the global coordinate system comprises a roll angle, a pitch angle, and a yaw angle of the electronic device; and

the processor is configured to reset the yaw angle of the electronic device to a predetermined value to align the yaw angle with the user.

19. The device of claim 13, wherein:

in performing the motion recognition, the processor is configured to obtain at least one motion recognition result representing the user'"'"'s motion in the 3D space; and

the processor is further configured to trigger at least one user-defined macro function corresponding to the at least one motion recognition result.

20. The device of claim 19, wherein:

the device further comprises a user-defined gesture database; and

the processor is configured to obtain the at least one motion recognition result using the user-defined gesture database.