MOBILE DEVICES WITH MOTION GESTURE RECOGNITION

• US 20090265671A1
• Filed: 10/15/2008
• Published: 10/22/2009
• Est. Priority Date: 04/21/2008
• Status: Abandoned Application

• Current Assignee: Invensense Incorporated (TDK Corporation)
• Original Assignee: Invensense Incorporated (TDK Corporation)
• Inventors: GU, ANJIA, JIANG, JOSEPH, SACHS, DAVID, NASIRI, STEVEN S.
• Application Number:
  US12/252,322
• Publication Number:
  US 20090265671A1
• Time in Patent Office: Days
• US Class Current: 715/863
• CPC Class Codes: G06F 2200/1636 Sensing arrangement for detection of a tap gesture on the housing
• CPC Class Codes: G06F 2200/1637 Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: H04M 2250/12 including a sensor for measuring a physical value, e.g. temperature or motion telephones including GPS signal receivers H04M2250/10

Abstract

Mobile devices using motion gesture recognition. In one aspect, processing motion to control a portable electronic device includes receiving, on the device, sensed motion data derived from motion sensors of the device and based on device movement in space. The motion sensors include at least three rotational motion sensors and at least three accelerometers. A particular operating mode is determined to be active while the movement of the device occurs, the mode being one of multiple different operating modes of the device. Motion gesture(s) are recognized from the motion data from a set of motion gestures available for recognition in the active operating mode. Each of the different operating modes, when active, has a different set of gestures available. State(s) of the device are changed based on the recognized gestures, including changing output of a display screen on the device.

Claims

1. A method for processing motion of a portable electronic device to control the portable electronic device, the method comprising:

receiving, on the portable electronic device, sensed motion data derived from motion sensors of the portable electronic device, wherein the sensed motion data is based on movement of the portable electronic device in space, the motion sensors providing six-axis motion sensing and including at least three rotational motion sensors and at least three accelerometers;

determining, on the portable electronic device, a particular operating mode that is active while the movement of the portable electronic device occurs, wherein the particular operating mode is one of a plurality of different operating modes available in the operation of the portable electronic device;

recognizing, on the portable electronic device, one or more motion gestures from the motion data, wherein the one or more motion gestures are recognized from a set of a plurality of motion gestures that are available for recognition in the active operating mode of the portable electronic device, and wherein each of the different operating modes of the portable electronic device, when active, has a different set of motion gestures available for recognition; and

changing one or more states of the portable electronic device based on the one or more recognized motion gestures, including changing output of a display screen on the portable electronic device.

2. The method of claim 1 wherein the one or more gestures includes a shake gesture, the shake gesture detected from the sensed motion data that describes motion of the portable electronic device in one angular direction and includes a magnitude that is at least a threshold level above a background noise level.

3. The method of claim 1 wherein the one or more gestures include a tap gesture, the tap gesture detected from the sensed motion data that describes motion of the portable electronic device as a pulse of movement of the device in space.

4. The method of claim 3 wherein the pulse of the tap gesture is detected by examining peaks in the motion sensor data above a background noise level, the tap gesture having a magnitude that is at least a threshold level above the background noise level, and including rejecting spikes in the motion sensor data at the end of the movement of the motion sensor device conesponding to the gesture.

5. The method of claim 1 wherein the one or more gestures includes a circle gesture, the circle gesture detected from the sensed motion data that describes motion of the portable electronic device in an approximate circular movement in space.

6. The method of claim 1 wherein the one or more gestures include a character gesture, the character gesture detected from sensed motion data that describes a combination of at least one linear movement and at least one approximately circular movement of the portable electronic device in space.

7. The method of claim 1 further comprising:

receiving an enter mode control signal indicating a motion control of the portable electronic device has been activated by a user;

in response to receiving the enter mode control signal, entering a motion mode of the portable electronic device that allows the sensed motion data to be used for recognizing the one or more motion gestures; and

exiting the motion mode of the portable electronic device based on an exit event determined by the portable electronic device.

8. The method of claim 7 further comprising ignoring additional sensed motion data derived from the motion sensors for the purpose of detecting gestures from the additional sensed motion data, while the portable electronic device is not in the motion mode.

9. The method of claim 8 wherein the portable electronic device stays in the motion mode only while the enter mode control signal is maintained by the user continuing to activate the motion control, and wherein the motion mode is exited in response to receiving an exit mode control signal, the exit mode control signal corresponding to the user releasing the motion control.

10. The method of claim 7 wherein the portable electronic device stays in the motion mode after the user has clicked the motion control, and wherein the exit event is detecting a predefined exit gesture in the sensed motion data.

11. The method of claim 7 wherein the portable electronic device stays in the motion mode after the user has clicked the motion control, and wherein the exit event is a completion of one of the one or more gestures.

12. The method of claim 1 wherein the detected one or more gestures are used to move an image on a display screen of the portable electronic device, the image moving in a direction corresponding to a direction of motion of the portable electronic device as detected in the motion data.

13. A portable electronic device for sensing motion gestures, the portable electronic device comprising:

a plurality of motion sensors providing sensed data based on movement of the portable electronic device in space, the motion sensors providing six-axis motion sensing and including at least three rotational motion sensors and at least three accelerometers;

a display screen; and

one or more processors, wherein at least one of the processors;

receives motion data derived from the sensed data provided by the motion sensors;

determines a particular operating mode that is active while the movement of the portable electronic device occurs, wherein the particular operating mode is one of a plurality of different operating modes available in the operation of the portable electronic device;

recognizes one or more motion gestures from the motion data, wherein the one or more motion gestures are recognized from a set of a plurality of motion gestures that are available for recognition in the active operating mode of the portable electronic device, and wherein each of the different operating modes of the portable electronic device, when active, has a different set of motion gestures available for recognition; and

changes one or more states of the portable electronic device based on the one or more recognized motion gestures, including changing output of the display screen.

14. The portable electronic device of claim 13 wherein the one or more motion gestures includes a shake gesture, the shake gesture detected from the motion data that describes motion of the portable electronic device in one angular direction and includes a magnitude that is at least a threshold level above a background noise level.

15. The portable electronic device of claim 13 wherein the one or more motion gestures include a tap gesture, the tap gesture detected from the motion data that describes motion of the portable electronic device as a pulse of movement of the device in space, wherein the pulse of the tap gesture has a magnitude that is at least a threshold level above a background noise level, wherein spikes of magnitude in the motion data are rejected as the tap gesture if occurring at the end of movements of the portable electronic device in space.

16. The portable electronic device of claim 13 wherein the one or more motion gestures includes a circle gesture, the circle gesture detected from the motion data that describes motion of the portable electronic device in an approximate circular movement in space.

17. The portable electronic device of claim 13 wherein the one or more motion gestures include a character gesture, the character gesture detected from motion data that describes a combination of at least one linear movement and at least one approximately circular movement of the portable electronic device in space.

18. The portable electronic device of claim 13 further comprising a motion control activatable by a user of the portable electronic device, wherein at least one of the one or more processors:

receives an enter mode control signal indicating the motion control of the portable electronic device has been activated by a user;

in response to receiving the enter mode control signal, enters a motion mode of the portable electronic device that allows the motion data to be used for recognizing the one or more motion gestures; and

exits the motion mode of the portable electronic device based on an exit event determined by the processor,

wherein the at least one processor ignores additional sensed data from motion sensors for the purpose of detecting motion gestures from the additional sensed motion data, while the portable electronic device is not in the motion mode.

19. The portable electronic device of claim 18 wherein the at least one processor maintains the portable electronic device in the motion mode only while the enter mode control signal is maintained by the user continuing to activate the motion control, and wherein the at least one processor exits the motion mode in response to the user releasing the motion control.

20. The portable electronic device of claim 14 wherein the detected one or more motion gestures are used to move an image displayed on the display screen, the image moving in a direction corresponding to a direction of motion of the portable electronic device as detected in the motion data.

21. A method for recognizing a gesture performed by a user using a motion sensing device, the method comprising:

receiving motion sensor data in device coordinates indicative of motion of the device, the motion sensor data received from a plurality of motion sensors of the motion sensing device, the motion sensors including a plurality of rotational motion sensors and a plurality of linear motion sensors;

transforming the motion sensor data in the device coordinates to motion sensor data in world coordinates, the motion sensor data in the device coordinates describing motion of the device relative to a frame of reference of the device, and the motion sensor data in the world coordinates describing motion of the device relative to a frame of reference external to the device; and

detecting a gesture from the motion sensor data in the world coordinates.

22. The method of claim 21 further comprising transforming the motion sensor data from the world coordinates to local world coordinates, the motion sensor data in the local world coordinates describing motion relative to the body of the user of the device.

23. The method of claim 22 wherein the local world coordinates are determined by updating the world coordinates to track the motion of the motion sensing device when the motion sensing device is moved at a velocity below a predetermined threshold.

24. The method of claim 23 wherein the velocity below the predetermined threshold is derived from an angular velocity of the motion sensing device and a linear velocity of the motion sensing device.

25. The method of claim 23 wherein in response to the motion sensing device moving at a velocity above the predetermined threshold during the gesture, the local world coordinates are kept fixed during the gesture, the world coordinates being fixed at the last position and orientation of the motion sensing device before the gesture is determined to have started.

26. The method of claim 21 wherein the gesture is detected by extracting one or more data features from the motion sensor data and processing the one or more data features to detect the gesture, the data features comprising less data points than the motion sensor data over a portion of the motion sensor data including the data features.

27. The method of claim 26 wherein the one or more data features include at least one of:

a maximum magnitude or minimum magnitude of the motion sensor data;

a zero crossing of the motion sensor data from positive values to negative values or negative values to positive values; and

an integral of an interval defined by a graph of the motion sensor data.

28. The method of claim 26 wherein the gesture is detected by examining the motion sensor data for the one or more data features in terms of one of more of the following:

relative timing between the one or more data features, and

relative magnitudes between the one or more data features.

29. The method of claim 26 wherein the gesture is detected by timing each of the one or more data features and recognizing the gesture in response to the data features occuning within a predetermined time of each other.

30. The method of claim 26 wherein a plurality of the data features are peaks in the motion sensor data, and wherein the gesture is detected by at least one of:

selecting only the highest peak in the motion sensor data, and

examining a peak previous to the highest peak in the motion sensor data.

31. The method of claim 21 further comprising, after detecting the gesture, triggering a function of the motion sensing device, the function associated with the detected gesture, and further comprising testing at least one abort condition before triggering the associated function, wherein if the abort condition is met, the associated function is not triggered, wherein the abort condition includes a minimum amount of time preceding and following the gesture during which no significant movement of the motion sensing device has occured.

32. The method of claim 21 wherein detecting the gesture includes correlating an angular velocity during the motion of the motion sensing device with linear acceleration of the motion sensing device, and using the correlation to reject noise motion of the motion sensing device that is substantially all rotation.

33. A system for detecting gestures, the system including:

a plurality of motion sensors providing motion sensor data, the motion sensors including a plurality of rotational motion sensors and a plurality of linear motion sensors;

at least one feature detector, each feature detector operative to detect an associated data feature derived from the motion sensor data, each data feature being a characteristic of the motion sensor data, each feature detector outputting one or more feature values describing the detected data feature; and

at least one gesture detector, each gesture detector operative to detect a gesture associated with the gesture detector based on the one or more feature values.

34. The system of claim 33 wherein the at least one feature detector includes a peak feature detector operative to detect a peak in the motion sensor data.

35. The system of claim 33 wherein the at least one feature detector includes a zero crossing feature detector operative to detect a zero crossing in the motion sensor data, the zero crossing indicating a change in direction of motion in an axis of movement.

36. The system of claim 33 further comprising a processing block that processes the motion sensor data to provide the augmented motion data, the augmented motion data being in reference to world coordinates and the motion sensor data being in reference to device coordinates, and wherein each feature detector is operative to detect an associated data feature derived from the motion sensor data and the augmented motion data.

37. The system of claim 33 wherein the motion sensor data further includes sensor data from additional sensors of the motion sensing device, the additional sensors including at least one of a temperature sensor, a pressure sensor, and a compass.

38. The system of claim 33 wherein the rotational motion sensors include gyroscopes or compasses and the linear motion sensors include accelerometers.

39. A handheld electronic device operable in a plurality of motion-responsive operating modes, wherein at least one recognizable motion gesture corresponds to each motion-responsive operating mode, the electronic device comprising:

a set of motion sensors sensing rotational rate around at least three axes and linear acceleration along at least three axes;

a display; and

processing logic that, based on motion data derived from at least one of the motion sensors in response to a physical movement of the electronic device and a then-active operating mode of the device, is capable of recognizing at least one gesture corresponding to the physical movement and changing at least one operating mode of the device.