Context dependent gesture response

• US 7,301,529 B2
• Filed: 03/23/2004
• Issued: 11/27/2007
• Est. Priority Date: 03/23/2004
• Status: Active Grant

• Current Assignee: Fujitsu Limited
• Original Assignee: Fujitsu Limited
• Inventors: Adler, B. Thomas, Matsumoto, Hitoshi, Marvit, David L., Reinhardt, Albert H. M., Wilcox, Bruce A.
• Primary Examiner(s): Liang; Regina
• Application Number:
  US10/807,589
• Publication Number:
  US 20050212767A1
• Time in Patent Office: 1,344 Days
• Field of Search: 345156-158, 345/168, 345/179, 715/856, 715/857, 715/863, 715/864
• US Class Current: 345/156
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1684 Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 2200/1637 Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors

Abstract

A motion controlled handheld device includes a display having a viewable surface and operable to generate an image and a gesture database maintaining a plurality of gestures. Each gesture is defined by a motion of the device with respect to a first position of the device. The device includes a plurality of applications each having a plurality of predefined commands and a gesture mapping database comprising a plurality of command maps. Each of the command maps corresponds to a particular one of the applications and maps each of the predefined commands to one of the gestures. The device includes a motion detection module operable to detect motion of the handheld device within three dimensions and to identify components of the motion in relation to the viewable surface. The device also includes a control module operable to load one of the applications, to select one of the command maps corresponding to the loaded application, to track movement of the handheld device using the motion detection module, to compare the tracked movement against the gestures to determine a matching gesture, to identify, using the selected command map, the predefined command mapped to the matching gesture, and to perform the identified command using the loaded application.

Claims

1. A motion controlled handheld device comprising:

a display having a viewable surface and operable to generate an image;

a gesture database maintaining a plurality of gestures, each gesture defined by a motion of the device with respect to a first position of the device;

a plurality of applications each having a plurality of predefined commands;

a gesture mapping database comprising a plurality of command maps, each of the command maps corresponding to a particular one of the applications and mapping each of the predefined commands to one of the gestures;

a motion detection module operable to detect motion of the handheld device within three dimensions and to identify components of the motion in relation to the viewable surface;

a control module operable to load one of the applications, to select one of the command maps corresponding to the loaded application, to track movement of the handheld device using the motion detection module, to compare the tracked movement against the gestures to determine a matching gesture, to identify, using the selected command map, the predefined command mapped to the matching gesture, and to perform the identified command using the loaded application;

wherein a selected one of the applications has a first application state and a second application state;

wherein the command map associated with the selected application comprises a first mapping for the first application state and a second mapping for the second application state, the first mapping and the second mapping each mapping selected ones of the gestures to selected ones of the predefined commands of the selected application; and

wherein the first application state comprises viewing of a portion of an image of application data, and the second application state comprises viewing a hierarchical menu for performing operations with respect to the application data.

2. The motion controlled handheld device of claim 1, wherein the control module is further operable to load a second one of the applications, to select a second one of the command maps corresponding to the second loaded application, and to replace the first selected command map with the second selected command map.

3. The motion controlled handheld device of claim 2, wherein the matching gesture maps to a first predefined command using the first selected command map and to a second predefined command using the second selected command map.

4. The motion controlled handheld device of claim 1, wherein the first application state is associated with a first image type and the second application state is associated with a second image type.

5. The motion controlled handheld device of claim 1, further comprising:

a first accelerometer operable to detect acceleration along a first axis;

a second accelerometer operable to detect acceleration along a second axis, the second axis perpendicular to the first axis; and

a third accelerometer operable to detect acceleration along a third axis, the third axis perpendicular to the first axis and perpendicular to the second axis; and

wherein;

the gesture database further defines each of the gestures using a sequence of accelerations;

the motion detection module is further operable to detect motion of the device using accelerations measured by the first accelerometer, the second accelerometer, and the third accelerometer; and

the control module is further operable to match the accelerations measured by the motion detection module against gesture definitions in the gesture database to identify particular ones of the gestures.

6. A method for controlling a handheld device comprising:

generating an image on a viewable surface of the handheld device;

maintaining a gesture database comprising a plurality of gestures, each gesture defined by a motion of the device with respect to a first position of the device;

maintaining a plurality of applications each having a plurality of predefined commands;

maintaining a gesture mapping database comprising a plurality of command maps, each of the command maps corresponding to a particular one of the applications and mapping each of the predefined commands to one of the gestures;

loading one of the applications;

selecting one of the command maps corresponding to the loaded application;

tracking movement of the handheld device using the motion detection module in relation to the viewable surface;

comparing the tracked movement against the gestures to determine a matching gesture;

identifying, using the selected command map, the predefined command mapped to the matching gesture;

performing the identified command using the loaded application;

wherein a selected one of the applications has a first application state and a second

application state;

wherein the command map associated with the selected application comprises a first mapping for the first application state and a second mapping for the second application state, the first mapping and the second mapping each mapping selected ones of the gestures to selected ones of the predefined commands of the selected application; and

wherein the first application state comprises viewing of a portion of an image of application data, and the second application state comprises viewing a hierarchical menu for performing operations with respect to the application data.

7. The method of claim 6, further comprising, in response to a received command:

loading a second one of the applications;

selecting a second one of the command maps corresponding to the second loaded application; and

replacing the first selected command map with the second selected command map.

8. The method of claim 7, wherein the matching gesture maps to a first predefined command using the first selected command map and to a second predefined command using the second selected command map.

9. The method of claim 6, wherein the first application state is associated with a first image type and the second application state is associated with a second image type.

10. The method of claim 6, wherein the gesture database further defines each of the gestures using a sequence of accelerations;

the method further comprising;

detecting acceleration along a first axis;

detecting acceleration along a second axis, the second axis perpendicular to the first axis; and

detecting acceleration along a third axis, the third axis perpendicular to the first axis and perpendicular to the second axis;

detecting motion of the device using accelerations measured by the first accelerometer, the second accelerometer, and the third accelerometer; and

matching the accelerations against gesture definitions in the gesture database to identify potential indicated ones of the gestures.

11. Logic for controlling a handheld device, the logic embodied as a computer program stored on a computer readable medium and operable when executed to perform the steps of:

generating an image on a viewable surface of the handheld device;

maintaining a gesture database comprising a plurality of gestures, each gesture defined by a motion of the device with respect to a first position of the device;

maintaining a plurality of applications each having a plurality of predefined commands;

maintaining a gesture mapping database comprising a plurality of command maps, each of the command maps corresponding to a particular one of the applications and mapping each of the predefined commands to one of the gestures;

loading one of the applications;

selecting one of the command maps corresponding to the loaded application;

tracking movement of the handheld device using the motion detection module in relation to the viewable surface;

comparing the tracked movement against the gestures to determine a matching gesture;

identifying, using the selected command map, the predefined command mapped to the matching gesture;

performing the identified command using the loaded application;

wherein a selected one of the applications has a first application state and a second application state;

wherein the command map associated with the selected application comprises a first mapping for the first application state and a second mapping for the second application state, the first mapping and the second mapping each mapping selected ones of the gestures to selected ones of the predefined commands of the selected application; and

wherein the first application state comprises viewing of a portion of an image of application data, and the second application state comprises viewing a hierarchical menu for performing operations with respect to the application data.

12. The logic of claim 11, further operable, in response to a received command, to:

load a second one of the applications;

select a second one of the command maps corresponding to the second loaded application; and

replace the first selected command map with the second selected command map.

13. The logic of claim 12, wherein the matching gesture maps to a first predefined command using the first selected command map and to a second predefined command using the second selected command map.

14. The logic of claim 11, wherein the first application state is associated with a first image type and the second application state is associated with a second image type.

15. The logic of claim 11, wherein the gesture database further defines each of the gestures using a sequence of accelerations;

the logic further operable when executed to perform the steps of;

detecting acceleration along a first axis;

detecting acceleration along a second axis, the second axis perpendicular to the first axis; and

detecting acceleration along a third axis, the third axis perpendicular to the first axis and perpendicular to the second axis;

detecting motion of the device using accelerations measured by the first accelerometer, the second accelerometer, and the third accelerometer; and

matching the accelerations against gesture definitions in the gesture database to identify potential indicated ones of the gestures.

16. A motion controlled handheld device comprising:

means for generating an image on a viewable surface of the handheld device;

means for maintaining a gesture database comprising a plurality of gestures, each gesture defined by a motion of the device with respect to a first position of the device;

means for maintaining a plurality of applications each having a plurality of predefined commands;

means for maintaining a gesture mapping database comprising a plurality of command maps, each of the command maps corresponding to a particular one of the applications and mapping each of the predefined commands to one of the gestures;

means for loading one of the applications;

means for selecting one of the command maps corresponding to the loaded application;

means for tracking movement of the handheld device using the motion detection module in relation to the viewable surface;

means for comparing the tracked movement against the gestures to determine a matching gesture;

means for identifying, using the selected command map, the predefined command mapped to the matching gesture;

means for performing the identified command using the loaded application;

wherein a selected one of the applications has a first application state and a second application state;

wherein the command map associated with the selected application comprises a first mapping for the first application state and a second mapping for the second application state, the first mapping and the second mapping each mapping selected ones of the gestures to selected ones of the predefined commands of the selected application; and

wherein the first application state comprises viewing of a portion of an image of application data, and the second application state comprises viewing a hierarchical menu for performing operations with respect to the application data.