Customizable gesture mappings for motion controlled handheld devices

• US 7,365,736 B2
• Filed: 03/23/2004
• Issued: 04/29/2008
• Est. Priority Date: 03/23/2004
• Status: Active Grant

• Current Assignee: Fujitsu Limited
• Original Assignee: Fujitsu Limited
• Inventors: Wilcox, Bruce A., Adler, B. Thomas, Matsumoto, Hitoshi, Marvit, David L., Reinhardt, Albert H. M.
• Primary Examiner(s): Liang; Regina
• Application Number:
  US10/807,560
• Publication Number:
  US 20050212751A1
• Time in Patent Office: 1,498 Days
• Field of Search: 345156-158, 345/168, 345/179, 715/856, 715/857, 715/863, 715/864
• US Class Current: 345/156
• CPC Class Codes: G06F 1/1613 for portable computers cooling arrangements therefor G06F1/203; constructional details or arrangements for pocket calculators, electronic agendas or books G06F15/0216; constructional details of portable telephone sets: with several bodies H04M1/0202
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 2200/1637 Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors

Abstract

A motion controlled handheld device includes a display having a viewable surface and operable to generate an image and a gesture database maintaining a plurality of predefined gestures. Each gesture is defined by a motion of the device with respect to a first position of the device. The device includes an application having a plurality of predefined commands and a motion detection module operable to detect motion of the handheld device within three dimensions and to identify components of the motion in relation to the viewable surface. The device includes a user interface operable to receive user input associating selected ones of the gestures with corresponding ones of the commands and a gesture mapping database comprising a command map for the application. The command map comprises mappings of the selected gestures to the corresponding commands as indicated by the user input. The device also includes a control module operable to load the application, to track movement of the handheld device using the motion detection module, to compare the tracked movement against the gestures to determine a matching one of the gestures, to identify, using the command map, the command mapped to the matching gesture, and to perform the identified command using the application.

Claims

1. A motion controlled handheld device comprising:

a display having a viewable surface and operable to generate an image;

a gesture database maintaining a plurality of predefined gestures, each gesture defined by a motion of the device with respect to a first position of the device;

an application having a plurality of predefined commands;

a motion detection module operable to detect motion of the handheld device within three dimensions and to identify components of the motion in relation to the viewable surface;

a user interface operable to receive user input associating selected ones of the gestures with corresponding ones of the commands;

a gesture mapping database comprising a command map for the application, the command map comprising mappings of the selected gestures to the corresponding commands as indicated by the user input;

a control module operable to load the application, to track movement of the handheld device using the motion detection module, to compare the tracked movement against the gestures to determine a matching one of the gestures, to identify, using the command map, the command mapped to the matching gesture, and to perform the identified command using the application;

wherein the control module is further operable to;

detect an indication to record a new gesture;

detect a stabilization of the components of the motion of the device;

upon detecting the stabilization, determine a base reference position;

record movement of the device with respect to the base reference position;

detect an indication to stop recording the new gesture;

define the new gesture using the recorded movement of the device with respect to the base reference position; and

store the new gesture in the gesture database; and

wherein the user interface is further operable to receive user input associating the new gesture with at least one of the commands.

2. The motion controlled handheld device of claim 1, wherein:

the user input identifies a particular one of the gestures and a plurality of the predefined commands;

the command map further comprises a mapping of the identified gesture to the identified plurality of the predefined commands; and

the control module is further operable, in response to the matching gesture being the particular one of the gestures, to perform each of the plurality of the predefined commands.

3. The motion controlled handheld device of claim 1, wherein the application has a first application state and a second application state, and the command map for the application, in response to a same gesture, maps a first predefined command in the first application state and a second predefined command in the second application state.

4. The motion controlled handheld device of claim 1, wherein detecting the indication to stop recording comprises detecting a second stabilization of the movement of the device.

5. The motion controlled handheld device of claim 1, wherein detecting the indication to record comprises a selection of an input associated with a user interface of the device.

6. The motion controlled handheld device of claim 1, further comprising:

a first accelerometer operable to detect acceleration along a first axis;

a second accelerometer operable to detect acceleration along a second axis, the second axis perpendicular to the first axis; and

a third accelerometer operable to detect acceleration along a third axis, the third axis perpendicular to the first axis and perpendicular to the second axis; and

wherein;

the gesture database further defines each of the predefined gestures using a sequence of accelerations;

the motion detection module is further operable to detect motion of the device using accelerations measured by the first accelerometer, the second accelerometer, and the third accelerometer; and

the control module is further operable to match the accelerations measured by the motion detection module against gesture definitions in the gesture database to identify particular ones of the predefined gestures.

7. A method for controlling a handheld device comprising:

generating an image on a viewable surface of the handheld device;

maintaining a gesture database comprising a plurality of predefined gestures, each gesture defined by a motion of the device with respect to a first position of the device;

maintaining an application having a plurality of predefined commands;

receiving user input associating selected ones of the gestures with corresponding ones of the commands;

maintaining a gesture mapping database comprising a command map for the application and updating the command map to comprise mappings of the selected gestures to the corresponding commands as indicated by the user input; and

loading the application;

tracking movement of the handheld device using the motion detection module in relation to the viewable surface;

comparing the tracked movement against the gestures to determine a matching gesture;

identifying, using the selected command map, the predefined command mapped to the matching gesture;

performing the identified command using the loaded application

detecting an indication to record a new gesture;

detecting a stabilization of the components of the motion of the device;

upon detecting the stabilization, determining a base reference position;

recording movement of the device with respect to the base reference position;

detecting an indication to stop recording the new gesture;

defining the new gesture using the recorded movement of the device with respect to the base reference position;

storing the new gesture in the gesture database; and

receiving user input associating the new gesture with at least one of the commands.

8. The method of claim 7, wherein:

the user input identifies a particular one of the gestures and a plurality of the predefined commands;

the command map further comprises a mapping of the identified gesture to the identified plurality of the predefined commands; and

further comprising in response to the matching gesture being the particular one of the gestures, performing each of the plurality of the predefined commands.

9. The method of claim 7, wherein the application has a first application state and a second application state, and the command map for the application, in response to a same gesture, maps a first predefined command in the first application state and a second predefined command in the second application state.

10. The method of claim 7, wherein detecting the indication to stop recording comprises detecting a second stabilization of the movement of the device.

11. The method of claim 7, wherein detecting the indication to record comprises a selection of an input associated with a user interface of the device.

12. The method of claim 7, wherein the gesture database further defines each of the gestures using a sequence of accelerations;

the method further comprising;

detecting acceleration along a first axis;

detecting acceleration along a second axis, the second axis perpendicular to the first axis; and

detecting acceleration along a third axis, the third axis perpendicular to the first axis and perpendicular to the second axis;

detecting motion of the device using accelerations measured by the first accelerometer, the second accelerometer, and the third accelerometer; and

matching the accelerations against gesture definitions in the gesture database to identify potential indicated ones of the predefined gestures.

13. Logic for controlling a handheld device, the logic embodied as a computer program stored on a computer readable medium and operable when executed to perform the steps of:

generating an image on a viewable surface of the handheld device;

maintaining a gesture database comprising a plurality of predefined gestures, each gesture defined by a motion of the device with respect to a first position of the device;

maintaining an application having a plurality of predefined commands;

receiving user input associating selected ones of the gestures with corresponding ones of the commands;

maintaining a gesture mapping database comprising a command map for the application and updating the command map to comprise mappings of the selected gestures to the corresponding commands as indicated by the user input; and

loading the application;

tracking movement of the handheld device using the motion detection module in relation to the viewable surface;

comparing the tracked movement against the gestures to determine a matching gesture;

identifying, using the selected command map, the predefined command mapped to the matching gesture;

performing the identified command using the loaded application

detecting an indication to record a new gesture;

detecting a stabilization of the components of the motion of the device;

upon detecting the stabilization, determining a base reference position;

recording movement of the device with respect to the base reference position;

detecting an indication to stop recording the new gesture;

defining the new gesture using the recorded movement of the device with respect to the base reference position;

storing the new gesture in the gesture database; and

receiving user input associating the new gesture with at least one of the commands.

14. The logic of claim 13, wherein:

the user input identifies a particular one of the gestures and a plurality of the predefined commands;

the command map further comprises a mapping of the identified gesture to the identified plurality of the predefined commands; and

further operable when executed;

in response to the matching gesture being the particular one of the gestures, to perform each of the plurality of the predefined commands.

15. The logic of claim 13, wherein the application has a first application state and a second application state, and the command map for the application, in response to a same gesture, maps a first predefined command in the first application state and a second predefined command in the second application state.

16. The logic of claim 13, wherein detecting the indication to stop recording comprises detecting a second stabilization of the movement of the device.

17. The logic of claim 13, wherein the gesture database further defines each of the gestures using a sequence of accelerations;

the logic further operable when executed to perform the steps of;

detecting acceleration along a first axis;

detecting acceleration along a second axis, the second axis perpendicular to the first axis; and

detecting acceleration along a third axis, the third axis perpendicular to the first axis and perpendicular to the second axis;

detecting motion of the device using accelerations measured by the first accelerometer, the second accelerometer, and the third accelerometer; and

matching the accelerations against gesture definitions in the gesture database to identify potential indicated ones of the predefined gestures.

18. A motion controlled handheld device comprising:

means for generating an image on a viewable surface of the handheld device;

means for maintaining a gesture database comprising a plurality of predefined gestures, each gesture defined by a motion of the device with respect to a first position of the device;

means for maintaining an application having a plurality of predefined commands;

means for receiving user input associating selected ones of the gestures with corresponding ones of the commands;

means for maintaining a gesture mapping database comprising a command map for the application and updating the command map to comprise mappings of the selected gestures to the corresponding commands as indicated by the user input; and

means for loading the application;

means for tracking movement of the handheld device using the motion detection module in relation to the viewable surface;

means for comparing the tracked movement against the gestures to determine a matching gesture;

means for identifying, using the selected command map, the predefined command mapped to the matching gesture;

means for performing the identified command using the loaded application;

means for detecting an indication to record a new gesture;

means for detecting a stabilization of the components of the motion of the device;

means for, upon detecting the stabilization, determining a base reference position;

means for recording movement of the device with respect to the base reference position;

means for detecting an indication to stop recording the new gesture;

means for defining the new gesture using the recorded movement of the device with respect to the base reference position;

means for storing the new gesture in the gesture database; and

means for receiving user input associating the new gesture with at least one of the commands.