Gesture based user interface supporting preexisting symbols

• US 8,692,764 B2
• Filed: 06/29/2010
• Issued: 04/08/2014
• Est. Priority Date: 03/23/2004
• Status: Active Grant

• Current Assignee: Fujitsu Limited
• Original Assignee: Fujitsu Limited
• Inventors: Marvit, David L., Matsumoto, Hitoshi, Reinhardt, Albert H. M.
• Primary Examiner(s): Liang, Regina
• Application Number:
  US12/826,470
• Publication Number:
  US 20100328201A1
• Time in Patent Office: 1,379 Days
• Field of Search: 345156-158, 345/168, 345/169, 345/173, 348/155, 348/169, 348/734, 715856-864
• US Class Current: 345/156
• CPC Class Codes: G06F 1/1613 for portable computers cooling arrangements therefor G06F1/203; constructional details or arrangements for pocket calculators, electronic agendas or books G06F15/0216; constructional details of portable telephone sets: with several bodies H04M1/0202
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 2200/1637 Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0236 using selection techniques to select from displayed items
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors

Abstract

A motion controlled handheld device includes a display having a viewable surface and operable to generate an image and a gesture database maintaining a plurality of gestures. Each gesture is defined by a motion of the device with respect to a first position of the device. The gestures comprise symbol gestures each corresponding to a character from a preexisting character set. The device includes an application database maintaining at least one application and a gesture mapping database comprising a gesture input map for the application. The gesture input map comprises mappings of the symbol gestures to corresponding inputs for the application. The device includes a motion detection module operable to detect motion of the handheld device within three dimensions and to identify components of the motion in relation to the viewable surface. The device also includes a control module operable to load the application, to track movement of the handheld device using the motion detection module, to compare the tracked movement against the symbol gestures to identify a matching symbol gesture, to identify, using the gesture input map, the corresponding input mapped to the matching symbol gesture, and to provide the corresponding input to the application.

Claims

1. A handheld motion control system comprising:

a motion detector within a handheld motion device, the motion detector having a three-axis acceleration sensor;

a device locator within the handheld motion device, the device locator configured to;

receive motion information obtained by the motion detector within the handheld motion device, wherein the motion information comprises a set of points traversed by the handheld motion device between a starting point base reference position and a stopping point;

identify and select a device to be controlled from among a plurality of devices based on the motion information and device selection information, the device selection information correlating different motion information with different devices of the plurality of devices; and

wherein the starting point base reference position is associated with an orientation of the handheld motion device relative to the device identified and selected to be controlled;

a wireless communication interface configured to communicate device commands for the selected device, the device commands communicated from the handheld motion device to a processing apparatus of the selected device configured to obtain the information from the wireless communication interface to process the device commands; and

wherein the motion detector is operable to allow a user to reset the starting point base reference position of the handheld motion device.

2. The motion control system according to claim 1, wherein the handheld device further comprises a camera with an axis of the camera along a third axis perpendicular to each of a first axis and a second axis of the three-axis acceleration sensor.

3. The motion control system according to claim 2, wherein the camera is configured to provide information used for a distance measurement.

4. The motion control system according to claim 1, wherein the handheld device further comprises a camera with an axis of the camera in a direction along a longitudinal side of the handheld device.

5. The motion control system according to claim 4, wherein the camera is configured to provide information used for a distance measurement.

6. The motion control system according to claim 1, wherein the handheld device further comprises a camera with an axis of the camera of the handheld device along an axis that is parallel to a mounting surface of the three-axis acceleration sensor.

7. The motion control system according to claim 6, wherein the camera is configured to provide information used for a distance measurement.

8. The motion control system according to claim 1, wherein the motion detector of the handheld device includes a processor operable to allow a user to repeatedly selectively engage and disengage motion sensitivity.

9. The motion control system according to claim 1, wherein the handheld device is operable to provide feedback information to notify a user that a particular motion of the handheld device is recognized as a particular gesture.

10. The motion control system according to claim 9, wherein the handheld device is operable to provide the feedback information in an audio format.

11. The motion control system according to claim 9, wherein the handheld device is operable to provide the feedback information in a vibratory format.

12. The motion control system according to claim 1, wherein the processing apparatus is a component of a DVD player.

13. The motion control system according to claim 1, wherein the processing apparatus is a component of a television.

14. The motion control system according to claim 1, wherein the handheld device is connected to other computer peripheral equipment.

15. The motion control system according to claim 1, wherein the handheld device further comprises a unit that notifies a user of output.

16. The motion control system according to claim 1, wherein the handheld device further comprises a unit that notifies a user of a battery level.

17. The motion control system according to claim 1, wherein the handheld device further comprises a unit that notifies via a sound.

18. The motion control system according to claim 1, wherein the handheld device further comprises a user interface.

19. The motion control system according to claim 1, wherein the handheld device further comprises a unit that gives a vibratory feedback.

20. The motion control system according to claim 1, wherein the processing apparatus comprises a calendar application.

21. The motion control system according to claim 1, wherein the processing apparatus is operable to provide electronic mail functionality.

22. The motion control system according to claim 1, wherein the processing apparatus is operable to provide a navigation menu.

23. The motion control system according to claim 1, wherein the processing apparatus is operable to provide time and date information.

24. The motion control system according to claim 1, wherein the processing apparatus further comprises a gesture mapping database.

25. The motion control system according to claim 1, wherein the processing apparatus further comprises a gesture database.

26. The motion control system according to claim 25, wherein the gesture database is operable to record a gesture made by a user.

27. The motion control system according to claim 1, wherein the wireless communication interface configured to communicate device commands for the selected device is further configured to communicate a device specific command to the selected device.

28. A motion control system comprising:

a camera;

a motion detector having a three-axis acceleration sensor the motion detector is operable to detect a tilt and translation motion based on information obtained by the three-axis acceleration sensor and the camera, wherein the information obtained by the three axis acceleration sensor and the camera comprises a set of points traversed by the handheld motion device between a starting point base reference position and a stopping point; and

a device locator configured to identify and select a device to be controlled from among a plurality of devices based on a direction of movement of the motion detector; and

wherein the starting point base reference position is associated with an orientation of the handheld motion device relative to the device identified and selected to be controlled;

a wireless communication interface configured to communicate information obtained by the motion detector to a processing apparatus of the selected device configured to obtain the information from the wireless communication interface to process the information.

29. A method for controlling a motion control system including a handheld device and a processing apparatus, the method comprising:

detecting movement of the handheld device by using a motion detector within the handheld motion device, the motion detector having a three-axis acceleration sensor operable to detect a set of points traversed by the handheld motion device between a starting point base reference position and a stopping point;

identifying and selecting a device to be controlled from among a plurality of devices via a device locator within the handheld motion device based on the detected movement of the handheld device and device selection information maintained by the handheld device, the device selection information correlating different motion information with different devices of the plurality of devices, wherein the starting point base reference position is associated with an orientation of the handheld motion device relative to the device identified and selected to be controlled;

communicating device commands for the selected device, the device commands communicated from the handheld motion device to the processing apparatus through a wireless communication interface of the identified and selected device-for processing of the commands by the processing apparatus; and

resetting, by a user, the starting point base reference position of the handheld device.

30. A method for controlling a motion control system including a handheld device and a processing apparatus, the method comprising:

receiving information from a three-axis acceleration sensor;

receiving information from a camera;

detecting movement of the handheld device via a motion detector operable to detect a tilt and translation motion based on the information received from the three-axis acceleration sensor and the camera, wherein the information received from the three axis acceleration sensor and the camera comprises a set of points traversed by the handheld motion device between a starting point base reference position and a stopping point;

identifying and selecting a device to be controlled from among a plurality of devices based on a direction of movement of the motion detector, wherein the starting point base reference position is associated with an orientation of the handheld motion device relative to the device identified and selected to be controlled; and

communicating information obtained by the motion detector to a processing apparatus of the selected device configured to obtain the information from the wireless communication interface to process the information.

31. The method of claim 30, wherein an axis of the camera is along a third axis perpendicular to each of a first axis and a second axis of the three-axis acceleration sensor.

32. The method of claim 31, wherein the camera is configured to provide information used for a distance measurement.

33. The method of claim 30, wherein an axis of the camera is in a direction along a longitudinal side of the handheld device.

34. The method of claim 33, wherein the camera is configured to provide information used for a distance measurement.

35. The method of claim 30, wherein an axis of the camera of the handheld device is along an axis that is parallel to a mounting surface of the three-axis acceleration sensor.

36. The method of claim 35, wherein the camera is configured to provide information used for a distance measurement.

37. The method of claim 30, further comprising allowing a user to repeatedly selectively engage and disengage motion sensitivity.

38. The method of claim 30, further comprising providing feedback information to notify a user that a particular motion of the handheld device is recognized as a particular gesture.

39. The method of claim 38, wherein providing the feedback information comprises providing the feedback information in an audio format.

40. The method of claim 38, wherein providing the feedback information comprises providing the feedback information in a vibratory format.

41. The method of claim 30, wherein the processing apparatus is a component of a DVD player.

42. The method of claim 30, wherein the processing apparatus is a component of a television.

43. The method of claim 30, further comprising connecting the handheld device to other computer peripheral equipment.

44. The method of claim 30, further comprising notifying a user of output.

45. The method of claim 30, further comprising notifying a user of a battery level.

46. The method of claim 30, further comprising notifying a user via a sound.

47. The method of claim 30, further comprising displaying a user interface.

48. The method of claim 30, further comprising giving a vibratory feedback.

49. The method of claim 30, further comprising providing a calendar application.

50. The method of claim 30, further comprising resetting a base reference position of the handheld device.

51. The method of claim 30, further comprising providing electronic mail functionality.

52. The method of claim 30, further comprising providing a navigation menu.

53. The method of claim 30, further comprising providing time and date information.

54. The method of claim 30, further comprising maintaining a gesture mapping database.

55. The method of claim 30, further comprising maintaining a gesture database.

56. The method of claim 55, further comprising recording a gesture made by a user, the gesture recorded in the gesture database is operable to.

57. A handheld motion control system comprising:

a three-axis acceleration sensor;

a camera;

a motion detector configured to determine a direction of movement based on information obtained from the three-axis acceleration sensor and the camera, wherein the information obtained from the three axis acceleration sensor and the camera comprises a set of points traversed by the handheld motion device between a starting point base reference position and a stopping point;

a device locator configured to identify and select a device to be controlled from among a plurality of devices based on a direction of movement of the motion detector; and

wherein the starting point base reference position is associated with an orientation of the handheld motion device relative to the device identified and selected to be controlled;

a wireless communication interface configured to communicate information obtained by the motion detector to a processing apparatus of the selected device configured to obtain the information from the wireless communication interface to process the information.