Motion controlled remote controller

• US 7,990,365 B2
• Filed: 06/29/2010
• Issued: 08/02/2011
• Est. Priority Date: 03/23/2004
• Status: Active Grant

• Current Assignee: Fujitsu Limited
• Original Assignee: Fujitsu Limited
• Inventors: Adler, B. Thomas, Matsumoto, Hitoshi, Marvit, David L.
• Primary Examiner(s): Liang; Regina
• Application Number:
  US12/826,439
• Publication Number:
  US 20110050569A1
• Time in Patent Office: 399 Days
• Field of Search: 345156-158, 345/168, 345/169, 715856-864
• US Class Current: 345/156
• CPC Class Codes: G06F 16/176 Support for shared access to files; File sharing support
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G08C 17/00 Arrangements for transmitting signals characterised by the use of a wireless electrical link
• CPC Class Codes: G08C 2201/12 Power saving techniques of remote control or controlled devices
• CPC Class Codes: G08C 2201/32 Remote control based on movements, attitude of remote control device
• CPC Class Codes: G08C 2201/33 Remote control using macros, scripts
• CPC Class Codes: G08C 2201/34 Context aware guidance
• CPC Class Codes: G08C 2201/42 Transmitting or receiving remote control signals via a network
• CPC Class Codes: G08C 2201/60 Security, fault tolerance
• CPC Class Codes: G08C 2201/61 Password, biometric
• CPC Class Codes: G08C 2201/70 Device selection
• CPC Class Codes: G08C 2201/92 Universal remote control
• CPC Class Codes: G08C 2201/93 Remote control using other portable devices, e.g. mobile phone, PDA, laptop

Abstract

A handheld device includes a display having a viewable surface and operable to generate an image indicating a currently controlled remote device and a gesture database maintaining a plurality of remote command gestures. Each remote command gesture is defined by a motion of the device with respect to a first position of the handheld device. The device includes a gesture mapping database comprising a mapping of each of the remote command gestures to an associated command for controlling operation of the remote device and a motion detection module operable to detect motion of the handheld device within three dimensions and to identify components of the motion in relation to the viewable surface. The device includes a control module operable to track movement of the handheld device using the motion detection module, to compare the tracked movement against the remote command gestures to determine a matching gesture, and to identify the one of the commands corresponding to the matching gesture. The device also includes a wireless interface operable to transmit the identified command to a remote receiver for delivery to the remote device.

Claims

1. A motion controlled handheld device, comprising:

a first accelerometer operable to detect acceleration along a first axis;

a second accelerometer operable to detect acceleration along a second axis, the second axis perpendicular to the first axis;

a tilt detection component operable to detect rotation having a component around at least one of the first axis and the second axis, the tilt detection component comprising;

a first camera operable to generate a first video stream;

a video analysis module operable to detect a direction of motion based on the first video stream;

a range finder operable to determine distance information including a distance between the device and an object in the first video stream, wherein the video analysis module is further operable to determine a magnitude of translation of the device using the distance;

a display operable to present a current image;

a motion tracking module operable to track motion of the device in three dimensions using the first accelerometer, the second accelerometer, and the tilt detection component; and

a controller operable to generate the current image and to modify the current image in response to the motion of the device.

2. The motion controlled handheld device of claim 1, further comprising a second camera operable to generate a second video stream.

3. The motion controlled handheld device of claim 2, wherein:

first camera and the second camera are oriented along a same axis and directed in opposite directions; and

the video analysis module is further operable to distinguish between tilt and translation based on the first video stream and the second video stream.

4. The motion controlled handheld device of claim 1, wherein the motion tracking module is further operable to determine that no significant motion is taking place.

5. The motion controlled handheld device of claim 1, wherein the motion tracking module is further operable to identify and amplify a dominant motion.

6. The motion controlled handheld device of claim 1, wherein the motion tracking module is further operable to identify and minimize a minor motion.

7. The motion controlled handheld device of claim 1:

further comprising a device application configured to generate a graphical user interface that is displayed as the current image on the display; and

wherein the motion tracking module is further operable to select for amplification at least one motion component of the tracked motion of the device and to select for minimization at least one motion component of the tracked motion of the device, wherein the selections are based on the currently displayed graphical user interface.

8. A method for controlling a handheld device, comprising:

detecting acceleration along a first axis;

detecting acceleration along a second axis, the second axis perpendicular to the first axis;

generating a first video stream;

detecting a direction of motion based on the first video stream;

determining distance information including a distance between the handheld device and an object in the first video stream;

determining a magnitude of translation of the handheld device using the distance information;

detecting rotation having a component around at least one of the first axis and the second axis;

generating and presenting a current image;

tracking motion of the device in three dimensions based on the acceleration along the first axis and the second axis, the magnitude of translation, and the rotation; and

modifying the current image in response to the tracked motion of the handheld device.

9. The method for controlling a handheld device of claim 8, further comprising generating a second video stream.

10. The method for controlling a handheld device of claim 9:

wherein the first video stream is generated by a first camera and the second video stream is generated by a second camera, and the first and second cameras are oriented along a same axis and directed in opposite directions; and

further comprising distinguishing between tilt and translation based on the first video stream and the second video stream.

11. The method for controlling a handheld device of claim 8, further comprising determining that no significant motion is taking place.

12. The method for controlling a handheld device of claim 8, further comprising identifying and amplifying a dominant motion.

13. The method for controlling a handheld device of claim 8, further comprising identifying and minimizing a minor motion.

14. The method for controlling a handheld device of claim 8, further comprising:

generating a graphical user interface that is displayed as the current image on the display;

selecting for amplification at least one motion component of the tracked motion of the handheld device; and

selecting for minimization at least one motion component of the tracked motion of the handheld device, the selections are based on the currently displayed graphical user interface.

15. Computer readable instructions encoded in a non-transitory tangible computer readable medium, that when executed by a processor of a handheld device are configured to:

detect acceleration along a first axis;

detect acceleration along a second axis, the second axis perpendicular to the first axis;

generate a first video stream;

detect a direction of motion based on the first video stream;

determine distance information including a distance between the handheld device and an object in the first video stream;

determine a magnitude of translation of the handheld device using the distance information;

detect rotation having a component around at least one of the first axis and the second axis;

generate and presenting a current image;

track motion of the device in three dimensions based on the acceleration along the first axis and the second axis, the magnitude of translation, and the rotation; and

modify the current image in response to the tracked motion of the handheld device.

16. The computer readable instructions of claim 15, wherein the instructions are further configured to generate a second video stream.

17. The computer readable instructions of claim 16, wherein:

the first video stream is generated by a first camera and the second video stream is generated by a second camera, and the first and second cameras are oriented along a same axis and directed in opposite directions; and

the instructions are further configured to distinguish between tilt and translation based on the first video stream and the second video stream.

18. The computer readable instructions of claim 15, wherein the instructions are further configured to determine that no significant motion is taking place.

19. The computer readable instructions of claim 15, wherein the instructions are further configured to identify and amplify a dominant motion.

20. The computer readable instructions of claim 15, wherein the instructions are further configured to identify and minimize a minor motion.

21. The computer readable instructions of claim 15, wherein the instructions are further configured to:

generate a graphical user interface that is displayed as the current image on the display;

select for amplification at least one motion component of the tracked motion of the handheld device; and

select for minimization at least one motion component of the tracked motion of the handheld device, the selections are based on the currently displayed graphical user interface.