Motion control used as controlling device

• US 9,007,299 B2
• Filed: 09/30/2011
• Issued: 04/14/2015
• Est. Priority Date: 07/14/2006
• Status: Expired due to Fees

• Current Assignee: Wei Yen
• Original Assignee: AiLive, Inc.
• Inventors: Zhu, Youding, Powers, III, William Robert, Yen, Wei, Chakravorty, Anupam, Wilkinson, Dana, Reynolds, Stuart, Kay, Robert, Musick, JR., Charles
• Primary Examiner(s): Sharifi-Tafreshi, Koosha
• Application Number:
  US13/249,251
• Publication Number:
  US 20120256835A1
• Time in Patent Office: 1,292 Days
• Field of Search: 345/156, 345/158, 345/161, 345/163, 345/173, 345/157, 345/169, 463/36, 463/7, 463/37, 715/863
• US Class Current: 345/156
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/2145 the surface being also a display device, e.g. touch screens
• CPC Class Codes: A63F 13/22 Setup operations, e.g. calibration, key configuration or button assignment
• CPC Class Codes: A63F 13/24 Constructional details thereof, e.g. game controllers with detachable joystick handles
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 2300/1018 Calibration; Key and button assignment
• CPC Class Codes: A63F 2300/1043 being characterized by constructional details
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/1075 using a touch screen
• CPC Class Codes: A63F 2300/6027 using adaptive systems learning from user actions, e.g. for skill level adjustment
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: G01C 21/3664 Details of the user input interface, e.g. buttons, knobs or sliders, including those provided on a touch screen; remote controllers; input using gestures
• CPC Class Codes: G06F 18/40 Software arrangements specially adapted for pattern recognition, e.g. user interfaces or toolboxes therefor
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06V 10/945 User interactive design; Environments; Toolboxes
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

Techniques for using a motion sensitive device as a controller are disclosed. A motion controller as an input/control device is used to control an existing electronic device (a.k.a., controlled device) previously configured for taking inputs from a pre-defined controlling device. The signals from the input device are in a different form from the pre-defined controlling device. According to one aspect of the present invention, the controlled device was designed to respond to signals from a pre-defined controlling device (e.g., a touch-screen device). The inputs from the motion controller are converted into touch-screen like signals that are then sent to the controlled device or programs being executed in the controlled device to cause the behavior of the controlled device to change or respond thereto, without reconfiguration of the applications running on the controlled device.

Claims

1. A system for motion-based control of a controlled device, the system comprising:

a motion sensitive device including a plurality of sensors generating sensor signals responsive to motion of the motion sensitive device, the sensor signals being sufficient to estimate positions and orientations of the motion sensitive device in six degrees of freedom, wherein at least one application, being executed on the controlled device, is preconfigured to be responsive to input signals from a touch-screen device;

a processing unit configured to facilitate the motion sensitive device to control the at least one application on the controlled device by converting a first set of signals from the motion sensitive device to a part of a second set of signals, the second set of signals including some signals substantially similar to the input signals from the touch-screen device, wherein the first set of signals includes the sensor signals; and

at least one external sensor reference, wherein the estimated positions and orientations of the motion sensitive device are corrected with respect to a signal from the external sensor reference before converting the first set of signals to the second set of signals.

2. The system as recited in claim 1, wherein said converting the first set of signals from the motion sensitive device to the part of the second set of signals includes a transformation from the first set of signals being non-touch-surface-like signals to the part of the second set of signals being touch-surface-like signals.

3. The system as recited in claim 2, wherein the transformation includes an operation of augmenting or replacing at least some of the touch-screen input signals of the touch-screen device.

4. The system as recited in claim 2, wherein the second set of signals is used to control one or more functions of the controlled device configured to be controlled by the touch-screen device.

5. The system as recited in claim 4, wherein the one or more functions include some or all of content searching, browsing, navigating, selecting and controlling an application that the touch-screen device is used to control in the controlled device.

6. The system as recited in claim 1, wherein the controlled device includes a display, and the at least one application is being displayed on the display of the controlled device.

7. The system as recited in claim 6, wherein the controlled device is coupled to the touch-screen device, and the at least one application is being executed in the touch-screen device.

8. The system as recited in claim 7, wherein the first set of signals is remotely received in the touch-screen device, behaviors of the controlled device are controlled by the motion sensitive device via the touch-screen device.

9. The system as recited in claim 6, wherein the processing unit is included in the controlled device, and the first set of signals is remotely received in the controlled device.

10. The system as recited in claim 1, wherein the estimated positions and orientations of the motion sensitive device are corrected with respect to a fixed sensor reference before converting the first set of signals to the second set of signals.

11. The system as recited in claim 10, wherein the second set of signals includes a series of signals from multi-touch interactions by a user with the motion sensitive device.

12. The system as recited in claim 1, wherein the external sensor takes images of the motion sensitive device.

13. The system as recited in claim 12, wherein the external sensor reference includes a video stream from a video camera.

14. The system as recited in claim 1, wherein the processing unit is further configured to map a frame of reference of the motion sensitive device to a frame of reference of a display of the controlled device.

15. The system as recited in claim 14, wherein the positions and orientations of the motion sensitive device in six degrees of freedom are continuously mapped to the frame of reference of the display to allow down-the-barrel pointing at the display to control at least one object being displayed on the display.

16. The system as recited in claim 1, wherein the touch-screen device is one of a tablet, a smart phone, and an electronic device with a touch-screen input element.

17. The system as recited in claim 16, wherein the controlled device is one of a television, a monitor, a projected graphics system and a computing device with a display.

18. The system as recited in claim 17, wherein the at least one application controls a digital video recorder (DVR), a streaming movie system, a game, a live TV, or a visual application accessible on a TV.

19. The system as recited in claim 1, wherein the sensor signals are first transformed to motion-based gestures before being further transformed to the part of the second set of signals to control the controlled device, allowing gesture-based control of the at least one application responsive to the input signals from the motion sensitive device.

20. A method claim for motion-based control of a controlled device, the method comprising:

generating in a motion sensitive device a set of sensor signals responsive to motion of the motion sensitive device, the sensor signals from a plurality of sensors enclosed in the motion sensitive device and being sufficient to estimate positions and orientations of the motion sensitive device in six degrees of freedom;

converting a first set of signals from the motion sensitive device to a part of a second set of signals to control at least one application being executed on the controlled device, the second set of signals including some signals substantially similar to touch-screen input signals from a touch-screen device, wherein the first set of signals includes the sensor signals, and the controlled device is designed to be responsive to the touch-screen input signals from the touch-screen device; and

receiving a signal from at least one external sensor reference, wherein estimated positions and orientations of the motion sensitive device are corrected with respect to the signal before converting the first set of signals to the second set of signals.

21. The method as recited in claim 20, wherein said converting a first set of signals from the motion sensitive device to a part of a second set of signals includes a transformation from the first set of signals being non-touch-surface-like signals to the second set of signals being touch-surface-like signals.

22. The method as recited in claim 21, wherein the transformation includes an operation of augmenting or replacing at least some of the touch-screen input signals of the touch-screen device.

23. The method as recited in claim 22, wherein the second set of signals is used to control one or more functions of the controlled device that is configured to be controlled by the touch-screen device.

24. The method as recited in claim 23, wherein the one or more functions include some or all of content searching, browsing, navigating, selecting and controlling an application that the touch-screen device is used to control in the controlled device.

25. The method as recited in claim 20, wherein the controlled device includes a display, and the at least one application is being displayed on the display of the controlled device.

26. The method as recited in claim 25, wherein the controlled device is coupled to the touch-screen device, and the at least one application is being executed in the touch-screen device.

27. The method as recited in claim 26, wherein the first set of signals is remotely received in the touch-screen device, behaviors of the controlled device are controlled by the motion sensitive device via the touch-screen device.

28. The method as recited in claim 20, wherein the controlled device includes a processing unit, and the first set of signals is remotely received in the controlled device to be converted to the second set of signals.

29. The method as recited in claim 20, wherein the estimated positions and orientations of the motion sensitive device are corrected with respect to a fixed sensor reference before converting the first set of signals to a part of the second set of signals.

30. The method as recited in claim 29, wherein the second set of signals includes a series of signals from multi-touch interactions by a user with the motion sensitive device.

31. The method as recited in claim 20, wherein the at least one external sensor takes images of the motion sensitive device.

32. The method as recited in claim 31, wherein the external sensor reference includes a video stream from a video camera.

33. The method as recited in claim 20, further comprising mapping a frame of reference of the motion sensitive device to a frame of reference of a display of the controlled device.

34. The method as recited in claim 20, wherein the controlled device includes a display, and further comprising mapping continuously the positions and orientations of the motion sensitive device in six degrees of freedom to a frame of reference of the display to allow down-the-barrel pointing at the display to control at least one object being displayed in the display.

35. The method as recited in claim 34, wherein the touch-screen device is one of a tablet, a smart phone, and an electronic device with a touch screen input element.

36. The system as recited in claim 35, wherein the display is one of a television, a monitor, a projected graphics system and a computing device with a display.

37. The system as recited in claim 36, wherein the at least one application controls a digital video recorder (DVR), a streaming movie system, a game, a live TV, or a visual application accessible on a TV.

38. A method for motion-based control of a controlled device, the method comprising:

receiving a first set of sensor signals from sensors in a motion sensitive device, the sensor signals being sufficient to estimate positions and orientations of the motion sensitive device in six degrees of freedom;

receiving a second set of sensor signals from a secondary sensor that provides a fixed frame of reference;

determining, from the first and second sets of signals, pointing locations of the motion sensitive device relative to the fixed frame of reference, wherein said determining pointing locations of the motion sensitive device comprises;

correcting the estimated positions and orientations of the motion sensitive device with respect to the second sets of signals; and

deriving the pointing locations of the motion sensitive device from the corrected estimated positions and orientations of the motion sensitive device with respect to the fixed frame of reference; and

producing a set of control signals from a transformation to control the controlled device, wherein the transformation is configured to transform at least some of the control signals to some of the control signals substantially similar to input signals from a predefined device provided to control the controlled device.

39. The method as recited in claim 38, wherein the secondary sensor is an external sensor disposed with respect to a display of the controlled device.

40. The method as recited in claim 39, wherein the external sensor is an LED sensor, a magnetometer or a camera.

41. The method as recited in claim 38, further comprising mapping the pointing locations of the motion sensitive device to corresponding positions on a display of the controlled device.

42. The method as recited in claim 38, wherein the secondary sensor is provided to monitor an area substantially covering movements of the motion sensitive device.

43. The method as recited in claim 42, wherein the sensors in the motion sensitive device include a plurality of inertial sensors.

44. The method as recited in claim 43, wherein at least one application is being executed and displayed on the display of the controlled device.

45. The method as recited in claim 43, wherein at least one application being executed in the controlled device is designed to listen for and respond specifically to touch-surface-like signals, and wherein the at least some of the control signals are the touch-surface-like signals.

46. The method as recited in claim 45, wherein the at least one application is executed with one or more software components to perform mouse functions in responding to the at least some of the control signals.

47. The method as recited in claim 43, wherein at least one application being executed in the controlled device is designed to listen for and respond specifically to one or more of touch-screen-like signals, button-like signals, joystick-like signals, and text-like signals.

48. The method as recited in claim 47, wherein the touch-surface-like signals contain information representing 2D coordinates on a 2D surface.

49. The method as recited in claim 38, wherein the transformation is provided as a convenience to application programmers who are now freed from having to consider a large variety of input devices subsequently used to control the controlled device.

50. The method as recited in claim 49, wherein the transformation is configured to determine dynamically the first set of signals from the motion sensitive device based upon a contextual identifier before the first set of signals is transformed.

51. The method as recited in claim 49, wherein the transformation is configured to transform joystick or button signals of the motion sensitive device into at least some of the control signals.

52. The system as recited in claim 43, wherein the controlled device is coupled to a touch-screen device, and the at least one application is being executed in the touch-screen device, and wherein the first and second sets of signals are remotely received in the touch-screen device, behaviors of the controlled device is controlled by the motion sensitive device via the touch-screen device.

53. The method as recited in claim 52, wherein said generating a set of control signals from a mapping to control the controlled device comprises:

augmenting or replacing at least some of touch-screen inputs of the touch-screen device to subsequently control behavior in at least one application being executed on the controlled device.

54. The method as recited in claim 38, further comprising causing the motion sensitive device to initiate an initial calibration setup before the first set of sensor signals from sensors in the motion sensitive device is used to estimate the positions and orientations of the motion sensitive device in six degrees of freedom.

55. The method as recited in claim 54, wherein the pointing locations are transformed to navigate and interact with a menu displayed on the controlled device.

56. The method as recited in claim 55, wherein absolute pointing locations are combined with button press inputs to the motion sensitive device, and transformed to provide multi-touch touch-screen inputs to the controlled device.

57. The method as recited in claim 54, wherein a context from the controlled device includes information pertaining to an application being displayed on a display of the controlled device and a current pointing location of the motion sensitive device, and further comprising modifying any subsequent touch-screen inputs that are used to control the application.

58. The method as recited in claim 57, wherein a hierarchy of contexts is available for different categories of applications so that other applications run immediately without recalibration of the motion sensitive device.

59. The method as recited in claim 38, wherein the set of control signals includes a series of signals from multi-touch interactions by a user with the motion sensitive device.

60. The method as recited in claim 59, wherein the series of signals include dual signals simulating a stationary touch with constant position at a pointed position by the motion sensitive device when a button on the motion sensitive device is first depressed, followed by dragging the motion sensitive device through the pointed position until the button is released.

61. The method as recited in claim 60, wherein the series of signals allow one-button activations by the user to simulate multi-touch signals sufficient to allow several common uses in a touch screen device for zooming in or out on a display or rotating an image being displayed.

62. The method as recited in claim 38, wherein the control signals includes signals from one or more other motion sensitive devices pointing at several positions of the display simultaneously, the positions are then combined to produce a multi-touch selection.