Spatial, multi-modal control device for use with spatial operating system

• US 9,823,747 B2
• Filed: 05/27/2010
• Issued: 11/21/2017
• Est. Priority Date: 02/08/2006
• Status: Expired due to Fees

• Current Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Original Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Inventors: Underkoffler, John S., Sparrell, Carlton, Belker, Harald, Kramer, Kwindla Hultman
• Primary Examiner(s): JOSEPH, DENNIS P
• Application Number:
  US12/789,129
• Publication Number:
  US 20110025603A1
• Time in Patent Office: 2,735 Days
• Field of Search: 345156-184
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 40/107 Static hand or arm

Abstract

A system comprising an input device includes a detector coupled to a processor. The detector detects an orientation of the input device. The input device has multiple modal orientations corresponding to the orientation. The modal orientations correspond to multiple input modes of a gestural control system. The detector is coupled to the gestural control system and automatically controls selection of an input mode in response to the orientation.

Claims

1. A system comprising:

an input device; and

a detector coupled to a processor and detecting an orientation of the input device and gestures of the input device, wherein the input device has a plurality of modal orientations corresponding to the orientation, wherein the plurality of modal orientations correspond to a plurality of control modes of a gestural control system that is remote to the input device and the detector, wherein the detector is coupled via a communication channel to the gestural control system and automatically controls selection of a control mode of the plurality of control modes in response to the orientation, wherein each control mode assigns a different set of operations to the gestures from a plurality of sets of operations that include two-dimensional control operations of a remote object and three-dimensional control operations of the remote object,

wherein a first side of the input device corresponds to a first modal orientation of the plurality of modal orientations and a first control mode of the plurality of control modes,

wherein a second side of the input device corresponds to a second modal orientation of the plurality of modal orientations and a second control mode of the plurality of control modes,

wherein an orientation transition comprising rotating the input device causes a change from the first control mode to the second control mode,

wherein the detector detects an event of the input device, wherein the event corresponds to at least one of a three-space location and a three-space orientation of the input device, wherein the event corresponds to an application of a first type,

wherein the processor generates data sequences comprising input device event data specifying the event and state information of the event,

wherein the input device event data and state information are type-specific data having a type corresponding to a first application of the gestural control system,

wherein the processor forms a data capsule to include the data sequences, wherein the data capsule has a data structure comprising an application-independent representation of the data sequences,

wherein the processor places the data capsule in a repository,

wherein a second event running under an application of a second type searches the repository and identifies a correspondence between the data capsule and the second event,

wherein the application of the second type executes an operation corresponding to the second event using contents of the data sequences of the data capsule.

2. The system of claim 1, wherein the detector interprets and translates the orientation of the input device into input signals of the gestural control system.

3. The system of claim 2, wherein the detector interprets and translates orientation transitions of the input device between the plurality of modal orientations into input signals of the gestural control system.

4. The system of claim 1, wherein the detecting of the orientation comprises detecting an absolute three-space location of an instantaneous state of the input device at a point in time and space.

5. The system of claim 1, wherein the detector tracks orientation transitions of the input device between the plurality of modal orientations.

6. The system of claim 5, wherein the orientation transitions are rotational transitions about an axis of the input device.

7. The system of claim 5, wherein the orientation transitions are rotational transitions about a plurality of axes of the input device.

8. The system of claim 5, comprising applying hysteresis to the selection of the control mode during the orientation transitions.

9. The system of claim 1, wherein the detector tracks instantaneous orientation in real-time.

10. The system of claim 9, wherein the detector translates raw tracking data of the input device into six degrees of spatial orientation.

11. The system of claim 1, wherein the detector tracks instantaneous position of the input device in real-time.

12. The system of claim 11, wherein the detector translates raw tracking data of the input device into six degrees of spatial position.

13. The system of claim 1, comprising at least one input sensor, wherein the input sensor is positioned on the input device.

14. The system of claim 13, wherein the detector translates raw input sensor position data into input sensor state.

15. The system of claim 13, wherein the detector translates raw input sensor position data into input sensor transition data.

16. The system of claim 13, wherein position of the input sensor controls the plurality of control modes.

17. The system of claim 1, wherein the input device comprises the detector.

18. The system of claim 1, wherein the detector is a remote detector that is remote to the input device.

19. The system of claim 1, wherein the detector comprises an input device detector component carried by the input device and a remote detector component that is remote to the input device.

20. The system of claim 1, wherein the input device is a hand-held input device.

21. The system of claim 1, wherein the detector detects the gesture as an absolute three-space location of an instantaneous state of the input device at a point in time and space.

22. The system of claim 21, wherein the plurality of control modes comprise a direct manipulation mode in which the instantaneous state of the gesture is used for direct manipulation of an application element of a component coupled to the gestural control system.

23. The system of claim 21, wherein the plurality of control modes comprise a meta-manipulation mode in which the instantaneous state of the gesture is used for direct manipulation of a set of application elements of a component coupled to the gestural control system.

24. The system of claim 21, wherein the plurality of control modes comprise a three-dimensional manipulation mode in which the instantaneous state of the gesture is used for three-dimensional manipulation of an application element of a component coupled to the gestural control system.

25. The system of claim 21, wherein the gestural control system controls a three-space object coupled to the gestural control system.

26. The system of claim 25, comprising controlling the three-space object through three translational degrees of freedom and three rotational degrees of freedom.

27. The system of claim 26, wherein the controlling comprises a direct coupling between motion of the input device and the three-space object.

28. The system of claim 26, wherein the controlling includes an indirect coupling between motion of the input device and the three-space object.

29. The system of claim 26, wherein the three-space object is presented on a display device coupled to the processor.

30. The system of claim 26, comprising controlling movement of the three-space object by mapping the plurality of control modes of the input device to a plurality of object translations of the three-space object.

31. The system of claim 30, wherein the mapping includes a direct mapping between the plurality of control modes and the plurality of object translations.

32. The system of claim 30, wherein the mapping includes an indirect mapping between the plurality of control modes and the plurality of object translations.

33. The system of claim 30, wherein the mapping includes correlating positional offsets of the plurality of control modes to positional offsets of the object translations of the three-space object.

34. The system of claim 30, wherein the mapping includes correlating positional offsets of the input device to translational velocity of the object translations of the three-space object.

35. The system of claim 26, comprising controlling movement of the three-space object by mapping a linear gesture of the input device to a linear translation of the three-space object.

36. The system of claim 26, comprising controlling movement of the three-space object by mapping a rotational gesture of the input device to a rotational translation of the three-space object.

37. The system of claim 26, comprising controlling movement of the three-space object by mapping a linear gesture of the input device to a rotational translation of the three-space object.

38. The system of claim 26, comprising controlling movement of the three-space object by mapping a rotational gesture of the input device to a linear translation of the three-space object.

39. The system of claim 26, wherein the detecting comprises detecting when an extrapolated position of the input device intersects virtual space, wherein the virtual space comprises space depicted on a display device coupled to the gestural control system.

40. The system of claim 1, wherein the tracking is electromagnetic field (EMF) tracking.

41. The system of claim 40, wherein the input device includes the detector coupled to circuitry, wherein the detector comprises a plurality of coils.

42. The system of claim 41, comprising a field generator remote to the input device, wherein the field generator generates an EMF that induces signals in the plurality of coils.

43. The system of claim 42, comprising a plurality of field generators remote to the input device, wherein each field generator induces signals in the plurality of coils of the input device when the input device is proximate to the field generator that induces the signals.

44. The system of claim 42, wherein the detector detects the orientation of the input device using the EMF signals induced in the plurality of coils.

45. The system of claim 44, wherein the detector detects a position of the input device using the signals induced in the plurality of coils.

46. The system of claim 41, wherein the input device comprises a transmitter coupled to the processor, wherein the transmitter is a wireless transmitter.

47. The system of claim 46, wherein the transmitter communicates the orientation of the input device to the gestural control system.

48. The system of claim 46, wherein the transmitter communicates a position of the input device to the gestural control system.

49. The system of claim 46, wherein the input device comprises at least one input sensor, wherein the transmitter communicates state of the at least one input sensor to the gestural control system.

50. The system of claim 1, wherein the tracking is optical tracking.

51. The system of claim 50, comprising at least one tag connected to the input device.

52. The system of claim 51, wherein the at least one tag comprises a plurality of tags connected to a front region of the input device.

53. The system of claim 51, wherein the tracking includes dynamically detecting a position of the at least one tag.

54. The system of claim 53, wherein the tracking includes detecting position of a set of tags coupled to a region of the input device.

55. The system of claim 54, wherein each tag of the set of tags include a pattern, wherein each pattern of each tag of the set of tags is different than any pattern of any remaining tag of the plurality of tags.

56. The system of claim 55, wherein each tag includes a first pattern and a second pattern, wherein the first pattern is common to any tag of the set of tags and the second pattern is different between at least two tags of the set of tags.

57. The system of claim 54, wherein the set of tags form a plurality of patterns on the input device.

58. The system of claim 51, wherein the at least one tag comprises a set of infrared (IR) light-emitting diodes (LEDs) and a set of retro-reflective dots.

59. The system of claim 58, wherein the input device includes a microprocessor coupled to circuitry, wherein the circuitry is coupled to the set of IR LEDs.

60. The system of claim 59, comprising at least one input sensor coupled to the circuitry, wherein the at least one input sensor is positioned on the input device.

61. The system of claim 60, wherein the at least one input sensor controls a state of the set of IR LEDs.

62. The system of claim 61, wherein the state of each IR LED corresponds to at least one control mode of the plurality of control modes.

63. The system of claim 51, wherein the at least one tag comprises at least one tracking dot connected to the input device.

64. The system of claim 63, wherein the at least one tracking dot comprises at least one infrared (IR) light-emitting diode (LED).

65. The system of claim 63, wherein the at least one tracking dot comprises at least one retro-reflective dot.

66. The system of claim 63, wherein the at least one tracking dot comprises at least one of an infrared (IR) light-emitting diode (LED) and a retro-reflective dot.

67. The system of claim 63, wherein the at least one tracking dot comprises a set of IR LEDs and a set of retro-reflective dots.

68. The system of claim 1, wherein the input device comprises a plurality of sides, wherein each side of the plurality of sides corresponds to a modal orientation of the plurality of modal orientations.

69. The system of claim 68, wherein each side of the plurality of sides corresponds to a control mode of the plurality of control modes.

70. The system of claim 68, wherein each side of the plurality of sides is assigned a control mode to which the side corresponds.

71. The system of claim 68, wherein each side of the plurality of sides is dynamically assigned a control mode based on a context.

72. The system of claim 68, wherein at least one side of the plurality of sides comprises an input sensor.

73. The system of claim 68, wherein each side of the plurality of sides comprises an input sensor.

74. The system of claim 68, wherein each input sensor of each side is dynamically assigned a function based on the orientation.

75. The system of claim 68, wherein each input sensor of each side is dynamically assigned a function based on a context.

76. The system of claim 68, wherein the plurality of sides includes three sides, wherein the input device has a triangular cross-section, wherein each side of the plurality of sides corresponds to a modal orientation of the plurality of modal orientations.

77. The system of claim 76, wherein each side of the plurality of sides comprises a input sensor.

78. The system of claim 1, wherein rotating the input device comprises rotating the input device around a longitudinal axis of the input device.

79. The system of claim 1, wherein rotating the input device comprises rotating the input device around a longitudinal axis of the input device more than a threshold amount relative to the first side of the input device.

80. The system of claim 1, wherein rotating the input device comprises rotating the input device around a longitudinal axis of the input device more than a threshold amount relative to a center of the first side of the input device.

81. The system of claim 1, wherein the detector is constructed to:

responsive to detection of rotation of the input device to a first orientation of the input device that corresponds to a 2D control mode that assigns the set of two-dimensional control operations to gestures of the input device, automatically control selection of the 2D control mode at the gestural control system via the communication channel, and

responsive to detection of rotation of the input device to a second orientation of the input device that corresponds to a 3D control mode that assigns the set of three-dimensional control operations to gestures of the input device, automatically control selection of the 3D control mode at the gestural control system via the communication channel.