Spatial, multi-modal control device for use with spatial operating system

• US 8,665,213 B2
• Filed: 05/27/2010
• Issued: 03/04/2014
• Est. Priority Date: 02/08/2006
• Status: Expired due to Fees

• Current Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Original Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Inventors: Underkoffler, John S., Sparrell, Carlton, Belker, Harald, Kramer, Kwindla Hultman
• Primary Examiner(s): Shankar, Vijay
• Application Number:
  US12/789,302
• Publication Number:
  US 20110018803A1
• Time in Patent Office: 1,377 Days
• Field of Search: 345156-169, 345173-179, 345204-215, 178 1801- 1807, 178 1901- 1906, 178 2001- 2004
• US Class Current: 345/158
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 40/107 Static hand or arm

Abstract

A system comprising an input device includes a detector coupled to a processor. The detector detects an orientation of the input device. The input device has multiple modal orientations corresponding to the orientation. The modal orientations correspond to multiple input modes of a gestural control system. The detector is coupled to the gestural control system and automatically controls selection of an input mode in response to the orientation.

Claims

1. A method comprising:

detecting and translating an orientation in which an input device is currently operated and transitions between a plurality of orientations of the input device, wherein the input device comprises a hand-held housing that includes a processor;

automatically controlling selection of an input mode of the plurality of input modes in response to the orientation and the transitions; and

controlling a three-space object coupled to the gestural control system according to the input mode;

wherein the correspondence comprises a first side of the plurality of sides corresponding to a first input mode and a second side of the plurality of sides corresponding to a second input mode, wherein the transition comprising rotating the input device around a longitudinal axis more than 120 degrees relative to a center of the first side causes a change from the first input mode to the second input mode.

2. The method of claim 1, wherein the detecting of the orientation comprises detecting an absolute three-space location of an instantaneous state of the input device at a point in time and space.

3. The method of claim 1, wherein the transitions are rotational transitions about an axis of the input device.

4. The method of claim 1, wherein the transitions are rotational transitions about a plurality of axes of the input device.

5. The method of claim 1, comprising applying hysteresis to the selection of the input mode during the transitions.

6. The method of claim 1, wherein the detecting comprises detecting instantaneous orientation in real-time.

7. The method of claim 6, wherein the translating comprises translating raw tracking data of the input device into six degrees of spatial orientation.

8. The method of claim 1, wherein the detecting comprises detecting instantaneous position of the input device in real-time.

9. The method of claim 8, wherein the translating comprises translating raw tracking data of the input device into six degrees of spatial position.

10. The method of claim 1, comprising at least one input sensor, wherein the input sensor is positioned on the input device.

11. The method of claim 1, comprising detecting and translating into input sensor state raw input sensor position data of an input sensor of the input device.

12. The method of claim 11, comprising controlling the plurality of input modes in response to the input sensor state.

13. The method of claim 1, wherein the detecting is performed onboard the input device.

14. The method of claim 1, wherein the detecting is performed remote to the input device.

15. The method of claim 1, wherein the detecting is performed onboard the input device and performed remote to the input device.

16. The method of claim 1, wherein the input device is a hand-held input device.

17. The method of claim 1, wherein the detecting comprises detecting an absolute three-space location of an instantaneous state of the input device at a point in time and space.

18. The method of claim 17, wherein the plurality of input modes comprise a direct manipulation mode in which the instantaneous state is used for direct manipulation of an application element of a component coupled to the gestural control system.

19. The method of claim 17, wherein the plurality of input modes comprise a meta-manipulation mode in which the instantaneous state is used for direct manipulation of a set of application elements of a component coupled to the gestural control system.

20. The method of claim 17, wherein the plurality of input modes comprise a three-dimensional manipulation mode in which the instantaneous state is used for three-dimensional manipulation of an application element of a component coupled to the gestural control system.

21. The method of claim 17, comprising controlling via the gestural control system a three-space object coupled to the gestural control system.

22. The method of claim 21, comprising controlling the three-space object through three translational degrees of freedom and three rotational degrees of freedom.

23. The method of claim 22, wherein the controlling comprises a direct coupling between motion of the input device and the three-space object.

24. The method of claim 22, wherein the controlling includes an indirect coupling between motion of the input device and the three-space object.

25. The method of claim 22, comprising presenting the three-space object on a display device.

26. The method of claim 22, comprising controlling movement of the three-space object by mapping the plurality of input modes of the input device to a plurality of object translations of the three-space object.

27. The method of claim 26, wherein the mapping includes a direct mapping between the plurality of input modes and the plurality of object translations.

28. The method of claim 26, wherein the mapping includes an indirect mapping between the plurality of input modes and the plurality of object translations.

29. The method of claim 26, wherein the mapping includes correlating positional offsets of the plurality of input modes to positional offsets of the object translations of the three-space object.

30. The method of claim 26, wherein the mapping includes correlating positional offsets of the input device to translational velocity of the object translations of the three-space object.

31. The method of claim 22, comprising controlling movement of the three-space object by mapping a linear gesture of the input device to a linear translation of the three-space object.

32. The method of claim 22, comprising controlling movement of the three-space object by mapping a rotational gesture of the input device to a rotational translation of the three-space object.

33. The method of claim 22, comprising controlling movement of the three-space object by mapping a linear gesture of the input device to a rotational translation of the three-space object.

34. The method of claim 22, comprising controlling movement of the three-space object by mapping a rotational gesture of the input device to a linear translation of the three-space object.

35. The method of claim 22, wherein the detecting comprises detecting when an extrapolated position of the input device intersects virtual space, wherein the virtual space comprises space depicted on a display device coupled to the gestural control system.

36. The method of claim 1, wherein the detecting comprises an event of the input device, wherein the event corresponds to at least one of a three-space location and a three-space orientation of the input device, wherein the event corresponds to an application of a first type.

37. The method of claim 36, comprising generating data sequences comprising input device event data specifying the event and state information of the event, wherein the input device event data and state information are type-specific data having a type corresponding to a first application of the gestural control system.

38. The method of claim 37, comprising forming a data capsule to include the data sequences, wherein the data capsule has a data structure comprising an application-independent representation of the data sequences.

39. The method of claim 38, comprising placing the data capsule in a repository.

40. The method of claim 39, wherein a second event running under an application of a second type searches the repository and identifies a correspondence between the data capsule and the second event, wherein the application of the second type executes an operation corresponding to the second event using contents of the data sequences of the data capsule.

41. The method of claim 1, wherein the detecting uses electromagnetic field (EMF) tracking.

42. The method of claim 41, comprising generating an EMF that induces signals in a plurality of coils of the input device.

43. The method of claim 42, comprising generating an EMF using a plurality of field generators remote to the input device, wherein each field generator induces signals in the plurality of coils of the input device when the input device is proximate to the field generator that induces the signals.

44. The method of claim 42, wherein the detecting comprises detecting the orientation of the input device using the EMF signals induced in the plurality of coils.

45. The method of claim 44, wherein the detecting comprises detecting a position of the input device using the signals induced in the plurality of coils.

46. The method of claim 41, comprising communicating the orientation of the input device to the gestural control system.

47. The method of claim 41, comprising communicating a position of the input device to the gestural control system.

48. The method of claim 41, comprising communicating a state of at least one input sensor of the input device to the gestural control system.

49. The method of claim 1, wherein the detecting uses optical tracking.

50. The method of claim 49, comprising tracking at least one tag connected to the input device.

51. The method of claim 50, wherein the at least one tag comprises a plurality of tags connected to a front region of the input device.

52. The method of claim 50, wherein the tracking includes dynamically detecting a position of the at least one tag.

53. The method of claim 52, wherein the tracking includes detecting position of a set of tags coupled to a region of the input device.

54. The method of claim 53, wherein each tag of the set of tags include a pattern, wherein each pattern of each tag of the set of tags is different than any pattern of any remaining tag of the plurality of tags.

55. The method of claim 54, wherein each tag includes a first pattern and a second pattern, wherein the first pattern is common to any tag of the set of tags and the second pattern is different between at least two tags of the set of tags.

56. The method of claim 53, wherein the set of tags faun a plurality of patterns on the input device.

57. The method of claim 50, wherein the at least one tag comprises a set of infrared (IR) light-emitting diodes (LEDs) and a set of retro-reflective dots.

58. The method of claim 57, comprising controlling a state of the set of IR LEDs via at least one input sensor of the input device.

59. The method of claim 58, wherein the state of each IR LED corresponds to at least one input mode of the plurality of input modes.

60. The method of claim 50, wherein the at least one tag comprises at least one tracking dot connected to the input device.

61. The method of claim 60, wherein the at least one tracking dot comprises at least one infrared (IR) light-emitting diode (LED).

62. The method of claim 60, wherein the at least one tracking dot comprises at least one retro-reflective dot.

63. The method of claim 60, wherein the at least one tracking dot comprises at least one of an infrared (IR) light-emitting diode (LED) and a retro-reflective dot.

64. The method of claim 60, wherein the at least one tracking dot comprises a set of IR LEDs and a set of retro-reflective dots.

65. The method of claim 1, comprising associating each side of a plurality of sides of the input device with an orientation of a plurality of orientations.

66. The method of claim 65, comprising defining a correspondence between each side of the plurality of sides and an input mode of the plurality of input modes.

67. The method of claim 65, comprising assigning each side of the plurality of sides to an input mode.

68. The method of claim 65, comprising dynamically assigning each side of the plurality of sides to an input mode based on a context.

69. The method of claim 65, comprising dynamically assigning a function to an input sensor of a side based on the orientation.

70. The method of claim 65, comprising dynamically assigning a function to an input sensor of each side based on a context.

71. The method of claim 65, wherein the plurality of sides includes three sides, wherein the input device has a triangular cross-section, comprising associating each side of the plurality of sides to an orientation of the plurality of orientations.

72. The method of claim 65, comprising defining a correspondence between each side of the plurality of sides and an input mode of the plurality of input modes.

73. The method of claim 1, comprising applying a hysteresis band to the selection of the input mode during the transition.

74. The method of claim 73, wherein the hysteresis band is at least one of equal to and greater than approximately 30 degrees.

75. The method of claim 73, wherein the hysteresis band is programmable.

76. A method comprising:

automatically detecting a gesture of an input device from gesture data received via a detector, wherein the gesture data is absolute three-space orientation data of an instantaneous state of the input device at a point in time and space, and identifying the gesture using only the gesture data;

translating the gesture to a gesture signal; and

automatically controlling selection of an input mode of a plurality of input modes of the input device in response to the gesture signal;

wherein the correspondence comprises a first side of the plurality of sides corresponding to a first input mode and a second side of the plurality of sides corresponding to a second input mode, wherein the transition comprising rotating the input device around a longitudinal axis more than 120 degrees relative to a center of the first side causes a change from the first input mode to the second input mode.

77. A method comprising:

detecting an orientation of an input device having a plurality of modal orientations corresponding to the orientation, wherein the plurality of modal orientations correspond to a plurality of input modes of a gestural control system;

automatically controlling selection of an input mode of the plurality of input modes in response to the orientation; and

controlling a three-space object coupled to the gestural control system according to the input mode;

wherein the correspondence comprises a first side of the plurality of sides corresponding to a first input mode and a second side of the plurality of sides corresponding to a second input mode, wherein the transition comprising rotating the input device around a longitudinal axis more than 120 degrees relative to a center of the first side causes a change from the first input mode to the second input mode.