MULTI-PROCESS INTERACTIVE SYSTEMS AND METHODS

• US 20100127968A1
• Filed: 10/14/2009
• Published: 05/27/2010
• Est. Priority Date: 04/24/2008
• Status: Active Grant

• Current Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Original Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Inventors: Underkoffler, John S., Kramer, Kwindla Hultman
• Granted Patent: US 9,052,970 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications
• CPC Class Codes: G06F 9/544 Buffers; Shared memory; Pipes

Abstract

A multi-process interactive system is described. The system includes numerous processes running on a processing device. The processes include separable program execution contexts of application programs, such that each application program comprises at least one process. The system translates events of each process into data capsules. A data capsule includes an application-independent representation of event data of an event and state information of the process originating the content of the data capsule. The system transfers the data messages into pools or repositories. Each process operates as a recognizing process, where the recognizing process recognizes in the pools data capsules comprising content that corresponds to an interactive function of the recognizing process and/or an identification of the recognizing process. The recognizing process retrieves recognized data capsules from the pools and executes processing appropriate to contents of the recognized data capsules.

Claims

1. A method comprising:

detecting from gesture data a gesture made by a body, wherein the gesture data is received via a detector;

executing a plurality of processes on a processing device, the plurality of processes generating events, the events including a set of events representing the gesture;

translating the events of each process of the plurality of processes into data capsules;

transferring the data capsules into a plurality of pools;

a set of processes of the plurality of processes operating as recognizing processes, the recognizing processes recognizing in the plurality of pools data capsules comprising content that corresponds to the gesture; and

the recognizing processes retrieving recognized data capsules from the plurality of pools and generating a gesture signal from the recognized data capsules by compositing contents of the recognized data capsules to form the gesture signal, wherein the gesture signal represents the gesture.

2. The method of claim 1, wherein the plurality of processes includes separable program execution contexts of a spatial operating application.

3. The method of claim 1, wherein the gesture data is absolute three-space location data of an instantaneous state of a user at a point in time and space.

4. The method of claim 1, comprising identifying the gesture using only the gesture data.

5. The method of claim 1, wherein the detecting includes at least one of detecting a location of the body, detecting an orientation of the body, and detecting motion of the body.

6. The method of claim 1, comprising identifying the gesture, wherein the identifying includes identifying a pose and an orientation of a portion of the body.

7. The method of claim 1, wherein the detecting includes detecting at least one of a first set of appendages and a second set of appendages of the body.

8. The method of claim 1, wherein the detecting includes dynamically detecting a position of at least one tag coupled to the body.

9. The method of claim 8, wherein the detecting includes detecting position of a set of tags coupled to the body.

10. The method of claim 9, wherein each tag of the set of tags includes a pattern, wherein each pattern of each tag of the set of tags is different than any pattern of any remaining tag of the plurality of tags.

11. The method of claim 1, wherein the detecting includes dynamically detecting and locating a marker on the body.

12. The method of claim 11, wherein the detecting includes detecting position of a set of markers coupled to the body.

13. The method of claim 11, wherein the set of markers form a plurality of patterns on the body.

14. The method of claim 11, wherein the detecting includes detecting position of a plurality of appendages of the body using a set of markers coupled to each of the appendages.

15. The method of claim 1, wherein the translating comprises translating information of the gesture to a gesture notation.

16. The method of claim 15, wherein the gesture notation represents a gesture vocabulary, and the gesture signal comprises communications of the gesture vocabulary.

17. The method of claim 16, wherein the gesture vocabulary represents in textual form instantaneous pose states of kinematic linkages of the body.

18. The method of claim 16, wherein the gesture vocabulary represents in textual form an orientation of kinematic linkages of the body.

19. The method of claim 16, wherein the gesture vocabulary represents in textual form a combination of orientations of kinematic linkages of the body.

20. The method of claim 16, wherein the gesture vocabulary includes a string of characters that represent a state of kinematic linkages of the body.

21. The method of claim 20, wherein the kinematic linkage is at least one first appendage of the body.

22. The method of claim 21, comprising assigning each position in the string to a second appendage, the second appendage connected to the first appendage.

23. The method of claim 22, comprising assigning characters of a plurality of characters to each of a plurality of positions of the second appendage.

24. The method of claim 23, wherein the plurality of positions is established relative to a coordinate origin.

25. The method of claim 24, comprising establishing the coordinate origin using a position selected from a group consisting of an absolute position and orientation in space, a fixed position and orientation relative to the body irrespective of an overall position and heading of the body, and interactively in response to an action of the body.

26. The method of claim 23, comprising assigning characters of the plurality of characters to each of a plurality of orientations of the first appendage.

27. The method of claim 21, wherein the detecting comprises detecting when an extrapolated position of the body intersects virtual space, wherein the virtual space comprises space depicted on a display device coupled to the at least one processing device.

28. The method of claim 27, comprising controlling a virtual object in the virtual space when the extrapolated position intersects the virtual object.

29. The method of claim 28, wherein the controlling comprises controlling a position of the virtual object in the virtual space in response to the extrapolated position in the virtual space.

30. The method of claim 28, wherein the controlling comprises controlling attitude of the virtual object in the virtual space in response to the gesture.

31. The method of claim 1, comprising controlling scaling of the detecting and controlling to generate coincidence between virtual space and the physical space, wherein the virtual space comprises space depicted on a display, wherein the physical space comprises space inhabited by the body.

32. The method of claim 31, comprising controlling at least one virtual object in the virtual space in response to movement of at least one physical object in the physical space.

33. The method of claim 1, comprising controlling a component using the gesture signal, the component coupled to the at least one processing device.

34. The method of claim 33, wherein controlling the component comprises controlling a three-space object in six degrees of freedom simultaneously by mapping the gesture to a three-space object.

35. The method of claim 33, wherein controlling the component comprises controlling a three-space object through three translational degrees of freedom and three rotational degrees of freedom.

36. The method of claim 35, wherein the three-space object is presented on a display device coupled to the at least one processing device.

37. The method of claim 35, wherein the three-space object is a remote system coupled to the computer.

38. The method of claim 35, comprising controlling movement of the three-space object by mapping the gesture to a plurality of object translations of the three-space object.

39. The method of claim 38, wherein the mapping includes a direct mapping between the gesture and the plurality of object translations.

40. The method of claim 38, wherein the mapping includes an indirect mapping between the gesture and the plurality of object translations.

41. The method of claim 1, wherein a data capsule includes an application-independent representation of event data of an event and state information of the process originating the data message.

42. The method of claim 1, comprising forming an interactive application from the plurality of processes by coordinating operations of each of the plurality of processes using the data capsules and the plurality of pools.

43. The method of claim 1, comprising coordinating operations of the plurality of processes using at least one of the data capsules and the plurality of pools.

44. The method of claim 1, comprising dividing an application program into a set of processes, wherein the plurality of processes includes the set of processes.

45. The method of claim 1, comprising a process generating an output by interactively processing a plurality of retrieved data capsules of at least one pool of the plurality of pools.

46. The method of claim 1, wherein the plurality of processes includes separable program execution contexts of a plurality of application programs, wherein each application program comprises at least one process.

47. The method of claim 1, comprising executing the plurality of processes in parallel.

48. The method of claim 1, comprising executing a first set of processes in parallel, and executing a second set of processes in sequence, wherein the plurality of processes includes the first set of processes and the second set of processes.

49. The method of claim 1, wherein the events represent process inputs.

50. The method of claim 1, wherein the events represent process outputs.

51. The method of claim 1, wherein the events comprise a user interface event.

52. The method of claim 1, wherein the events comprise a graphics event.

53. The method of claim 1, wherein the events represent process state.

54. The method of claim 53, wherein the state of a process represents the interactive function of the process, wherein the interactive function of the process is exposed to the plurality of processes as contents of the data capsules.

55. The method of claim 54, comprising defining an application programming interface (API) of the plurality of processes by contents of the data capsules instead of defining the API by function calls.

56. The method of claim 55, wherein the contents of the data capsules are application-independent and recognizable by the plurality of processes.

57. The method of claim 1, wherein the at least one processing device comprises a plurality of processing devices.

58. The method of claim 57, wherein at least one first set of processes of the plurality of processes are running under at least one first set of processing devices of the plurality of processing devices and at least one second set of processes of the plurality of processes are running under at least one second set of processing devices of the plurality of processing devices.

59. The method of claim 1, wherein the plurality of processes includes a first process.

60. The method of claim 59, wherein the translating comprises transforming events of the first process into at least one data sequence comprising first process event data specifying the event and state information of the event.

61. The method of claim 60, wherein the first process event data and state information are type-specific data having a type corresponding to an application of the first process.

62. The method of claim 61, wherein the translating comprises forming the data capsule to include the at least one data sequence, the data capsule having a data structure comprising an application-independent representation of the at least one data sequence.

63. The method of claim 59, wherein the plurality of processes includes a second process.

64. The method of claim 63, wherein the translating comprises transforming state change events of the second process into at least one data sequence comprising second process event data specifying the event and state information of the event.

65. The method of claim 64, wherein the second process event data and state information are type-specific data having a type corresponding to an application of the second process.

66. The method of claim 65, wherein the translating comprises forming the data capsule to include the at least one data sequence, the data capsule having a data structure comprising an application-independent representation of the at least one data sequence.

67. The method of claim 63, wherein the recognizing process is the second process, the retrieving comprising the second process retrieving the recognized data capsules from the plurality of pools and executing processing appropriate to contents of the recognized data capsules.

68. The method of claim 67, wherein the contents of the recognized data capsules are data representing state information of the first process.

69. The method of claim 68, wherein the translating comprises transforming the contents of the recognized data capsules into at least one new data sequence, the at least one new data sequence representing at least one of an event of the first process and an event of the second process.

70. The method of claim 69, wherein the at least one new data sequence comprises event data specifying the event and state information of at least one of the first process and the second process.

71. The method of claim 70, wherein the event data and state information of the at least one of the first process and the second process are type-specific data having a type corresponding to an application of the at least one of the first process and the second process.

72. The method of claim 71, wherein the translating comprises forming the data capsule to include the at least one new data sequence, the data capsule having a data structure comprising an application-independent representation of the at least one new data sequence.

73. The method of claim 72, wherein the plurality of processes uses the at least one new data sequence.

74. The method of claim 1, wherein the processing appropriate to contents of the recognized data capsules comprises rendering a graphical object, wherein the graphical object is rendered on a display of the at least one processing device.

75. The method of claim 74, wherein the rendering comprises direct rendering in which the plurality of processes draw directly to a graphics layer of the at least one processing device, wherein the plurality of pools is used for coordination among the plurality of processes as appropriate to the rendering.

76. The method of claim 74, wherein the rendering comprises the plurality of processes:

transferring data capsules comprising rendering commands into the plurality of pools; and

retrieving the rendering commands from the plurality of pools, interpreting the rendering commands, and driving a graphics layer of the at least one processing device in response to the rendering commands.

77. The method of claim 74, wherein the rendering comprises the plurality of processes:

rendering to pixel buffers;

transferring raw frame data into the plurality of pools, the raw frame data resulting from the rendering to the pixel buffers; and

retrieving the raw frame data from the plurality of pools and combining the raw frame data for use in driving a graphics layer of the at least one processing device.

78. The method of claim 1, comprising:

detecting an event of the plurality of processes;

generating at least one data sequence comprising event data specifying the event and state information of the event, wherein the event data and state information are type-specific data having a type corresponding to an application of the at least one processing device; and

forming a data capsule to include the at least one data sequence, the data capsule having a data structure comprising an application-independent representation of the at least one data sequence.

79. The method of claim 1, wherein the generating of the at least one data sequence comprises:

generating a first respective data set that includes first respective event data;

generating a second respective data set that includes second respective state information; and

forming a first data sequence to include the first respective data set and the second respective data set.

80. The method of claim 79, wherein the generating of the first respective data set includes forming the first respective data set to include identification data of the at least one processing device, the identification data including data identifying the at least one processing device.

81. The method of claim 79, wherein the generating of the at least one data sequence comprises:

generating a first respective data set that includes first respective event data;

generating a second respective data set that includes second respective state information; and

forming a second data sequence to include the first respective data set and the second respective data set.

82. The method of claim 81, wherein the generating of the first respective data set includes generating a first respective data set offset, wherein the first respective data set offset points to the first respective data set of the second data sequence.

83. The method of claim 81, wherein the generating of the second respective data set includes generating a second respective data set offset, wherein the second respective data set offset points to the second respective data set of the second data sequence.

84. The method of claim 79, wherein the first respective data set is a description list, the description list including a description of the data.

85. The method of claim 1, wherein the event data is a tagged byte-sequence representing typed data.

86. The method of claim 85, wherein the event data includes a type header and a type-specific data layout.

87. The method of claim 1, wherein the state information is a tagged byte-sequence representing typed data.

88. The method of claim 87, wherein the state information includes a type header and a type-specific data layout.

89. The method of claim 1, comprising:

generating at least one offset; and

forming the data capsule to include the at least one offset.

90. The method of claim 89, comprising:

generating a first offset having a first variable length;

wherein the first offset points to the event data of a first data sequence of the at least one data sequence.

91. The method of claim 89, comprising:

generating a second offset having a second variable length;

wherein the second offset points to the state information of a first data sequence of the at least one data sequence.

92. The method of claim 89, comprising:

forming a first code path through the data capsule using a first offset of the at least one offset;

forming a second code path through the data capsule using a second offset of the at least one offset;

wherein the first code path and the second code path are different paths.

93. The method of claim 89, wherein at least one of the first offset and the second offset include metadata, the metadata comprising context-specific metadata corresponding to a context of the application.

94. The method of claim 1, comprising:

generating a header that includes a length of the data capsule;

forming the data capsule to include the header.

95. The method of claim 1, comprising transferring the data capsule to a pool of the plurality of pools.

96. The method of claim 95, comprising:

detecting a second event of the at least one processing device;

searching the plurality of pools for data capsules corresponding to the second event.

97. The method of claim 96, comprising:

identifying a correspondence between the data capsule and the second event;

extracting the data capsule from the pool in response to the identifying; and

executing on behalf of the at least one processing device a processing operation corresponding to the second event in response to contents of the data capsule, wherein the at least one processing device corresponds to an application of a first type and a second application of a second type.

98. The method of claim 95, wherein the plurality of pools is coupled to a plurality of applications, the plurality of pools including a plurality of data capsules corresponding to the plurality of applications, the plurality of pools providing access to the plurality of data capsules by the plurality of applications, wherein at least two applications of the plurality of applications are different applications.

99. The method of claim 95, wherein the plurality of pools provides state caching of a plurality of data capsules.

100. The method of claim 95, wherein the plurality of pools provides linear sequencing of a plurality of data capsules.

101. The method of claim 1, wherein the data structure is untyped.

102. The method of claim 1, wherein the data structure of the data capsule provides a platform-independent representation of the event data and the state information.

103. The method of claim 1, wherein the data structure of the data capsule provides platform-independent access to the event data and the state information.

104. The method of claim 1, wherein the transferring comprises transferring the data capsule from a first application having a first application type to at least one second application having at least one second application type, wherein the first application type is different than the second application type, wherein the generating of the at least one data sequence was executed by the first application, the method comprising maintaining intact the at least one data sequence of the data capsule during the transferring.

105. The method of claim 104, comprising using the at least one data sequence during operations of the second application.

106. The method of claim 1, comprising generating a first data set that includes event data and identification data of a source device of the at least one processing device, the device event data including data specifying an event registered by the source device, the identification data including data identifying the source device.

107. The method of claim 106, comprising generating a second data set that includes a full set of state information of the event, wherein each of the first data set and the second data set comprise typed data bundles in a type-specific data layout.

108. The method of claim 107, wherein the translating comprises encapsulating the first data set and the second data set by forming a data capsule to include the first data set and the second data set, wherein the data capsule has a data structure comprising an application-independent representation of the at least one data sequence.

109. The method of claim 1, comprising:

detecting an event of a first processing device running under an application of a first type;

generating data sequences comprising event data of the first processing device, the event data specifying the event and state information of the event, wherein the event data and state information are type-specific data having a type corresponding to the application;

forming a data capsule to include the data sequences, the data capsule having a data structure comprising an application-independent representation of the data sequences;

detecting a second event of a second processing device running under at least one second application having at least one second type, wherein the second type is different from the first type, wherein the at least one processing device comprises the first processing device and the second processing device;

identifying a correspondence between the data capsule and the second event; and

executing an operation in response to the second event using contents of the data sequences of the data capsule.

110. The method of claim 109, wherein the generating of the data sequences comprises:

generating a first data set that includes the event data;

generating a second data set that includes the state information; and

forming a first data sequence to include the first data set and the second data set.

111. The method of claim 109, wherein the event data is a tagged byte-sequence representing typed data.

112. The method of claim 111, wherein the event data includes a type header and a type-specific data layout.

113. The method of claim 109, wherein the state information is a tagged byte-sequence representing typed data.

114. The method of claim 113, wherein the state information includes a type header and a type-specific data layout.

115. The method of claim 109, comprising:

generating at least one offset; and

forming the data capsule to include the at least one offset.

116. The method of claim 115, comprising:

generating a first offset having a first variable length, wherein the first offset points to the event data of a first data sequence of the at least one data sequence; and

generating a second offset having a second variable length, wherein the second offset points to the state information of a first data sequence of the at least one data sequence.

117. The method of claim 115, comprising:

forming a first code path through the data capsule using a first offset of the at least one offset;

forming a second code path through the data capsule using a second offset of the at least one offset;

wherein the first code path and the second code path are different paths.

118. The method of claim 115, wherein at least one of the first offset and the second offset include metadata, the metadata comprising context-specific metadata corresponding to a context of the application, on

119. The method of claim 109, comprising transferring the data capsule to a pool of the plurality of pools.

120. The method of claim 119, comprising:

searching the plurality of pools for data capsules corresponding to the second event; and

extracting the data capsule from the pool in response to the identifying of the correspondence.

121. The method of claim 119, wherein the plurality of pools is coupled to the application and the at least one second application, the plurality of pools including a plurality of data capsules corresponding to the application and the at least one second application, the plurality of pools providing access to the plurality of data capsules by the application and the at least one second application.

122. The method of claim 119, wherein the plurality of pools provides state caching of a plurality of data capsules.

123. The method of claim 119, wherein the plurality of pools provides linear sequencing of a plurality of data capsules.

124. The method of claim 109, wherein the data structure is untyped.

125. The method of claim 109, wherein the data structure of the data capsule provides a platform-independent representation of the event data and the state information.

126. The method of claim 109, wherein the data structure of the data capsule provides platform-independent access to the event data and the state information.

127. A method comprising:

executing a plurality of processes on a processing device, the plurality of processes including separable program execution contexts of a plurality of application programs, wherein each application program comprises at least one process;

translating events of each process of the plurality of processes into data messages, wherein a data message includes an application-independent representation of event data of an event and state information of the process originating the data message;

transferring the data messages into at least one pool of a plurality of pools;

coordinating among the processes, the coordinating including each process of the plurality of processes coordinating with peer processes of the plurality of processes by retrieving from the plurality of pools the state information of the peer processes; and

generating an output of the plurality of processes by interactively combining a set of data messages of at least one pool of the plurality of pools.

128. A system comprising:

a detector for receiving gesture data that represents a gesture made by a body; and

a processor coupled to the detector, the processor automatically detecting the gesture from the gesture data, the processor executing a plurality of processes, the plurality of processes generating events that include a set of events representing the gesture, the processor translating the events of each process of the plurality of processes into data capsules, the processor transferring the data capsules into a plurality of pools, wherein a set of processes of the plurality of processes function as recognizing processes, the recognizing processes recognizing in the plurality of pools data capsules comprising content that corresponds to the gesture, the recognizing processes retrieving recognized data capsules from the plurality of pools and generating a gesture signal from the recognized data capsules by compositing contents of the recognized data capsules to form the gesture signal, wherein the gesture signal represents the gesture.