Multi-process interactive systems and methods

• US 8,370,383 B2
• Filed: 10/14/2009
• Issued: 02/05/2013
• Est. Priority Date: 02/08/2006
• Status: Active Grant

• Current Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Original Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Inventors: Kramer, Kwindla H., Underkoffler, John S.
• Primary Examiner(s): PANNALA, SATHYANARAYA R
• Application Number:
  US12/579,354
• Publication Number:
  US 20100131549A1
• Time in Patent Office: 1,210 Days
• Field of Search: 345/904, 318/516, 701/200, 385/15, 707/769
• US Class Current: 707/769
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications
• CPC Class Codes: G06F 9/544 Buffers; Shared memory; Pipes

Abstract

A multi-process interactive system is described. The system includes numerous processes running on a processing device. The processes include separable program execution contexts of application programs, such that each application program comprises at least one process. The system translates events of each process into data capsules. A data capsule includes an application-independent representation of event data of an event and state information of the process originating the content of the data capsule. The system transfers the data messages into pools or repositories. Each process operates as a recognizing process, where the recognizing process recognizes in the pools data capsules comprising content that corresponds to an interactive function of the recognizing process and/or an identification of the recognizing process. The recognizing process retrieves recognized data capsules from the pools and executes processing appropriate to contents of the recognized data capsules.

Claims

1. A method comprising:

executing a plurality of processes on at least one processing device, the plurality of processes including separable program execution contexts of a plurality of application programs, wherein each application program comprises at least one process;

translating events of each process of the plurality of processes into data capsules, wherein a data capsule includes an application-independent representation of event data of an event and state information of the process originating the data capsule;

transferring the data capsules into a plurality of pools;

each process operating as a recognizing process, the recognizing process recognizing in the plurality of pools data capsules comprising at least one of content that corresponds to an interactive function of the recognizing process and an identification of the recognizing process; and

the recognizing process retrieving recognized data capsules from the plurality of pools and executing processing appropriate to contents of the recognized data capsules.

2. The method of claim 1, comprising forming an interactive application from the plurality of processes by coordinating operations of each of the plurality of processes using the data capsules and the plurality of pools.

3. The method of claim 1, comprising coordinating operations of the plurality of processes using at least one of the data capsules and the plurality of pools.

4. The method of claim 1, comprising dividing an application program into a set of processes, wherein the plurality of processes includes the set of processes.

5. The method of claim 1, comprising a process generating an output by interactively processing a plurality of retrieved data capsules of at least one pool of the plurality of pools.

6. The method of claim 1, comprising executing the plurality of processes in parallel.

7. The method of claim 1, comprising executing a first set of processes in parallel, and executing a second set of processes in sequence, wherein the plurality of processes includes the first set of processes and the second set of processes.

8. The method of claim 1, wherein the events represent process inputs.

9. The method of claim 1, wherein the events represent process outputs.

10. The method of claim 1, wherein the events comprise a user interface event.

11. The method of claim 1, wherein the events comprise a graphics event.

12. The method of claim 1, wherein the events represent process state.

13. The method of claim 12, wherein the state of a process represents the interactive function of the process, wherein the interactive function of the process is exposed to the plurality of processes as contents of the data capsules.

14. The method of claim 13, comprising defining an application programming interface (API) of the plurality of processes by contents of the data capsules instead of defining the API by function calls.

15. The method of claim 14, wherein the contents of the data capsules are application-independent and recognizable by the plurality of processes.

16. The method of claim 1, wherein the at least one processing device comprises a plurality of processing devices.

17. The method of claim 16, wherein at least one first set of processes of the plurality of processes are running under at least one first set of processing devices of the plurality of processing device and at least one second set of processes of the plurality of processes are running under at least one second set of processing devices of the plurality of processing devices.

18. The method of claim 1, wherein the plurality of processes includes a first process.

19. The method of claim 18, wherein the translating comprises transforming events of the first process into at least one data sequence comprising first process event data specifying the event and state information of the event.

20. The method of claim 19, wherein the first process event data and state information are type-specific data having a type corresponding to an application of the first process.

21. The method of claim 20, wherein the translating comprises forming the data capsule to include the at least one data sequence, the data capsule having a data structure comprising an application-independent representation of the at least one data sequence.

22. The method of claim 18, wherein the plurality of processes includes a second process.

23. The method of claim 22, wherein the translating comprises transforming state change events of the second process into at least one data sequence comprising second process event data specifying the event and state information of the event.

24. The method of claim 23, wherein the second process event data and state information are type-specific data having a type corresponding to an application of the second process.

25. The method of claim 24, wherein the translating comprises forming the data capsule to include the at least one data sequence, the data capsule having a data structure comprising an application-independent representation of the at least one data sequence.

26. The method of claim 22, wherein the recognizing process is the second process, the retrieving comprising the second process retrieving the recognized data capsules from the plurality of pools and executing processing appropriate to contents of the recognized data capsules.

27. The method of claim 26, wherein the contents of the recognized data capsules are data representing state information of the first process.

28. The method of claim 27, wherein the translating comprises transforming the contents of the recognized data capsules into at least one new data sequence, the at least one new data sequence representing at least one of an event of the first process and an event of the second process.

29. The method of claim 28, wherein the at least one new data sequence comprises event data specifying the event and state information of at least one of the first process and the second process.

30. The method of claim 29, wherein the event data and state information of the at least one of the first process and the second process are type-specific data having a type corresponding to an application of the at least one of the first process and the second process.

31. The method of claim 30, wherein the translating comprises forming the data capsule to include the at least one new data sequence, the data capsule having a data structure comprising an application-independent representation of the at least one new data sequence.

32. The method of claim 31, wherein the plurality of processes uses the at least one new data sequence.

33. The method of claim 1, wherein the plurality of processes includes an input process, the input process receiving input events from an input device.

34. The method of claim 33, wherein the translating comprises transforming the input events of the input device into at least one data sequence comprising input device event data specifying the event and state information of the event.

35. The method of claim 34, wherein the input device event data and state information are type-specific data having a type corresponding to an application of the source device.

36. The method of claim 35, wherein the translating comprises forming the data capsule to include the at least one data sequence, the data capsule having a data structure comprising an application-independent representation of the at least one data sequence.

37. The method of claim 1, wherein the plurality of processes includes a pointer process.

38. The method of claim 37, wherein the recognizing process is the pointer process, the retrieving comprising the pointer process retrieving the recognized data capsules from the plurality of pools and executing processing appropriate to contents of the recognized data capsules.

39. The method of claim 38, wherein the contents of the recognized data capsules are data representing input events from an input process.

40. The method of claim 38, wherein the contents of the recognized data capsules are data representing a position on a display where a user of the at least one processing device is directing a pointer object.

41. The method of claim 40, wherein the translating comprises transforming the contents of the recognized data capsules into at least one new data sequence, the at least one new data sequence defining a position of the pointer object with respect to a display.

42. The method of claim 41, wherein the at least one new data sequence comprises pointer process event data specifying the event and state information of the pointer process event.

43. The method of claim 42, wherein the pointer process event data and state information are type-specific data having a type corresponding to an application of the pointer process.

44. The method of claim 43, wherein the translating comprises forming the data capsule to include the at least one new data sequence, the data capsule having a data structure comprising an application-independent representation of the at least one new data sequence.

45. The method of claim 44, wherein the plurality of processes uses the at least one new data sequence in rendering the pointer object on the display.

46. The method of claim 1, wherein the plurality of processes includes a graphical process.

47. The method of claim 46, wherein the translating comprises transforming state change events of the graphical process into at least one data sequence comprising graphical process event data specifying the event and state information of the event.

48. The method of claim 47, wherein the graphical process event data and state information are type-specific data having a type corresponding to an application of the graphical process.

49. The method of claim 48, wherein the translating comprises forming the data capsule to include the at least one data sequence, the data capsule having a data structure comprising an application-independent representation of the at least one data sequence.

50. The method of claim 46, wherein the recognizing process is the graphical process, the retrieving comprising the graphical process retrieving the recognized data capsules from the plurality of pools and executing processing appropriate to contents of the recognized data capsules.

51. The method of claim 50, wherein the contents of the recognized data capsules are data representing state information of another process of the plurality of processes.

52. The method of claim 51, wherein the state information includes information of at least one of spatial state and modal state.

53. The method of claim 50, wherein the contents of the recognized data capsules are data representing a position on a display where a user of the at least one processing device is directing a pointer object.

54. The method of claim 53, wherein the position of the pointer object is within a boundary of a graphical object, wherein the graphical object is rendered by the graphical process.

55. The method of claim 53, wherein the translating comprises transforming the contents of the recognized data capsules into at least one new data sequence, the at least one new data sequence representing at least one of the graphical object, the pointer object, and an overlap of the pointer object and the boundary.

56. The method of claim 55, wherein the at least one new data sequence comprises graphical process event data specifying the event and state information of the graphical process event.

57. The method of claim 56, wherein the graphical process event data and state information are type-specific data having a type corresponding to an application of the graphical process.

58. The method of claim 57, wherein the translating comprises forming the data capsule to include the at least one new data sequence, the data capsule having a data structure comprising an application-independent representation of the at least one new data sequence.

59. The method of claim 58, wherein the plurality of processes uses the at least one new data sequence in rendering at least one of the graphical object and the pointer object on the display.

60. The method of claim 1, wherein the processing appropriate to contents of the recognized data capsules comprises rendering a graphical object, wherein the graphical object is rendered on a display of the at least one processing device.

61. The method of claim 60, wherein the rendering comprises direct rendering in which the plurality of processes draw directly to a graphics layer of the at least one processing device, wherein the plurality of pools is used for coordination among the plurality of processes as appropriate to the rendering.

62. The method of claim 60, wherein the rendering comprises the plurality of processes:

transferring data capsules comprising rendering commands into the plurality of pools; and

retrieving the rendering commands from the plurality of pools, interpreting the rendering commands, and driving a graphics layer of the at least one processing device in response to the rendering commands.

63. The method of claim 60, wherein the rendering comprises the plurality of processes:

rendering to pixel buffers;

transferring raw frame data into the plurality of pools, the raw frame data resulting from the rendering to the pixel buffers; and

retrieving the raw frame data from the plurality of pools and combining the raw frame data for use in driving a graphics layer of the at least one processing device.

64. The method of claim 1, comprising:

detecting an event of the plurality of processes;

generating at least one data sequence comprising event data specifying the event and state information of the event, wherein the event data and state information are type-specific data having a type corresponding to an application of the at least one processing device; and

forming a data capsule to include the at least one data sequence, the data capsule having a data structure comprising an application-independent representation of the at least one data sequence.

65. The method of claim 64, wherein the generating of the at least one data sequence comprises:

generating a first respective data set that includes first respective event data;

generating a second respective data set that includes second respective state information; and

forming a first data sequence to include the first respective data set and the second respective data set.

66. The method of claim 65, wherein the generating of the first respective data set includes forming the first respective data set to include identification data of the at least one processing device, the identification data including data identifying the at least one processing device.

67. The method of claim 65, wherein the generating of the at least one data sequence comprises:

generating a first respective data set that includes first respective event data;

generating a second respective data set that includes second respective state information; and

forming a second data sequence to include the first respective data set and the second respective data set.

68. The method of claim 67, wherein the generating of the first respective data set includes generating a first respective data set offset, wherein the first respective data set offset points to the first respective data set of the second data sequence.

69. The method of claim 67, wherein the generating of the second respective data set includes generating a second respective data set offset, wherein the second respective data set offset points to the second respective data set of the second data sequence.

70. The method of claim 65, wherein the first respective data set is a description list, the description list including a description of the data.

71. The method of claim 64, wherein the event data is a tagged byte-sequence representing typed data.

72. The method of claim 71, wherein the event data includes a type header and a type-specific data layout.

73. The method of claim 64, wherein the state information is a tagged byte-sequence representing typed data.

74. The method of claim 73, wherein the state information includes a type header and a type-specific data layout.

75. The method of claim 64, comprising:

generating at least one offset; and

forming the data capsule to include the at least one offset.

76. The method of claim 75, comprising:

generating a first offset having a first variable length;

wherein the first offset points to the event data of a first data sequence of the at least one data sequence.

77. The method of claim 75, comprising:

generating a second offset having a second variable length;

wherein the second offset points to the state information of a first data sequence of the at least one data sequence.

78. The method of claim 75, comprising:

forming a first code path through the data capsule using a first offset of the at least one offset;

forming a second code path through the data capsule using a second offset of the at least one offset;

wherein the first code path and the second code path are different paths.

79. The method of claim 75, wherein at least one of the first offset and the second offset include metadata, the metadata comprising context-specific metadata corresponding to a context of the application.

80. The method of claim 64, comprising:

generating a header that includes a length of the data capsule;

forming the data capsule to include the header.

81. The method of claim 64, comprising transferring the data capsule to a pool of the plurality of pools.

82. The method of claim 81, comprising:

detecting a second event of the at least one processing device;

searching the plurality of pools for data capsules corresponding to the second event.

83. The method of claim 82, comprising:

identifying a correspondence between the data capsule and the second event;

extracting the data capsule from the pool in response to the identifying; and

executing on behalf of the at least one processing device a processing operation corresponding to the second event in response to contents of the data capsule, wherein the at least one processing device corresponds to an application of a first type and a second application of a second type.

84. The method of claim 81, wherein the plurality of pools is coupled to a plurality of applications, the plurality of pools including a plurality of data capsules corresponding to the plurality of applications, the plurality of pools providing access to the plurality of data capsules by the plurality of applications, wherein at least two applications of the plurality of applications are different applications.

85. The method of claim 81, wherein the plurality of pools provides state caching of a plurality of data capsules.

86. The method of claim 81, wherein the plurality of pools provides linear sequencing of a plurality of data capsules.

87. The method of claim 64, wherein the data structure is untyped.

88. The method of claim 64, wherein the data structure of the data capsule provides a platform-independent representation of the event data and the state information.

89. The method of claim 64, wherein the data structure of the data capsule provides platform-independent access to the event data and the state information.

90. The method of claim 64, wherein the transferring comprises transferring the data capsule from a first application having a first application type to at least one second application having at least one second application type, wherein the first application type is different than the second application type, wherein the generating of the at least one data sequence was executed by the first application, the method comprising maintaining intact the at least one data sequence of the data capsule during the transferring.

91. The method of claim 90, comprising using the at least one data sequence during operations of the second application.

92. The method of claim 64, comprising generating a first data set that includes event data and identification data of a source device of the at least one processing device, the device event data including data specifying an event registered by the source device, the identification data including data identifying the source device.

93. The method of claim 92, comprising generating a second data set that includes a full set of state information of the event, wherein each of the first data set and the second data set comprise typed data bundles in a type-specific data layout.

94. The method of claim 93, wherein the translating comprises encapsulating the first data set and the second data set by forming a data capsule to include the first data set and the second data set, wherein the data capsule has a data structure comprising an application-independent representation of the at least one data sequence.

95. The method of claim 64, comprising:

detecting an event of a first processing device running under an application of a first type;

generating data sequences comprising event data of the first processing device, the event data specifying the event and state information of the event, wherein the event data and state information are type-specific data having a type corresponding to the application;

forming a data capsule to include the data sequences, the data capsule having a data structure comprising an application-independent representation of the data sequences;

detecting a second event of a second processing device running under at least one second application having at least one second type, wherein the second type is different from the first type, wherein the at least one processing device comprises the first processing device and the second processing device;

identifying a correspondence between the data capsule and the second event; and

executing an operation in response to the second event using contents of the data sequences of the data capsule.

96. The method of claim 95, wherein the generating of the data sequences comprises:

generating a first data set that includes the event data;

generating a second data set that includes the state information; and

forming a first data sequence to include the first data set and the second data set.

97. The method of claim 95, wherein the event data is a tagged byte-sequence representing typed data.

98. The method of claim 97, wherein the event data includes a type header and a type-specific data layout.

99. The method of claim 95, wherein the state information is a tagged byte-sequence representing typed data.

100. The method of claim 99, wherein the state information includes a type header and a type-specific data layout.

101. The method of claim 95, comprising:

generating at least one offset; and

forming the data capsule to include the at least one offset.

102. The method of claim 101, comprising:

generating a first offset having a first variable length, wherein the first offset points to the event data of a first data sequence of the at least one data sequence; and

generating a second offset having a second variable length, wherein the second offset points to the state information of a first data sequence of the at least one data sequence.

103. The method of claim 101, comprising:

forming a first code path through the data capsule using a first offset of the at least one offset;

forming a second code path through the data capsule using a second offset of the at least one offset;

wherein the first code path and the second code path are different paths.

104. The method of claim 101, wherein at least one of the first offset and the second offset include metadata, the metadata comprising context-specific metadata corresponding to a context of the application.

105. The method of claim 95, comprising transferring the data capsule to a pool of the plurality of pools.

106. The method of claim 105, comprising:

searching the plurality of pools for data capsules corresponding to the second event; and

extracting the data capsule from the pool in response to the identifying of the correspondence.

107. The method of claim 105, wherein the plurality of pools is coupled to the application and the at least one second application, the plurality of pools including a plurality of data capsules corresponding to the application and the at least one second application, the plurality of pools providing access to the plurality of data capsules by the application and the at least one second application.

108. The method of claim 105, wherein the plurality of pools provides state caching of a plurality of data capsules.

109. The method of claim 105, wherein the plurality of pools provides linear sequencing of a plurality of data capsules.

110. The method of claim 95, wherein the data structure is untyped.

111. The method of claim 95, wherein the data structure of the data capsule provides a platform-independent representation of the event data and the state information.

112. The method of claim 95, wherein the data structure of the data capsule provides platform-independent access to the event data and the state information.

113. A method comprising:

dividing an application program into a plurality of processes;

generating a portion of an output of the application program using a process of the plurality of processes;

encapsulating the portion of the output in a first data capsule and transferring the first data capsule to at least one of a plurality of pools, wherein the plurality of pools comprise a plurality of data capsules received from the plurality of processes;

accessing the plurality of pools and retrieving an input for a second process of the plurality of processes, wherein the input is in a second data capsule of the plurality of data capsules; and

coordinating processing among the plurality of processes using the plurality of data capsules and the plurality of pools.

114. A system comprising:

at least one processing device, the processing device executing a plurality of processes, the plurality of processes including separable program execution contexts of a plurality of application programs, wherein each application program comprises at least one process; and

a plurality of pools coupled to the at least one processing device;

the at least one processing device translating events of each process of the plurality of processes into data capsules and transferring the data capsules into a plurality of pools, wherein a data capsule includes an application-independent representation of event data of an event and state information of the process originating the data capsule;

each process operating as a recognizing process, the recognizing process recognizing in the plurality of pools data capsules comprising at least one of content that corresponds to an interactive function of the recognizing process and an identification of the recognizing process;

the recognizing process retrieving recognized data capsules from the plurality of pools and executing processing appropriate to contents of the recognized data capsules.