Multi-process interactive systems and methods

• US 9,495,228 B2
• Filed: 02/05/2013
• Issued: 11/15/2016
• Est. Priority Date: 02/08/2006
• Status: Active Grant

• Current Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Original Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Inventors: Kramer, Kwindla Hultman, Underkoffler, John S.
• Primary Examiner(s): Reyes, Mariela
• Assistant Examiner(s): Almani, Mohsen
• Application Number:
  US13/759,472
• Publication Number:
  US 20130312009A1
• Time in Patent Office: 1,379 Days
• Field of Search: 707/769
• US Class Current: 1/1
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications
• CPC Class Codes: G06F 9/544 Buffers; Shared memory; Pipes

Abstract

A multi-process interactive system is described. The system includes numerous processes running on a processing device. The processes include separable program execution contexts of application programs, such that each application program comprises at least one process. The system translates events of each process into data capsules. A data capsule includes an application-independent representation of event data of an event and state information of the process originating the content of the data capsule. The system transfers the data messages into pools or repositories. Each process operates as a recognizing process, where the recognizing process recognizes in the pools data capsules comprising content that corresponds to an interactive function of the recognizing process and/or an identification of the recognizing process. The recognizing process retrieves recognized data capsules from the pools and executes processing appropriate to contents of the recognized data capsules.

Claims

1. A method comprising:

executing a plurality of processes on at least one processing device;

translating events of each process of the plurality of processes into data capsules;

transferring the data capsules into a plurality of pools;

recognizing in the plurality of pools data capsules comprising content that corresponds to an interactive function of the plurality of processes; and

retrieving recognized data capsules from the plurality of pools and executing processing appropriate to contents of the recognized data capsules.

2. The method of claim 1, wherein the plurality of processes include separable program execution contexts of a plurality of application programs, wherein each application program comprises at least one process.

3. The method of claim 1, comprising forming an interactive application from the plurality of processes by coordinating operations of each of the plurality of processes using the data capsules and the plurality of pools.

4. The method of claim 1, comprising coordinating operations of the plurality of processes using at least one of the data capsules and the plurality of pools.

5. The method of claim 1, comprising a process generating an output by interactively processing a plurality of retrieved data capsules of at least one pool of the plurality of pools.

6. The method of claim 1, wherein the events represent at least one of process inputs, process outputs, a user interface event, and a graphics event.

7. The method of claim 1, wherein the events represent process state.

8. The method of claim 7, wherein the state of a process represents the interactive function of the process, wherein the interactive function of the process is exposed to the plurality of processes as contents of the data capsules.

9. The method of claim 8, comprising defining an application programming interface (API) of the plurality of processes by contents of the data capsules instead of defining the API by function calls.

10. The method of claim 9, wherein the contents of the data capsules are application-independent and recognizable by the plurality of processes.

11. The method of claim 10, wherein the contents of the data capsules includes event data of an event and state information of the process originating the data capsule.

12. The method of claim 1, wherein the plurality of processes includes an input process, the input process receiving input events from an input device.

13. The method of claim 1, wherein the plurality of processes includes a pointer process.

14. The method of claim 1, wherein the plurality of processes includes a graphical process.

15. The method of claim 14, wherein the translating comprises transforming state change events of the graphical process into at least one data sequence comprising graphical process event data specifying the event and state information of the event.

16. The method of claim 15, wherein the graphical process event data and state information are type-specific data having a type corresponding to an application of the graphical process.

17. The method of claim 16, wherein the translating comprises forming the data capsule to include the at least one data sequence, the data capsule having a data structure comprising an application-independent representation of the at least one data sequence.

18. The method of claim 14, wherein the recognizing process is the graphical process, the retrieving comprising the graphical process retrieving the recognized data capsules from the plurality of pools and executing processing appropriate to contents of the recognized data capsules.

19. The method of claim 18, wherein the contents of the recognized data capsules are data representing state information of another process of the plurality of processes.

20. The method of claim 19, wherein the state information includes information of at least one of spatial state and modal state.

21. The method of claim 18, wherein the contents of the recognized data capsules are data representing a position on a display where a user of the at least one processing device is directing a pointer object.

22. The method of claim 21, wherein the position of the pointer object is within a boundary of a graphical object, wherein the graphical object is rendered by the graphical process.

23. The method of claim 21, wherein the translating comprises transforming the contents of the recognized data capsules into at least one new data sequence, the at least one new data sequence representing at least one of the graphical object, the pointer object, and an overlap of the pointer object and the boundary.

24. The method of claim 23, wherein the at least one new data sequence comprises graphical process event data specifying the event and state information of the graphical process event.

25. The method of claim 24, wherein the graphical process event data and state information are type-specific data having a type corresponding to an application of the graphical process.

26. The method of claim 25, wherein the translating comprises forming the data capsule to include the at least one new data sequence, the data capsule having a data structure comprising an application-independent representation of the at least one new data sequence.

27. The method of claim 26, wherein the plurality of processes uses the at least one new data sequence in rendering at least one of the graphical object and the pointer object on the display.

28. The method of claim 1, wherein the processing appropriate to contents of the recognized data capsules comprises rendering a graphical object, wherein the graphical object is rendered on a display of the at least one processing device.

29. The method of claim 28, wherein the rendering comprises direct rendering in which the plurality of processes draw directly to a graphics layer of the at least one processing device, wherein the plurality of pools is used for coordination among the plurality of processes as appropriate to the rendering.

30. The method of claim 28, wherein the rendering comprises the plurality of processes:

transferring data capsules comprising rendering commands into the plurality of pools; and

retrieving the rendering commands from the plurality of pools, interpreting the rendering commands, and driving a graphics layer of the at least one processing device in response to the rendering commands.

31. The method of claim 28, wherein the rendering comprises the plurality of processes:

rendering to pixel buffers;

transferring raw frame data into the plurality of pools, the raw frame data resulting from the rendering to the pixel buffers; and

retrieving the raw frame data from the plurality of pools and combining the raw frame data for use in driving a graphics layer of the at least one processing device.

32. The method of claim 1, comprising:

detecting an event of the plurality of processes;

generating at least one data sequence comprising event data specifying the event and state information of the event, wherein the event data and state information are type-specific data having a type corresponding to an application of the at least one processing device; and

forming a data capsule to include the at least one data sequence, the data capsule having a data structure comprising an application-independent representation of the at least one data sequence.

33. The method of claim 32, wherein the generating of the at least one data sequence comprises:

generating a first respective data set that includes first respective event data;

generating a second respective data set that includes second respective state information; and

forming a first data sequence to include the first respective data set and the second respective data set.

34. The method of claim 33, wherein the first respective data set is a description list, the description list including a description of the data.

35. The method of claim 32, comprising:

generating at least one offset; and

forming the data capsule to include the at least one offset.

36. The method of claim 35, comprising:

generating a first offset having a first variable length;

wherein the first offset points to the event data of a first data sequence of the at least one data sequence.

37. The method of claim 35, comprising:

generating a second offset having a second variable length;

wherein the second offset points to the state information of a first data sequence of the at least one data sequence.

38. The method of claim 35, comprising:

forming a first code path through the data capsule using a first offset of the at least one offset;

forming a second code path through the data capsule using a second offset of the at least one offset;

wherein the first code path and the second code path are different paths.

39. The method of claim 35, wherein at least one of the first offset and the second offset include metadata, the metadata comprising context-specific metadata corresponding to a context of the application.

40. The method of claim 32, comprising transferring the data capsule to a pool of the plurality of pools.

41. The method of claim 40, comprising:

detecting a second event of the at least one processing device;

searching the plurality of pools for data capsules corresponding to the second event.

42. The method of claim 41, comprising:

identifying a correspondence between the data capsule and the second event;

extracting the data capsule from the pool in response to the identifying; and

executing on behalf of the at least one processing device a processing operation corresponding to the second event in response to contents of the data capsule, wherein the at least one processing device corresponds to an application of a first type and a second application of a second type.

43. The method of claim 32, wherein the transferring comprises transferring the data capsule from a first application having a first application type to at least one second application having at least one second application type, wherein the first application type is different than the second application type, wherein the generating of the at least one data sequence was executed by the first application, the method comprising maintaining intact the at least one data sequence of the data capsule during the transferring.

44. The method of claim 43, comprising using the at least one data sequence during operations of the second application.

45. A system comprising:

a memory device comprising a plurality of pools; and

a processing device coupled to the memory device and executing a plurality of processes, wherein the processing device translates events of each process of the plurality of processes into data capsules and transfers the data capsules into the plurality of pools, wherein each process of the plurality of processes comprises a recognizing process that recognizes in the plurality of pools data capsules comprising content that corresponds to an interactive function of the recognizing process, and retrieves recognized data capsules from the plurality of pools and executes processing appropriate to contents of the recognized data capsules.

46. A method comprising:

executing a plurality of processes that include separable program execution contexts of a plurality of application programs;

translating events of each process of the plurality of processes into data messages that include state information of the process originating the data message;

transferring the data messages into a plurality of pools;

coordinating among the plurality of processes, wherein a process coordinates with at least one peer process by retrieving from the plurality of pools the state information of the peer process; and

generating an output of the plurality of processes by interactively combining a set of data messages of at least one pool of the plurality of pools.