Realtime kernel

• US 9,055,131 B2
• Filed: 10/03/2013
• Issued: 06/09/2015
• Est. Priority Date: 10/24/2007
• Status: Active Grant

• Current Assignee: Sococo, Inc.
• Original Assignee: Social Communications Company
• Inventors: Van Wie, David, Altmaier, Joseph
• Primary Examiner(s): SHINGLES, KRISTIE D
• Application Number:
  US14/045,562
• Publication Number:
  US 20140136726A1
• Time in Patent Office: 614 Days
• Field of Search: 709/220
• US Class Current: 1/1
• CPC Class Codes: A63F 13/12 involving interaction between a plurality of game devices, e.g. transmisison or distribution systems protocols for networked virtual reality, networked games H04L29/06034
• CPC Class Codes: A63F 13/30 Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: H04L 12/00 Data switching networks interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units G06F13/00
• CPC Class Codes: H04L 65/00 Network arrangements, protocols or services for supporting real-time applications in data packet communication real-time or near real-time messaging, e.g. instant messaging [IM] H04L51/04; selective video distribution H04N21/00
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/60 Network streaming of media packets
• CPC Class Codes: H04L 65/764 at the destination reformatting of additional data in video clients H04N21/4355
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04L 67/1059 Inter-group management mechanisms, e.g. splitting, merging or interconnection of groups
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04L 67/141 Setup of application sessions admission control or resource allocation in data switching networks H04L47/70
• CPC Class Codes: H04L 67/34 involving the movement of software or configuration parameters network booting or remote initial program loading [RIPL] G06F9/4416
• CPC Class Codes: H04N 21/00 Selective content distribution, e.g. interactive television or video on demand [VOD] real-time bi-directional transmission of motion video data H04N7/14 ; broadcast or conference over packet switching networks H04L12/18

Abstract

A realtime kernel supports realtime communications between communicants operating on respective network nodes. The realtime kernel handles the complex tasks of connecting to communicants, virtual areas, and other network resources, switching those connections in response to user inputs, and mixing realtime data streams. The realtime kernel enables developers to focus on developing high-level communications functionality instead of low-level plumbing code. The realtime kernel imposes relatively low computational resource requirements so that realtime communications performance can be achieved using a wide range of computing devices and network connections that currently are available.

Claims

1. A method of realtime data stream handling, comprising by a local network node:

establishing at least one realtime data stream connection between the local network node and a remote network node;

processing at least one realtime data stream sourced by the remote network node over the realtime data stream connection, wherein the processing comprises processing the at least one realtime data stream through one or more realtime data processing operations to produce a resultant data stream;

wherein the processing comprises instantiating processing objects ones of which are operable to perform respective ones of the data processing operations, building a directed graph form ones of the instantiated processing objects, and processing the realtime data stream through the directed graph;

monitoring the processing; and

in response to a determination based on the monitoring that the processing deviates from a performance target, modifying the processing in accordance with a realtime performance targeting routine.

2. The method of claim 1, wherein the modifying comprises omitting the processing of one or more portions of the realtime data stream in response to a determination that the processing fails to satisfy the performance target.

3. The method of claim 1, wherein the modifying comprises omitting one or more of the realtime data processing operations in response to a determination that the processing fails to satisfy the performance target.

4. The method of claim 3, wherein the omitting comprises omitting one or more of the data processing operations characterized by respective performance values that are outside the performance target.

5. The method of claim 3, wherein ones of the data processing operations are assigned respective priority values, and the omitting comprises preferentially omitting one or more of the data processing operations based on the assigned priority values.

6. The method of claim 1, wherein the modifying comprises replacing at least one of the realtime data processing operations with a different respective realtime data processing operation in response to a determination that the processing fails to satisfy the performance target.

7. The method of claim 1, wherein one or more of the realtime data processing operations are assigned respective priority values, the realtime performance targeting routine comprises a heuristic that determines ones of the data processing operations to omit based on the assigned priority values, and the modifying is performed in accordance with the heuristic.

8. The method of claim 7, wherein the heuristic determines ones of the data processing operations to omit based on a weighting of the assigned priority values by a time-based performance statistic.

9. The method of claim 1, wherein the modifying comprises iteratively modifying the processing until the processing is within the specified performance target.

10. The method of claim 1, wherein the modifying comprises pruning one or more of the instantiated processing objects from the directed graph.

11. The method of claim 10, wherein ones of the processing objects are assigned respective priority values, and the pruning comprises removing ones of the instantiated processing objects from the directed graph based on the assigned priority values.

12. The method of claim 11, wherein the pruning comprises removing from the directed graph ones of the instantiated processing objects assigned respective priority values that fail to satisfy a priority threshold.

13. The method of claim 1, wherein the processing comprises building a second directed graph from ones of the instantiated processing objects, and processing through the second directed graph a second realtime data stream sourced by one of the local network node and the remote network node.

14. The method of claim 13, wherein the first and second directed graphs are assigned respective priority values, and the modifying comprises preferentially modifying one of the first and second directed graphs based on the assigned priority values.

15. The method of claim 14, wherein the modifying comprises omitting the processing of the one of the first and second realtime data streams through the respective one of the first and second directed graphs assigned a lowest priority value.

16. The method of claim 1, wherein the processing comprises processing through the directed graph a second realtime data stream sourced by one of the local network node and the remote network node.

17. The method of claim 16, wherein the first and second realtime data streams are assigned respective priority values, and the modifying comprises preferentially modifying the processing of one of the first and second realtime data streams based on the assigned priority values.

18. The method of claim 1, wherein the establishing comprises establishing respective realtime data stream connections between the local network node and multiple remote network nodes, and the processing comprises processing through the directed graph realtime data streams sourced by respective ones of the remote network nodes.

19. The method of claim 18, wherein the realtime data streams are assigned respective priority values, and the modifying comprises preferentially modifying the processing of one or more of the realtime data streams based on the assigned priority values.

20. The method of claim 18, wherein the directed graph comprises multiple directed chains of respective ones of the instantiated processing objects, and the processing comprises processing through each of the directed chains a respective one of the realtime data streams each of which is sourced by a respective one of the local network node and the remote network nodes.

21. The method of claim 20, wherein the modifying comprises iteratively modifying the processing until the processing is within the specified performance target, and during each iteration the modifying comprises performing one or more of (i) removing one or more of the chains from the directed graph and (ii) pruning one or more of the instantiated processing objects from the directed graph.

22. The method of claim 1, wherein the performance target comprises a time-based threshold on production of the resultant data stream.

23. The method of claim 1, further comprising by the local network node receiving from a remote network node one or more stream handing instructions comprising assignments of respective priority values to the one or more realtime data processing operations, wherein the modifying comprises modifying the processing based on the assigned priority values.

24. The method of claim 1, wherein the modifying comprises reducing a computational resource load to a lower level in response to a determination that the processing fails to satisfy the performance target.

25. The method of claim 24, wherein the modifying comprises increasing the computational resource load from the lower level in response to a determination that the processing satisfies the performance target.

26. The method of claim 1, wherein the realtime data stream is packetized into frames, and the monitoring is performed on each of each of the frames.

27. The method of claim 26, wherein the processing comprises processing the frames of the realtime data stream during each of successive fixed length intervals that are set in accordance with a local clock.

28. The method of claim 27, further comprising by the local network node, synchronizing the local clock with a remote master clock service.

29. The method of claim 1, wherein the monitoring comprises monitoring utilization of at least one processor of the local network node.

30. The method of claim 1, wherein the monitoring comprises monitoring bandwidth utilization by at least one networking resource of the local network node.

31. The method of claim 1, further comprising, by the local network node, generating a human-perceptible output responsive to the resultant data stream.

32. The method of claim 31, wherein the local network node and the remote network node are associated with respective objects in a virtual area, and the human-perceptible output is a visualization of the objects in the virtual area on a display.

33. The method of claim 1, wherein the establishing comprises creating a peer-to-peer network connection with the remote network node.

34. The method of claim 33, wherein the creating is performed by the local network node in accordance with instructions received by the local network node from a server network node.

35. The method of claim 34, wherein the instructions comprise an application programming interface (API) call to a stream connection software component operable on the local network node to perform the establishing.

36. The method of claim 34, wherein the instructions comprise an application programming interface (API) call to a stream handler software component operable on the local network node to perform the processing.

37. The method of claim 34, wherein the instructions comprise an application programming interface (API) call to a scheduler software component operable on the local network node to perform the monitoring and the modifying.

38. The method of claim 34, wherein the instructions are associated with a context that is shared by the local network node and the remote network node.

39. The method of claim 38, wherein the context is defined by a virtual area in which users of the local and remote network nodes are copresent.

40. The method of claim 39, further comprising by the local network node:

based on instructions received from the server network node, displaying indications of the presence of the user of the remote network node in the virtual area.

41. The method of claim 39, further comprising by the local network node:

based on instructions received from the server network node, displaying a visualization of the virtual area.

42. A local network node, comprising:

a computer-readable medium storing computer-readable instructions; and

a data processing unit coupled to the memory, operable to execute the instructions, and based at least in part on the execution of the instructions operable to perform operations comprising

establishing at least one realtime data stream connection between the local network node and at least one remote network node,

processing at least one realtime data stream sourced by the remote network node over the realtime data stream connection, wherein the processing comprises processing the at least one realtime data stream through one or more realtime data processing operations to produce a resultant data stream;

wherein the processing comprises instantiating processing objects ones of which are operable to perform respective ones of the data processing operations, building a directed graph from ones of the instantiated processing objects, and processing the realtime data stream through the directed graph;

monitoring the processing, and

in response to a determination based on the monitoring that the processing deviates from a performance target, modifying the processing in accordance with a realtime performance targeting routine.

43. At least one non-transitory computer-readable medium having computer-readable program code embodied therein, the computer-readable program code adapted to be executed by a local network node to implement a method comprising:

establishing at least one realtime data stream connection between the local network node and at least one remote network node;

processing at least one realtime data stream sourced by the remote network node over the realtime data stream connection, wherein the processing comprises processing the at least one realtime data stream through one or more realtime data processing operations to produce a resultant data stream;

wherein the processing comprises instantiating processing objects ones of which are operable to perform respective ones of the data processing operations, building a directed graph from ones of the instantiated processing objects, and processing the realtime data stream through the directed graph;

monitoring the processing; and

in response to a determination based on the monitoring that the processing deviates from a performance target, modifying the processing in accordance with a realtime performance targeting routine.