Realtime kernel

• US 8,578,000 B2
• Filed: 12/04/2009
• Issued: 11/05/2013
• Est. Priority Date: 12/05/2008
• Status: Active Grant

• Current Assignee: Sococo, Inc.
• Original Assignee: Social Communications Company
• Inventors: Van Wie, David, Altmaier, Joseph
• Primary Examiner(s): SHINGLES, KRISTIE D
• Application Number:
  US12/630,973
• Publication Number:
  US 20100146085A1
• Time in Patent Office: 1,432 Days
• Field of Search: 709/220, 709/238, 709/249, 709/250
• US Class Current: 709/220
• CPC Class Codes: A63F 13/12 involving interaction between a plurality of game devices, e.g. transmisison or distribution systems protocols for networked virtual reality, networked games H04L29/06034
• CPC Class Codes: A63F 13/30 Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: H04L 12/00 Data switching networks interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units G06F13/00
• CPC Class Codes: H04L 65/00 Network arrangements, protocols or services for supporting real-time applications in data packet communication real-time or near real-time messaging, e.g. instant messaging [IM] H04L51/04; selective video distribution H04N21/00
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/60 Network streaming of media packets
• CPC Class Codes: H04L 65/764 at the destination reformatting of additional data in video clients H04N21/4355
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04L 67/1059 Inter-group management mechanisms, e.g. splitting, merging or interconnection of groups
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04L 67/141 Setup of application sessions admission control or resource allocation in data switching networks H04L47/70
• CPC Class Codes: H04L 67/34 involving the movement of software or configuration parameters network booting or remote initial program loading [RIPL] G06F9/4416
• CPC Class Codes: H04N 21/00 Selective content distribution, e.g. interactive television or video on demand [VOD] real-time bi-directional transmission of motion video data H04N7/14 ; broadcast or conference over packet switching networks H04L12/18

Abstract

A realtime kernel supports realtime communications between communicants operating on respective network nodes. The realtime kernel handles the complex tasks of connecting to communicants, virtual areas, and other network resources, switching those connections in response to user inputs, and mixing realtime data streams. The realtime kernel enables developers to focus on developing high-level communications functionality instead of low-level plumbing code. The realtime kernel imposes relatively low computational resource requirements so that realtime communications performance can be achieved using a wide range of computing devices and network connections that currently are available.

Claims

1. A computer-implemented method of realtime data stream handling in a network communication environment comprising a network service implemented by at least one server network node and supporting realtime communications between respective network nodes of a user and other communicants associated with respective communicant objects in one or more virtual areas, comprising by the user'"'"'s network node:

parsing a specification of a realtime stream handler from one or more stream handling instructions received from the network service in connection with a particular one of the one or more virtual areas, wherein the parsing comprises parsing an input source identifier, an output sink identifier, and a respective identifier of each of one or more data processing objects from the one or more stream handing instructions, wherein the one or more stream handling instructions are based on realtime position information describing respective positions of respective ones of the communicant objects in the particular virtual area;

instantiating realtime stream handling objects corresponding to the parsed identifiers;

creating a directed graph from the instantiated realtime stream handling objects in accordance with the specification;

receiving a realtime data stream from an input source corresponding to the input source identifier; and

producing a resultant data stream at an output sink corresponding to the output sink identifier, wherein the producing comprises processing the realtime data stream through the directed graph, wherein the producing is based on configuration parameter values determined from the realtime position information and comprises dynamically configuring one or more of the instantiated realtime stream handling objects in the directed graph with the configuration parameter values.

2. The method of claim 1, wherein the parsing comprises extracting identifiers of data processing objects from the one or more stream handling instructions, and the creating comprises assembling the directed graph from instantiations of realtime stream handling objects corresponding to the extracted identifiers.

3. The method of claim 1, wherein the parsing comprises extracting identifiers of audio processing objects from the one or more stream handling instructions, and the processing comprises processing the realtime data stream through one or more instantiated audio processing objects corresponding to respective ones of the extracted identifiers.

4. The method of claim 1, wherein the instantiating comprises querying an object library for the respective identifiers, retrieving objects from the object library in response to the querying, and instantiating the retrieved objects.

5. The method of claim 4, wherein the instantiating comprises issuing to a processing object application program interface (API) procedure calls comprising respective ones of the identifiers.

6. The method of claim 1, wherein the producing comprises

determining configuration parameter values from realtime state information specified in one or more of the stream handling instructions, and

dynamically configuring one or more of the instantiated realtime stream handling objects in the directed graph with the configuration parameter values.

7. The method of claim 1, wherein:

the receiving comprises receiving a second realtime data stream from a second input source corresponding to a second input source identifier in the specification; and

the producing comprises processing the second realtime data stream through the directed graph.

8. The method of claim 7, wherein the instantiating comprises instantiating a realtime stream mixing object corresponding to a respective one of the parsed identifiers, the creating comprises incorporating the instantiated realtime stream mixing object into the directed graph, and the producing comprises executing the instantiated realtime stream mixing object to combine the first realtime data stream and the second realtime data stream.

9. The method of claim 8, wherein the first input source corresponds to a definition of an incoming realtime data stream from a first one of the other communicants'"'"' network node, and the second input source corresponds to a definition of an incoming realtime data stream from a second one of the other communicants'"'"' network node.

10. The method of claim 1, wherein:

the parsing comprises parsing a second specification of a second realtime stream handler from the one or more stream handling instructions, the second specification comprising a second input source identifier, a second output sink identifier, and a respective identifier of each of one or more data processing objects;

the creating comprises creating a second directed graph from ones of the instantiated realtime stream handling objects in accordance with the second specification;

the receiving comprises receiving a second realtime data stream from a second input source corresponding to the second input source identifier, wherein the first and second realtime data streams are different in terms of data type; and

the producing comprises processing the second realtime data stream through the second directed graph to a second output sink corresponding to the second output sink identifier.

11. The method of claim 10, wherein the first realtime data stream and the second realtime data stream are processed concurrently.

12. The method of claim 1, wherein at least one of the instantiated realtime stream handling objects encapsulates a respective call to a driver module controlling a local hardware component of the user'"'"'s network node.

13. The method of claim 1, wherein the user'"'"'s network node is associated with a first object in the particular virtual area, and the receiving comprises receiving the realtime data stream from one of the other communicants'"'"' network node that is associated with a second object in the particular virtual area.

14. The method of claim 13, wherein the receiving comprises receiving a second realtime data stream from a second one of the other communicants'"'"' network node that is associated with a third object in the particular virtual area, the creating comprises building the directed graph with an instantiated mixing object corresponding to a respective one of the parsed identifiers, and the producing of the resultant data stream comprises executing the instantiated mixing object to combine the first realtime data stream and the second realtime data stream.

15. The method of claim 13, wherein:

the parsing comprises parsing a second specification of a second realtime stream handler from the one or more stream handling instructions, the second specification comprising a second input source identifier, a second output sink identifier, and a respective identifier of each of one or more data processing objects;

the creating comprises creating a second directed graph from ones of the instantiated realtime stream handling objects in accordance with the second specification;

the receiving comprises receiving a second realtime data stream from a second input source corresponding to the second input source identifier, wherein the second input source corresponds to a definition of an incoming realtime data stream from a second one of the other communicants'"'"' network node and the first and second realtime data streams are different in terms of data type; and

the producing comprises processing the second realtime data stream through the second directed graph to a second output sink corresponding to the second output sink identifier.

16. The method of claim 13, wherein:

the parsing comprises extracting identifiers of graphic object appearance processing objects from the one or more stream handling instructions; and

the producing comprises processing the realtime data stream through one or more instantiated graphic object appearance processing objects corresponding to respective ones of the extracted identifiers, and dynamically configuring the instantiated graphic object appearance processing objects to control a visual representation of the first object in terms of at least one of orientation, position, movement, and pose based at least in part on the position of the first object in the particular virtual area.

17. The method of claim 1, wherein the one or more stream handling instructions specify the realtime stream handler without reference to function.

18. The method of claim 1, wherein the one or more stream handling instructions specify the realtime stream handler without reference to stream type.

19. The method of claim 1, wherein the producing comprises traversing the directed graph from the output sink, through each of one or more data processing objects, to the input source during each of successive fixed length intervals.

20. The method of claim 1, wherein the parsing comprises parsing one or more records in the one or more stream handling instructions, each of the one or more records comprises a respective definition type, a respective definition length, and at least one of a definition-type-specific field and a respective definition-type-specific definition.

21. The method of claim 1, further comprising generating a human-perceptible output based at least in part on the resultant data stream.

22. The method of claim 1, wherein the parsing of respective identifier of each of one or more data processing objects comprises parsing respective identifiers of multiple media content processing objects from the one or more stream handing instructions.

23. The method of claim 22, wherein the realtime data stream is a realtime audio data stream, and the parsing comprises parsing respective identifiers of multiple audio content processing objects from the one or more stream handling instructions.

24. The method of claim 23, wherein the producing comprises processing the realtime audio data stream through instantiated realtime audio stream handling objects corresponding to the parsed identifiers of the audio content processing objects, and further comprising outputting human perceptible audio output from the resultant data stream.

25. The method of claim 24, wherein at least one of the audio stream handling objects is a volume adjusting realtime audio stream handling object.

26. The method of claim 24, wherein at least one of the audio stream handling objects is a Doppler shifting realtime audio stream handling object.

27. The method of claim 1, further comprising passing the resultant data stream to a hardware component of the user'"'"'s local node that generates a human perceptible output from the resultant data stream.

28. Apparatus, comprising:

a computer-readable medium storing computer-readable instructions; and

a data processing unit coupled to the memory, operable to execute the instructions, and based at least in part on the execution of the instructions operable to perform operations in a network communication environment comprising a network service implemented by at least one server network node and supporting realtime communications between the apparatus and other network nodes associated with respective objects in one or more virtual areas, the operations comprising

parsing a specification of a realtime stream handler from one or more stream handling instructions received from the network service in connection with a particular one of the one or more virtual areas, wherein the parsing comprises parsing an input source identifier, an output sink identifier, and a respective identifier of each of one or more data processing objects from the one or more stream handing instructions, wherein the one or more stream handling instructions are based on realtime position information describing respective positions of respective ones of the objects in the particular virtual area,

instantiating realtime stream handling objects corresponding to the parsed identifiers,

creating a directed graph from the instantiated realtime stream handling objects in accordance with the specification,

receiving a realtime data stream from an input source corresponding to the input source identifier, and

producing a resultant data stream at an output sink corresponding to the output sink identifier, wherein the producing comprises processing the realtime data stream through the directed graph, wherein the producing is based on configuration parameter values determined from the realtime position information and comprises dynamically configuring one or more of the instantiated realtime stream handling objects in the directed graph with the configuration parameter values.

29. At least one non-transitory computer-readable medium having computer-readable program code embodied therein, the computer-readable program code adapted to be executed by a computer to implement a method in a network communication environment comprising a network service implemented by at least one server network node and supporting realtime communications between the computer and other network nodes associated with respective objects in one or more virtual areas, the method comprising:

parsing a specification of a realtime stream handler from one or more stream handling instructions received from the network service in connection with a particular one of the one or more virtual areas, wherein the parsing comprises parsing an input source identifier, an output sink identifier, and a respective identifier of each of one or more data processing objects from the one or more stream handing instructions, wherein the one or more stream handling instructions are based on realtime position information describing respective positions of respective ones of the objects in the particular virtual area;

instantiating realtime stream handling objects corresponding to the parsed identifiers;

creating a directed graph from the instantiated realtime stream handling objects in accordance with the specification;

receiving a realtime data stream from an input source corresponding to the input source identifier; and

producing a resultant data stream at an output sink corresponding to the output sink identifier, wherein the producing comprises processing the realtime data stream through the directed graph, wherein the producing is based on configuration parameter values determined from the realtime position information and comprises dynamically configuring one or more of the instantiated realtime stream handling objects in the directed graph with the configuration parameter values.