Client application integrating web browsing and network data stream processing for realtime communications

• US 9,069,851 B2
• Filed: 03/01/2012
• Issued: 06/30/2015
• Est. Priority Date: 01/15/2009
• Status: Active Grant

• Current Assignee: Sococo, Inc.
• Original Assignee: Social Communications Company
• Inventors: Van Wie, David, Altmaier, Joseph
• Primary Examiner(s): Ng, Amy
• Assistant Examiner(s): Kim, Sang H
• Application Number:
  US13/409,344
• Publication Number:
  US 20120226987A1
• Time in Patent Office: 1,216 Days
• Field of Search: 715/753, 715/757, 715/765
• US Class Current: 1/1
• CPC Class Codes: G06F 16/95 Retrieval from the web
• CPC Class Codes: H04L 41/22 comprising specially adapted graphical user interfaces [GUI]
• CPC Class Codes: H04L 47/283 in response to processing delays, e.g. caused by jitter or round trip time [RTT]
• CPC Class Codes: H04L 65/60 Network streaming of media packets

Abstract

A graphical user interface is displayed. The graphical user interface includes at least one communication control for managing communications with other network nodes and at least one browser control for navigating to different network resource addresses. Human perceptible realtime output is generated from realtime communication data relating to a realtime communication session with at least one other client network node. In the process of generating of at least some of the human perceptible realtime output, operation of the client network node is managed based on at least one performance target. In the graphical user interface, a graphical representation of a network resource identified in connection with the browser control is displayed.

Claims

1. A method performed by a client network node executing a client application integrating web browsing and network data stream processing for realtime communications, the method comprising:

on a display, displaying a graphical user interface providing a single point of control for realtime communications and web browsing interactions, wherein the displaying comprises displaying in the graphical user interface at least one control for establishing a presence in a virtual area;

requesting a presence in a particular virtual area identified in connection with the control and establishing a realtime communication session with at least one other client network node that is present in the particular virtual area;

transmitting outgoing data streams to and receiving incoming data streams from one or more other network nodes over respective network links via the realtime communications and web browsing interactions, the incoming and outgoing data streams comprising one or more time-critical types of data streams and one or more non-time-critical types of data streams, wherein the transmitting and receiving comprise, by the client network node, transmitting and receiving one or more time-critical types of data streams with each at least one other client network node and executing the client application to interface the client network node with network protocols on the client network node;

processing the one or more time-critical types of data streams and the one or more nontime-critical types of data streams;

monitoring at least one metric characterizing operation of the client network node during the processing of at least one of the time-critical types of the data streams; and

in response to a determination that the at least one metric fails to satisfy at least one operational target, dynamically adjusting the operation of the client network node with respect to at least one of the time-critical types of data streams and the non-time-critical types data streams in accordance with a realtime performance targeting process.

2. The method of claim 1, wherein the displaying comprises, by the client network node, executing the client application to render the graphical user interface.

3. The method of claim 1, wherein the processing, the monitoring, and the adjusting are specified in executable code of the client application.

4. The method of claim 1, further comprising, by the client network node, establishing a realtime communication session with at least one other client network node, determining presence data indicating availabilities of communicants operating the client network nodes to communicate, and displaying graphical indications of the presence data in the graphical user interface.

5. The method of claim 4, wherein the presence data comprises visual cues indicating current communication states of communicants in a realtime communication session.

6. The method of claim 1, further comprising, by the client network node, generating sound from a realtime audio data stream received from another client network node, wherein the generating comprises, by the client network node, executing the client application to interface with programmatic interfaces of sound generating resources of the client network.

7. The method of claim 1, wherein the modifying the processing to manage loads on resources of the client network node.

8. The method of claim 7, wherein the adjusting comprises dynamically configuring one or more components for processing ones of the time-critical data streams.

9. The method of claim 1, wherein:

the processing comprises, by the client network node, producing realtime communication data from at least one of the time-critical types of data streams;

the at least one monitored metric comprises a production rate of the realtime communication data;

the operational target comprises a time-based operational target on the production rate of the realtime communication data; and

the adjusting is performed by the client network node in response to a determination that the production rate of the realtime communication data fails to satisfy the time-based operational target.

10. The method of claim 9, wherein the time-based operational target comprises a threshold rate of production of the realtime communication data.

11. The method of claim 9, wherein the time-based operational target comprises a target latency for producing the realtime communication data.

12. The method of claim 9, wherein the adjusting comprises omitting processing of or more portions of the at least one time-critical type of data streams.

13. The method of claim 9, wherein the adjusting comprises omitting performance of one or more specified realtime data processing operations on the at least one time-critical data stream.

14. The method of claim 9, wherein the adjusting comprises performing on the at least one time-critical data stream a replacement set of one or more realtime data processing operations different from a default set of one or more realtime data processing operations designated for processing the at least one time-critical data stream.

15. The method of claim 1, wherein the adjusting comprises dynamically adjusting allocation of network resources of the client network node between the at least one time-critical types of data streams and the at least one non-time-critical types of the data streams.

16. The method of claim 15, wherein the adjusting comprises by the client network node dynamically adjusting the allocation of bandwidth available to the client network node between at least one of the time-critical types of data streams and at least one of the non-time-critical types of data streams.

17. The method of claim 16, wherein the adjusting comprises by the client network node delaying delivery of the at least one of the non-time-critical types of data streams.

18. The method of claim 1, wherein the adjusting comprises by the client network node throttling one or more of the non-time-critical types of data streams.

19. The method of claim 1, further comprising by the client network node:

capturing an image of a graphical representation of an information resource displayed in the graphical user interface; and

transmitting the captured image to a particular one of the other network nodes.

20. The method of claim 19, wherein the managing comprises preferentially allocating bandwidth available on a network link with the particular network node to transmission of a time-critical data stream over transmission of the captured image.

21. The method of claim 1, wherein the at least one operational target comprises at least one of a time-based threshold in the processing of respective ones of the data streams into resultant data streams and a quality of service level for one or more designated content types of respective ones of the data streams.

22. The method of claim 1, wherein:

the at least one monitored metric comprises utilization of at least one resource of the client network node;

the operational target comprises a utilization target for the at least one resource of the client network node; and

the adjusting is performed by the client network node in response to a determination that the monitored resource utilization fails to satisfy the utilization target.

23. The method of claim 22, wherein the at least one resource of the client network node comprises a processing resource of the client network node.

24. The method of claim 22, wherein the at least one resource of the client network node comprises a memory resource of the client network node.

25. The method of claim 22, wherein the at least one resource of the client network node comprises a networking resource of the client network node.

26. The method of claim 25, wherein the networking resource comprises bandwidth.

27. The method of claim 1, wherein the one or more non-time critical types of data streams are one or more of non-streaming web content data stream, text chat data streams, file sharing data streams, and screen share data streams.

28. The method of claim 1, wherein the one or more time-critical types of data streams are one or more of audio data streams and video data streams.

29. The method of claim 1, further comprising, by the client network node, executing the client application to interface with programmatic interfaces of resources of the client network node to perform operations comprising:

converting locally generated human perceptible input into data streams to be processed into outgoing data streams for transmission over respective ones of the network links; and

converting processed ones data streams received over respective ones of the network links into locally generated human perceptible output.

30. The method of claim 1, wherein the at least one operational target comprises a target delay level specified for at least one of the time-critical types of data streams.

31. The method of claim 1, wherein the at least one operational target comprises a target jitter level specified for at least one of the time-critical types of data streams.

32. The method of claim 1, wherein the at least one operational target comprises a target packet loss level specified for at least one of the time-critical types of data streams.

33. The method of claim 1, further comprising, by the client network node, executing the client application to interface the client network node with device drivers on the client network node.

34. The method of claim 1, wherein the transmitting and receiving of one or more time-critical types of data streams are performed in accordance with a User Datagram Protocol (UDP).

35. The method of claim 1, wherein the graphical user interface comprises at least one communication control for managing realtime communications with other network nodes and at least one browser control for navigating to different network resource addresses, the communication and browser controls being responsive to user inputs.

36. The method of claim 35, wherein the displaying comprises displaying the graphical realtime output from realtime communication data relating to a realtime communication session with at least one other client network node in a first display area of the graphical user interface, and displaying the a graphical representation of the network resource identified in connection with the browser control in a second display area of the graphical user interface.

37. The method of claim 35, wherein the at least one browsing control relates to at least one of:

navigating to a specified uniform resource identifier (URI) address of an information resource;

traversing a navigation stack of URI addresses; and

reloading a graphic representation of the information resource in the graphical user interface.

38. Apparatus, comprising:

a memory storing a client application integrating web browsing and network data stream processing for realtime communications; and

a processor coupled to the memory, operable to execute the client application, and based at least in part on the execution of the client application operable to perform operations comprising

on a display, displaying a graphical user interface providing a single point of control for realtime communications and web browsing interactions, wherein the displaying comprises displaying in the graphical user interface at least one control for establishing a presence in a virtual area;

requesting a presence in a particular virtual area identified in connection with the control and establishing a realtime communication session with at least one other client network node that is present in the particular virtual area;

transmitting outgoing data streams to and receiving incoming data streams from one or more other network nodes over respective network links via the realtime communications and web browsing interactions, the incoming and outgoing data streams comprising one or more time-critical types of data streams and one or more non-time-critical types of data streams, wherein the transmitting and receiving comprise, by the client network node, transmitting and receiving one or more time-critical types of data streams with each at least one other client network node and executing the client application to interface the client network node with network protocols on the client network node;

processing the one or more time-critical types of data streams and the one or more non-time-critical types of data streams;

monitoring at least one metric characterizing operation of the client network node during the processing of at least one of the time-critical types of the data streams; and

in response to a determination that the at least one metric fails to satisfy at least one operational target, dynamically adjusting the operation of the client network node with respect to at least one of the time-critical types of data streams and the non-time-critical types data streams in accordance with a realtime performance targeting process.

39. At least one non-transitory computer-readable medium comprising a client application integrating web browsing and network data stream processing for realtime communications, the client application being adapted to be executed by a computer to perform operations comprising:

on a display, displaying a graphical user interface providing a single point of control for realtime communications and web browsing interactions, wherein the displaying comprises displaying in the graphical user interface at least one control for establishing a presence in a virtual area;

requesting a presence in a particular virtual area identified in connection with the control and establishing a realtime communication session with at least one other client network node that is present in the particular virtual area;

transmitting outgoing data streams to and receiving incoming data streams from one or more other network nodes over respective network links via the realtime communications and web browsing interactions, the incoming and outgoing data streams comprising one or more time-critical types of data streams and one or more non-time-critical types of data streams, wherein the transmitting and receiving comprise, by the client network node, transmitting and receiving one or more time-critical types of data streams with each at least one other client network node and executing the client application to interface the client network node with network protocols on the client network node;

processing the one or more time-critical types of data streams and the one or more nontime-critical types of data streams;

monitoring at least one metric characterizing operation of the client network node during the processing of at least one of the time-critical types of the data streams; and

in response to a determination that the at least one metric fails to satisfy at least one operational target, dynamically adjusting the operation of the client network node with respect to at least one of the time-critical types of data streams and the non-time-critical types data streams in accordance with a realtime performance targeting process.