Adaptive Data Transmission

• US 20120089568A1
• Filed: 09/02/2011
• Published: 04/12/2012
• Est. Priority Date: 09/03/2010
• Status: Active Grant

• Current Assignee: Cloud Software Group
• Original Assignee: TIBCO Software Incorporated (Cloud Software Group)
• Inventors: Manley, Stephen, Malaiyandi, Prasanna Kumar, Pautet, Laurent
• Granted Patent: US 9,191,323 B2
• Time in Patent Office: Days
• US Class Current: 707/634
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: H04L 47/10 Flow control; Congestion control
• CPC Class Codes: H04L 49/9005 using dynamic buffer space allocation

Abstract

Methods, program products, and systems implementing adaptive data transmission are disclosed. A universal data collector can act as an intermediary between a log producer and a log processor. The universal data collector can be connected to the log producer through a first, reliable network. The universal data collector can be connected to the log processor through a second, less reliable network. The universal data collector can perform a multi-phase commit of data to ensure that the log data are saved by the log processor. The universal data collector can smooth the data transmission from the universal data collector to the log processor through a network, including adaptively buffering the log data according to available network bandwidth of the network.

Claims

1. A method executed by a computing device, comprising:

receiving log data from one or more log producers to be transmitted to a log processor;

subject to a transmission constraint of transmitting the received log data to the log processor, storing at least a portion of the received log data into a buffer configured to adapt to the transmission constraint; and

transmitting the stored portion of the log data to the log processor in accordance with the transmission constraint.

2. The method of claim 1, wherein:

receiving log data from the one or more log producers includes receiving log data from one or more log producing devices through a first network; and

transmitting the stored log data to the log processor includes transmitting the stored log data to a remote log processing device through a second network.

3. The method of claim 2, wherein the first network includes a local area network (LAN) and the second network includes a wide area network (WAN).

4. The method of claim 2, wherein the transmission constraint includes an available network bandwidth of the second network for transmitting the log data.

5. The method of claim 4, wherein storing the received log data into the buffer configured to adapt to the transmission constraint includes storing the portion of the log data such that other portions of the log data are transmitted to the log processor using the available network bandwidth.

6. The method of claim 4, wherein the buffer is configured to have a size that is determined based on a rate of receiving the log data and the available network bandwidth.

7. The method of claim 4, wherein transmitting the stored portion of the log data to the log processor in accordance with the transmission constraint includes delaying transmitting the stored portion of the log data to the log processor until the available network bandwidth permits the stored portion of the log data to be transmitted.

8. The method of claim 1, wherein the transmission constraint of transmitting the received log data to the log processor includes a reliability of the log processor.

9. The method of claim 8, wherein the reliability of the log processor is determined using an acknowledgement from the log processor, the acknowledgment indicating that transmitted log data have been successfully stored on a storage device.

10. The method of claim 9, wherein transmitting the stored portion of the log data to the log processor includes transmitting the stored portion of the log data before receiving the acknowledgement.

11. The method of claim 9, further comprising transmitting to the log processor an instruction instructing the log processor to send the acknowledgement to the computing device after a specified quantity of log data have been successfully stored on the storage device.

12. The method of claim 8, wherein determining the reliability of the log processor includes receiving a heart beat signal from the log processor, the heart beat signal indicating that the log processor is capable of receiving and processing transmitted log data.

13. The method of claim 8, wherein transmitting the stored portion of the log data to the log processor in accordance with the transmission constraint includes delaying transmitting the stored portion of the log data to the log processor until the computing device receives a signal from the log processor indicating that the log processor is capable of receiving and processing transmitted log data.

14. The method of claim 1, wherein transmitting the stored portion of the log data to the log processor includes transmitting information indicating that the stored portion of the log data have been stored for a period of time.

15. The method of claim 1, further comprising:

determining a first feature indicator, the first feature indicator indicating that one or more first features of processing the log data are available on the computing device;

receiving a second feature indicator from the log processor, the second feature indicator indicating that one or more second features of processing the log data are available on the log processor;

determining that at least one first feature does not correspond to the one or more second features; and

configuring the computing device such that the first feature that does not correspond to the one or more second features is not applied in processing the log data to be transmitted to the log processor.

16. A computer program product stored on a storage device, operable to cause one or more processors to perform operations comprising:

receiving log data from one or more log producers to be transmitted to a log processor;

subject to a transmission constraint of transmitting the received log data to the log processor, storing at least a portion the received log data into a buffer configured to adapt to the transmission constraint; and

transmitting the stored portion of the log data to the log processor in accordance with the transmission constraint.

17. A system comprising:

a computing device configured to perform operations comprising;

receiving log data from one or more log producers to be transmitted to a log processor;

subject to a transmission constraint of transmitting the received log data to the log processor, storing at least a portion the received log data into a buffer configured to adapt to the transmission constraint; and

transmitting the stored portion of the log data to the log processor in accordance with the transmission constraint.