Adaptive data transmission
First Claim
1. A method, comprising:
- receiving, by a data collector comprising one or more computer processors, log data from one or more log producers to be transmitted to a log processor through a communications network, the log processor being configured to provide an acknowledgement to the data collector for each unit of log data received from the data collector;
subject to a transmission constraint of transmitting the received log data to the log processor, transmitting, by the data collector, a first portion of the received log data to the log processor and storing, by the data collector, a second portion of the received log data into a buffer configured to adapt to the transmission constraint, the transmission constraint including;
a first threshold on bandwidth usage and a second threshold on log processor response time; and
rules on adapting a size of the buffer or computing resources allocated to the log processor when the thresholds are reached; and
adapting the data collector or the log processor for transmitting the stored portion of the log data to the log processor in accordance with the rules in the transmission constraint, including;
upon determining, by a collection manager comprising one or more computer processors, that a bandwidth utilization rate of the communication network reached the first threshold, instructing, by the collection manager, the data collector to adapt to the bandwidth utilization rate by increasing the size of the buffer, increasing the size of the buffer including allocating additional storage space for the buffer wherein, when the bandwidth utilization rate no longer reaches the first threshold, the additional storage space is reallocated for other uses, and wherein determining that the bandwidth utilization rate reached the first threshold comprises determining, by the collection manager, that traffic of transmitting the first portion of the received log data from the data collector to the log processor has been utilizing full available bandwidth of the communication network for a given period of time; and
upon determining, by the collection manager, that a promptness for providing the acknowledgement by the log processor satisfies the second threshold, configuring the log processor to allocate more computing resources to process log data transmitted from the data collector.
15 Assignments
0 Petitions
Accused Products
Abstract
Methods, program products, and systems implementing adaptive data transmission are disclosed. A universal data collector can act as an intermediary between a log producer and a log processor. The universal data collector can be connected to the log producer through a first, reliable network. The universal data collector can be connected to the log processor through a second, less reliable network. The universal data collector can perform a multi-phase commit of data to ensure that the log data are saved by the log processor. The universal data collector can smooth the data transmission from the universal data collector to the log processor through a network, including adaptively buffering the log data according to available network bandwidth of the network.
-
Citations
20 Claims
-
1. A method, comprising:
-
receiving, by a data collector comprising one or more computer processors, log data from one or more log producers to be transmitted to a log processor through a communications network, the log processor being configured to provide an acknowledgement to the data collector for each unit of log data received from the data collector; subject to a transmission constraint of transmitting the received log data to the log processor, transmitting, by the data collector, a first portion of the received log data to the log processor and storing, by the data collector, a second portion of the received log data into a buffer configured to adapt to the transmission constraint, the transmission constraint including; a first threshold on bandwidth usage and a second threshold on log processor response time; and rules on adapting a size of the buffer or computing resources allocated to the log processor when the thresholds are reached; and adapting the data collector or the log processor for transmitting the stored portion of the log data to the log processor in accordance with the rules in the transmission constraint, including; upon determining, by a collection manager comprising one or more computer processors, that a bandwidth utilization rate of the communication network reached the first threshold, instructing, by the collection manager, the data collector to adapt to the bandwidth utilization rate by increasing the size of the buffer, increasing the size of the buffer including allocating additional storage space for the buffer wherein, when the bandwidth utilization rate no longer reaches the first threshold, the additional storage space is reallocated for other uses, and wherein determining that the bandwidth utilization rate reached the first threshold comprises determining, by the collection manager, that traffic of transmitting the first portion of the received log data from the data collector to the log processor has been utilizing full available bandwidth of the communication network for a given period of time; and upon determining, by the collection manager, that a promptness for providing the acknowledgement by the log processor satisfies the second threshold, configuring the log processor to allocate more computing resources to process log data transmitted from the data collector. - View Dependent Claims (2, 3, 4, 5, 6)
-
-
7. A non-transitory storage device storing a computer program product, the computer program product, when executed, operable to cause a data collector and a collection manager each comprising one or more processors to perform operations comprising:
-
receiving, by the data collector, log data from one or more log producers to be transmitted to a log processor through a communications network, the log processor being configured to provide an acknowledgement to the data collector for each unit of log data received from the data collector; subject to a transmission constraint of transmitting the received log data to the log processor, transmitting, by the data collector, a first portion of the received log data to the log processor and storing, by the data collector, a second portion the received log data into a buffer configured to adapt to the transmission constraint, the transmission constraint including; a first threshold on bandwidth usage and a second threshold on log processor response time; and rules on adapting a size of the buffer or computing resources allocated to the log processor when the thresholds are reached; and adapting the data collector or the log processor for transmitting the stored portion of the log data to the log processor in accordance with the rules in the transmission constraint, including; upon determining, by the collection manager, that a bandwidth utilization rate of the communication network reached the first threshold, instructing, by the collection manager, the data collector to adapt to the bandwidth utilization rate by increasing the size of the buffer, increasing the size of the buffer including allocating additional storage space for the buffer wherein, when the bandwidth utilization rate no longer reaches the first threshold, the additional storage space is reallocated for other uses, and wherein determining that the bandwidth utilization rate reached the first threshold comprises determining, by the collection manager, that traffic of transmitting the first portion of the received log data from the data collector to the log processor has been utilizing full available bandwidth of the communication network for a given period of time; and upon determining, by the collection manager, that a promptness for providing the acknowledgement by the log processor satisfies the second threshold, configuring the log processor to allocate more computing resources to process log data transmitted from the data collector. - View Dependent Claims (8, 9, 10, 11, 12, 13, 14)
-
-
15. A system comprising:
-
a data collector and a collection manager each comprising one or more computer processors; and a non-transitory storage device storing a computer program product, the computer program product, when executed, operable to cause the data collector and collection manager to perform operations comprising; receiving, by the data collector, log data from one or more log producers to be transmitted to a log processor through a communications network, the log processor being configured to provide an acknowledgement to the data collector for each unit of log data received from the data collector; subject to a transmission constraint of transmitting the received log data to the log processor, transmitting, by the data collector, a first portion of the received log data to the log processor and storing, by the data collector, a second portion the received log data into a buffer configured to adapt to the transmission constraint, the transmission constraint including; a first threshold on bandwidth usage and a second threshold on log processor response time; and rules on adapting a size of the buffer or computing resources allocated to the log processor when the thresholds are reached; and adapting the data collector or the log processor for transmitting the stored portion of the log data to the log processor in accordance with the rules in the transmission constraint, including; upon determining, by the collection manager, that a bandwidth utilization rate of the communication network reached the first threshold, instructing, by the collection manager, the data collector to adapt to the bandwidth utilization rate by increasing the size of the buffer, increasing the size of the buffer including allocating additional storage space for the buffer wherein, when the bandwidth utilization rate no longer reaches the first threshold, the additional storage space is reallocated for other uses, and wherein determining that the bandwidth utilization rate reached the first threshold comprises determining, by the collection manager, that traffic of transmitting the first portion of the received log data from the data collector to the log processor has been utilizing full available bandwidth of the communication network for a given period of time; and upon determining, by the collection manager, that a promptness for providing the acknowledgement by the log processor satisfies the second threshold, configuring the log processor to allocate more computing resources to process log data transmitted from the data collector. - View Dependent Claims (16, 17, 18, 19, 20)
-
Specification