Managing large volumes of event data records

• US 10,103,964 B2
• Filed: 06/17/2016
• Issued: 10/16/2018
• Est. Priority Date: 06/17/2016
• Status: Active Grant

• Current Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Sheleheda, Daniel G., Alexander, Samuel Norman, Hardenbergh, John H., Harten, Joseph, Pace, James, Spielman, Chaim
• Primary Examiner(s): Thompson, Jr., Otis L
• Application Number:
  US15/185,152
• Publication Number:
  US 20170366440A1
• Time in Patent Office: 851 Days
• Field of Search: None
• CPC Class Codes: H04L 41/0613 based on the type or category of the network elements
• CPC Class Codes: H04L 41/0622 based on time
• CPC Class Codes: H04L 41/069 using logs of notifications; Post-processing of notifications
• CPC Class Codes: H04L 43/02 Capturing of monitoring data
• CPC Class Codes: H04L 43/024 by adaptive sampling
• CPC Class Codes: H04L 43/0876 Network utilisation, e.g. volume of load or congestion level
• CPC Class Codes: H04L 43/16 Threshold monitoring
• CPC Class Codes: H04L 63/1408 by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/1458 Denial of Service

Abstract

A network device that operates as an analysis platform for analysis of event data records that can provide a flexible approach to event data record aggregation. For example, aggregation can be flexibly turned on or off and dynamically adjusted based on event record volume and other factors such as network capacity or throughput. Devices that are instructed to aggregate records can also be instructed to archive the raw records, e.g., to maintain a full fidelity log of events. Devices can further be instructed to utilize a mixed queue approach to determine an order to deliver those records that includes both older records and newer records.

Claims

1. A network device, comprising:

a processor; and

a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, comprising;

receiving first data comprising first raw event data records describing events of a first period logged by devices of a network from a record collection device that collects the first data from the devices;

determining that a volume of the first raw event data records exceeds a defined threshold, wherein the defined threshold is a rate of change threshold that is exceeded in response to a determination that slopes of a number of first raw event data records plotted over time increase for multiple consecutive time intervals;

instructing the record collection device to perform a first level of aggregation, aggregating on a first data key, on second raw event data records describing events of a second period logged by the devices; and

receiving, from the record collection device, second data comprising an aggregated event data record representing an aggregation of the second raw event data records.

2. The network device of claim 1, wherein the defined threshold is a first defined threshold, and wherein the operations further comprise determining that the volume of the first raw event data records exceeds a second defined threshold that is a volumetric threshold that is exceeded in response to a volumetric maximum determined as a first derivative function of the volume of the first raw event data records over time.

3. The network device of claim 1, wherein the first level of aggregation is configured based on the volume of the first raw event data records.

4. The network device of claim 1, wherein the volume is a first volume and the operations further comprise:

determining that a second volume of aggregated event data records, comprising the aggregated event data record, exceeds the defined threshold; and

instructing the record collection device to perform a second level of aggregation, aggregating on a second data key that is not the first data key, on the second raw event data records.

5. The network device of claim 1, wherein the operations further comprise determining a level of aggregation, ranging from no aggregation to a defined maximum level of aggregation, to apply over a defined time window to the second raw event data records, as a function of network throughput between the record collection device and the network device and a measure of volume of event data records to be transmitted to the network device.

6. The network device of claim 5, wherein the operations further comprise adjusting the defined time window as a function of a defined measure of volume of the event data records to be received.

7. The network device of claim 1, wherein the operations further comprise instructing the record collection device to store the second raw event data records that are aggregated to generate the aggregated event data record.

8. The network device of claim 7, wherein the operations further comprise in response to a determination that available throughput exists between the record collection device and the network device, instructing the record collection device to transmit the second raw event data records.

9. The network device of claim 1, wherein the operations further comprise instructing the record collection device to order event data records transmitted to the network device according to a mixed ordering approach comprising transmitting a first portion of the event data records according to first-in-first-out transmissions and transmitting a second portion of the event data records according to last-in-last-out transmissions.

10. The network device of claim 9, wherein the operations further comprise instructing the record collection device to adjust a ratio of the first portion and the second portion.

11. A network device, comprising:

a processor; and

a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, comprising;

receiving, from a record collection device, first data comprising an aggregated event data record that is aggregated to a first level of aggregation based on a first data key and represents an aggregation of first raw event data records describing events of a first period logged by devices of a network;

determining that a volume of aggregated event data records, comprising the aggregated event data record, is less than a defined threshold;

instructing the record collection device to terminate the aggregation of the first raw event data records and to transmit second data comprising second raw event data records describing events of a second period logged by the devices;

receiving the second data; and

instructing the record collection device to perform a second level of aggregation that aggregates based on a second data key that is not the first data key and operates on the second raw event data records.

12. The network device of claim 11, wherein the operations further comprise determining a level of aggregation, ranging from no aggregation to a defined maximum level of aggregation, to apply over a defined time window to the first raw event data records, as a function of network throughput between the record collection device and the network device and a measure of volume of event data records to be transmitted to the network device.

13. The network device of claim 11, wherein the operations further comprise instructing the record collection device to archive the first raw event data records that are aggregated to generate the aggregated event data records.

14. The network device of claim 11, wherein the operations further comprise instructing the record collection device to arrange event data records transmitted to the network device according to a mixed ordering approach comprising transmitting a first portion of the event data records according to a first-in-first-out queue of records to be transmitted and transmitting a second portion of the event data records according to last-in-last-out queue of records to be transmitted.

15. The network device of claim 11, wherein the operations further comprise instructing the record collection device to order event data records transmitted to the network device according to a mixed ordering approach comprising transmitting a first portion of the event data records according to first-in-first-out transmissions and transmitting a second portion of the event data records according to last-in-last-out transmissions.

16. A method, comprising:

receiving, by a network device comprising a processor, first data comprising first event data records describing events of a first period logged by devices of a network, wherein the first event data records are not aggregated;

determining, by the network device, a volume of the first event data records exceeds a defined threshold, wherein the defined threshold is a threshold applicable to a rate of change that is exceeded in response to a determination that slopes of a number of first raw event data records plotted over time have increased for at least two consecutive time intervals;

determining, by the network device, aggregation data indicative of an aggregation profile to apply to second event data records describing events of a second period logged by the devices, wherein the aggregation profile comprises configurable elements comprising aggregation key data representing a key employable to aggregate the second event data records and aggregation time data representing a defined time interval employable to aggregate the second event data records; and

receiving, by the network device, second data comprising an aggregated representation of the second event data records, wherein the aggregated representation corresponds to the aggregation profile.

17. The method of claim 16, further comprising configuring, by the network device, the configurable elements of the aggregation profile as a function of a network throughput measure and a measure of volume of event data records to be transmitted to the network device.

18. The method of claim 16, further comprising transmitting, by the network device, archive instruction data to a source device representing a device from which the second data is received, wherein the archive instruction data comprises an indication that the source device is to archive the second event data records.

19. The method of claim 16, further comprising transmitting, by the network device, format instruction data to a source device representing a device from which the second data is received, wherein the format instruction data comprises an indication that the source device is to transmit the second data according to a mixed ordering approach comprising transmitting a first portion of the second data according to a first-in-first-out approach and transmitting a second portion of the second data according to a last-in-last-out approach.

20. The method of claim 16, wherein the defined threshold is a first defined threshold, and the method further comprising determining, by the network device, that the volume of the first raw event data records exceeds a second defined threshold that is a volumetric threshold that is exceeded in response to a volumetric maximum determined as a first derivative function of the volume of the first raw event data records over time.