Systems and methods for dynamic aggregation of data and minimization of data loss

• US 10,657,018 B1
• Filed: 08/26/2019
• Issued: 05/19/2020
• Est. Priority Date: 08/26/2019
• Status: Active Grant

• Current Assignee: Coupang Corp.
• Original Assignee: Coupang Corp.
• Inventors: Nam, Il Won, Park, Seung Hoon, Kim, Ki Young
• Primary Examiner(s): Mehrmanesh, Elmira
• Application Number:
  US16/551,091
• Time in Patent Office: 267 Days
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3068 where the reporting involves data format conversion
• CPC Class Codes: G06F 11/3075 the data filtering being achieved in order to maintain consistency among the monitored data, e.g. ensuring that the monitored data belong to the same timeframe, to the same system or component
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3466 Performance evaluation by tracing or monitoring
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/906 Clustering; Classification
• CPC Class Codes: G06F 2201/835 Timestamp

Abstract

A computer-implemented system for dynamic aggregation of data and minimization of data loss is disclosed. The system may be configured to perform instructions for: aggregating information from a plurality of networked systems by collecting a set of data from the networked systems, the set of data comprising data associated with a predetermined period of time and comprising one or more central variables that are included in data associated with more than one networked systems of the plurality of networked systems and one or more associated variables that describe one or more aspects of the central variables; retrieving one or more data transformation rules based on a relational map among the central variables and the associated variables; and aggregating the first set of data into one or more master data structures corresponding to the central variables based on the data transformation rules.

Claims

1. A computer-implemented system for dynamic aggregation of data and minimization of data loss, the system comprising:

a memory storing instructions; and

at least one processor configured to execute the instructions for;

aggregating information from a plurality of networked systems by;

collecting a first set of data at a first time point from the networked systems, the first set of data comprising data associated with a predetermined period of time and comprising one or more central variables that are included in data associated with more than one networked systems of the plurality of networked systems and one or more associated variables that describe one or more aspects of the central variables, each of the central variables and the associated variables comprising a corresponding value;

retrieving one or more data transformation rules based on a relational map among the central variables and the associated variables;

aggregating the first set of data into one or more master data structures corresponding to the central variables based on the data transformation rules, each of the one or more master data structures comprising one or more data fields that correspond to one of the central variables and a subset of the associated variables;

determining that a second subset of the networked systems is available after a gap of time; and

updating one or more timestamps corresponding to the networked systems based on the first time point; and

generating one or more data reports based on the master data structures,

wherein collecting the first set of data from the networked systems further comprises collecting a second set of data from the second subset of the networked systems, the second set of data comprising data associated with the gap of time, and

wherein aggregating the first set of data into the master data structures further comprises aggregating the second set of data into the master data structures.

2. The computer-implemented system of claim 1, wherein the data transformation rules define how a value in the first set of data corresponds to one or more data fields of the master data structures.

3. The computer-implemented system of claim 1, wherein a subset of the data transformation rules assigns values of one or more of the associated variables to more than one data field of the master data structures.

4. The computer-implemented system of claim 1, wherein the processor is configured to repeat the instructions for aggregating information from the networked systems at a predetermined interval, wherein the predetermined interval is less than or equal to the predetermined period.

5. The computer-implemented system of claim 4, wherein the predetermined interval is equal to or less than one minute.

6. The computer-implemented system of claim 1, wherein aggregating the information from the networked systems further comprises:

determining that a first subset of the networked systems is unavailable; and

updating one or more timestamps corresponding to the networked systems,

wherein aggregating the first set of data into the master data structures omits a subset of the data transformation rules that correspond to a subset of data corresponding to the first subset of the networked systems, and

wherein updating the timestamps omits updating timestamps that correspond to the first subset of the networked systems.

7. The computer-implemented system of claim 1, wherein aggregating the first set of data comprises:

sorting the first set of data in sequence based on time;

iterating through the sorted first set of data chronologically to replace existing values of one or more data fields in the master data structures with the values from the sorted first set of data.

8. The computer-implemented system of claim 1, wherein the relational map is determined based on one or more data profiles of the networked systems, the data profiles comprising at least one of one or more metadata, one or more definitions of the associated variables, and a data element synonym registry.

9. The computer-implemented system of claim 1, wherein generating the data reports comprises analyzing the master data structures to determine one or more performance metrics associated with the networked systems.

10. A computer-implemented method for dynamic aggregation of data and minimization of data loss, the method comprising:

aggregating information from a plurality of networked systems by;

collecting a first set of data at a first time point from the networked systems, the first set of data comprising data associated with a predetermined period of time and comprising one or more central variables that are included in data associated with more than one networked systems of the plurality of networked systems and one or more associated variables that describe one or more aspects of the central variables, each of the central variables and the associated variables comprising a corresponding value;

retrieving one or more data transformation rules based on a relational map among the central variables and the associated variables;

aggregating the first set of data into one or more master data structures corresponding to the central variables based on the data transformation rules, each of the one or more master data structures comprising one or more data fields that correspond to one of the central variables and a subset of the associated variables;

determining that a second subset of the networked systems is available after a gap of time; and

updating one or more timestamps corresponding to the networked systems based on the first time point; and

generating one or more data reports based on the master data structures,

wherein collecting the first set of data from the networked systems further comprises collecting a second set of data from the second subset of the networked systems, the second set of data comprising data associated with the gap of time, and

wherein aggregating the first set of data into the master data structures further comprises aggregating the second set of data into the master data structures.

11. The computer-implemented method of claim 10, wherein the data transformation rules define how a value in the first set of data corresponds to one or more data fields of the master data structures.

12. The computer-implemented method of claim 10, wherein a subset of the data transformation rules assigns values of one or more of the associated variables to more than one data fields of the master data structures.

13. The computer-implemented method of claim 10, wherein the processor is configured to repeat the instructions for aggregating information from the networked systems at a predetermined interval, wherein the predetermined interval is less than or equal to the predetermined period.

14. The computer-implemented method of claim 10, wherein aggregating the information from the networked systems further comprises:

determining that a first subset of the networked systems is unavailable; and

updating one or more timestamps corresponding to the networked systems,

wherein aggregating the first set of data into the master data structures omits a subset of the data transformation rules that correspond to a subset of data corresponding to the first subset of the networked systems, and

wherein updating the timestamps omits updating timestamps that correspond to the first subset of the networked systems.

15. The computer-implemented method of claim 10, wherein aggregating the first set of data comprises:

sorting the first set of data in sequence based on time;

iterating through the sorted first set of data chronologically to replace existing values of one or more data fields in the master data structures with the values from the sorted first set of data.

16. The computer-implemented method of claim 10, wherein the relational map is determined based on one or more data profiles of the networked systems, the data profiles comprising at least one of one or more metadata, one or more definitions of the associated variables, and a data element synonym registry.

17. The computer-implemented method of claim 10, wherein generating the data reports comprises analyzing the master data structures to determine one or more performance metrics of the networked systems.

18. A computer-implemented system for dynamic aggregation of data and minimization of data loss, the system comprising:

a memory storing instructions; and

at least one processor configured to execute the instructions for;

aggregating information from a plurality of networked systems by;

transmitting a data request to the networked systems at a predetermined interval;

receiving a first set of data at a first time point from a first subset of networked systems, the first subset of networked systems having a first set of corresponding timestamps from an immediately preceding time point, and the first set of data comprising data associated with a period of time between the immediately preceding time point and the first time point;

receiving a second set of data at the first time point from a second subset of networked systems, the second subset of networked systems having a second set of corresponding timestamps from a second time point older than the immediately preceding time point, and the second set of data comprising data associated with a period between the second time point and the first time point;

receiving a third set of data at the first time point from a third subset of networked systems, the third subset of networked systems having a third set of corresponding timestamps from the immediately preceding time point, and the third set of data indicating that the third subset of networked systems are not available,

wherein the first set of data and the second set of data comprise one or more central variables that are included in data from more than one networked systems of the plurality of networked systems and one or more associated variables that describe one or more aspects of the central variables;

retrieving one or more data transformation rules based on a relational map among the central variables and the associated variables;

aggregating the first and second sets of data into one or more master data structures corresponding to the central variables based on the data transformation rules, each of the one or more master data structures comprising one or more data fields that correspond to one of the central variables and a subset of the associated variables; and

updating the first and second sets of corresponding timestamps based on the first time point; and

generating one or more data reports based on the master data structures.