Systems and methods for storing time-series data

• US 10,168,921 B1
• Filed: 11/20/2015
• Issued: 01/01/2019
• Est. Priority Date: 10/10/2007
• Status: Active Grant

• Current Assignee: United Services Automobile Association
• Original Assignee: United Services Automobile Association
• Inventors: Becker, Glen A.
• Primary Examiner(s): Truong, Cam Y T
• Application Number:
  US14/948,003
• Time in Patent Office: 1,138 Days
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/2291 User-Defined Types; Storage management thereof
• CPC Class Codes: G06F 16/2452 Query translation
• CPC Class Codes: G06F 16/24565 Triggers; Constraints
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/0685 Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays

Abstract

In various embodiments, a method for maintaining and analyzing a rolling history of performance data for a computer network is described. In one embodiment, one year'"'"'s data is stored, sorted by object and date. New data is appended to the end of the file each day and an index by object is stored. Using this method, data for twelve full months can be provided in object order, and data for one partial month that was added daily in date order can also be provided.

Claims

1. A computer-implemented method comprising:

receiving, by a volatile memory coupled to one or more server computers, a set of data from a first computer comprising a set of objects;

storing, by the volatile memory based on an organization'"'"'s policy, the set of data into a first set of memory locations of the volatile memory according to a first sort-order, wherein the first sort-order sorts the set of data according to an object type of each respective object in the set of objects;

in response to determining that a performance parameter satisfies a condition during a time period, storing, by a non-volatile memory coupled to the one or more server computers, the set of data into a second set of memory locations of the non-volatile memory according to a second sort-order, wherein a physical datastore comprises the volatile memory and the non-volatile memory, wherein the second sort-order sorts the set of data according to a timestamp of each respective object in the set of objects, wherein the storing the set of data into the non-volatile memory further comprises;

updating, by a server coupled to the non-volatile memory, a memory location of existing data stored in the non-volatile memory upon storing the set of data into the second set of memory locations according to the second sort-order, and wherein the memory location of the existing data is updated to a third set of memory locations within the non-volatile memory in accordance with the second sort-order and the third set of memory locations is configured to be purged;

receiving, by the volatile memory coupled to the one or more server computers, a new set of data; and

upon receiving the new set of data, updating the set of data by;

sorting, by the one or more server computers, the new set of data according to object types of the new set of data and according to timestamps of the new set of data;

merging, by the one or more server computers, the sorted new set of data and the data within the second set of memory locations, thereby generating merged data being sorted according to object types and timestamps, wherein the merging the sorted new set of data is performed when processing resources are below a threshold value; and

updating, by the one or more server computers, indexes corresponding to memory locations of the merged data to reflect new locations of the merged data.

2. The method according to claim 1, wherein storing the set of data into the non-volatile memory further comprises:

transforming, by a server coupled to at least one of the non-volatile memory and the volatile memory, the set of data from the first sort-order to the second sort-order.

3. The method according to claim 1, wherein the set of data received from the first computer comprises a set of objects associated with an application executed by the first computer.

4. The method according to claim 1, further comprising accessing, by a server computer coupled to the volatile memory, an object stored at a memory location in the volatile memory according to the object type.

5. The method according to claim 1, wherein a magnetic disk comprises at least a portion of the non-volatile memory.

6. The method according to claim 1, wherein the volatile memory is a type of a random access memory.

7. The method according to claim 1, wherein the condition is based on at least one of:

a time period, and a characteristic of the set of data received from the first computer.

8. A computing system comprising:

a volatile memory coupled to one or more server computers, the volatile memory configured to receive a set of data from a first computer comprising a set of objects, and store the set of data into a first set of memory locations of the volatile memory according to a first sort-order, wherein the first sort-order sorts the set of data according to an object type of each respective object in the set of objects;

a non-volatile memory coupled to one or more server computers, the non-volatile memory configured to store the set of data into a second set of memory locations of the non-volatile memory according to a second sort-order, wherein a physical datastore comprises the volatile memory and the non-volatile memory, and wherein the second sort-order sorts the set of data according to a timestamp of each respective object in the set of objects; and

at least one server computer coupled to at least one of the volatile memory and the volatile memory, and configured to;

in response to determining that a performance parameter satisfies a condition during a time period, store the set of data from the first set of memory locations of the volatile memory into the second set of memory locations of the non-volatile memory, wherein the storing the set of data into the non-volatile memory further comprises;

updating, by a server coupled to the non-volatile memory, a memory location of existing data stored in the non-volatile memory upon storing the set of data into the second set of memory locations according to the second sort-order, wherein the memory location of the existing data is updated to a third set of memory locations within the non-volatile memory in accordance with the second sort-order and the third set of memory locations is configured to be purged;

upon the at least one server computer detecting a triggering condition associated with the set of data when the set of data is stored in the volatile memory, receive a new set of data; and

upon receiving the new set of data, update the set of data by;

sorting the new set of data according to object types of the new set of data and according to the timestamps of new set of data;

merging the sorted new set of data and the data within the second set of memory locations, thereby generating merged data being sorted according to object types and timestamps, wherein the merging the sorted new set of data is performed when processing resources are below a threshold value, and

updating indexes corresponding to memory locations of the merged data to reflect new locations of the merged data.

9. The system according to claim 8, wherein the at least one server computer is further configured to transform the set of data from the first sort-order to the second sort-order upon detecting the triggering condition.

10. The system according to claim 8, wherein the set of data received from the first computer comprises a set of objects associated with an application executed by the first computer.

11. The system according to claim 8, wherein the volatile memory comprises a type of random access memory.

12. The system according to claim 8, wherein a magnetic disk comprises at least a portion of the non-volatile memory.

13. The system according to claim 8, wherein the triggering condition is based on at least one of:

a time period, and a characteristic of the set of data received from the first computer.