Systems and Methods for Reducing Data Storage Overhead

• US 20200134077A1
• Filed: 10/30/2018
• Published: 04/30/2020
• Est. Priority Date: 10/30/2018
• Status: Active Grant

• Current Assignee: Elasticsearch Global B.V. (Elastic N.V.)
• Original Assignee: Elasticsearch Global B.V. (Elastic N.V.)
• Inventors: Tong, Zachary
• Granted Patent: US 10,997,196 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/2246 Trees, e.g. B+trees
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database

Abstract

Systems and methods for reducing data storage overhead are disclosed herein. In some embodiments, a system includes a rollup service that converts a raw data set into a rolled up index that takes up less storage than the raw data but is created in such a way that the rolled up index can be queried so as to generate responses that will substantially correspond to responses that would be generated using the raw data.

Claims

1. A method, comprising:

obtaining raw data from one or more systems providing one or more services, the raw data comprising data elements having corresponding fields and metrics, the data elements being obtained according to a first time interval, the raw data having a storage size;

generating a rollup index of the raw data based on a first set of rollup parameters that comprise selections of the fields and metrics by;

searching for matching ones of the data elements of the raw data that correspond to the first set of rollup parameters;

grouping the data elements together based on the matching;

flattening the data elements using a tree structure; and

rolling up the data elements into the rollup index using the tree structure based on a second time interval that is larger than the first time interval; and

storing the rollup index in a storage space, the rollup index being reduced in size relative to the raw data so as to reduce an amount of the storage space required to store rollup index relative to storing the raw data.

2. The method according to claim 1, wherein when the first time interval is measured in milli-seconds the second time interval is measured in any of days, weeks, months, years, or other time frames that are larger than seconds.

3. The method according to claim 1, further comprising storing the raw data along with the rollup index.

4. The method according to claim 3, wherein the raw data that is stored comprises only a last segment of the raw data covering a latter portion of the first time interval.

5. The method according to claim 4, wherein the rollup index comprises additional flattened data elements selected using a second set of rollup parameters.

6. The method according to claim 5, wherein the second set of rollup parameters are orthogonal in content to the first set of rollup parameters.

7. The method according to claim 6, further comprising:

receiving a query having query parameters;

parsing the query parameters;

searching the query parameters against the rollup index; and

generating a response using the rollup index .

8. The method according to claim 7, further comprising initially searching the last segment of the raw data for portions of the data elements that correspond to the query parameters prior to searching the rollup index.

9. The method according to claim 7, wherein the response is generated relative to a third time interval that is larger than the second time interval.

10. The method according to claim 1, wherein the rollup index is continually updated as the one or more systems generate new data elements.

11. A system, comprising:

a rollup service configured to;

obtain raw data from one or more systems providing one or more services, the raw data comprising data elements having corresponding fields and metrics, the data elements being obtained according to a first time interval, the raw data having a storage size;

generate a rollup index of the raw data based on a first set of rollup parameters that comprise selections of the fields and metrics by;

searching for matching ones of the data elements of the raw data that correspond to the first set of rollup parameters;

grouping the data elements together based on the matching;

flattening the data elements; and

rolling up the data elements for placement into the rollup index based on a second time interval that is larger than the first time interval; and

store the rollup index in a storage space, the rollup index being reduced in size relative to the raw data so as to reduce an amount of the storage space required to store rollup index relative to storing the raw data; and

a search service endpoint configured to;

receive a query having query parameters;

parse the query parameters;

search the query parameters against the rollup index; and

generate a response using the rollup index.

12. The system according to claim 11, wherein the search service endpoint configured to is further configured to initially search a last segment of the raw data for portions of the data elements that correspond to the query parameters prior to searching the rollup index.

13. The system according to claim 12, wherein the response is generated relative to a third time interval that is larger than the second time interval.

14. The system according to claim 13, wherein the rollup index is continually updated as the one or more systems generate new data elements.

15. The system according to claim 14, wherein when the first time interval is measured in seconds the second time interval is measured in any of days, weeks, months, years, or other time frames that are larger than seconds.

16. The system according to claim 15, wherein the rollup service is configured to store the raw data along with the rollup index.

17. A method, comprising:

obtaining raw historical data for a computing system, the raw historical data being according to a first time interval;

converting the raw historical data into a rollup index, the rollup index comprising aggregations of data elements of the raw historical data which are grouped according to at least one field and at least one metric regarding the at least one field; and

generating a query response for a query using the rollup index, the query response substantially corresponding to a query response when executed against the raw historical data due to the conversion of the raw historical data into the rollup index.

18. The method according to claim 17, wherein the raw historical data is converted into the rollup index using flattened aggregation trees.

19. The method according to claim 17, wherein the aggregations of data elements in the rollup index are obtained relative to a second time interval that is greater than the first time interval, wherein the query response is generated relative to a third time interval that is greater than or equal to the second time interval.