SYSTEM AND METHOD FOR OPTIMIZING STORAGE OF MULTI-DIMENSIONAL DATA IN DATA STORAGE

• US 20160019249A1
• Filed: 09/04/2014
• Published: 01/21/2016
• Est. Priority Date: 07/18/2014
• Status: Active Grant

• Current Assignee: Wipro Limited
• Original Assignee: Wipro Limited
• Inventors: Gunjan, Abhishek
• Granted Patent: US 9,465,831 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/2264 Multidimensional index structures
• CPC Class Codes: G06F 16/2282 Tablespace storage structures; Management thereof
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP

Abstract

This technology relates to method and optimization systems for optimizing storage of multi-dimensional data in data storage. The method comprises analyzing a plurality of queries received over period of time from one or more client machines. Then, a query pattern is determined from plurality of queries. Based on query pattern dimensionality of data is identified for determining data storage strategy. The dimensionality is characterized into 11 dimensions comprising 4 standard level dimensions and 7 higher level dimensions. A highest dimension out of 7 higher dimensions is parallel data storage dimension. Based on storage strategy, at least one of data and columns of a table is segmented in data storage. Next, data is stored in remote data storage when data is an element of last higher level dimension. Then, higher level dimensions are fragmented into one or more smaller level dimensions when data is element greater than 11 dimensions.

Claims

1. A method for optimizing storage of multidimensional data, the method comprising:

analyzing, by an optimization system, a plurality of queries received over a period of time from one or more client machines;

determining, by the optimization system, a query pattern from the plurality of queries;

identifying, by the optimization system, dimensionality of data based on the query pattern and determining a data storage strategy based at least in part on the dimensionality, wherein the dimensionality is characterized into 11 dimensions comprising 4 standard level dimensions and 7 higher level dimensions; and

performing, based on the determined data storage strategy and by the optimization system, at least one of;

segmenting at least one of the data or columns of a table stored in a data storage based on the identified dimensionality;

storing the data in a remote data storage when the data is an element of a last one of the higher level dimensions;

or

fragmenting the higher level dimensions into one or more lower level dimensions when the data is an element greater than the 11 dimensions.

2. The method as claimed in claim 1, wherein one or more of the plurality of queries is a Structured Query Language (SQL) based query or a Not Only Structured Query Language (NoSQL) based query.

3. The method as claimed in claim 1, wherein the query pattern is further determined based on parameters comprising type of the queries, location from where the queries are initiated, time at which the queries are received, frequency of the queries, correlation between columns of the queries, correlation between constraints of the queries, volume of data retrieved on each of the queries, frequency of data usage, complexity of join queries, keys used in the queries, or dataset retrieved from a demography of the queries.

4. The method as claimed in claim 1 further comprising storing, by the optimization system, metadata of the data, the table, and the dimensionality of the data in a master table.

5. The method as claimed in claim 1 further comprising providing, by the optimization system, one or more alerts based on at least one of a failure of the optimization system and the at least one secondary optimization system, a syntax error in defining one or more of the queries, an unavailability of space for storing the data, an unavailability of data for one or more of the queries, or an increase of query volume more than a predefined volume.

6. The method as claimed in claim 1 further comprising archiving, by the optimization system, the data based on a lifespan of the data, wherein the lifespan of the data is determined from the query pattern.

7. An optimization system, comprising:

a processor; and

a memory communicatively coupled to the processor, wherein the memory stores processor-executable instructions which, on execution by the processor, cause the processor to;

analyzing a plurality of queries received over a period of time from one or more client machines;

determining a query pattern from the plurality of queries;

identifying dimensionality of data based on the query pattern and determining a data storage strategy based at least in part on the dimensionality, wherein the dimensionality is characterized into 11 dimensions comprising 4 standard level dimensions and 7 higher level dimensions; and

performing, based on the determined data storage strategy, at least one of;

segmenting at least one of the data or columns of a table stored in a data storage based on the identified dimensionality;

storing the data in a remote data storage when the data is an element of a last one of the higher level dimensions;

or

fragmenting the higher level dimensions into one or more lower level dimensions when the data is an element greater than the 11 dimensions.

8. The optimization system as claimed in claim 7, wherein one or more of the plurality of queries is a Structured Query Language (SQL) based query or a Not Only Structured Query Language (NoSQL) based query.

9. The optimization system as claimed in claim 7, wherein the query pattern is further determined based on parameters comprising type of the queries, location from where the queries are initiated, time at which the queries are received, frequency of the queries, correlation between columns of the queries, correlation between constraints of the queries, volume of data retrieved on each of the queries, frequency of data usage, complexity of join queries, keys used in the queries, or dataset retrieved from a demography of the queries.

10. The optimization system as claimed in claim 7, wherein the memory further stores processor-executable instructions which, on execution by the processor, cause the processor to store metadata of the data, the table, and the dimensionality of the data in a master table.

11. The optimization system as claimed in claim 7, wherein the memory further stores processor-executable instructions which, on execution by the processor, cause the processor to provide one or more alerts based on at least one of a failure of the optimization system and the at least one secondary optimization system, a syntax error in defining one or more of the queries, an unavailability of space for storing the data, an unavailability of data for one or more of the queries, or an increase of query volume more than a predefined volume.

12. The optimization system as claimed in claim 7, wherein the memory further stores processor-executable instructions which, on execution by the processor, cause the processor to archive the data based on a lifespan of the data, wherein the lifespan of the data is determined from the query pattern.

13. A non-transitory computer readable medium having stored thereon instructions for optimizing storage of multidimensional data comprising executable code which when executed by a processor, cause the processor to perform steps comprising:

analyzing a plurality of queries received over a period of time from one or more client machines;

determining a query pattern from the plurality of queries;

identifying dimensionality of data based on the query pattern and determining a data storage strategy based at least in part on the dimensionality, wherein the dimensionality is characterized into 11 dimensions comprising 4 standard level dimensions and 7 higher level dimensions; and

performing, based on the determined data storage strategy, at least one of;

segmenting at least one of the data or columns of a table stored in a data storage based on the identified dimensionality;

storing the data in a remote data storage when the data is an element of a last one of the higher level dimensions;

or

fragmenting the higher level dimensions into one or more lower level dimensions when the data is an element greater than the 11 dimensions.

14. The medium as claimed in claim 13, wherein one or more of the plurality of queries is a Structured Query Language (SQL) based query or a Not Only Structured Query Language (NoSQL) based query.

15. The medium as claimed in claim 13, wherein the query pattern is further determined based on parameters comprising type of the queries, location from where the queries are initiated, time at which the queries are received, frequency of the queries, correlation between columns of the queries, correlation between constraints of the queries, volume of data retrieved on each of the queries, frequency of data usage, complexity of join queries, keys used in the queries, or dataset retrieved from a demography of the queries.

16. The medium as claimed in claim 13, further having stored thereon instructions further comprising executable code which when executed by the processor, cause the processor to perform steps further comprising storing metadata of the data, the table, and the dimensionality of the data in a master table.

17. The medium as claimed in claim 13, further having stored thereon instructions further comprising executable code which when executed by the processor, cause the processor to perform steps further comprising providing one or more alerts based on at least one of a failure of the optimization system and the at least one secondary optimization system, a syntax error in defining one or more of the queries, an unavailability of space for storing the data, an unavailability of data for one or more of the queries, or an increase of query volume more than a predefined volume.

18. The medium as claimed in claim 13, further having stored thereon instructions further comprising executable code which when executed by the processor, cause the processor to perform steps further comprising archiving the data based on a lifespan of the data, wherein the lifespan of the data is determined from the query pattern.