System and method for analysis and management of data distribution in a distributed database environment

• US 10,303,702 B2
• Filed: 02/06/2015
• Issued: 05/28/2019
• Est. Priority Date: 02/07/2014
• Status: Active Grant

• Current Assignee: Ignite ScaleArc Solutions, Inc.
• Original Assignee: Ignite ScaleArc Solutions, Inc.
• Inventors: Levari, Doron, Dvorak, Petr, Vexler, Vladimir
• Primary Examiner(s): Tran, Anhtai V
• Application Number:
  US14/615,903
• Publication Number:
  US 20150227521A1
• Time in Patent Office: 1,572 Days
• Field of Search: None
• CPC Class Codes: G06F 16/278 Data partitioning, e.g. horizontal or vertical partitioning

Abstract

A system and method for analysis and management of data distribution in a distributed database environment. A method may include analyzing information related to transactions between an application and a plurality of data storage systems to produce analysis results. Analysis results may be used to produce a set of rules for defining logical data chunks. A logical data chunk may include at least a portion a data set stored in the plurality of data storage systems. A data distribution policy may be defined based on the set of rules. Other embodiments are described and claimed.

Claims

1. A method comprising:

collecting statements from an application directed to a database located in at least one data server in a plurality of data servers, wherein the statements collected include a command stream;

producing analysis results by analyzing information including the collected statements, wherein producing the analysis results includes;

identifying in the command stream of the collected statements patterns of command that appear together; and

identifying related tables in the database from the identified patterns of command; and

using the analysis results to produce a data distribution policy having one or more rules for defining logical data chunks of the database that include at least the related tables and for distributing and managing the logical data chunks for storage among the plurality of data servers in a horizontally partitioned database.

2. The method of claim 1 wherein producing analysis results further comprises analyzing metadata related to data sets stored in the plurality of data servers and accessed by the application.

3. The method of claim 1, further comprising defining and storing a plurality of logical data chunks according to the data distribution policy, wherein no logical data chunk of the plurality of logical data chunks is distributed between more than one data server in the plurality of data servers.

4. The method of claim 1 further comprising defining and storing a plurality of logical data chunks according to the data distribution policy, wherein the data of one of the logical data chunks of the plurality of logical data chunks are stored entirely on a single one of the data servers of the plurality of data servers.

5. The method of claim 1, further comprising defining and storing a plurality of logical data chunks according to the data distribution policy, wherein the data of a first one of the logical data chunks of the plurality of logical data chunks are stored entirely on a first one of the data servers of the plurality of data servers, and the data of a second one of the logical data chunks of the plurality of logical data chunks are stored entirely on a second one of the data servers of the plurality of data servers.

6. The method of claim 1 further comprising defining a particular logical data chunk according to the data distribution policy, wherein the particular logical data chunk includes portions of related data in a plurality of data sets.

7. The method of claim 1 further comprising defining a particular logical data chunk according to the data distribution policy, wherein the data of the particular logical data chunk are stored in a single shard.

8. The method of claim 1, further comprising calculating a first efficiency score for the data distribution policy and presenting the first efficiency score and policy to a user.

9. The method of claim 1 wherein the information analyzed to produce analysis results further comprises at least one of:

activity per data set, join operations, database schema, structures of data sets, and size of data sets.

10. The method of claim 1 wherein the step of collecting statements is performed by one or more of:

a collection unit, a network sniffer, a log file, and a database proxy.

11. The method of claim 1 further comprising selectively replacing and obfuscating entries in the collected statements to produce normalized aggregated statements, wherein the information analyzed to produce analysis results includes the normalized aggregated statements.

12. The method of claim 1 further comprising defining a particular logical data chunk according to the data distribution policy, wherein the particular logical data chunk includes all portions of data sets accessed in a set or sequence of statements.

13. The method of claim 1, further comprising graphically presenting a distribution and usage of data sets to a user and defining a logical data chunk based on input from the user.

14. The method of claim 1, further comprising defining a particular logical data chunk according to the data distribution policy, and including at least portions of a first data set and at least portions of a second data set in the particular logical data chunk upon determining the first data set is linked to the second data set based on at least one of:

a foreign key in the first data set, name matching between the first and second data sets, both the first and second data set being referred to in a request, both the first and second data set being referred to in a flow of statements, and data from the first and second data sets being joined in a join operation.

15. The method of claim 1, further comprising graphically presenting to a user, for a data set, at least one of:

data access patterns related to the data set, join operations related to the data set, transactions referencing the data set, and size of the data set and data growth pattern of the data set.

16. The method of claim 1, further comprising graphically presenting to a user, for the data distribution policy, efficiency metrics related to at least one of:

data access to a selected shard, cross-shard data access, global table reads, global table writes, a specific statement, total statements and blocked statements.

17. The method of claim 8 further comprising calculating a second efficiency score for a second data distribution policy and presenting the second efficiency score to the user.

18. An article comprising a computer-readable storage medium, having stored in a non-transitory manner thereon instructions that, when executed by a controller, cause the controller to:

collect statements from an application directed to a database located in at least one data server in a plurality of data servers, wherein the statements collected include a command stream;

produce analysis results by analyzing information including the collected statements, wherein to produce the analysis results includes;

to identify in the command stream of the collected statements patterns of command that appear together; and

to identify related tables in the database from the identified patterns of command; and

use the analysis results to produce a data distribution policy having one or more rules for defining logical data chunks of the database that include at least the related tables and for distributing and managing the logical data chunks for storage among the plurality of data servers in a horizontally partitioned database.

19. The article of claim 18 wherein producing analysis results further comprises analyzing metadata related to data sets stored in the plurality of data servers and accessed by the application.

20. The article of claim 18, wherein the instructions further cause the controller to define and store a plurality of logical data chunks according to the data distribution policy, wherein no logical data chunk of the plurality of logical data chunks is distributed between more than one data server in the plurality of data servers.

21. The article of claim 18 wherein the instructions further cause the controller to define and store a plurality of logical data chunks according to the data distribution policy, wherein the data of one of the logical data chunks of the plurality of logical data chunks are stored entirely on a single one of the data servers of the plurality of data servers.

22. The article of claim 18, wherein the instructions further cause the controller to define and store a plurality of logical data chunks according to the data distribution policy, wherein the data of a first one of the logical data chunks of the plurality of logical data chunks are stored entirely on a first one of the data servers of the plurality of data servers, and the data of a second one of the logical data chunks of the plurality of logical data chunks are stored entirely on a second one of the data servers of the plurality of data servers.

23. The article of claim 18 wherein the instructions further cause the controller to define a particular logical data chunk according to the data distribution policy, wherein the particular logical data chunk includes portions of related data in a plurality of data sets.

24. The article of claim 18 wherein the instructions further cause the controller to define a particular logical data chunk according to the data distribution policy, wherein the data of the particular logical data chunk are stored in a single shard.

25. The article of claim 18, wherein the instructions further cause the controller to calculate a first efficiency score for the data distribution policy and present the first efficiency score and policy to a user.

26. The article of claim 18 wherein the information analyzed to produce analysis results further comprises at least one of:

activity per data set, join operations, database schema, structures of data sets, and size of data sets.

27. The article of claim 18 wherein the step of collecting statements is performed by one or more of:

a collection unit, a network sniffer, a log file, and a database proxy.

28. The article of claim 18 wherein the instructions further cause the controller to selectively replace and obfuscate entries in the collected statements to produce normalized aggregated statements, wherein the information analyzed to produce analysis results includes the normalized aggregated statements.

29. The article of claim 18 wherein the instructions further cause the controller to define a particular logical data chunk according to the data distribution policy, wherein the particular logical data chunk includes all portions of data sets accessed in a set or sequence of statements.

30. The article of claim 18, wherein the instructions further cause the controller to graphically present a distribution and usage of data sets to a user and define a logical data chunk based on input from the user.

31. The article of claim 18, wherein the instructions further cause the controller to define a particular logical data chunk according to the data distribution policy, and include at least portions of a first data set and at least portions of a second data set in the particular logical data chunk upon determining the first data set is linked to the second data set based on at least one of:

a foreign key in the first data set, name matching between the first and second data sets, both the first and second data set being referred to in a request, both the first and second data set being referred to in a flow of statements, and data from the first and second data sets being joined in a join operation.

32. The article of claim 18, wherein the instructions further cause the controller to graphically present to a user, for a data set, at least one of:

data access patterns related to the data set, join operations related to the data set, transactions referencing the data set, and size of the data set and data growth pattern of the data set.

33. The article of claim 18, wherein the instructions further cause the controller to graphically present to a user, for the data distribution policy, efficiency metrics related to at least one of:

data access to a selected shard, cross-shard data access, global table reads, global table writes, a specific statement, total statements and blocked statements.

34. The article of claim 25 wherein the instructions further cause the controller to calculate a second efficiency score for a second data distribution policy and present the second efficiency score to the user.

35. A system for analysis and management of data distribution in a distributed database environment, comprising:

a processor;

a first agent running on the processor for collecting statements between an application and a database located in at least one data server in a plurality of data servers, wherein the statements collected include a command stream; and

a second agent running on the processor for;

analysis results by analyzing information including the collected statements, wherein

producing the analysis results includes;

identifying in the command stream of the collected statements patterns of command that appear together; and

identifying related tables in the database from the identified patterns of command; and

using the analysis results to produce a data distribution policy having one or more rules for defining logical data chunks of the database that include at least the related tables and for distributing and managing the logical data chunks for storage among the plurality of data servers in a horizontally partitioned database.