Internet database system

• US 6,523,036 B1
• Filed: 08/01/2000
• Issued: 02/18/2003
• Est. Priority Date: 08/01/2000
• Status: Expired due to Term

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Dantz Development Corporation (StorCentric, Inc.)
• Inventors: Tanner, Jeffrey C., Lau, Patrick Wing Sang, Friedman, Marc T., Mullally, Joseph P., Hickman, Robert C., Bobbitt, Jared E.
• Primary Examiner(s): CHOULES, JACK M
• Application Number:
  US09/629,652
• Time in Patent Office: 931 Days
• Field of Search: 707/1, 707/10, 707/201, 709/214
• US Class Current: 707/704
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: Y10S 707/955 Object-oriented
• CPC Class Codes: Y10S 707/966 Distributed

Abstract

An incrementally-scalable database system and method. The system architecture enables database servers to be scaled by adding resources, such as additional servers, without requiring that the system be taken offline. Such scaling includes both adding one or more computer servers to a given server cluster, which enables an increase in database read transaction throughput, and adding one or more server clusters to the system configuration, which provides for increased read and write transaction throughput. The system also provides for load balancing read transactions across each server cluster, and load balancing write transactions across a plurality of server clusters. The system architecture includes an application server layer including one or more computers on which an application program(s) is running, a database server layer comprising two or more server clusters that each include two or more computer servers with replicated data, and an intermediate “virtual transaction” layer that includes at least two computers that facilitate database transactions with one or more databases operating in the database server layer. Data in the database(s) are evenly distributed across the server clusters in fragmented mutually exclusive subsets of data based on a hashing function. An application program interface is provided so as to enable application programs to perform a full range of database transactions without regard for where data is stored, or what database(s) is operating in the database server layer.

Claims

1. A method for incrementally scaling a database server system on which a database is served, comprising

storing a plurality of database instances on a first server cluster comprising at least two computer servers, each having a duplicate copy of said database instance, said database instance comprising a database server application and data corresponding to the database, including metadata that defines a structure of the database and table data upon which database transactions are performed;

adding a second server cluster comprising at least two computer servers, each computer having installed thereon an instance of the database server application and the metadata;

migrating a portion of the table data from the first server cluster to the second server cluster by moving the portion of table data from each of said at least two computer servers in the first server cluster to said at least two computer servers in the second cluster while enabling database transactions on other portions of the table data remaining on the first server cluster, each of said at least two computer servers in the second cluster containing identical table data after the portion of the table data is migrated; and

enabling database transactions across both the first and second clusters after the portion of table data has been migrated to said at least two computer servers in the second server cluster.

2. The method of claim 1, further comprising providing read transaction load balancing across each of the first and second server clusters such that read transactions corresponding to table data stored on a given server cluster are evenly distributed among the computer servers in that server cluster.

3. The method of claim 1, further comprising providing write transaction load balancing across the first and second server clusters after the portion of table data has been migrated such that write transactions are evenly distributed among the first and second server clusters.

4. The method of claim 1, further comprising:

adding a new computer server to one of the server clusters, said new computer server having installed thereon an instance of the database server application;

copying metadata and table data from at least one other computer server in the server cluster to which the new computer server is added so as to replicate the data from the other computer servers in the server cluster while enabling database transactions on the server cluster; and

distributing future database transactions across all computer servers, including the new computer server, in the server cluster to which the new computer server is added.

5. The method of claim 1, wherein the table data comprise a plurality of record objects, further comprising:

enabling a write transaction to add, modify or delete a record object on one of said first or second server clusters; and

processing the write transaction in substantial synchrony across all computer servers in the server cluster in which the data object is stored.

6. The method of claim 1, wherein the table data comprises a plurality of record objects, further comprising partitioning the table data by:

assigning each record object to a fragment; and

distributing record objects across each of the first, and second clusters based on the fragment each record object is assigned to.

7. The method of claim 6, wherein the record objects are assigned to fragments based on a hash function that operates on each record object.

8. The method of claim 6, further comprising providing fragment mapping data that maps each fragment to the server cluster on which its records are stored.

9. The method of claim 6, wherein migrating the portion of the table data from the first server cluster to the second server cluster comprises migrating at least one fragment of table data at a time while enabling database transactions on all other fragments of table data in the first server cluster.

10. The method of claim 9, wherein migrating the fragments of table data further comprises:

selecting a fragment of table data to be migrated;

writelocking the fragment of table data to prevent write transactions from being performed on record objects in the fragment of table data;

shipping the fragment of table data to each computer server in the second server cluster;

updating fragment mapping data to indicate the fragment of table data has been moved to the second cluster; and

releasing locks on the fragment of table data so as to re-enable write transactions to be performed on record objects in the fragment of table data.

11. The method of claim 10, further comprising:

rejecting transaction requests to access record objects corresponding to the fragment of table data while it is being shipped to the second server cluster; and

providing a message in accord with a rejected transaction request identifying that the fragment of table data now resides on the second server cluster.

12. The method of claim 1, wherein the database server system has an initial configuration comprising the first server cluster, further comprising:

determining if the portion of table data is successfully migrated to the second server cluster; and

restoring the database server system back to its initial configuration so as to enable database transactions on only the first server cluster if it is determined that the data was not successfully migrated to the second server cluster.

13. A method for incrementally scaling a database server system on which a database comprising table data is served, comprising

storing a first partition of table data on a first server cluster comprising at least two computer servers, each having a duplicate copy of said first partition of table data;

storing a second partition of table data on a second server cluster comprising at least two computer servers, each having a duplicate copy of said second partition of table data;

adding a third server cluster comprising at least two computer servers;

migrating portion(s) of table data stored on the first and/or second server clusters to the third server cluster while enabling database transactions on other portions of table data remaining on the first and second server clusters; and

enabling database transactions across all of the first, second and third server clusters after the portions of table data have been migrated to the third server cluster.

14. The method of claim 13, further comprising providing read transaction load balancing across each server cluster such that read transactions are evenly distributed among the computer servers in that server cluster.

15. The method of claim 13, further comprising providing write transaction load balancing across the server clusters in the system such that write transactions are evenly distributed among the server clusters.

16. The method of claim 13, further comprising:

adding a new computer server to one of the server clusters;

copying metadata and table data from another computer server in the server cluster to which the new computer server is added so as to replicate the table data on said another computer while enabling database transactions on the server cluster; and

distributing future database transactions across all computer servers, including the new computer server, in the server cluster to which the new computer server is added.

17. The method of claim 13, wherein the table data comprise a plurality of record objects, further comprising:

enabling a write transaction to add, modify or delete a record object on one of said first, second, or third server clusters; and

processing the write transaction in substantial synchrony across all computer servers in the server cluster in which the data object is stored.

18. The method of claim 13, wherein migrating the portion(s) of data to the third server cluster comprises copying fragments of data to all of said at least two computer servers in the third server cluster.

19. The method of claim 13, wherein the data comprises a plurality of record objects, further comprising partitioning the table data in the database by:

assigning each record object to a fragment; and

distributing record objects across each of the first, second, and third clusters based on the fragment each record object is assigned to.

20. The method of claim 19, wherein the record objects are assigned to fragments based on a hash function that operates on each record object.

21. The method of claim 19, further comprising providing fragment mapping data that maps each fragment to the server cluster on which its records are stored.

22. The method of claim 19, wherein migrating the fragments of data to the third server cluster comprises creating any database tables and any associated indices and stored procedures relating to any fragments of data that are to be migrated on each computer server in the third cluster.

23. The method of claim 19, wherein migrating the portion(s) of table data to the third server cluster further comprises:

selecting a fragment of table data to be migrated;

writelocking the fragment of table data to prevent write transactions from being performed on record objects in the fragment of table data;

shipping the fragment of table data to each computer server in the third server cluster;

updating fragment mapping data to indicate the fragment of table data has been moved to the third cluster; and

releasing locks on the fragment of table data so as to re-enable write transactions to be performed on record objects in the fragment of table data.

24. The method of claim 13, wherein the portion(s) of table data that are migrated originally resides on a source server cluster and shipping the portion(s) of data to each computer server in the third server cluster comprises:

copying the portion(s) of table data from a computer server in the source server cluster to each computer server in the third server cluster; and

deleting the portion(s) of table data from all of the computer servers in the source server cluster.

25. The method of claim 23, wherein a plurality of fragments of table data are concurrently migrated to the computer servers in the third server cluster.

26. The method of claim 23, further comprising:

rejecting transaction requests to access record objects corresponding to the fragment of table data while it is being shipped to the third server cluster; and

providing a message in accord with a rejected transaction request identifying that the fragment of table data now resides on the third server cluster.

27. The method of claim 13, wherein the database server system has an initial configuration comprising the first and second server clusters, further comprising:

determining if the portion(s) of table data is successfully migrated to the third server cluster; and

restoring the database server system back to its initial configuration so as to enable database transactions on only the first and second server clusters if it is determined that the portion(s) of table data was not successfully migrated to the third server cluster.

28. The method of claim 13, further comprising providing an application program interface that enables an application program to perform transactions on record objects in the database without requiring the application program to consider where those record objects are stored on the database server system.

29. The method of claim 13, wherein the database comprises a database management system (DBMS) comprising a native interface language, further comprising providing an application program interface that enables an application program to interact with the database without directly using the native interface language of the DBMS.

30. A scalable database server system comprising:

(a) an application server layer comprising at least one application server computer on which an application program is running;

(b) a scalable database server layer including at least two server clusters on which a database comprising a plurality of record objects is served, each server cluster including at least one computer server, said plurality of record objects being distributed across said at least two server clusters based on an adaptive partitioning scheme;

(c) an intermediate layer disposed between and in communication with the application server layer and the database server layer comprising of one or more computers, each computer having a plurality of instructions executing thereon that enable the application program to perform write transactions on selected record objects stored on said at least two server clusters, whereby the write transactions are load balanced across said at least two server clusters.

31. The system of claim 30, wherein said database comprises an database management system (DBMS) having a native interface language, further including an application program interface (API) component comprising a plurality of executable instructions running on said at least one application server computer, said API component enabling the application program to interact with the database without directly using the native interface language of the DBMS.

32. The system of claim 31, wherein said plurality of record objects are partitioned into a plurality of fragments, each record object being assigned to a fragment, and further wherein the API component includes fragment mapping data that maps each fragment to the server cluster on which its records are stored.

33. The system of claim 30, further comprising a configuration management component that provides information concerning the configuration of the database server layer comprising a plurality of instructions executing on at least one of said at least two computers in the intermediate layer.

34. The system of claim 33, wherein said plurality of record objects are partitioned into a plurality of fragments, each record object being assigned to a fragment, and further wherein the configuration management component maintains fragment mapping data that maps each fragment to the server cluster on which its records are stored.

35. The system of claim 30, wherein the database data are partitioned into a plurality of fragments, whereby each record object is assigned to a fragment, and further wherein fragment mapping data is stored on each of said at least two server clusters such that each server cluster knows what record objects are stored on it.

36. The system of claim 30, wherein each server cluster comprises at least two computer servers and data is replicated across all of the computer servers in a given server cluster such that each computer server in the server cluster has an identical set of record objects stored thereon.

37. The system of claim 36, further including a load balancing module executing on said at least one application server computer that evenly distributes read transactions across said at least two computer servers in each server cluster.

38. The system of claim 30, wherein each computer server in each server cluster is connected in communication with a respective storage device on which data and computer executable instructions are stored in a shared-nothing configuration.

39. A scalable database server system comprising:

(a) a scalable database server layer including at least one server cluster on which a database comprising a plurality of record objects is served, each of said at least one server cluster including at least two computer servers, each computer server in a given server cluster having a duplicative copy of at least a portion of said plurality of record objects stored thereon;

(b) an intermediate layer in communication with the database server layer comprising at least one computer having a plurality of instructions executing thereon that manages a distribution of said plurality of record objects across said at least one server cluster; and

(c) an application server layer in communication with the intermediate layer and the database server layer, comprising at least one application server computer on which an application program is running and including an application program interface (API) component comprising a plurality of executable instructions running on said at least one application server computer that enables interactions between the application program and the database, said API component providing load balancing of database read transactions across said at least two computer servers in each of said at least one server cluster.

40. The system of claim 39, wherein said database comprises an database management system (DBMS) having a native interface language, further wherein the API component enables the application program to interact with the database without directly using the native interface language of the DBMS.

41. The system of claim 39, further comprising a configuration management component that provides information concerning the configuration of the database server layer comprising a plurality of instructions executing on at least one of said at least one computers in the intermediate layer.

42. The system of claim 39, wherein each computer server in each server cluster is connected in communication with a respective storage device on which data and computer executable instructions are stored in a shared-nothing configuration.

43. A scalable datastore system comprising:

(a) an application server layer comprising at least one application server computer on which an application program is running;

(b) a scalable database server layer comprising at least two server clusters on which a database including a plurality of record objects are served, said plurality of record objects being distributed across said at least two server clusters, each server cluster including at least two computer servers having stored thereon identical sets of record objects;

(c) an intermediate layer disposed between and in communication with the application server layer and the database server layer comprising at least two computers, at least one of said at least two computers having a configuration management component comprising a plurality of instructions executing thereon that provides information concerning the configuration of the database server layer including what server cluster each record object is stored on.

44. The system of claim 43, wherein said database comprises an database management system (DBMS) having a native interface language, further including an application program interface (API) component comprising a plurality of executable instructions running on said at least one application server computer, said API component enabling the application program to interact with the database without directly using the native interface language of the DBMS.

45. The system of claim 44, wherein said plurality of record objects are partitioned into a plurality of fragments, each record object being assigned to a fragment, and further wherein the API component includes fragment mapping data that maps each fragment to the server cluster on which its records are stored.

46. The system of claim 43, wherein said plurality of record objects are partitioned into a plurality of fragments, each record object being assigned to a fragment, and further wherein the configuration management component maintains fragment mapping data that maps each fragment to the server cluster on which its records are stored.

47. The system of claim 43, wherein the database data are partitioned into a plurality of fragments, whereby each record object is assigned to a fragment, and further wherein fragment mapping data is stored on each of said at least two server clusters such that each server cluster knows which fragments are assigned to the server cluster and which fragments are not.

48. The system of claim 43, further including a load balancing module executing on said at least one application server computer that evenly distributes read transactions across said at least two computer servers in each server cluster.

49. The system of claim 43, wherein each computer server in each server cluster is connected in communication with a respective storage device on which data and computer executable instructions are stored in a shared-nothing configuration.

50. The system of claim 43, wherein at least two of said at least two computers in the intermediate layer have stored thereon executable instructions comprising the configuration management component and a first instance of the configuration management component is executing on a first computer in the intermediate layer, and further wherein if execution of the configuration management component on the first computer is disrupted, execution of a second instance of the configuration management component automatically begins on another computer in the intermediate layer.