Distributed system for application processing

• US 9,619,295 B1
• Filed: 05/21/2009
• Issued: 04/11/2017
• Est. Priority Date: 05/21/2008
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Lyle, Michael P., Ross, Robert F., Geiger, Robert L., Vekhter, Daniel
• Primary Examiner(s): Strange, Aaron
• Application Number:
  US12/454,840
• Time in Patent Office: 2,882 Days
• Field of Search: 709226
• CPC Class Codes: G06F 16/162 Delete operations erasing in storage systems G06F3/0652
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/2282 Tablespace storage structures; Management thereof
• CPC Class Codes: G06F 9/5083 Techniques for rebalancing the load in a distributed system
• CPC Class Codes: H04L 41/30 Decision processes by autonomous network management units using voting and bidding
• CPC Class Codes: H04L 67/1001 for accessing one among a plurality of replicated servers
• CPC Class Codes: H04L 67/1008 based on parameters of servers, e.g. available memory or workload monitoring of computer activity G06F11/30
• CPC Class Codes: H04L 67/1034 Reaction to server failures by a load balancer
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A system that includes multiple nodes is disclosed. A first node in the system has a first local storage and a first portion of a distributed index. A second node in the system has a second local storage and a second portion of the distributed index. A third node in the system has a third local storage and a third portion of the distributed index. Each of the first, second, and third nodes includes application logic. If any one of the nodes is removed from the system, neither application nor data availability is impacted.

Claims

1. A system, comprising:

a first node having a first local storage and a first portion of a distributed index;

a second node having a second local storage and a second portion of the distributed index;

a third node having a third local storage and a third portion of the distributed index; and

wherein each of the first, second, and third nodes includes application logic;

wherein in an event that any one of the first, second, and third nodes is removed from the system, one or more nodes that remain available are configured to cooperate to ensure that the one or more nodes that remain available are configured to perform an application workload associated with the application logic of the removed node in a manner that is transparent to an end user with which the application workload is associated and to store a distributed index portion stored previously by the removed node;

wherein the first, second, and third nodes comprise participants associated with a commitment process;

a processor configured to;

receive a request for journal synchronization; and

perform the commitment process, wherein to perform the commitment process includes to;

collect votes from the first, second, and third nodes; and

determine whether the commitment process is successful based at least in part on the votes;

wherein in an event that the commitment process is successful, remove at least a portion of historical vote data; and

wherein in an event that the commitment process is not successful, do not remove the portion of historical vote data.

2. The system of claim 1, wherein the first, second, and third local storage each represent a location at which a new piece of data can be stored and wherein the location at which the new piece of data is stored is determined based at least in part on a factor.

3. The system of claim 2, wherein the factor includes at least one of the following:

node availability, administrative policy, or redundancy requirements.

4. The system of claim 1, wherein data stored in the first node is associated with at least one data attribute.

5. The system of claim 4, wherein the at least one data attribute is correlated with a historical usage pattern.

6. The system of claim 4, wherein the at least one data attribute is correlated with an expected usage pattern.

7. The system of claim 1, wherein data is dynamically reassigned from the first node to the second node based on changes in a factor.

8. The system of claim 1, wherein each of the first, second, and third nodes is configured to include a load balancing module.

9. The system of claim 8, wherein the load balancing module is configured to use a Domain Name System (DNS).

10. The system of claim 8, wherein the load balancing module is configured to use hypertext transfer protocol (HTTP) redirect.

11. The system of claim 1, wherein each of the first, second, and third nodes is configured to include an atomic commitment module.

12. The system of claim 11, wherein the atomic commitment module is configured to use a Paxos protocol.

13. The system of claim 1, wherein the processor is further configured to determine the portion of historical vote data to remove based at least in part on a specified revision identifier associated with an instance of the commitment process.

14. A method, comprising:

storing at a first system a first portion of a global index, wherein the first portion is included in a plurality of portions of the global index, and wherein the first portion is stored by at least one other of a plurality of systems, wherein each of the plurality of systems includes application logic;

determining that in an event that one of the plurality of systems is removed, one or more of the plurality of systems that remain available cooperate to ensure that the one or more of the plurality of systems that remain available perform an application workload associated with the application logic of the removed system in a manner that is transparent to an end user with which the application workload is associated and to store a global index portion stored previously by the removed system;

wherein the first system and at least a subset of the plurality of systems comprise participants associated with a commitment process;

receiving a request for journal synchronization; and

performing the commitment process, wherein performing the commitment process includes;

collecting votes from the first system and the subset of the plurality of systems; and

determining whether the commitment process is successful based at least in part on the votes;

wherein in an event that the commitment process is successful, removing at least a portion of historical vote data and

wherein in an event that the commitment process is not successful, not removing the portion of historical vote data.

15. The method of claim 14, further comprising:

determining that the first portion of the global index stored on the first system exceeds a fair share of the global index; and

requesting another system to move a boundary of a second portion of the global index stored on the other system to decrease a size of the first portion of the global index.

16. A non-transitory computer readable medium storing computer-executable instructions for:

storing at a first system a first portion of a global index, wherein the first portion is included in a plurality of portions of the global index, and wherein the first portion is stored by at least one other of a plurality of systems, wherein each of the plurality of systems includes application logic;

determining that in an event that one of the plurality of systems is removed, one or more of the plurality of systems that remain available cooperate to ensure that the one or more of the plurality of systems that remain available perform an application workload associated with the application logic of the removed system in a manner that is transparent to an end user with which the application workload is associated and to store a global index portion stored previously by the removed system;

wherein the first system and at least a subset of the plurality of systems comprise participants associated with a commitment process;

receiving a request for journal synchronization; and

performing the commitment process, wherein the computer-executable instructions for performing the commitment process includes computer-executable instructions for;

collecting votes from the first system and the subset of the plurality of systems; and

determining whether the commitment process is successful based at least in part on the votes;

wherein in an event that the commitment process is successful, removing at least a portion of historical vote data; and

wherein in an event that the commitment process is not successful, not removing the portion of historical vote data.

17. The non-transitory computer readable medium of claim 16, further storing computer-executable instructions for:

determining that the first portion of the global index stored on the first system exceeds a fair share of the global index; and

requesting another system to move a boundary of a second portion of the global index stored on the other system to decrease a size of the first portion of the global index.

18. An apparatus comprising:

means for storing at a first system a first portion of a global index, wherein the first portion is included in a plurality of portions of the global index, and wherein the first portion is stored by at least one other of a plurality of systems, wherein each of the plurality of systems includes application logic;

means for determining that in an event that one of the plurality of systems is removed, one or more of the plurality of systems that remain available cooperate to ensure that the one or more of the plurality of systems that remain available perform an application workload associated with the application logic of the removed system in a manner that is transparent to an end user with which the application workload is associated and to store a global index portion stored previously by the removed system;

wherein the first system and at least a subset of the plurality of systems comprise participants associated with a commitment process;

means for receiving a request for journal synchronization; and

means for performing the commitment process including;

means for collecting votes from the first system and the subset of the plurality of systems; and

means for determining whether the commitment process is successful based at least in part on the votes,

wherein the means for performing the commitment process is configured to remove at least a portion of historical vote data if the commitment process is successful and to not remove the portion of historical vote data if the commitment process is not successful.

19. The apparatus of claim 18, further comprising:

means for determining that the first portion of the global index stored on the first system exceeds a fair share of the global index; and

means for requesting another system to move a boundary of a second portion of the global index stored on the other system to decrease a size of the first portion of the global index.