Implementing failover processes between storage stamps

• US 8,751,863 B2
• Filed: 05/23/2011
• Issued: 06/10/2014
• Est. Priority Date: 05/23/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Calder, Bradley Gene, Nilakantan, Niranjan, Srivastav, Shashwat, Wu, Jiesheng, Skjolsvold, Arild Einar, Mazeev, Maxim, Abbasi, Abdul Rafay, Mainali, Shane, Khatri, Hemal, Wang, Ju, Uddaraju, Padmanabha Chakravarthy, Rigas, Leonidas
• Primary Examiner(s): Iqbal, Nadeem
• Application Number:
  US13/113,795
• Publication Number:
  US 20120303999A1
• Time in Patent Office: 1,114 Days
• Field of Search: 714/6.1, 714/6.12, 714/6.13, 714/6.2, 714/6.23, 714/6.3, 714/6.31, 714/6.32
• US Class Current: 714/6.3
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 11/1662 the resynchronized component or unit being a persistent storage device re-synchronization of failed mirror storage G06F11/2082; rebuild or reconstruction of parity RAID storage G06F11/1008
• CPC Class Codes: G06F 11/2082 Data synchronisation
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups

Abstract

Embodiments of the present invention relate to invoking and managing a failover of a storage account between partitions within a distributed computing environment, where each partition represents a key range of data for the storage account. The partitions affected by the failover include source partitions hosted on a primary storage stamp and destination partitions hosted on a secondary storage stamp, where the storage account'"'"'s data is being actively replicated from the primary to the secondary storage stamp. Upon receiving a manual or automatic indication to perform the failover, configuring the source partitions to independently perform flush-send operations (e.g., distributing pending messages as a group) and then configuring the destination partitions to independently perform flush-replay operations (e.g., aggressively replaying currently pending transactions). Upon completing the flush-replay operations, designating the secondary storage stamp as a new primary storage stamp such that live traffic is directed to the new primary storage stamp.

Claims

1. A computer-implemented method in a distributed environment utilizing a processor and memory for invoking a clean failover of a storage account residing on storage stamps within the distributed computing environment, the method comprising:

providing a primary storage stamp that includes one or more source partitions configured for accepting live traffic and for sending messages, to one or more destination partitions in order to advance replication to a secondary storage stamp, wherein messages comprise transactions;

providing the secondary storage stamp that includes the one or more destination partitions configured to replay the transactions of the messages;

performing a failover between the primary and the secondary storage stamp for a subset of the data on the primary stamp, wherein the failover is an anticipated transition from the primary storage stamp to the secondary storage stamp;

requesting that the one or more source partitions attempt to independently carry out a flush-send operation, wherein the flush-send operation involves distributing pending messages at the primary storage stamp to the one or more destination partitions as a group, wherein pending messages comprise logged messages at primary storage stamp; and

reconfiguring the one or more destination partitions to independently carry out a flush-replay operation, wherein the flush-replay operation involves aggressively replaying the transactions of at least the pending messages received at the one or more destination partitions.

2. The method of claim 1, further comprising disabling the primary storage stamp from processing live traffic for the subset of data involved within the failover during the flush-send operation.

3. The method of claim 1, wherein, upon substantially completing the flush-replay operation, designating the secondary storage stamp as a new primary storage stamp.

4. The method of claim 1, wherein, upon substantially completing the flush-replay operation, designating the primary storage stamp as a new secondary storage stamp.

5. The method of claim 1, wherein the flush-replay operation commences on the one or more destination partitions incident to a substantial completion of the flush-send operation on the one or more destination partitions.

6. The method of claim 1, further comprising, upon an individual destination partition substantially completing the flush-send operation, allowing the individual destination partition to complete flush-replay and then commence accepting live traffic prior to others of the one or more destination partitions.

7. The method of claim 1, further comprising, reconfiguring the primary storage stamp to redirect live traffic from the client to the secondary storage stamp for the data involved in the failover.

8. The method of claim 7, further comprising, initiating an update to a table at a domain name server (DNS), wherein the DNS table is updated to reflect that the primary storage stamp is designated as the new secondary storage stamp and that the secondary storage stamp is designated as the new primary storage stamp.

9. The method of claim 8, further comprising maintaining the redirect reconfiguration of the primary storage stamp for a predefined retaining period to allow for propagation of the update throughout the DNS table.

10. The method of claim 1, further comprising providing a location service that is communicatively coupled to a first account control unit (ACU) running on the primary storage stamp and to a second ACU running on the secondary storage stamp.

11. The method of claim 10, wherein reconfiguring the one or more source partitions comprises receiving a message from the location service at the first ACU, wherein the message instructs the first ACU to update a first table of accounts residing on the primary storage stamp, wherein the first table of accounts governs whether the one or more source partitions are actively participating in a failover, a replication, or neither.

12. The method of claim 11, wherein reconfiguring the one or more destination partitions comprises receiving a message from the location service at the second ACU, wherein the message instructs the second ACU to update a second table of accounts residing on the secondary storage stamp, wherein the second table of accounts governs whether the one or more destinations partitions are actively participating in a failover, a replication, or neither.

13. The method of claim 1, wherein the primary and secondary storage stamps exist within a single geo-location such that the failover occurs internal to the geo-location.

14. The method of claim 1, wherein the primary and secondary storage stamps exist in geo-locations that are geographically remote with respect to one another, and wherein the failover results in designating a new primary storage stamp that is external to a geo-location of the primary storage stamp.

15. The method of claim 1, further comprising triggering the performance of the failover upon the storage system deciding to carry out a failover or upon receiving an indication from a client, wherein receiving an indication from the client involves exposing to the client a control that allows for manually triggering the failover.

16. The method of claim 1, exposing to the client a set of options that allow for modifying a policy that controls aspects of the failover, wherein the failover aspects that are controlled by the policy comprise an identity of a storage stamp to be a target of the failover.

17. The method of claim 16, wherein the failover aspects that are controlled by the policy comprise whether the failover is to be automatically triggered upon an occurrence of specified conditions, and wherein one of the specified conditions includes exposing a delay timer that allows customers to specify a maximum amount of time between experiencing a disaster and automatically performing the failover.

18. One or more computer storage device having computer-executable instructions embodied thereon, that when executed by a computing system having a processor and memory, cause the computing system to perform a method for implementing an abrupt failover of a client'"'"'s storage account from a primary storage stamp to a destination storage stamp, the method comprising:

experiencing a disaster at a first geo-location, wherein the primary storage stamp for the storage account exists within the first geo-location;

automatically triggering a failover from the primary storage stamp to the destination storage stamp in accordance with a predefined policy, wherein the destination storage stamp for the storage account exist in a second geo-location that is geographically remote from the first geo-location, and wherein the failover comprises;

(a) disconnecting the primary storage stamp from taking client requests;

(b) ensuring that an ongoing replication between the primary storage stamp and the destination storage stamp is severed such that the communication of transactions therebetween is interrupted; and

(c) replaying the transactions held in a geo message log (GML) on the destination storage stamp that were delivered prior to experiencing the disaster; and

bootstrapping the storage account from the destination storage stamp to a third storage stamp, wherein bootstrapping comprises;

generating a log of incoming changes to the destination storage stamp while replicating a copy of the storage account to the third storage stamp; and

updating the third storage stamp using the log of incoming messages.

19. The media of claim 18, wherein replaying the transactions held in the GML on the destination storage stamp that were delivered prior to experiencing the disaster comprises:

committing the transactions remaining in the GML after detecting a latest commitID provided as part of the failover;

replaying the remaining transactions held in the GML, thereby allowing data within the destination storage stamp to be strongly consistent against data within the primary storage stamp upon completion of the failover.

20. A computer system within a distributed networking environment for conducting a clean failover for a storage account, the system comprising:

a primary storage stamp that includes one or more source partitions that represent a key range of initial data associated with a storage account, wherein the one or more source partitions are configured for accepting live traffic and for sending transactions to one or more destination partitions for carrying out replication thereto;

a secondary storage stamp that includes the one or more destination partitions that represent a key range of replicated data associated with the storage account, wherein the one or more destination partitions are configured to replay the transactions such that the replicated data substantially minors content of the initial data; and

a location service for managing the storage account based on a domain name service (DNS) storage namespace mapping to the storage account, wherein the storage account specifies the primary storage stamp and the second storage stamp for the storage account, and wherein the location service is further configured for managing partitions independently of each other for invoking changes to the configuration of the one or more source partitions and for invoking changes to the configuration of the one or more destination partitions upon receiving an indication to failover from the primary storage stamp to the secondary storage stamp, wherein invoking changes to implement the failover comprises;

(a) instructing the one or more source partitions to refrain from accepting live traffic;

(b) instructing the one or more source partitions to independently perform a flush-send operation that attempts to flush the transactions remaining at the partitions to the secondary storage stamp;

(c) instructing the one or more destination partitions to independently perform a flush-replay operation that replays pending transactions at the secondary storage stamp;

(d) upon an individual destination partition, of the one or more destination partitions, substantially completing the flush-send operation, allowing the individual destination partition to commence accepting live traffic prior to others of the one or more destination partitions; and

(e) upon each of the one or more destination partitions substantially completing the flush-send operation, designating the secondary storage stamp as a new primary storage stamp for the set of data failed over.