Time optimized secure traceable migration of massive quantities of data in a distributed storage system

• US 8,261,033 B1
• Filed: 09/01/2009
• Issued: 09/04/2012
• Est. Priority Date: 06/04/2009
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: Bycast Inc. (NetApp, Inc.)
• Inventors: Slik, David, Montour, Michael, Austin, Jay, Cheng, Angela, Lam, Alvin, Lampert, Markus
• Primary Examiner(s): Lo, Kenneth
• Application Number:
  US12/552,173
• Time in Patent Office: 1,099 Days
• Field of Search: 711/162, 711/111
• US Class Current: 711/162
• CPC Class Codes: G06F 3/0605 by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/0647 Migration mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

A method and system for migrating massive amounts of data in a secure and traceable fashion using a distributed storage system is described. Data shuttle storage devices moves large quantities of content from one source location to one or more target locations by first acting in local communication with the source location, then being physically moved to a location where a shuttle storage device may be in local communication with the target location. This migration does not compromise data accessibility, reliability, or security. Dynamic configurable policy-driven data placement specifies the number and location of multiple copies of each digital data object. Each digital data object replica remains traceable even as the hardware onto which it is stored is being relocated. The content remains accessible at any time.

Claims

1. A method for physically moving digital data objects from a source location to at least one target location in a distributed storage system, wherein the digital data objects are stored in accordance with information lifecycle management policies, the method comprising:

registering, with a storage grid control system that maintains data object lifecycle management information and storage node information in a distributed storage network, that at least one shuttle storage device is in local communication at a source location;

duplicating, onto the at least one shuttle storage device, digital data objects in accordance with data object lifecycle management information associated with the digital data objects;

registering, with the storage grid control system, the presence of the digital data objects duplicated onto the at least one shuttle storage device, wherein registering comprises obtaining verification object metadata that allows validation of the integrity of the digital data objects duplicated onto the at least one shuttle storage device;

physically moving the at least one shuttle storage device from the source location to a target location; and

registering, with the storage grid control system, that the at least one shuttle storage device is in local communication at the target location.

2. The method of claim 1 wherein:

registering, with the storage grid control system, that at least one shuttle storage device is in local communication at the source location comprises registering at least three shuttle storage devices in local communication at the source location;

duplicating, onto the at least one shuttle storage device, the digital data objects specified in the information lifecycle management policies comprises duplicating onto the at least three shuttle storage devices the digital data objects specified in the information lifecycle management policies;

physically moving the at least one shuttle storage device from the source location to a target location comprises physically moving a first of the at least three shuttle storage devices to a first target location and physically moving a second of the at least three shuttle storage devices to a second target location; and

registering, with the storage grid control system, that the at least one shuttle storage device is in local communication at the target location comprises registering with the storage grid control system that the first of the at least three shuttle storage devices is in local communication at the first target location and registering with the storage grid control system that the second of the at least three shuttle storage devices is in local communication at the second target location.

3. The method of claim 1 further comprising verifying object metadata registered with the storage grid control system prior to physically moving the at least one shuttle storage device from the source location to a target location.

4. The method of claim 1 where digital data objects are encrypted before the shuttle storage device onto which they were duplicated is physically moved.

5. The method of claim 4 where the storage grid control system stores encryption keys as object metadata associated with the encrypted digital data objects.

6. The method of claim 1 where digital data objects are compressed before the shuttle storage device onto which they were duplicated is physically moved.

7. The method of claim 1 further comprising verifying, after duplicating a digital data object to a shuttle storage device, the integrity of the content of the digital data object prior to physically moving the shuttle storage device onto which the digital data object was duplicated.

8. The method of claim 1 further comprising verifying, after physically moving a shuttle storage device to the target location, the integrity of the content of the digital data objects on that shuttle storage device.

9. The method of claim 8 further comprising quarantining the shuttle storage device until after verification of the digital data objects on the shuttle storage device, where quarantining the shuttle storage device comprises restricting access to the shuttle storage device.

10. The method of claim 8 further comprising deleting digital data objects from the source location once the corresponding digital data objects have been verified to be received at the target location.

11. The method of claim 1 further comprising verifying that all of the digital data objects that were duplicated from the source location still exist on the shuttle storage device after it arrives at the target location.

12. The method of claim 11 further comprising verifying that no additional digital data objects exist on the shuttle storage device after it arrives at the target location.

13. The method of claim 12 further comprising quarantining the shuttle storage device until after verification that no additional digital data objects exist on the shuttle storage device, where quarantining the shuttle storage device comprises restricting access to the shuttle storage device.

14. The method of claim 1 wherein the digital data objects being physically moved from the source location remain accessible throughout the distributed storage system while the shuttle storage device is being physically moved.

15. The method of claim 1 further comprising generating an audit trail corresponding to the digital data objects.

16. The method of claim 15 wherein the audit trail includes governance information for the digital data objects.

17. The method of claim 15 wherein the audit trail includes chain of custody information for the digital data objects.

18. The method of claim 1 wherein the method is initiated by human action.

19. The method of claim 1 wherein the method is initiated automatically.

20. The method of claim 19 wherein the automatic initiation of the method occurs based upon time considerations.

21. The method of claim 19 wherein the automatic initiation of the method occurs based upon storage capacity considerations.

22. The method of claim 19 wherein the automatic initiation of the method occurs based upon the connection of a shuttle storage device to the source location.

23. The method of claim 1 further comprising duplicating the digital data objects on the shuttle storage device to a local storage device when the shuttle storage device arrives at the target location.

24. The method of claim 23 further comprising deleting the digital data objects from the shuttle storage device after duplicating the digital data objects on the shuttle storage device to the local storage device.

25. The method of claim 1 further comprising displaying, to a user, a visual representation of the information storage management policies and attributes of the shuttle storage device.

26. The method of claim 1 further comprising transmitting one or more notifications intended for a human recipient which contain information relevant to the status of a shuttle storage device.

27. The method of claim 1 wherein the at least one shuttle storage device comprises a magnetic spinning disk, magnetic tape, optical disc, solid state storage device or other digital storage device.

28. The method of claim 1 wherein the at least one shuttle storage device comprises a digital storage device enclosed in a passive physical storage shelf.

29. The method of claim 1 wherein the at least one shuttle storage device comprises a digital storage device enclosed in an active physical storage shelf comprising an electrical, optical or wireless interconnect that provides access to the enclosed digital storage devices when connected to the distributed storage system.

30. The method of claim 29 wherein the shuttle storage device further comprises a processing element, and a local memory element.

31. A distributed storage system configured for physically moving digital data objects from a source location to at least one target location in a distributed storage system, wherein the digital data objects are stored in accordance with information lifecycle management policies, the distributed storage system comprising:

one or more computer processors; and

at least one non-transitory computer-readable storage medium having stored thereon a plurality of instructions configured to cause the one or more computer processors to perform operations comprising;

registering, with a storage grid control system that maintains data object lifecycle management information and storage node information in a distributed storage network, that at least one shuttle storage device is in local communication at a source location;

duplicating, onto the at least one shuttle storage device, digital data objects in accordance with data object lifecycle management information associated with the digital data objects;

registering, with the storage grid control system, the presence of the digital data objects duplicated onto the at least one shuttle storage device, wherein registering comprises obtaining verification object metadata that allows validation of the integrity of the digital data objects duplicated onto the at least one shuttle storage device;

wherein the at least one shuttle storage device is configured to be physically moved from the source location to a target location; and

registering, with the storage grid control system, that the at least one shuttle storage device is in local communication at the target location.