Bulk updates and tape synchronization

• US 7,720,892 B1
• Filed: 06/30/2006
• Issued: 05/18/2010
• Est. Priority Date: 06/30/2006
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Healey, Michael W. Jr., Cooper, John D., Yang, Bin, Pocas, Jamie, Beaverson, Arthur
• Primary Examiner(s): Wong; Don
• Assistant Examiner(s): Nguyen; Kim T
• Application Number:
  US11/480,323
• Time in Patent Office: 1,418 Days
• Field of Search: 707/204, 707/205
• US Class Current: 707/831
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 11/1456 Hardware arrangements for backup
• CPC Class Codes: G06F 11/1461 Backup scheduling policy
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 16/2386 Bulk updating operations data conversion details G06F16/258

Abstract

Described are techniques for processing a request to store data. The request to store data in accordance with a single instance storage technique is received. A schedule for storing the data in accordance with information included in the request is determined. Data for processing is requested in accordance with the single instance storage technique in accordance with the schedule.

Claims

1. A method for processing a request to store data comprising:

receiving the request to store a set of data from multiple devices in accordance with a single instance storage technique, said request being sent from a first component to a second component;

determining, by a scheduler included in the second component, a schedule in accordance with information included in the request, the schedule indicating an ordering of the multiple devices, data of the set being requested and processed for storage in accordance with the ordering; and

requesting, by said second component from the first component, data of the set for processing in accordance with the single instance storage technique, wherein data of the set is requested from the first component by the second component in an order in accordance with said ordering indicated by the schedule, said scheduler of said second component controls when data of the set is transmitted from said first component to said second component by said second component issuing one or more requests for data of the set as data is processed by the second component for storage in accordance with the schedule, wherein, when a data segment of the set is stored in accordance with the schedule, first processing is performed including;

partitioning said data segment into one or more partitions;

for each of said one or more partitions, storing a copy of data for said each partition in a storage pool if said copy is not already in the storage pool, and otherwise incrementing a reference count for said each partition by one rather than storing said copy; and

determining a virtualized representation of said data segment based on tokens corresponding to said one or more partitions.

2. The method of claim 1, wherein said request includes attribute information for one or more tape devices, said attribute information for each of said tape devices including tape size information.

3. The method of claim 2, wherein said tape devices are virtual tape devices.

4. The method of claim 2, wherein said attribute information for each of said tape devices includes attribute information about a data source, the attribute information comprising media format information and backup application information.

5. The method of claim 2, wherein said schedule is determined using one or more attributes for each tape device and resource information, said resource information including resource availability information and resource cost information for processing a defined unit of a tape device of a particular class, each class being defined using one or more of said attributes.

6. The method of claim 5, wherein said schedule is determined in accordance with one or more scheduling criteria.

7. The method of claim 6, wherein said scheduling criteria includes scheduling tapes having a same approximate duration time for processing in a same batch, tapes in a same batch being scheduled for processing at a same time.

8. The method of claim 2, wherein said request specifies one or more tapes having a processing priority indicating an ordering in which tapes of said request are scheduled for processing.

9. The method of claim 8, wherein tapes of differing priorities are not scheduled for processing within the same batch, and wherein a batch includes tapes scheduled for processing at a same time.

12. The method of claim 5, further comprising:

determining one or more batches of tape devices for processing using the single instance storage technique at a same time in accordance with one or more of said attributes of each of said tape devices;

for each tape device in each batch, allocating a processor and one or more units of storage for storing portions of received data of said each tape device, said processor and said one or more units of storage being allocated for a duration of a process processing said each tape device in said each batch.

13. The method of claim 5, wherein said resource availability information includes at least one of:

a number of processors, information about storage availability of a data storage system within which said data is being stored, processor speed, a number of servers and per server resource information including a number of processors of each server and an amount of memory on each server.

14. The method of claim 5, wherein said resource cost information for processing a defined unit of a tape device of a particular class includes at least one of:

an amount of memory, a number of CPU cycles and an amount of storage.

15. The method of claim 1, further comprising:

determining, in accordance with one or more job parameters included in said request related to job duration, job start time and estimated available storage, whether said request can be serviced.

10. A method for processing a request to store data comprising:

receiving the request to store a set of data from multiple devices in accordance with a single instance storage technique, said request being sent from a first component to a second component;

determining, by a scheduler included in the second component, a schedule in accordance with information included in the request, the schedule indicating an ordering of the multiple devices, data of the set being requested and processed for storage in accordance with the ordering; and

requesting, by said second component from the first component, data of the set for processing in accordance with the single instance storage technique, wherein data of the set is requested from the first component by the second component in an order in accordance with said ordering indicated by the schedule, said scheduler of said second component controls when data of the set is transmitted from said first component to said second component by said second component issuing one or more requests for data of the set as data is processed by the second component for storage in accordance with the schedule, and further comprising processing a received data segment in accordance with the schedule by;

partitioning said data segment into one or more portions;

determining a hash value for each of said portion; and

determining whether the hash value for each of said portions is included in a storage directory, said directory including a hash value for each stored data portion.

11. The method of claim 10, wherein, if the hash value is included in the storage directory, determining that a copy of the data portion corresponding to the hash value is already stored on a device and incrementing a reference count in the directory for the hash value, and

if the hash value is not included in the storage directory, determining that a copy of the data portion corresponding to the hash value is not already stored on a device, and wherein the method further including:

creating an entry in the storage directory having the hash value;

allocating storage at a location on a device for the data portion; and

updating the entry of the storage directory to include the location and a reference count of 1.

16. A system for processing a request to store data comprising:

a first component that sends the request to store a set of data from multiple devices in accordance with a single instance storage technique, said request being sent from the first component to a second component;

one or more data storage systems for storing data thereon; and

the second component including a computer readable medium comprising executable code stored thereon, said computer readable medium comprising;

executable code of a scheduler that determines a schedule in accordance with information included in the request, the schedule indicating an ordering of the multiple devices, data of the set being requested and processed for storage in accordance with the ordering;

executable code that requests, by said second component from the first component, data of the set for processing using the single instance storage technique wherein data of the set is requested from the first component by the second component in an order in accordance with said ordering indicated by the schedule, said scheduler of said second component controls when data of the set is transmitted from said first component to said second component by said second component issuing one or more requests for data of the set as data is processed by the second component for storage in accordance with the schedule; and

executable code that performs first processing when a data segment of the set is stored in accordance with the schedule, said first processing including;

partitioning said data segment into one or more partitions,

for each of said one or more partitions, storing a copy of data for said each partition in a storage pool if said copy is not already in the storage pool, and otherwise incrementing a reference count for said each partition by one rather than storing said copy; and

determining a virtualized representation of said data segment based on tokens corresponding to said one or more partitions.

17. The system of claim 16, wherein said second component is an appliance.

18. The system of claim 16, wherein said request includes attribute information for one or more tape devices, said attribute information for each of said tape devices including tape size information.

19. The system of claim 18, wherein said schedule is determined using one or more attributes for each tape device and resource information, said resource information including resource availability information and resource cost information for processing a defined unit of a tape device of a particular class, each class being defined using one or more of said attributes.

20. A computer readable medium comprising code stored thereon for processing a request to store data, the computer readable medium comprising code that:

receives the request to store a set of data from multiple devices in accordance with a single instance storage technique, said request being sent from a first component to a second component;

determines, by a scheduler included in the second component, a schedule in accordance with information included in the request, the schedule indicating an ordering of the multiple devices, data of the set being requested and processed for storage in accordance with the ordering;

requests, by said second component from the first component, data of the set for processing in accordance with the single instance storage technique, wherein data of the set is requested from the first component by the second component in an order in accordance with said ordering indicated by the schedule, said scheduler of said second component controls when data of the set is transmitted from said first component to said second component by said second component issuing one or more requests for data of the set as data is processed by the second component for storage in accordance with the schedule; and

performs first processing when a data segment of the set is stored in accordance with the schedule, said first processing including;

partitioning said data segment into one or more partitions;

for each of said one or more partitions, storing a copy of data for said each partition in a storage pool if said copy is not already in the storage pool, and otherwise incrementing a reference count for said each partition by one rather than storing said copy; and

determining a virtualized representation of said data segment based on tokens corresponding to said one or more partitions.