Optimized selection and accessing of stored files to avoid mount and position thrashing

• US 6,886,019 B1
• Filed: 05/15/2000
• Issued: 04/26/2005
• Est. Priority Date: 05/15/2000
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Dawson, Colin Scott, Cannon, David Maxwell, Martin, Howard Newton
• Primary Examiner(s): Robinson, Greta
• Assistant Examiner(s): PANNALA, SATHYANARAYA R
• Application Number:
  US09/571,673
• Time in Patent Office: 1,807 Days
• Field of Search: 707200-205, 711/111, 711/162, 714/6
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 11/1458 Management of the backup or restore process
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 2201/80 Database-specific techniques
• CPC Class Codes: G06F 3/0611 in relation to response time
• CPC Class Codes: G06F 3/0647 Migration mechanisms
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/0683 Plurality of storage devices
• CPC Class Codes: Y10S 707/99955 Archiving or backup

Abstract

A method for generating file copies with minimal mounting and positioning of storage volumes involves receiving a request to generate file copies specifying file selection criteria, identifying matching files meeting the selection criteria, locating the matching files on their storage volumes, and copying the files to a copy set. Determination of file copying order is optimized by placing greater emphasis on relative storage locations of matching files than on the order in which their copies are requested. Once a storage volume is mounted, all files stored on that volume are matched against the pending requests including future requests and if found to be the subject of a request, are copied from the mounted volume in optimal position-sensitive order. Copying from secondary storage is deferred until all copying from primary storage is complete.

Claims

1. A method for transferring files stored on multiple separate storage volumes using information from a list in such a way as to minimize mounting and positioning of the storage volumes, the method comprising:

receiving a request specifying selection criteria for selecting a first file to be included in a copy set, wherein the request includes a file order;

identifying a first matching file meeting the selection criteria;

locating the first matching file on one of the storage volumes, wherein locating the first matching file on one of the storage volumes comprises searching entries in a storage reference list, first for a storage pool identifier, then for a bfid, then for a storage volume identifier, and then for a position within the storage volume that has the first matching file;

identifying any other matching files on the storage volume that has the first matching file;

determining a copying order of the first matching file and the any other matching files on the storage volume that has the first matching file, wherein the file order in the request is ignored when determining the copying order, and wherein the copying order is determined by, volume, then position, and then bfid; and

copying the first matching file and the any other matching files identified on the storage volume that has the first matching file, from the storage volume that has the first matching file to a copy set according to the determined copying order, wherein no matching files are duplicated in the copy set; and

wherein the copying operation further comprises determining if the first matching file cannot be accessed due to a media defect, and if the first matching file cannot be accessed due to a media defect, then the copying operation does not include copying the first matching file to the copy set, and the method further comprises determining if the first matching file resides on a secondary volume, and if the first matching file resides on a secondary volume, then the method further comprises adding the secondary volume to a deferred processing list.

2. The method of claim 1 wherein the method further comprises transferring a file on the deferred processing list from the secondary volume.

11. The method of claim 2, wherein the method further comprises constructing, with an inventory component, while the copy set is generated, a temporary table which contains an entry for each file that has been copied to the copy set, to avoid duplication of files in the copy set.

12. The method of claim 11, wherein the other matching files are files that are encountered on the storage volume before encountering the first matching file, and wherein the method further comprises determining if any of the other matching files are on an inventory list.

13. The method of claim 12:

wherein the first matching file is copied immediately if the storage volume that has the first matching file is a random access media; and

wherein the method further comprises mounting the storage volume that has the first matching file, if the storage volume that has the first matching file is a sequential media.

3. A method for transferring files stored on multiple separate storage volumes using information from a list in such a way as to minimize mounting and positioning of the storage volumes, the method comprising:

receiving a request specifying selection criteria for selecting a first file to be included in a copy set, wherein the request includes a file order;

identifying a first matching file meeting the selection criteria;

locating the first matching file on one of the storage volumes, wherein locating the first matching file on one of the storage volumes comprises searching entries in a storage reference list, first for a storage pool identifier, then for a bfid, then for a storage volume identifier, and then for a position within the storage volume that has the first matching file;

identifying any other matching files on the storage volume that has the first matching file;

determining a copying order of the first matching file and the any other matching files on the storage volume that has the first matching file, wherein the file order in the request is ignored when determining the copying order, and wherein the copying order is determined by, volume, the position, and then bfid; and

copying the first matching file and the any other matching files identified on the storage volume that has the first matching file, from the storage volume to a copy set according to the determined copying order, wherein no matching files are duplicated in the copy set; and

wherein the storage volumes comprise removable and non removable sequential access volumes and random access volumes; and

wherein the storage volume that has the first matching file is a sequential access volume, and wherein the copying operation comprises copying the first matching file and the any other matching files to the copy set in an order in which they are stored on the sequential access storage volume; and

wherein the first matching file stored on the storage volume is a primary copy of the first matching file; and

wherein a secondary copy of the first matching file is stored on a sequential access volume;

and further comprising deferring copying from the sequential access volume having the secondary copy of the first matching file until all copying from the storage volume that has the primary copy of the first matching file has been completed.

4. The method of claim 3 wherein a matching file that spans from a first volume to a second volume is copied from the first volume and then from the second volume, and all matching files on the second volume are copied before the second volume is dismounted.

14. The method of claim 3, wherein the copying operation further comprises determining if the first matching file cannot be accessed due to a media defect, and if the first matching file cannot be accessed due to a media defect, then the copying operation does not include copying the first matching file to the copy set, and the method further comprises adding the secondary volume to a deferred processing list.

5. An apparatus comprising:

a computer having data storage volumes connected thereto; and

one or more computer programs, performed by the computer, for generating a copy set in such a way as to minimize mounting and positioning of the storage volumes, the generating comprising;

receiving a request specifying selection criteria for selecting a first file to be included in a copy set, wherein the request includes a file order;

identifying a first matching file meeting the selection criteria;

locating the first matching file on one of the storage volumes, wherein locating the first matching file on one of the storage volumes comprises searching entries in a storage reference list, first for a storage pool identifier, then for a bfid, then for a storage volume identifier, and then for a position within the storage volume that has the first matching file;

identifying any other matching files on the storage volume that has the first matching file;

determining a copying order of the first matching file and the any other matching files on the storage volume that has the first matching file, wherein the file order in the request is ignored when determining the copying order, and wherein the copying order is determined by, volume, then position, and then bfid; and

copying the first matching file and the any other matching files identified on the storage volume that has the first matching file, from the storage volume that has the first matching file to a copy set according to the determined copying order, wherein no matching files are duplicated in the copy set; and

wherein the copying operation further comprises determining if the first matching file cannot be accessed due to a media defect, and if the first matching file cannot be accessed due to a media defect, then the copying operation does not include copying the first matching file to the copy set, and the generating further comprises determining if the first matching file resides on a secondary volume, and if the first matching file resides on a secondary volume, then the generating further comprises adding the secondary volume to a deferred processing list.

6. The apparatus of claim 5 wherein the generating further comprises transferring a file on the deferred processing list from a secondary volume.

15. The apparatus of claim 5, wherein the generating further comprises constructing, with an inventory component, while the copy set is generated, a temporary table which contains an entry for each file that has been copied to the copy set, to avoid duplication of files in the copy set, and

wherein the other matching files are files that are encountered on the storage volume before encountering the first matching file, and

wherein the generating further comprises determining if any of the other matching files are on an inventory list.

7. An apparatus comprising:

a computer having data storage volumes connected thereto; and

one or more computer programs, performed by the computer, for generating a copy set in such a way as to minimize mounting and positioning of the storage volumes, the generating comprising;

receiving a request specifying selection criteria for selecting a first file to be included in a copy set, wherein the request includes a file order;

identifying a first matching file meeting the selection criteria;

locating the first matching file on one of the storage volumes, wherein locating the first matching file on one of the storage volumes comprises searching entries in a storage reference list, first for a storage pool identifier, then for a bfid, then for a storage volume identifier, and then for a position within the storage volume that has the first matching file;

identifying any other matching files on the storage volume that has the first matching file;

determining a copying order of the first matching file and the any other matching files on the storage volume that has the first matching file, wherein the file order in the request is ignored when determining the copying order, and wherein the copying order is determined by, volume, then position, and then bfid; and

copying the first matching file and the any other matching files identified on the storage volume that has the first matching file, from the storage volume to a copy set according to the determined copying order, wherein no matching files are duplicated in the copy set; and

wherein the storage volumes comprise removable and non removable sequential access volumes and random access volumes; and

wherein the storage volume that has the first matching file is a sequential access volume, and wherein the copying operation comprises copying the first matching file and the any other matching files to the copy set in an order in which they are stored on the sequential access storage volume; and

wherein the first matching file stored on the storage volume is a primary copy of the first matching file; and

wherein a secondary copy of the first matching file is stored on a sequential access volume;

and further comprising deferring copying from the sequential access volume having the secondary copy of the first matching file until all copying from the storage volume that has the primary copy of the first matching file has been completed.

16. The apparatus of claim 7, wherein the copying operation further comprises determining if the first matching file cannot be accessed due to a media defect, and if the first matching file cannot be accessed due to a media defect, then the copying operation does not include copying the first matching file to the copy set, and the generating further comprises adding the secondary volume to a deferred processing list.

8. An article of manufacture comprising a computer program carrier readable by a computer and embodying one or more instructions executable by the computer for generating a copy set in such a way as to minimize mounting and positioning of storage volumes, the generating comprising:

receiving a request specifying selection criteria for selecting a first file to be included in a copy set, wherein the request includes a file order;

identifying a first matching file meeting the selection criteria;

locating the first matching file on one of the storage volumes, wherein locating the first matching file on one of the storage volumes comprises searching entries in a storage reference list, first for a storage pool identifier, then for a bfid, then for a storage volume identifier, and then for a position within the storage volume that has the first matching file;

identifying any other matching files on the storage volume that has the first matching file;

determining a copying order of the first matching file and the any other matching files on the storage volume that has the first matching file, wherein the file order in the request is ignored when determining the copying order, and wherein the copying order is determined by, volume, then position, and then bfid; and

copying the first matching file and the any other matching files identified on the storage volume that has the first matching file, from the storage volume to a copy set according to the determined copying order, wherein no matching files are duplicated in the copy set; and

wherein the copying operation further comprises determining if the first matching file cannot be accessed due to a media defect, and if the first matching file cannot be accessed due to a media defect, then the copying operation does not include copying the first matching file to the copy set, and the generating further comprises determining if the first matching file resides on a secondary volume, and if the first matching file resides on a secondary volume, then the generating further comprises adding the secondary volume to a deferred processing list.

9. The article of claim 8 wherein the generating further comprises transferring a file on the deferred processing list from a secondary volume.

17. The article of claim 8, wherein the generating further comprises constructing, with an inventory component, while the copy set is generated, a temporary table which contains an entry for each file that has been copied to the copy set, to avoid duplication of files in the copy set, and

wherein the other matching files are files that are encountered on the storage volume before encountering the first matching file, and

wherein the generating further comprises determining if any of the other matching files are on an inventory list.

10. An article of manufacture comprising a computer program carrier readable by a computer and embodying one or more instructions executable by the computer for generating a copy set in such a way as to minimize mounting and positioning of storage volumes, the generating comprising:

receiving a request specifying selection criteria for selecting a first file to be included in a copy set, wherein the request includes a file order;

identifying a first matching file meeting the selection criteria;

locating the first matching file on one of the storage volumes, wherein locating the first matching file on one of the storage volumes comprises searching entries in a storage reference list, first for a storage pool identifier, then for a bfid, then for a storage volume identifier, and then for a position within the storage volume that has the first matching file;

identifying any other matching files on the storage volume that has the first matching file;

determining a copying order of the first matching file and the any other matching files on the storage volume that has the first matching file, wherein the file order in the request is ignored when determining the copying order, and wherein the copying order is determined by, volume, then position, and then bfid; and

copying the first matching file and the any other matching files identified on the storage volume that has the first matching file, from the storage volume to a copy set according to the determined copying order, wherein no matching files are duplicated in the copy set; and

wherein the storage volumes comprise removable and non removable sequential access volumes and random access volumes; and

wherein the storage volume that has the first matching file is a sequential access volume, and wherein the copying operation comprises copying the first matching file and the any other matching files to the copy set in an order in which they are stored on the sequential access storage volume; and

wherein the first matching file stored on the storage volume is a primary copy of the first matching file; and

wherein a secondary copy of the first matching file is stored on a sequential access volume;

and further comprising deferring copying from the sequential access volume having the secondary copy of the first matching file until all copying from the storage volume that has the primary copy of the first matching file has been completed.

18. The article of claim 10, wherein the copying operation further comprises determining if the first matching file cannot be accessed due to a media defect, and if the first matching file cannot be accessed due to a media defect, then the copying operation does not include copying the first matching file to the copy set, and the generating further comprises adding the secondary volume to a deferred processing list.