Managing data storage using storage policy specifications

• US 9,984,079 B1
• Filed: 01/13/2012
• Issued: 05/29/2018
• Est. Priority Date: 01/13/2012
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Stickle, Thomas C.
• Primary Examiner(s): Truong, Dennis
• Application Number:
  US13/350,658
• Time in Patent Office: 2,328 Days
• Field of Search: 707821, 707626, 707634, 707636, 707694, 707812, 707652
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1456 Hardware arrangements for backup
• CPC Class Codes: G06F 16/122 using management policies backup systems G06F11/1446; file migration policies for HSM systems G06F16/185
• CPC Class Codes: G06F 16/1827 Management specifically adapted to NAS management of storage area networks [SAN] G06F3/067
• CPC Class Codes: G06F 16/1844 Management specifically adapted to replicated file systems
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0649 Lifecycle management
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0685 Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/568 Storing data temporarily at an intermediate stage, e.g. caching

Abstract

Techniques are described for managing data storage using defined data storage management policies. In some situations, data storage may be managed using multiple supported storage mechanisms, such as different storage mechanisms of different types and/or in different locations. As one example, the described techniques may be performed to manage data that is available to a software program executing on a computer system, such as by caching a subset of the available data on one or more storage mechanisms to enhance later retrieval times of that data subset by the software program. In this example, the multiple supported storage mechanisms may include one or more storage mechanisms local to the computer system and one or more storage mechanisms remote from the computer system, and a defined data storage management policy for the software program may define particular types of data to store on particular storage mechanisms in particular manners.

Claims

1. A computer-implemented method comprising:

receiving, by a configured computer system, a defined data storage management policy for use with an indicated program during execution on the configured computer system, wherein the defined data storage management policy specifies first criteria for storing a first group of data on a remote online storage service that is separated from the configured computer system by one or more computer networks, specifies second criteria for storing a second group of data on a distributed online memory cache stored using volatile memory of multiple computer systems, and specifies third criteria for storing a third group of data on a distributed file system stored on multiple non-volatile storage devices accessible to the configured computer system via a computer network to which the configured computer system is attached;

initiating, by the configured computer system, and prior to receiving any data access request for use by the indicated program of the first group of data, of the second group of data, or of the third group of data, the execution of the indicated program, including preloading data for subsequent use by the indicated program by;

determining, by the configured computer system, that the specified first criteria are satisfied, and initiating storing the first group of data on the online storage service;

determining, by the configured computer system, that the specified second criteria are satisfied, and initiating storing the second group of data on the distributed online memory cache; and

determining, by the configured computer system, that the specified third criteria are satisfied, and initiating storing the third group of data on the distributed file system; and

retrieving and using, during the execution of the indicated program and in response to one or more data access requests for use of at least some of the preloaded data by the indicated program, the at least some preloaded data.

2. The method of claim 1 wherein the retrieving of the at least some preloaded data in response to the one or more access requests includes retrieving, by the configured computer system, the stored first group of data from the online storage service and storing the retrieved first group of data on storage local to the configured computer system for use by the executing indicated program.

3. The method of claim 2 wherein the retrieving of the stored first group of data is based at least in part on one or more requests from one or more users of the executing indicated program, and wherein the defined data storage management policy is used to enable access by the one or more users to the first group of data.

4. The method of claim 2 wherein the determining that the first, second and third specified criteria are satisfied is performed by a data storage manager system executing on the configured computer system that is distinct from the executing indicated program.

5. The method of claim 2 wherein the first group of data is specified in the defined data storage management policy by specifying a type of data to which the first group of data belongs, and wherein the indicated program uses, during the initiated execution, data of multiple types that include the first type and one or more other types.

6. The method of claim 2 wherein the indicated program uses, during the initiated execution, multiple groups of data that include the first, second and third groups of data and further include one or more additional groups of data that are not specified in the defined data storage management policy.

7. The method of claim 1 wherein the first group of data is specified in the defined data storage management policy by specifying one or more metadata tags, and wherein the method further comprises determining, by the configured computing system, that the first group of data has at least one of the one or more metadata tags.

8. The method of claim 1 further comprising selecting, by the configured computing system and based on the defined data storage management policy, storage mechanisms for use with indicated groups of data from multiple available storage mechanisms that include the online storage service and the distributed online memory cache and the distributed file system and that further include a non-volatile storage device local to the configured computer system.

9. The method of claim 8 wherein the non-volatile storage device local to the computer system is used at least in part as a storage cache for one or more other of the multiple available storage mechanisms.

10. The method of claim 9 wherein the retrieving of the at least some preloaded data in response to the one or more data access requests includes providing, by the configured computer system to the indicated program, at least some of the first group of data from the online storage service, at least some of the second group of data from the distributed online memory cache, and at least some of the third group of data from the distributed file system.

11. The method of claim 1 wherein the configured computer system is part of a group of multiple computer systems that operate together in a distributed manner to each locally store a specified subset of multiple groups of data.

12. A non-transitory computer-readable medium having stored contents that cause a computer system to:

receive, by the computer system, a defined data storage management policy for use in preloading data to later be used by an indicated program during execution on the computer system, wherein the defined data storage management policy specifies first criteria for selecting a first storage mechanism of multiple accessible storage mechanisms on which to preload a first group of data of a first type and specifies second criteria for selecting a second storage mechanism of the multiple accessible storage mechanisms on which to preload a second group of data of a second type;

store, by the computer system at a time of initiating the execution of the indicated program;

the first group of data on the first storage mechanism in accordance with the first specified criteria and to preload the stored first group of data for later use by the indicated program during the execution, wherein the first storage mechanism is selected based on the first specified criteria to be a distributed online memory cache using volatile memory of multiple computer systems; and

the second group of data on the second storage mechanism in accordance with the second specified criteria and to preload the stored second group of data for later use by the indicated program during the execution, wherein the second storage mechanism is selected based on the second specified criteria to be a distributed file system stored on multiple storage devices accessible to the computer system via a computer network; and

retrieve and use, during the execution of the indicated program on the computer system, the stored first and second groups of data, including to provide the retrieved first and second groups of data to the indicated program for use.

13. The non-transitory computer-readable medium of claim 12 wherein the multiple accessible storage mechanisms further include a non-volatile storage device local to the computer system, and wherein the stored contents include software instructions that, when executed, further cause the computer system to store, in accordance with the third criteria specified in the defined data storage management policy and as part of initiating the execution of the indicated program on the computer system, a third group of data of a third type on the non-volatile storage device to preload the stored third group of data for later use by the indicated program, and to retrieve, at a later time while the indicated program is executing, the stored third group of data for use by the executing indicated program.

14. The non-transitory computer-readable medium of claim 13 wherein the stored contents further cause the computer system to identify, before the preloading of the first and second groups of data, the first and second groups of data as being of different first and second types based on differences in type of data in the first and second groups of data or in locations of the first and second groups of data, and to perform the storing of the first and second groups of data as part of the initiating of the execution of the indicated program.

15. A computer system, comprising:

one or more processors; and

one or more memories with software instructions that, when executed by at least one of the one or more processors, cause the at least one processor to manage data storage for the computer system, the managing of the data storage including;

receiving a defined data storage management policy for use in preloading data to later be used by an indicated program, wherein the defined data storage management policy specifies a first group of data of a first type to be preloaded on a first storage mechanism of multiple storage mechanisms accessible to the computer system, and specifies a second group of data of a second type to be preloaded on a second storage mechanism of the multiple storage mechanisms;

initiating execution of the indicated program and using the defined data storage management policy, including;

storing, before any request for the first group of data for the indicated program, the first group of data on the first storage mechanism to preload the stored first group of data for later use by the indicated program during the execution, wherein the first storage mechanism is selected to be a distributed file system stored on multiple storage devices accessible to the computer system via a computer network; and

storing, before any request for the second group of data for the indicated program, the second group of data on the second storage mechanism to preload the stored second group of data for later use by the indicated program during the execution, wherein the second storage mechanism is selected to be an online memory cache distributed across volatile memory of multiple computer systems; and

retrieving, for the indicated program and during the execution, at least one of the stored first and second groups of data for use by the indicated program.

16. The computer system of claim 15 wherein the defined data storage management policy further specifies first criteria for storing data of the first type and second criteria for storing data of the second type, wherein storing of the first group of data is further performed based at least in part on the first criteria being determined to be satisfied, and wherein storing of the second group of data is further performed based at least in part on the second criteria being determined to be satisfied.

17. The computer system of claim 15 further comprising a non-volatile storage device local to the computer system and a network connection operable to access an online storage service remote from the computing system over one or more networks, and wherein the software instructions, when executed, further cause the at least one processor to:

preload, before the indicated program begins executing, a third group of data of a third type on the non-volatile storage device in accordance with the defined data storage management policy;

preload, before the indicated program begins executing, a fourth group of data of a fourth type on the online storage service; and

retrieve, during the execution of the indicated program, the stored third and fourth groups of data for use by the indicated program.

18. The computer system of claim 17 wherein the software instructions, when executed, further cause the at least one processor to identify the first and second groups of data as being of different first and second types based on differences in a type of data in the first and second groups of data and in locations of the first and second groups of data before preloading, and to perform the storing of the first and second groups of data as part of the initiating of the execution and before the indicated program begins executing and before the indicated program makes any requests for data.