Chunk Monitoring

• US 20180004430A1
• Filed: 01/30/2015
• Published: 01/04/2018
• Est. Priority Date: 01/30/2015
• Status: Abandoned Application

• Current Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Original Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Inventors: Merritt, Alexander M., Milojicic, Dejan S.
• Application Number:
  US15/545,880
• Publication Number:
  US 20180004430A1
• Time in Patent Office: Days
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3466 Performance evaluation by tracing or monitoring
• CPC Class Codes: G06F 2201/81 Threshold
• CPC Class Codes: G06F 2201/88 Monitoring involving counting
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0611 in relation to response time
• CPC Class Codes: G06F 3/0631 by allocating resources to storage systems
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/60 Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources admission control or resource allocation H04L47/70

Abstract

One example of a system includes a plurality of clients, a master chunk coordinator, and a plurality of chunk servers. Each client submits requests to access chunks of objects. The master chunk coordinator maintains chunk information for each object. Each chunk server includes a chunk monitor to monitor client requests, maintain chunk statistics for each chunk based on the monitoring, and transmit the chunk statistics for each chunk to the master chunk coordinator. The master chunk coordinator instructs the chunk servers to re-chunk objects, replicate chunks, migrate chunks, and resize chunks based on the chunk statistics to meet specified parameters.

Claims

1. A system comprising:

a plurality of clients, each client to submit requests to access chunks of objects;

a master chunk coordinator to maintain chunk information for each object; and

a plurality of chunk servers to store chunks, each chunk server including a chunk monitor to monitor client requests, maintain chunk statistics for each chunk based on the monitoring, and transmit the chunk statistics for each chunk to the master chunk coordinator,

wherein the master chunk coordinator instructs the chunk servers to re-chunk objects, replicate chunks, migrate chunks, and resize chunks based on the chunk statistics to meet specified parameters.

2. The system of claim 1, wherein the chunk statistics comprise hotness, reuse, read/write ratio, concurrent demand, and relatedness.

3. The system of claim 1, wherein the chunk information for each object comprises static object information and dynamic object information, the dynamic object information comprising a chunk list indicating each chunk of the object, ordering of the chunks, each chunk replica, and each chunk location.

4. The system of claim 3, wherein the chunk list comprises static chunk information and dynamic chunk information for each chunk, the dynamic chunk information comprising reuse, read/write ratio, and concurrent demand statistics for the chunk.

5. The system of claim 1, wherein the master chunk coordinator and the plurality of chunk servers provide a distributed active in-memory object store.

6. A machine-readable storage medium encoded with instructions, the instructions executable by a processor of a system to cause the system to:

receive from a client application quality-of-service parameters for storing objects in a distributed active in-memory object store comprising a plurality of chunk servers;

monitor requests from the client application to access chunks of objects; and

instruct the chunk servers to re-chunk objects, replicate chunks, migrate chunks, and resize chunks to distribute chunks of objects across the plurality of chunk servers based on the monitoring to maintain compliance with the quality-of-service parameters.

7. The machine-readable storage medium of claim 6, wherein the instructions are executable by the processor to further cause the system to:

receive chunk statistics for each chunk in response to the monitoring, the chunk statistics comprising hotness, reuse, read/write ratio, concurrent demand, and relatedness; and

re-chunk objects, replicate chunks, migrate chunks, and resize chunks based on the chunk statistics to maintain compliance with the quality-of-service parameters.

8. The machine-readable storage medium of claim 7, wherein the instructions are executable by the processor to further cause the system to:

replicate a chunk in response to hotness of the chunk exceeding a threshold value.

9. The machine-readable storage medium of claim 7, wherein the instructions are executable by the processor to further cause the system to:

identify a chunk server having the lowest load and directing write accesses for a chunk to a chunk replica on the identified chunk server in response to concurrent write accesses to the chunk exceeding a threshold value.

10. The machine-readable storage medium of claim 7. wherein the instructions are executable by the processor to further cause the system to:

migrate chunk replicas for a set of objects from a larger set of chunk servers to a smaller set of chunk servers in response to a client repeatedly accessing the set of objects.

11. A method comprising:

receiving requests from clients to access chunks of objects stored in a distributed active in-memory object store;

maintaining chunk information for each object n the distributed active in-memory object store;

monitoring the client requests;

maintaining chunk statistics for each chunk based on the monitoring; and

re-chunking objects, replicating chunks, migrating chunks, and resizing chunks based on the chunk statistics to meet specified parameters.

12. The method of claim 11, wherein maintaining chunk statistics comprises maintaining hotness, reuse, read/write ratio, concurrent demand, and relatedness statistics for each chunk.

13. The method of claim 12, further comprising:

splitting a larger chunk into a plurality of smaller chunks based on access characteristics in response to the larger chunk having a hotness above a threshold value and read and write accesses to the larger chunk hitting disjoint subsets of the larger chunk.

14. The method of claim 12, further comprising:

splitting a larger chunk into a plurality of smaller chunks using boundaries observed by client accesses in response to the larger chunk having a hotness above a threshold value where the hotness is attributed to a subset region of the larger chunk.

15. The method of claim 12, further comprising:

migrating a chunk from a chunk server having a higher load to a chunk server having a lower load in response to a read/write ratio of that chunk exceeding a first threshold value; and

deleting replica chunks while maintaining at least one such chunk replica in response to a read/write ratio of a chunk falling below a second threshold value.