Managing distributed system performance using accelerated data retrieval operations

• US 10,067,830 B1
• Filed: 03/30/2017
• Issued: 09/04/2018
• Est. Priority Date: 02/08/2013
• Status: Active Grant

• Current Assignee: Quantcast
• Original Assignee: Quantcast
• Inventors: Molina-Estolano, Esteban V., Rus, Silvius V.
• Primary Examiner(s): Meky, Moustafa M
• Assistant Examiner(s): Hurst, Andrea
• Application Number:
  US15/475,008
• Time in Patent Office: 523 Days
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1088 Reconstruction on already foreseen single or plurality of spare disks
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0611 in relation to response time
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0647 Migration mechanisms
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A distributed system is adapted to manage the performance of distributed processes. In one aspect, multiple stripes associated with a data item are stored in a distributed storage. The stored stripes include one or more stripes of redundancy information for the data item. A distributed process including at least one task is performed. During performance of the distributed process, a determination is made as to whether to perform an accelerated data retrieval operation. Responsive to a determination to perform an accelerated data retrieval operation, at least one of the one or more stripes of redundancy information for the data item is requested from the distributed storage. Other stripes associated with the data item may also be requested from the distributed storage. After a sufficient subset of stripes associated with the data item is received, the data item is reconstructed using the subset.

Claims

1. A method for managing performance of a distributed system, the method comprising:

storing, in a plurality of storage devices of a distributed storage, a plurality of stripes associated with a data item, the plurality of stripes generated according to a coding scheme, wherein the coding scheme generates a number of stripes associated with the data item that is more than a minimum number of stripes needed to assemble the data item, and wherein the plurality of stripes includes at least one redundancy stripe;

performing a task that requires retrieval of the data item;

responsive to performance information associated with the task;

accelerating retrieval by requesting more than the minimum number of stripes needed to assemble the data item from at least two of the plurality of storage devices of the distributed storage;

receiving at least the minimum number of stripes; and

assembling the data item from the minimum number of stripes according to the coding scheme.

2. The method of claim 1, wherein the plurality of stripes associated with the data item is generated according to an error correction coding scheme.

3. The method of claim 1, wherein the stripes of the plurality of stripes are each stored on a different storage device of the distributed storage.

4. The method of claim 1 further comprising:

storing a plurality of copies of the plurality of stripes associated with the data item in the distributed storage, and wherein accelerating retrieval comprises requesting the plurality of copies of the plurality of stripes associated with the data item from the distributed storage.

5. The method of claim 1, wherein the performance information associated with the task comprises current bandwidths of storage devices of the distributed storage.

6. The method of claim 1, wherein the performance information associated with the task comprises measured network traffic load.

7. A non-transitory computer readable storage medium executing computer program instructions for managing performance of a distributed system, the computer program instructions comprising instructions for:

storing, in a plurality of storage devices of a distributed storage, a plurality of stripes associated with a data item, the plurality of stripes generated according to a coding scheme, wherein the coding scheme generates a number of stripes associated with the data item that is more than a minimum number of stripes needed to assemble the data item, and wherein the plurality of stripes includes at least one redundancy stripe;

performing a task that requires retrieval of the data item;

responsive to performance information associated with the task;

accelerating retrieval by requesting more than the minimum number of stripes needed to assemble the data item from at least two of the plurality of storage devices of the distributed storage;

receiving at least the minimum number of stripes; and

assembling the data item from the minimum number of stripes according to the coding scheme.

8. The medium of claim 7, wherein the plurality of stripes associated with the data item is generated according to an error correction coding scheme.

9. The medium of claim 7, wherein the stripes of the plurality of stripes are each stored on a different storage device of the distributed storage.

10. The medium of claim 7, wherein the computer program instructions further comprise instructions for:

storing a plurality of copies of the plurality of stripes associated with the data item in the distributed storage, and wherein accelerating retrieval comprises requesting the plurality of copies of the plurality of stripes associated with the data item from the distributed storage.

11. The medium of claim 7, wherein the performance information associated with the task comprises current bandwidths of storage devices of the distributed storage.

12. The medium of claim 7, wherein the performance information associated with the task comprises measured network traffic load.

13. A system comprising:

a computer readable storage medium storing processor-executable computer program instructions for managing performance of a distributed system, the instructions comprising instructions for;

storing, in a plurality of storage devices of a distributed storage, a plurality of stripes associated with a data item, the plurality of stripes generated according to a coding scheme, wherein the coding scheme generates a number of stripes associated with the data item that is more than a minimum number of stripes needed to assemble the data item, and wherein the plurality of stripes includes at least one redundancy stripe;

performing a task that requires retrieval of the data item;

responsive to performance information associated with the task;

accelerating retrieval by requesting more than the minimum number of stripes needed to assemble the data item from at least two of the plurality of storage devices of the distributed storage;

receiving at least the minimum number of stripes; and

assembling the data item from the minimum number of stripes according to the coding scheme; and

a processor for executing the computer program instructions.

14. The system of claim 13, wherein the plurality of stripes associated with the data item is generated according to an error correction coding scheme.

15. The system of claim 13, wherein the stripes of the plurality of stripes are each stored on a different storage device of the distributed storage.

16. The system of claim 13, wherein the computer program instructions further comprise instructions for:

storing a plurality of copies of the plurality of stripes associated with the data item in the distributed storage, and wherein accelerating retrieval comprises requesting the plurality of copies of the plurality of stripes associated with the data item from the distributed storage.

17. The system of claim 13, wherein the performance information associated with the task comprises current bandwidths of storage devices of the distributed storage.

18. The system of claim 13, wherein the performance information associated with the task comprises measured network traffic load.