Apparatus, method and system for using real-time performance feedback for modeling and improving access to solid state media

• US 8,972,689 B1
• Filed: 02/02/2011
• Issued: 03/03/2015
• Est. Priority Date: 02/02/2011
• Status: Active Grant

• Current Assignee: Innovations In Memory LLC
• Original Assignee: Violin Memory Incorporated
• Inventors: de la Iglesia, Erik
• Primary Examiner(s): Elmore, Reba I
• Application Number:
  US13/019,617
• Time in Patent Office: 1,490 Days
• Field of Search: 711/149, 711/168, 710/7, 710/20, 710/310, 712/9, 712/206
• US Class Current: 711/168
• CPC Class Codes: G06F 12/0842 for multiprocessing or multitasking
• CPC Class Codes: G06F 12/0844 Multiple simultaneous or quasi-simultaneous cache accessing
• CPC Class Codes: G06F 12/1054 the data cache being concurrently physically addressed
• CPC Class Codes: G06F 3/0611 in relation to response time
• CPC Class Codes: G06F 3/0628 making use of a particular technique
• CPC Class Codes: G06F 3/0631 by allocating resources to storage systems
• CPC Class Codes: G06F 3/0653 Monitoring storage devices or systems
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/0688 Non-volatile semiconductor memory arrays

Abstract

A storage processor identifies latency of memory drives for different numbers of concurrent storage operations. The identified latency is used to identify debt limits for the number of concurrent storage operations issued to the memory drives. The storage processor may issue additional storage operations to the memory devices when the number of storage operations is within the debt limit. Storage operations may be deferred when the number of storage operations is outside the debt limit.

Claims

1. An apparatus, comprising:

a memory;

a processor configured to issue the storage commands to the memory based on a number of concurrent storage commands being serviced by the memory and based on an expected latency associated with the number of concurrent storage commands;

identify a performance curve for the memory, wherein the performance curve maps the number of concurrent storage commands with the expected latency;

issue the storage commands to the memory based on the performance curve;

identify a first section of the performance curve associated with overhead processing;

identify a second section of the performance curve associated with stalling in the memory;

issue the storage commands to the memory based on the number of concurrent storage commands associated with the first and second section of the performance curve;

identify a first slope for the first section of the performance curve;

identify a second slope for the second section of the performance curve;

issue the storage commands to the memory based on the first slope and the second slope;

identify changes in the performance curve;

dynamically change the number of concurrent storage commands issued to in the memory based on the changes in the performance curve;

identify a write limit based on the performance curve, wherein the write limit is associated with writing data into the memory;

measure a latency for one of the storage commands;

compare the measured latency to the write limit;

discontinue writing data to the memory in response to the latency being outside of the write limit;

identify a use limit based on the performance curve, wherein the use limit is associated with reading data from the memory;

compare the latency to the use limit; and

erase data in the memory in response to the latency being outside of the use limit.

2. The apparatus of claim 1, wherein the processor is further configured to:

receive a new storage command;

determine a new number of concurrent storage commands serviced by the memory with the new storage command;

determine a new expected latency for the new number of concurrent storage commands; and

issue the new storage command to the memory based on the new expected latency.

3. The apparatus of claim 1, further comprising:

a command queue configured to receive the storage commands;

thread logic circuitry configured to issue the storage commands to the memory; and

a command scheduler configured to assign the storage commands to the threads based on the number of concurrent storage commands being serviced by the threads and the number of concurrent storage commands being serviced by the memory.

4. A method, comprising:

receiving a read operation;

identifying a memory device associated with the read operation;

identifying a latency of the memory device for servicing the read operation;

identifying a device debt for the memory device, wherein the device debt is associated with a number of read operations being processed by the memory device; and

tracking average latencies for different numbers of concurrent read operations issued to the memory device;

identifying patterns in the average latencies;

identifying a limit for the device debt based on the patterns;

identifying a predicted latency for the read operation based on the patterns;

measuring an actual latency of the read operation;

comparing the actual latency with the predicted latency;

increasing the device debt based on the comparison of the actual latency with the predicted latency;

deferring the issuing of the read operation to the memory device when the device debt is outside the limit; and

issuing the read operation to the memory device based on the identified latency.

5. The method of claim 4, further comprising:

identifying a thread debt, wherein the thread debt is based on the device debt of different memory devices accessed by a thread; and

assigning the read operation to the thread based on the thread debt.

6. The method of claim 5, further comprising increasing the device debt and increasing the thread debt in response to assigning the read operation to the thread.

7. The method of claim 5, further comprising reducing the device debt and reducing the thread debt in response to completion of the read operation.

8. The method of claim 4, further comprising:

selecting candidate threads from among multiple threads for issuing the read operation;

identifying thread debts for the candidate threads, wherein the thread debts are associated with a number of read operations being serviced by disk drives accessed by the candidate threads; and

assigning the read operation to one of the candidate threads based on the thread debts.

9. The method of claim 8, wherein selecting the candidate threads comprises:

maintaining thread maps for the disk drives, wherein the thread maps identify which of the threads are accessing the disk drives;

identifying one of the disk drives associated with the read operation; and

identifying the threads not listed in the thread map for the identified one of the disk drives as the candidate threads.

10. A storage processor, comprising:

a command queue configured to;

receive storage commands; and

maintain threads configured to initiate the storage commands to memory devices;

a command scheduler configured to;

identify device debts for the memory devices, wherein the device debts are associated with a number of storage operations pending in the memory devices, the command scheduler further configured to assign the storage commands to the threads based on the device debts;

the storage processor further comprising logic circuitry configured to;

identify latency patterns for the memory devices for different numbers of concurrent storage operations;

identify latency limits for the memory devices based on the latency patterns;

measure storage access latencies for the memory devices; and

erase the memo devices when the storage access latencies for the memory devices are outside the latency limits for the memory devices.

11. The storage processor of claim 10, further comprising thread maps associated with the memory devices, wherein the thread maps identify which threads are accessing the memory devices.

12. The storage processor of claim 10, wherein the command scheduler is further configured to:

identify thread debts for the threads, wherein the thread debts are associated with the device debts of the memory devices accessed by the threads; and

assign the storage commands to the threads based on the thread debts.

13. The storage processor of claim 12, further comprising device maps associated with the threads, wherein the device maps identify which memory devices are being accessed by the threads.

14. The storage processor of claim 10, further comprising logic circuitry configured to:

measure latencies of the memory devices for different numbers of concurrent storage operations; and

identify concurrent storage access limits for the memory devices based on the latencies,

wherein the command scheduler is configured to assign the storage commands to the threads when the device debts for the memory devices accessed by the threads are within the concurrent storage access limits for the memory devices.

15. The storage processor of claim 10, wherein the command scheduler is configured to defer assigning the storage commands to the threads when the devices debts for the memory devices associated with the storage commands are outside of the concurrent storage access limits for the memory devices.

16. The storage processor of claim 10, wherein the command scheduler is configured to:

identify predicted storage access latencies for the memory devices associated with the storage commands based on the device debts for the associated memory devices; and

assign the storage commands to the threads based on the predicted storage access latencies.

17. The storage processor of claim 16, wherein the command scheduler is further configured to:

measure actual storage access latencies of the memory devices;

receive new storage commands;

assign the new storage commands to the threads; and

increase the device debts for the memory devices accessed by the new storage commands based on the predicted storage access latencies and the actual storage access latencies.

18. The storage processor of claim 10, wherein a first set of the memory devices are associated with targets addressed by the storage commands and a second set of the memory drives are configured to operate as a tiering or cache media and store copies of at least some of the data in the first set of memory devices.

19. The storage processor of claim 10 wherein the memory devices comprise solid state drives (SSDs) comprising multiple separately accessible memory elements configured to be accessed by multiple storage commands at the same time.

20. A storage processor, comprising:

a command queue configured to;

receive storage commands; and

maintain threads configured to initiate the storage commands to memory devices; and

a command scheduler configured to;

identify device debts for the memory devices, wherein the device debts are associated with a number of storage operations pending in the memory devices, the command scheduler further configured to assign the storage commands to the threads based on the device debts;

measure latencies of the memory devices for different numbers of concurrent storage operations; and

identify concurrent storage access limits for the memory devices based on the latencies,

wherein the command scheduler is configured to assign the storage commands to the threads when the device debts for the memory devices accessed by the threads are within the concurrent storage access limits for the memory devices; and

, to defer assigning the storage commands to the threads when the devices debts for the memory devices associated with the storage commands are outside of the concurrent storage access limits for the memory devices.

21. A storage processor, comprising:

a command queue configured to;

receive storage commands; and

maintain threads configured to initiate the storage commands to memory devices;

a command scheduler configured to;

identify device debts for the memory devices, wherein the device debts are associated with a number of storage operations pending in the memory devices, the command scheduler further configured to assign the storage commands to the threads based on the device debts;

identify predicted storage access latencies for the memory devices associated with the storage commands based on the device debts for the associated memory devices; and

assign the storage commands to the threads based on the predicted storage access latencies;

measure actual storage access latencies of the memory devices;

receive new storage commands;

assign the new storage commands to the threads; and

increase the device debts for the memory devices accessed by the new storage commands based on the predicted storage access latencies and the actual storage access latencies.