APPARATUS,SYSTEM,AND METHOD FOR MANAGING COMMANDS OF SOLID-STATE STORAGE USING BANK INTERLEAVE

• US 20110047356A2
• Filed: 12/06/2007
• Published: 02/24/2011
• Est. Priority Date: 12/06/2006
• Status: Active Grant

• Current Assignee: Unification Technologies LLC (Acacia Research Corporation)
• Original Assignee: Fusion-IO Incorporated (Western Digital Corporation)
• Inventors: Zappe, Michael, Lagerstedt, Bert, Flynn, David, Thatcher, Jonathan, Strasser, John
• Granted Patent: US 9,575,902 B2
• Time in Patent Office: Days
• US Class Current: 712/220
• CPC Class Codes: G06F 1/183 Internal mounting support structures, e.g. for printed circuit boards, internal connecting means for buses G06F13/409
• CPC Class Codes: G06F 11/108 Parity data distribution in semiconductor storages, e.g. in SSD
• CPC Class Codes: G06F 12/0246 in block erasable memory, e.g. flash memory
• CPC Class Codes: G06F 12/0804 with main memory updating G06F12/0806 takes precedence
• CPC Class Codes: G06F 12/0868 Data transfer between cache memory and other subsystems, e.g. storage devices or host systems
• CPC Class Codes: G06F 12/12 Replacement control
• CPC Class Codes: G06F 12/121 using replacement algorithms
• CPC Class Codes: G06F 12/123 with age lists, e.g. queue, most recently used [MRU] list or least recently used [LRU] list
• CPC Class Codes: G06F 13/28 using burst mode transfer, e.g. direct memory access DMA, cycle steal G06F13/32 takes precedence
• CPC Class Codes: G06F 13/4022 using switching circuits, e.g. switching matrix, connection or expansion network G06F13/4009 takes precedence
• CPC Class Codes: G06F 13/426 using an embedded synchronisation, e.g. Firewire bus, Fibre Channel bus, SSA bus
• CPC Class Codes: G06F 2211/002 Bus
• CPC Class Codes: G06F 2211/103 Hybrid, i.e. RAID systems with parity comprising a mix of RAID types
• CPC Class Codes: G06F 2212/1032 Reliability improvement, data loss prevention, degraded operation etc
• CPC Class Codes: G06F 2212/1044 Space efficiency improvement
• CPC Class Codes: G06F 2212/2022 Flash memory
• CPC Class Codes: G06F 2212/222 Non-volatile memory
• CPC Class Codes: G06F 2212/70 Details relating to dynamic memory management
• CPC Class Codes: G06F 2212/7203 Temporary buffering, e.g. using volatile buffer or dedicated buffer blocks
• CPC Class Codes: G06F 2212/7205 Cleaning, compaction, garbage collection, erase control
• CPC Class Codes: G06F 2212/7208 : Multiple device management, e.g. distributing data over multiple flash devices
• CPC Class Codes: G06F 3/0604 : Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0608 : Saving storage space on storage systems
• CPC Class Codes: G06F 3/0613 : in relation to throughput
• CPC Class Codes: G06F 3/0619 : in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0643 : Management of files
• CPC Class Codes: G06F 3/065 : Replication mechanisms
• CPC Class Codes: G06F 3/0652 : Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket
• CPC Class Codes: G06F 3/0656 : Data buffering arrangements
• CPC Class Codes: G06F 3/0659 : Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/0679 : Non-volatile semiconductor memory device, e.g. flash memory, one time programmable memory [OTP]
• CPC Class Codes: G06F 3/0685 : Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
• CPC Class Codes: G06F 3/0688 : Non-volatile semiconductor memory arrays
• CPC Class Codes: G06F 9/52 : Program synchronisation; Mutual exclusion, e.g. by means of semaphores
• CPC Class Codes: G06F 9/54 : Interprogram communication
• CPC Class Codes: H04L 67/02 : based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H05K 7/1444 : Complex or three-dimensional-arrangements; Stepped or dual mother boards
• CPC Class Codes: H05K 7/1487 : Blade assemblies, e.g. blade cases or inner arrangements within a blade
• CPC Class Codes: Y02D 10/00 : Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

An apparatus, system, and method are disclosed for efficiently managing commands in a solid-state storage device that includes a solid-state storage arranged in two or more banks. Each bank is separately accessible and includes two or more solid-state storage elements accessed in parallel by a storage input/output bus. The solid-state storage includes solid-state, non-volatile memory. The solid-state storage device includes a bank interleave that directs one or more commands to two or more queues, where the one or more commands are separated by command type into the queues. Each bank includes a set of queues in the bank interleave controller. Each set of queues includes a queue for each command type. The bank interleave controller coordinates among the banks execution of the commands stored in the queues, where a command of a first type executes on one bank while a command of a second type executes on a second bank.

Claims

1-22. -22. (canceled)

23. A method for storing data on non-volatile solid-state storage, the method comprising:

directing a solid-state storage command to a command queue of one of two or more banks of solid-state storage elements for a non-volatile solid-state storage device each bank configured to operate independently and each bank comprising one or more command queues; and

sequencing execution of the commands from the command queue of each bank such that a command of a first execution duration executes on a first bank concurrent with execution of a command of a second execution duration on a second bank.

24. The method of claim 23, wherein the command queues are first-in-first-out (FIFO) queues and wherein sequencing execution of the commands comprises ordering commands in each command queue for asynchronous execution by each bank.

25. The method of claim 23, wherein the bank comprises two or more non-volatile solid-state storage elements connected in parallel.

26. The method of claim 23, further comprising creating one or more subcommands from a command from the queue of the bank and coordinating execution of the subcommands such that a subcommand of a first type executes on a first bank concurrent with execution of a subcommand of a second type on a second bank.

27. A method for storing data on non-volatile solid-state storage, the method comprising:

providing a non-volatile solid-state storage device comprising two or more banks of solid-state storage elements, each bank configured to operate independently and each bank comprising a set of command queues;

directing a solid-state storage command to a command queue of one of the two or more banks based on a command type for each solid-state storage command; and

sequencing execution of the commands between the two or more banks such that a command of a first type executes on a first bank concurrent with execution of a command of a second type on a second bank.

28. The method of claim 27, wherein each set of queues for a bank comprise a queue for each command type and wherein sequencing execution of the commands comprises selecting commands from the queues of the set of queues such that a command of a first type executes on a first bank concurrent with execution of a command of a second type on a second bank.

29. The method of claim 27, wherein the bank comprises two or more non-volatile solid-state storage elements connected in parallel.

30. The method of claim 27, wherein the state storage elements are connected in parallel by an asynchronous storage input/output (“

I/O”

) bus that transports data packets comprising control information and data.

31. The method of claim 27, further comprising creating one or more subcommands from a command from a queue from the set of queues of the bank and coordinating execution of the subcommands such that a first subcommand executes on a first bank concurrent with execution of a second subcommand on a second bank.

32. The method of claim 27, wherein sequencing execution of the commands further comprises sequencing execution based on predicted command execution duration information of commands and subcommands.

33. The method of claim 27, wherein sequencing execution of the commands further comprises verifying execution based on status information received from the solid-state storage elements.

34. The method of claim 27, wherein the set of queues for a bank comprises at least a read queue to store read commands, a write queue to store write commands, an erase queue to store erase commands, and a management queue to store management commands.

35. The method of claim 27, wherein a read command reads data from a location within a page in the solid-state storage, a write command writes data to a storage write buffer, the storage write buffer located in the solid-state storage and comprising data for transfer to a designated page in the solid-state storage in response to a program command that programs data in the storage write buffer to the designated page, and an erase command to erase data in an erase block, an erase block comprising a plurality of pages in the solid-state storage.

36. The method of claim 27, wherein the command types further include a reset command to reset a bank and a read configuration register command to read a configuration register of a bank.

37. The method of claim 27, wherein sequencing execution of the commands further comprises coordinating execution of commands not stored in the queues with execution of commands stored in the queues.

38. The method of claim 27, further comprising

combining one or more data packets stored in a write buffer with a write command prior to sending the data packets and write command to the solid-state storage elements via a storage input/output (“

I/O”

) bus; and

receiving one or more data packets and corresponding status and management data and forwarding the data packets read from the solid-state storage elements to a read buffer and the corresponding status and management data to a management buffer.

39. The method of claim 27, wherein the non-volatile solid-state storage device further comprises a write data pipeline and a read data pipeline and wherein sequencing execution of the commands comprises coordinating reading data and writing data with the write data pipeline and the read data pipeline.

40. The method of claim 27, wherein the non-volatile solid-state storage device is configured in a dual-inline memory module (“

DIMM”

).

41. The method of claim 27, wherein the solid-state storage elements of the non-volatile solid-state storage device comprise flash memory.

42. The method of claim 27, further comprises sending commands to the solid-state storage elements asynchronously.

43. The method of claim 27, wherein storage locations within a solid-state storage element of the solid-state storage elements are accessed with a logical address mapped to one or more storage locations by corresponding physical addresses of the one or more storage locations, and wherein the method further comprises remapping a first logical address mapped to a first physical address to a second physical address wherein data commands sent to a storage location represented by the first logical address are directed to the first physical address prior to remapping and to the second physical address after remapping.

44. The method of claim 27, further comprising storing data in a write buffer and transferring data segments from the write buffer to a storage write buffer of a solid-state storage element during a write operation to fill a page of memory, and storing the data segments of the storage write buffer in a page within the solid-state storage element in response to a program command, the write buffer and the storage write buffer having a storage capacity equal to or greater than a page within the solid-state storage element.

45. A solid-state storage device comprising:

a solid-state storage arranged in two or more banks, each bank configured to operate independently and each bank comprising two or more solid-state storage elements accessed in parallel by a storage input/output (“

I/O”

) bus, the solid-state storage comprising non-volatile memory; and

a bank interleave controller that

directs a plurality of solid-state storage command to two or more queues, the plurality of commands separated by command type into the queues, each bank comprising a set of queues comprising a queue for each command type; and

coordinates among the banks, execution of the commands stored in the queues, wherein a command of a first type executes on one bank while a command of a second type executes on a second bank.

46. The solid-state storage device of claim 45, wherein the set of queues for a bank comprises at least a read queue to store read commands, a write queue to store write commands, an erase queue to store erase commands, and a management queue to store management commands.

47. A computer program product comprising a computer readable medium having computer usable program code executable to perform operations for storing data on non-volatile solid-state storage, the computer program product comprising:

directing a solid-state storage command to a command queue of one of two or more banks of solid-state storage elements for a non-volatile solid-state storage device based on a command type for each solid-state storage command, each bank configured to operate independently and each bank comprising one or more command queues; and

coordinating execution of the commands between the two or more banks such that a command of a first type executes on a first bank concurrent with execution of a command of a second type on a second bank.