IOC to IOC distributed caching architecture

• US 9,720,603 B1
• Filed: 03/17/2014
• Issued: 08/01/2017
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: BiTMICRO, LLC (BiTMICRO Networks, Inc.)
• Original Assignee: BiTMICRO Networks, Inc.
• Inventors: Salazar, Lawrence Moldez, Chiw, Bernard Sherwin Leung
• Primary Examiner(s): Tran, Jimmy H
• Assistant Examiner(s): Ma, Wing
• Application Number:
  US14/216,937
• Time in Patent Office: 1,233 Days
• Field of Search: 709214
• CPC Class Codes: G06F 12/0804 with main memory updating G06F12/0806 takes precedence
• CPC Class Codes: G06F 12/0813 with a network or matrix configuration
• CPC Class Codes: G06F 12/0868 Data transfer between cache memory and other subsystems, e.g. storage devices or host systems
• CPC Class Codes: G06F 2212/1016 Performance improvement
• CPC Class Codes: G06F 2212/1032 Reliability improvement, data loss prevention, degraded operation etc
• CPC Class Codes: G06F 2212/222 Non-volatile memory
• CPC Class Codes: G06F 2212/225 Hybrid cache memory, e.g. having both volatile and non-volatile portions
• CPC Class Codes: G06F 2212/284 being distributed
• CPC Class Codes: G06F 3/0613 in relation to throughput
• CPC Class Codes: G06F 3/0635 by changing the path, e.g. traffic rerouting, path reconfiguration
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

In an embodiment of the invention, a method which speeds up the transfer of data and increases the data throughput in an IO network comprised of Host Bus Adapters (HBAs)/IO bridges-switches, IO devices, and hosts is described. In the embodiment of the present invention, HBAs and IO bridges-switches utilize a multi-level cache composed of volatile memories (such as SRAM, SDRAM, etc.) and solid-state memories (such as flash, MRAM, etc.). These memories are used to cache the most recently accessed IO data by an active host or by another HBA/IO bridge-switch. Cache content can be from the local IO devices (the ones connected directly to the HBA/IO bridge-switch), from remote IO devices (the ones connected to different HBA/IO bridges/switches), or from both (a portion from local IO devices and another portion from remote IO devices). The combination of these caches from different HBAs/IO bridges-switches creates the cache for all IO devices in the entire network. Multiple copies of the same IO cache are possible depending on the transactions with the hosts.

Claims

1. An apparatus comprising:

a network comprising;

a first input-output (IO) device comprising a first IO storage device, a second IO device comprising a second IO storage device, and a third IO device comprising a third IO storage device,

a first host and a second host, and

a plurality of bridges-switches comprising a first bridge-switch, a second bridge-switch, and a third bridge-switch, wherein the first bridge-switch comprises a first cache memory, a first processor, a first memory direct memory access (DMA) control, and a first IO DMA controller,

wherein the first cache memory comprises a first solid state memory module and a first volatile memory module,

wherein the second bridge-switch comprises a second cache memory, a second processor, a second memory DMA control, and a second IO DMA controller,

wherein the second cache memory comprises a second solid state memory module and a second volatile memory module,

wherein the third bridge-switch comprises a third cache memory, a third processor, a third memory DMA control, and a third IO DMA controller,

wherein the third cache memory comprises a third solid state memory module and a third volatile memory module,

wherein the first host and the first IO storage device are both directly connected to the first bridge-switch,

wherein the second host and the second IO storage device are both directly connected to the second bridge-switch,

wherein the third IO storage device is directly connected to the third bridge-switch,

wherein the first bridge-switch is directly connected to the second bridge-switch and is directly connected to the third bridge-switch,

wherein the second bridge-switch is directly connected to the first bridge-switch and is directly connected to the third bridge-switch,

wherein the first host reads IO data from the third IO storage device via the third bridge-switch and the first bridge-switch, and/or wherein the first host writes IO data to the third IO storage device via the first bridge-switch and the third bridge-switch,

wherein the IO data is cached in the first cache memory and the third cache memory after the IO data is read from the third IO storage device or before the IO data is written to the third IO storage device,

wherein the second host reads the IO data, via the second bridge-switch, from the first cache memory or the third cache memory,

wherein the IO data is cached in the second cache memory after the IO data is read from the first cache memory or third cache memory by the second host so that the first host and the second host can read the IO data from the second cache memory,

wherein the network permits caching of the IO data that is distributed from the third IO storage device to at least three of the bridges-switches within the network so that the caching of the IO data in cache memories in the network results in high-performance and high-bandwidth cache operations that enhance data throughput, transfer rates, and cache hits.

2. The apparatus of claim 1, wherein one of the cache memories comprises a multi-level cache.

3. The apparatus of claim 1, wherein one of the cache memories comprises the volatile memory module and a plurality of solid state memory modules.

4. The apparatus of claim 1 wherein one of the cache memories comprises a plurality of volatile memory modules.

5. The apparatus of claim 1 further comprising another bridge-switch.

6. The apparatus of claim 1 wherein the first bridge-switch comprises cache control information and cache data lines.

7. The apparatus of claim 1,

wherein the third switch-bridge splits read data from the third IO storage device into a first data part that is cached in the first bridge-switch and a second data part that is cached in the second switch-bridge.

8. The apparatus of claim 7,

wherein the first switch-bridge sends a request to the second switch-bridge to transmit the second data part.

9. The apparatus of claim 7,

wherein the third switch-bridge instructs the second switch-bridge to transmit the second data part to the first bridge-switch.

10. The apparatus of claim 1,

wherein the first switch-bridge performs a write transaction to the second switch-bridge, and wherein the write transaction includes flush data that is cached by the second switch-bridge.

11. The apparatus of claim 10, wherein the first switch-bridge splits the flush data into a first data part and a second data part, wherein the first data part transmits along a first path from the first switch-bridge to the second switch-bridge, and the second data part transmits along a second path from the first switch-bridge to the second switch-bridge.

12. The apparatus of claim 1, wherein at least one of the switch-bridges is configured to receive a read data request to search a cache table to determine if a requested data is cached, and to return the requested data if the requested data is cached;

wherein at least one of the switch-bridges is configured to propagate the read request to an actual target if the requested data is not cached in the at least one of the switch-bridges and to cache the requested data before forwarding the requested data to a source of the read data request; and

wherein at least one of the switch-bridges is configured to search the cache table for a target location in response to write data request, to replace a cached copy with write data, and to inform at least another one of the bridges-switches having the cached copy of the write data.

13. A method comprising:

caching data that will be distributed within a network comprising;

a first input-output (IO) device comprising a first IO storage device, a second IO device comprising a second IO storage device, and a third IO device comprising a third IO storage device, a first host and a second host, and a plurality of bridges-switches comprising a first bridge-switch, a second bridge-switch, and a third bridge-switch, wherein the first bridge-switch comprises a first cache memory, a first processor, a first memory direct memory access (DMA) control, and a first IO DMA controller,

wherein the first cache memory comprises a first solid state memory module and a first volatile memory module,

wherein the second bridge-switch comprises a second cache memory, a second processor, a second memory DMA control, and a second IO DMA controller,

wherein the second cache memory comprises a second solid state memory module and a second volatile memory module,

wherein the third bridge-switch comprises a third cache memory, a third processor, a third memory DMA control, and a third IO DMA controller,

wherein the third cache memory comprises a third solid state memory module and a third volatile memory module,

wherein the first host and the first IO storage device are both directly connected to the first bridge-switch,

wherein the second host and the second IO storage device are both directly connected to the second bridge-switch,

wherein the third IO storage device is directly connected to the third bridge-switch,

wherein the first bridge-switch is directly connected to the second bridge-switch and is directly connected to the third bridge-switch,

wherein the second bridge-switch is directly connected to the first bridge-switch and is directly connected to the third bridge-switch,

the method further comprising;

reading, by the first host, IO data from the third IO storage device via the third bridge-switch and the first bridge-switch, and/or writing, by the first host, IO data to the third IO storage device via the first bridge-switch and the third bridge-switch,

caching the IO data in the first cache memory and the third cache memory after the IO data is read from the third IO storage device or before the IO data is written to the third IO storage device,

reading, by the second host, the IO data, via the second bridge-switch, from the first cache memory or the third cache memory,

caching the IO data in the second cache memory after the IO data is read from the first cache memory or third cache memory by the second host so that the first host and the second host can read the IO data from the second cache memory,

wherein the network permits caching of the IO data that is distributed from the third IO storage device to at least three of the bridges-switches within the network so that the caching of the IO data in cache memories in the network results in high-performance and high-bandwidth cache operations that enhance data throughput, transfer rates, and cache hits.

14. The method of claim 13, further comprising:

receiving a read data request to search a cache table to determine if a requested IO data is stored in the cache memories, and returning the requested IO data if the requested IO data is stored in one of the cache memories;

propagating the read request to an actual target if the requested IO data is not stored in one of the cache memories and caching the requested IO data before forwarding the requested IO data to a source of the read data request; and

searching the cache table for a target location in response to a write data request, replacing a cached copy with write data, and informing at least another one of the bridges-switches having the cached copy of the write data.

15. The method of claim 13 wherein one of the cache memories comprises a multi-level cache.

16. The method of claim 13, wherein one of the cache memories comprises the volatile memory module and a plurality of solid state memory modules.

17. The method of claim 13 wherein one of the cache memories comprises a plurality of volatile memory modules.

18. The method of claim 13 further comprising another bridge-switch.

19. The method of claim 13,

wherein the third switch-bridge splits read data from the third IO storage device into a first data part that will be cached in the first bridge-switch and a second data part that will be cached in the second switch-bridge.

20. An article of manufacture, comprising:

a non-transitory computer-readable medium having stored thereon instructions operable to permit an apparatus to;

cache data that is distributed within a network comprising;

a first input-output (IO) device comprising a first IO storage device, a second IO device comprising a second IO storage device, and a third IO device comprising a third IO storage device, a first host and a second host, and a plurality of bridges-switches comprising a first bridge-switch, a second bridge-switch, and a third bridge-switch, wherein the first bridge-switch comprises a first cache memory, a first processor, a first memory direct memory access (DMA) control, and a first IO DMA controller,

wherein the first cache memory comprises a first solid state memory module and a first volatile memory module,

wherein the second bridge-switch comprises a second cache memory, a second processor, a second memory DMA control, and a second IO DMA controller,

wherein the second cache memory comprises a second solid state memory module and a second volatile memory module,

wherein the third bridge-switch comprises a third cache memory, a third processor, a third memory DMA control, and a third IO DMA controller,

wherein the third cache memory comprises a third solid state memory module and a third volatile memory module,

wherein the first host and the first IO storage device are both directly connected to the first bridge-switch,

wherein the second host and the second IO storage device are both directly connected to the second bridge-switch,

wherein the third IO storage device is directly connected to the third bridge-switch,

wherein the first bridge-switch is directly connected to the second bridge-switch and is directly connected to the third bridge-switch,

wherein the second bridge-switch is directly connected to the first bridge-switch and is directly connected to the third bridge-switch,

read, by the first host, IO data from the third IO storage device via the third bridge-switch and the first bridge-switch, and/or writing, by the first host, IO data to the third IO storage device via the first bridge-switch and the third bridge-switch,

cache the IO data in the first cache memory and the third cache memory after the IO data is read from the third IO storage device or before the IO data is written to the third IO storage device,

read, by the second host, the IO data, via the second bridge-switch, from the first cache memory or the third cache memory,

cache the IO data in the second cache memory after the IO data is read from the first cache memory or third cache memory by the second host so that the first host and the second host can read the IO data from the second cache memory,

wherein the network permits caching of the IO data that is distributed from the third IO storage device to at least three of the bridges-switches within the network so that the caching of the IO data in cache memories in the network results in high-performance and high-bandwidth cache operations that enhance data throughput, transfer rates, and cache hits.