Multi-channel memory architecture for daisy chained arrangements of nodes with bridging between memory channels

• US 20060095592A1
• Filed: 10/29/2004
• Published: 05/04/2006
• Est. Priority Date: 10/29/2004
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: International Business Machines Corporation
• Inventors: Borkenhagen, John Michael
• Granted Patent: US 7,334,070 B2
• Time in Patent Office: Days
• US Class Current: 710/2
• CPC Class Codes: G06F 13/1673 using buffers
• CPC Class Codes: G06F 13/1684 using multiple buses

Abstract

Multiple memory channels of a multi-channel memory architecture are effectively bridged together to enable data traffic associated with various nodes in daisy chain arrangement to be communicated over both memory channels. For example, a daisy chain arrangement of nodes, such as FB-DIMM memory modules disposed in a first memory channel may be coupled to a second memory channel, with support for communicating data associated with one of the nodes over either or both of the first and second memory channels.

Claims

1. An apparatus, comprising:

a memory controller including first and second memory ports respectively configured to drive first and second memory channels;

a first plurality of Fully Buffered Dual Inline Memory Module (FB-DIMM) memory modules coupled to the first memory channel of the memory controller in a first daisy chain arrangement, each FB-DIMM memory module in the first plurality of FB-DIMM memory modules including a buffer device and a plurality of DRAM devices, and a last FB-DIMM memory module among the first plurality of FB-DIMM memory modules being disposed at an opposite end of the first daisy chain arrangement from the memory controller;

a second plurality of Fully Buffered Dual Inline Memory Module (FB-DIMM) memory modules coupled to the second memory channel of the memory controller in a second daisy chain arrangement, each FB-DIMM memory module in the second plurality of FB-DIMM memory modules including a buffer device and a plurality of DRAM devices, and a last FB-DIMM memory module among the second plurality of FB-DIMM memory modules being disposed at an opposite end of the second daisy chain arrangement from the memory controller; and

a bridging interconnect coupling together the last FB-DIMM memory modules from the first and second pluralities of FB-DIMM memory modules to enable data to be communicated over one of the first and second memory channels between the memory controller and a FB-DIMM memory module coupled to the other of the first and second memory channels.

2. A circuit arrangement for use in a multi-channel memory system of the type including first and second memory channels, wherein each memory channel is configured to couple a plurality of nodes to one another in a daisy chain arrangement, the circuit arrangement comprising:

a memory port configured to be coupled to the first memory channel; and

a control circuit coupled to the memory port and configured to communicate data associated with a node in the second memory channel through the memory port and over the first memory channel.

3. The circuit arrangement of claim 2, further comprising a second memory port configured to be coupled to the second memory channel, wherein the control circuit is coupled to the second memory port and configured to drive each of the first and second memory channels through the respective first and second memory ports.

4. The circuit arrangement of claim 3, wherein the control circuit is further configured to communicate data associated with a node in the first memory channel through the second memory port and over the second memory channel.

5. The circuit arrangement of claim 3, wherein the first and second memory ports are respectively configured to be coupled to first and second daisy chain arrangements of memory modules, and wherein the control circuit is configured to communicate data associated with a node in the second memory channel over the first memory channel by routing data between the first and second memory channels using a bridging interconnect coupling together last memory modules from each of the first and second daisy chain arrangements.

6. The circuit arrangement of claim 5, wherein the first and second memory ports are respectively configured to be coupled to first and second daisy chain arrangements of FB-DIMM memory modules.

7. The circuit arrangement of claim 5, wherein the control circuit is configured to load balance data traffic associated with nodes in the second memory channel by selectively directing first and second portions of the data traffic over the first and second memory ports.

8. The circuit arrangement of claim 5, wherein the control circuit is configured to communicate with a memory module disposed in the second memory channel but isolated from the second memory port by a discontinuity in the second daisy chain arrangement by directing data through the first memory port and over the first data channel and bridging interconnect.

9. The circuit arrangement of claim 5, wherein each of the first and second memory ports comprises a pair of unidirectional serial read and write ports, wherein each of the first and second memory channels comprises a read channel and a write channel, and wherein the bridging interconnect comprises first and second first unidirectional interconnects, the first unidirectional interconnect coupling the read channel of the first memory channel to the write channel of the second memory channel, and the second unidirectional interconnect coupling the write channel of the first memory channel to the read channel of the second memory channel.

10. The circuit arrangement of claim 2, wherein the memory port includes a read channel output port and a write channel input port configured to couple to a preceding node in a daisy chain arrangement disposed in the first memory channel, the circuit arrangement further comprising a read channel input port and a write channel output port configured to couple to a succeeding node in the daisy chain arrangement, and wherein the read channel input port and the write channel output port are configured to be coupled to a node in the second memory channel via a bridging interconnect when the read channel input port and the write channel output port are disposed in a last node in the daisy chain arrangement.

11. The circuit arrangement of claim 10, wherein the second memory channel includes a read channel and a write channel, wherein the read channel input port is configured to be coupled to the write channel of the second memory channel when disposed in a last node in the daisy chain arrangement, and wherein the write channel output port is configured to be coupled to the read channel of the second memory channel when disposed in a last node in the daisy chain arrangement.

12. The circuit arrangement of claim 11, wherein each of the read and write channels is unidirectional.

13. The circuit arrangement of claim 10, wherein the memory port and the control circuit are disposed in a memory module, and wherein the control circuit is configured to monitor both of the read and write channel input ports for incoming data associated with the memory module, and to route outgoing data associated with the memory module over both of the read and write channel output ports.

14. The circuit arrangement of claim 10, wherein the memory module is an FB-DIMM memory module, and wherein the control circuit is disposed in a buffer integrated circuit device disposed in the FB-DIMM memory module.

15. An integrated circuit device comprising the circuit arrangement of claim 2.

16. A memory controller comprising the circuit arrangement of claim 2.

17. A memory module, comprising:

a plurality of memory storage devices; and

an integrated circuit device including the circuit arrangement of claim 2.

18. The memory module of claim 17, wherein the integrated circuit device comprises a buffer.

19. The memory module of claim 17, wherein the memory module is an FB-DIMM module.

20. A program product, comprising a hardware definition program that defines the circuit arrangement of claim 2;

and a computer readable signal bearing medium bearing the hardware definition program, wherein the signal bearing medium includes at least one of a transmission type media and a recordable media.

21. An apparatus, comprising:

a memory controller including first and second memory ports respectively configured to drive first and second memory channels;

a daisy chain arrangement of memory modules disposed in the first memory channel and coupled at a first end to the first memory port; and

a bridging interconnect coupled between the second memory port and a second end of the daisy chain arrangement of memory modules to enable the memory controller to communicate data associated with a memory module in the daisy chain arrangement over the second memory channel.

22. The apparatus of claim 21, further comprising a second daisy chain arrangement of memory modules disposed in the second memory channel and coupled at a first end to the second memory port, wherein the bridging interconnect is coupled between a second end of the second daisy chain arrangement and the second end of the first daisy chain arrangement.

23. The apparatus of claim 21, wherein each of the first and second memory channels comprises a read channel and a write channel, and wherein the bridging interconnect is configured to couple the read channel of the first memory channel to the write channel of the second memory channel, and to couple the read channel of the second memory channel to the write channel of the first memory channel.

24. The apparatus of claim 23, wherein each memory module includes a control circuit configured to monitor both of the read and write channels in the first memory channel for incoming data associated with such memory module, and to route outgoing data associated with the memory module over both of the read and write channels in the first memory channel.

25. The apparatus of claim 24, wherein each of the read and write channels in the first memory channel is unidirectional.

26. The apparatus of claim 21, wherein the memory controller is configured to load balance data traffic associated with daisy chain arrangement of memory modules by selectively directing first and second portions of the data traffic over the first and second memory channels.

27. The apparatus of claim 21, wherein the memory controller is configured to communicate with a memory module disposed in the first memory channel but isolated from the first memory port by a discontinuity in the first daisy chain arrangement by directing data through the second memory port and over the second data channel and the bridging interconnect.

28. The apparatus of claim 27, wherein the memory controller is configured to detect the discontinuity.

29. The apparatus of claim 28, wherein the discontinuity includes at least one of a faulty memory module and a faulty interconnect in the first memory channel.

30. The apparatus of claim 21, wherein the memory controller and each memory module are FB-DIMM compatible.

31. A method of communicating data in a multi-channel memory system of the type including a memory controller and first and second memory channels, wherein the first memory channel includes a plurality of nodes coupled to one another in a daisy chain arrangement, the method comprising:

communicating over the first memory channel first data associated with a selected node among the plurality of nodes; and

communicating over the second memory channel second data associated with the selected node.

32. The method of claim 31, further comprising communicating over the second memory channel third data associated with a node disposed in a second daisy chain arrangement of node in the second memory channel.

33. The method of claim 31, wherein communicating the second data includes routing data between the first and second memory channels using a bridging interconnect coupling together last nodes from each of the first and second daisy chain arrangements.

34. The method of claim 33, wherein each node comprises an FB-DIMM memory module.

35. The method of claim 31, further comprising load balancing data traffic associated with nodes in the first daisy chain arrangement by selectively directing first and second portions of the data traffic over the first and second memory channels.

36. The method of claim 31, wherein communicating the second data occurs while the selected node is isolated from the memory controller over the first memory channel by a discontinuity in the first daisy chain arrangement.

37. A method of replacing a memory module in a memory architecture of the type comprising a plurality of memory modules arranged in a daisy chain arrangement and interconnected to a memory controller and to one another by a first memory channel, the method comprising:

communicating read and write data between the memory controller and the plurality of memory modules over the first memory channel;

powering down a selected memory module among the plurality of memory modules to enable replacement of the selected memory module with a replacement memory module;

while the selected memory module is powered down, communicating read and write data between the memory controller and at least one memory module disposed downstream of the selected memory module in the daisy chain arrangement over a second memory channel and a bridging interconnect that bridges the first and second memory channels; and

after replacement of the selected memory module with the replacement memory module, applying power to the replacement memory module.