Data exchanges among SMP physical partitions and I/O interfaces enterprise servers

• US 7,757,033 B1
• Filed: 07/12/2004
• Issued: 07/13/2010
• Est. Priority Date: 02/13/2004
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Habanero Holdings Incorporated
• Inventors: Mehrotra, Sharad, Nicolaou, Cosmos, Shah, Shreyas B., Lovett, Thomas Dean, Saraiya, Nakul Pratap
• Primary Examiner(s): PHAN, RAYMOND NGAN
• Application Number:
  US10/889,468
• Time in Patent Office: 2,192 Days
• Field of Search: 710 8- 19, 710104-105, 710306-317, 710 39- 41, 709/238, 709248-250, 709230-236
• US Class Current: 710/316
• CPC Class Codes: G06F 13/4022 using switching circuits, e.g. switching matrix, connection or expansion network G06F13/4009 takes precedence
• CPC Class Codes: H04L 49/357 Fibre channel switches
• CPC Class Codes: H04L 49/45 Arrangements for providing or supporting expansion
• CPC Class Codes: H04L 49/602 Multilayer or multiprotocol switching, e.g. IP switching
• CPC Class Codes: H04L 49/70 Virtual switches

Abstract

Pluggable modules communicate via a switch fabric dataplane accessible via a backplane. Various embodiments are comprised of varying numbers and arrangements of the pluggable modules in accordance with a system architecture that provides for provisioning virtual servers and clusters of servers from underlying hardware and software resources. The system architecture is a unifying solution for applications requiring a combination of computation and networking performance. Resources may be pooled, scaled, and reclaimed dynamically for new purposes as requirements change, using dynamic reconfiguration of virtual computing and communication hardware and software.

Claims

1. A method of communicating a packet, the method comprising:

converting a packet into cells within an input/output interface, each cell specifying a destination physical address corresponding to a first physical partition of a physically-partitionable symmetric multiprocessor;

transferring each cell through a switch in accordance with the destination address; and

forming the cells into a reconstructed packet within the first physical partition; and

wherein the physically-partitionable symmetric multiprocessor is programmable as a selected one of a first configuration having one physical partition and a second configuration having a plurality of physical partitions.

2. The method of claim 1, wherein the act of forming the cells into a reconstructed packet includes data transfers according to a direct memory access protocol.

3. The method of claim 1, wherein the input/output interface includes at least one of a network interface and a storage interface.

4. The method of claim 1, wherein the physically-partitionable symmetric multiprocessor is programmed in the first configuration and the first physical partition is the only physical partition.

5. The method of claim 1, wherein the physically-partitionable symmetric multiprocessor is programmed in the second configuration and the first physical partition is one of the plurality of physical partitions.

6. The method of claim 1, wherein the packet has a destination logical address and the destination physical address of each cell corresponding to the packet is based at least in part on the destination logical address of the packet.

7. A server, comprising:

a first input/output interface coupled to exchange a first set of packets with an included first packet-cell converter;

a first physically-partitionable symmetric multiprocessor having at least a first physical partition and coupled to exchange a second set of packets with an included second packet-cell converter;

a cell-based switch having ports coupled to switch cells between the packet-cell converters;

wherein a first packet received from the first input/output interface by the first packet-cell converter is reconstructed within the first physical partition by the second packet-cell converter, the first packet being a member of the first and second sets of packets; and

wherein a second packet received by the second packet-cell converter from a first process executing on the first physical partition is reconstructed within the first input/output interface by the first packet-cell converter, the second packet being a member of the first and second sets of packets.

8. The server of claim 7, wherein the exchange of the second set of packets includes data transfers according to a direct memory access protocol.

9. The server of claim 7, wherein forming the reconstructed first packet includes data transfers according to a direct memory access protocol.

10. The server of claim 7, wherein the first packet-cell converter generates a first subset of the cells and couples them to the switch via a first of the switch ports, each of the cells having a respectively associated destination physical address, the second packet-cell converter receives the first subset of the cells via a second of the switch ports in accordance with the destination physical address of each cell, and the reconstructed first packet is formed by the second packet-cell converter from the cells of the first subset.

11. The server of claim 7, wherein the cells are switched in accordance with a destination physical address respectively associated with each cell.

12. The server of claim 7, wherein the first input/output interface includes at least one of a network interface and a storage interface.

13. The server of claim 7, wherein:

the packets exchanged include packets received by and corresponding packets reconstructed by the packet-cell converters;

each of the packets has a destination logical address;

each packet-cell converter divides each received packet into a corresponding group of the cells that are subsequently coupled to an associated one of the switch ports;

each of the cells has a respective destination physical address based at least in part on the destination logical address of the corresponding packet;

each cell is conveyed via the switch in accordance with its destination physical address; and

each packet-cell converter forms each reconstructed packet from each corresponding group of the cells received from the switch.

14. The server of claim 7, wherein the first packet has a destination logical address and a destination physical address of each cell corresponding to the first packet is based at least in part on the destination logical address of the first packet.

15. The server of claim 13, wherein each destination physical address includes information used to determine the port of the switch from which the respective cell will egress.

16. A server, comprising:

a first input/output interface coupled to exchange a first set of packets with an included first packet-cell converter;

a first physically-partitionable symmetric multiprocessor having at least a first physical partition and coupled to exchange a second set of packets with an included second packet-cell converter;

a cell-based switch having ports coupled to switch cells between the packet-cell converters;

wherein a first packet received from the first input/output interface by the first packet-cell converter is reconstructed within the first physical partition by the second packet-cell converter, the first packet being a member of the first and second sets of packets; and

wherein the first physically-partitionable symmetric multiprocessor includes a second physical partition and a second packet received from the first input/output interface by the first packet-cell converter is reconstructed within the second physical partition by the second packet-cell converter, the second packet being a member of the first and second sets of packets.

17. A server, comprising:

a first input/output interface coupled to exchange a first set of packets with an included first packet-cell converter;

a first physically-partitionable symmetric multiprocessor having at least a first physical partition and coupled to exchange a second set of packets with an included second packet-cell converter;

a cell-based switch having ports coupled to switch cells between the packet-cell converters;

a second input/output interface coupled to exchange a third set of packets with an included third packet-cell converter;

wherein a first packet received from the first input/output interface by the first packet-cell converter is reconstructed within the first physical partition by the second packet-cell converter, the first packet being a member of the first and second sets of packets; and

wherein a second packet received by the third packet-cell converter is reconstructed within the first physical partition by the second packet-cell converter, the second packet being a member of the second and third sets of packets.

18. A server, comprising:

a first input/output interface coupled to exchange a first set of packets with an included first packet-cell converter;

a first physically-partitionable symmetric multiprocessor having at least a first physical partition and coupled to exchange a second set of packets with an included second packet-cell converter;

a cell-based switch having ports coupled to switch cells between the packet-cell converters;

a second physically-partitionable symmetric multiprocessor having at least a second physical partition and coupled to exchange a third set of packets with an included third packet-cell converter;

wherein a first packet received from the first input/output interface by the first packet-cell converter is reconstructed within the first physical partition by the second packet-cell converter, the first packet being a member of the first and second sets of packets; and

wherein a second packet received from the first input/output interface by the first packet-cell converter is reconstructed within the second physical partition by the third packet-cell converter, the second packet being a member of the first and third sets of packets.

19. The server of claim 18, wherein:

a third packet received by the second packet-cell converter from a first process executing on the first physical partition is reconstructed within the first input/output interface by the first packet-cell converter, the third packet being a member of the first and second sets of packets; and

a fourth packet received by the third packet-cell converter from a second process executing on the second physical partition is reconstructed within the first input/output interface first packet-cell converter, the fourth packet being a member of the first and third sets of packets.

20. A server, comprising:

a first input/output interface coupled to exchange a first set of packets with an included first packet-cell converter;

a first physically-partitionable symmetric multiprocessor having at least a first physical partition and coupled to exchange a second set of packets with an included second packet-cell converter;

a cell-based switch having ports coupled to switch cells between the packet-cell converters;

wherein a first packet received from the first input/output interface by the first packet-cell converter is reconstructed within the first physical partition by the second packet-cell converter, the first packet being a member of the first and second sets of packets; and

wherein the first physically-partitionable symmetric multiprocessor is programmable as a selected one of a first configuration having one physical partition and a second configuration having a plurality of physical partitions.

21. The server of claim 20, wherein the first physically-partitionable symmetric multiprocessor is programmed in the first configuration and the first physical partition is the only physical partition in the first physically-partitionable symmetric multiprocessor.

22. The server of claim 20, wherein the first physically-partitionable symmetric multiprocessor is programmed in the second configuration and the first physical partition is one of the plurality of physical partitions in the first physically-partitionable symmetric multiprocessor.

23. A server, comprising:

a cell-based switch;

a physically-partitionable symmetric multiprocessor having a first and a second physical partition and coupled to exchange packets with an included first packet-cell converter further coupled to exchange cells with the switch;

an input/output interface coupled to exchange packets with an included second packet-cell converter further coupled to exchange cells with the switch;

wherein at least one packet received by the first packet-cell converter from a first process executing on the first physical partition is transferred in cellified form via the switch from and to the first packet-cell converter and reconstructed within the second physical partition; and

wherein at least one packet received by the first packet-cell converter from a second process executing on the first physical partition is reconstructed within the input/output interface.

24. The server of claim 23, wherein the exchange of the packets with the first packet-cell converter and forming the at least one packet reconstructed within the second physical partition include data transfers according to a direct memory access protocol.

25. The server of claim 23, wherein the cells are switched in accordance with a destination physical address respectively associated with each cell.

26. The server of claim 23, wherein the input/output interface includes at least one of a network interface and a storage interface.

27. A server, comprising:

a cell-based switch;

a physically-partitionable symmetric multiprocessor having a first and a second physical partition and coupled to exchange packets with an included first packet-cell converter further coupled to exchange cells with the switch;

an input/output interface coupled to exchange packets with an included second packet-cell converter further coupled to exchange cells with the switch;

wherein at least one packet received by the first packet-cell converter from a first process executing on the first physical partition is transferred in cellified form via the switch from and to the first packet-cell converter and reconstructed within the second physical partition; and

wherein a first and a second packet received by the second packet-cell converter from the input/output interface are respectively reconstructed within the first physical partition and the second physical partition.

28. The server of claim 27, wherein the input/output interface includes at least one of a network interface and a storage interface.

29. The server of claim 27, wherein the cells are switched in accordance with a destination physical address respectively associated with each cell.

30. A method of directing a packet to a process executing on a symmetric multi-processor, the method comprising:

in an input/output interface, receiving a packet having a destination address;

dividing the packet into a plurality of cells wherein each cell has a fabric address specifying an egress port of a switch fabric and wherein each fabric address is based at least in part on the packet destination address;

routing each cell via the switch fabric in accordance with each cell fabric address;

forming the routed cells into a reconstructed packet directly provided to the process executing on the symmetric multi-processor; and

wherein the symmetric multi-processor is programmable as a selected one of a first configuration having one physical partition and a second configuration having a plurality of physical partitions, each of the configurations thereby having at least a first physical partition, and the process executes in the first physical partition.

31. The method of claim 30, wherein:

directly providing the reconstructed packet includes data transfers according to a direct memory access protocol.

32. The method of claim 30, wherein:

the input/output interface includes at least one of a network interface and a storage interface.

33. A method of directing a packet from a process executing on a symmetric multi-processor, the method comprising:

directly receiving from the process a packet having a destination address;

dividing the packet into a plurality of cells wherein each cell has a fabric address specifying an egress port of a switch fabric and wherein each fabric address is based at least in part on the packet destination address;

routing each cell via the switch fabric in accordance with each cell fabric address;

forming the routed cells into a reconstructed packet provided to an input/output interface; and

wherein the symmetric multi-processor is programmable as a selected one of a first configuration having one physical partition and a second configuration having a plurality of physical partitions, each of the configurations thereby having at least a first physical partition, and the process executes in the first physical partition.

34. The method of claim 33, wherein:

directly receiving the packet includes data transfers according to a direct memory access protocol.

35. The method of claim 33, wherein:

the input/output interface includes at least one of a network interface and a storage interface.

36. A method of inter-process communication comprising:

programmatically configuring a symmetric multi-processor as a first physical partition and a second physical partition, the symmetric multi-processor being programmatically configurable as either a single physical partition or a plurality of physical partitions;

directly receiving a packet from a send process executing on the first physical partition, the packet having a destination address;

dividing the packet into a plurality of cells wherein each cell has a fabric address specifying an egress port of a switch fabric and wherein each fabric address is based at least in part on the packet destination address;

routing each cell via the switch fabric in accordance with each cell fabric address; and

forming the routed cells into a reconstructed packet directly provided to a receive process executing on the second physical partition.

37. The method of claim 36, wherein:

directly receiving the packet includes data transfers according to a direct memory access protocol.

38. The method of claim 36, wherein:

directly providing the reconstructed packet includes data transfers according to a direct memory access protocol.

39. A method of performing communication within a server, the method comprising:

in a first compute module, having a first fabric interface connected to a switch fabric and a first symmetric multi-processor connected to the first fabric interface, the first fabric interface directly receiving a first-to-second packet from a first send process executing on the first symmetric multi-processor, the first fabric interface cellifying the first-to-second packet into first-to-second cells and emitting the first-to-second cells to the switch fabric, each cell having a respective fabric address specifying an egress port of the switch fabric, each fabric address being based at least in part on a first-to-second packet destination address included within the first-to-second packet;

in a second compute module, having a second fabric interface connected to the switch fabric and a second symmetric multi-processor connected to the second fabric interface, the second fabric interface directly receiving a second-to-first packet from a second send process executing on the second symmetric multi-processor, the second fabric interface cellifying the second-to-first packet into second-to-first cells and emitting the second-to-first cells to the switch fabric, each cell having a respective fabric address specifying an egress port of the switch fabric, each fabric address being based at least in part on a second-to-first packet destination address included within the second-to-first packet;

in the switch fabric, receiving the first-to-second cells and the second-to-first cells and routing each cell in accordance with its respective fabric address;

in the first compute module, the first compute fabric interface receiving the second-to-first cells, forming the received second-to-first cells into a reconstructed second-to-first packet, and directly providing the reconstructed second-to-first packet to a first receive process executing on the first symmetric multi-processor;

in the second compute module, the second compute fabric interface receiving the first-to-second cells, forming the received first-to-second cells into a reconstructed first-to-second packet, and directly providing the reconstructed first-to-second packet to a second receive process executing on the second symmetric multi-processor; and

wherein each of the first symmetric multi-processor and the second symmetric multi-processor are programmable as a selected one of at least a first configuration having one physical partition and a second configuration having a plurality of physical partitions.

40. The method of claim 39, wherein:

directly receiving the first-to-second packet and the second-to-first packet includes data transfers according to a direct memory access read protocol; and

directly providing the reconstructed first-to-second packet and the reconstructed second-to-first packet includes data transfers according to a direct memory access write protocol.

41. A method of performing communication within a server, and between the server and a plurality of elements, the elements including a first client coupled via a first network, a second client coupled via a second network, a first mass storage array coupled via a first storage network, and a second mass storage array coupled via a second storage network, the server including a switch fabric, the method comprising:

in each of a first network module and a second network module, each network module having a corresponding network interface adapted to couple respectively to the first network and the second network, each network module further having a corresponding network fabric interface connected to the switch fabric, receiving packets from the client coupled to the corresponding network interface via the corresponding network, and providing the packets to the corresponding network fabric interface;

in each of a first storage module and a second storage module, each storage module having a corresponding storage interface adapted to couple respectively to the first storage network and the second storage network, each storage module further having a corresponding storage fabric interface connected to the switch fabric, receiving packets from the storage array coupled to the corresponding storage interface via the corresponding storage network, and providing the packets to the corresponding storage fabric interface;

in each of a first compute module and a second compute module, each compute module having a corresponding compute fabric interface connected to the switch fabric and a corresponding symmetric multi-processor connected to the corresponding compute fabric interface, directly receiving packets from a corresponding send process executing on the corresponding symmetric multi-processor, and providing the packets to the corresponding compute fabric interface;

in each fabric interface, cellyfing the provided packets into cells, and emitting the cells to the switch fabric, each cell having a respective fabric address specifying an egress port of the switch fabric, each fabric address being based at least in part on a destination address included within the corresponding packet;

in the switch fabric, receiving cells and routing each cell in accordance with its respective fabric address;

in each network module, the corresponding network fabric interface receiving cells from the switch fabric, forming the received cells into reconstructed packets, and sending the reconstructed packets to the corresponding client via the corresponding network interface;

in each storage module, the corresponding storage fabric interface receiving cells from the switch fabric, forming the received cells into reconstructed packets, and sending the reconstructed packets to the corresponding storage array via the corresponding storage interface;

in each compute module, the corresponding compute fabric interface receiving cells from the switch fabric, forming the received cells into reconstructed packets, and directly providing the reconstructed packets to a receive process executing on the corresponding symmetric multi-processor; and

wherein each of the symmetric multi-processors are programmable as a selected one of at least a first configuration having one physical partition and a second configuration having a plurality of physical partitions.

42. The method of claim 41, wherein:

the packets received from the send process executing on the symmetric multi-processor of the first compute module include at least a first packet and a second packet having respective destination addresses corresponding to a first recipient and a different second recipient, each recipient being selected from the group consisting of the first compute module, the second compute module, the first network module, the second network module, the first storage module, and the second storage module.

43. The method of claim 41, wherein:

the packets provided by the first network module to the first network fabric interface include at least a first packet and a second packet having respective destination addresses corresponding to a first recipient and a different second recipient, each recipient being selected from the group consisting of the first compute module, the second compute module, the first network module, the second network module, the first storage module, and the second storage module.

44. The method of claim 41, wherein:

the packets provided by the first storage module to the first storage fabric interface include at least a first packet and a second packet having respective destination addresses corresponding to a first recipient and a different second recipient, each recipient being selected from the group consisting of the first compute module, the second compute module, the first network module, the second network module, the first storage module, and the second storage module.

45. A system including:

a switch fabric;

an input/output interface;

a symmetric multi-processor being programmable as a selected one of at least a first configuration having one physical partition and a second configuration having a plurality of physical partitions, at least a first of the physical partitions selectively executing a first process;

the input/output interface and the symmetric multi-processor being coupled to the switch fabric via respective fabric interfaces and whereby the input/output interface and the first process selectively exchange data by transmitting and receiving packets over the switch fabric via the fabric interfaces;

wherein each fabric interface has a transmit data path and a receive data path, the transmit data path including logic stages performing cellification and encapsulation of the packet data prior to transmitting, the receive data path including logic stages performing complementary operations to retrieve the packet data subsequent to receiving; and

wherein the packets transmitted to and received from the input/output interface by the first process are transferred between the first physical partition and the data paths of the symmetric multi-processor fabric interface via a dedicated packet interface directly coupling the symmetric multi-processor to the symmetric multi-processor fabric interface.

46. The system of claim 45, further including:

a mass storage device coupled to the input/output interface.

47. The system of claim 45, further including:

a network coupled to the input/output interface.

48. The system of claim 45, further including:

a mass storage device coupled to the input/output interface; and

a network coupled to the input/output interface.

49. The system of claim 45, wherein:

the symmetric multi-processor is programmed in the second configuration, and at least a second of the physical partitions selectively executing a second process; and

the input/output interface and the second process selectively exchange data by transmitting and receiving packets over the switch fabric via the fabric interfaces.

50. The system of claim 45, wherein:

the first physical partition includes a memory buffer accessible by the first process, the memory buffer being further accessible by the packet interface; and

the packet interface selectively reads the packet data from the memory buffer and selectively writes the packet data to the memory buffer.

51. The system of claim 45, wherein:

the transmit data path encapsulation includes providing a fabric address based in part on a packet destination address included within the packet; and

the switch fabric receives the fabric address and routes the cellified and encapsulated packet data according to the fabric address.

52. The system of claim 47, wherein:

the network includes at least one of a storage area network, a local area network, a metro area network, a wide area network, and the Internet.

53. A system including:

a switch fabric;

a network interface;

a storage interface;

a first symmetric multi-processor and a second symmetric multi-processor, each being programmable as a selected one of at least a first configuration having one physical partition and a second configuration having a plurality of physical partitions, at least a first of the physical partitions of the first symmetric multi-processor selectively executing a first process, and at least a first of the physical partitions of the second symmetric multi-processor selectively executing a second process;

the network interface, the storage interface, the first symmetric multi-processor, and the second symmetric multi-processor being coupled to the switch fabric via respective fabric interfaces and whereby the network interface, the storage interface, the first process, and the second process selectively exchange data by transmitting and receiving packets over the switch fabric via the fabric interfaces;

wherein each fabric interface has a transmit data path and a receive data path, the transmit data path including logic stages performing cellification and encapsulation of the packet data prior to transmitting, the receive data path including logic stages performing complementary operations to retrieve the packet data subsequent to receiving; and

wherein the packets transmitted to and received from the input/output interface by the first process are transferred between the first physical partition and the data paths of the symmetric multi-processor fabric interface via a dedicated packet interface directly coupling the symmetric multi-processor to the symmetric multi-processor fabric interface.

54. The system of claim 53, further including:

a mass storage device coupled to the storage interface.

55. The system of claim 53, further including:

a network coupled to the network interface.

56. The system of claim 53, further including:

a mass storage device coupled to the storage interface and

a network coupled to the network interface.

57. The system of claim 55, the system further including:

a client coupled to the network; and

wherein

the first process includes an application process executed on behalf of the client, the application process producing and consuming application data, and

the client selectively exchanges the application data with the first process via the network, the network interface, the fabric interface coupled to the network interface, the switch fabric, and the fabric interface coupled to the first symmetric multi-processor.

58. The system of claim 55, wherein:

the network includes at least one of a storage area network, a local area network, a metro area network, a wide area network, and the Internet.