Communicating Between Host Computers and Peripheral Resources in an Input/Output (I/O) Virtualization System

• US 20110055433A1
• Filed: 05/31/2010
• Published: 03/03/2011
• Est. Priority Date: 08/18/2009
• Status: Active Grant

• Current Assignee: SanDisk Technologies LLC (Western Digital Corporation)
• Original Assignee: Fusion-IO Incorporated (Western Digital Corporation)
• Inventors: Malwankar, Kiron, Kishore, Karagada R., Kirkpatrick, Peter E.
• Granted Patent: US 8,521,915 B2
• Time in Patent Office: Days
• US Class Current: 710/9
• CPC Class Codes: G06F 13/4022 using switching circuits, e.g. switching matrix, connection or expansion network G06F13/4009 takes precedence
• CPC Class Codes: G06F 2213/0026 PCI express
• CPC Class Codes: G06F 2213/0058 Bus-related hardware virtualisation

Abstract

An Input/Output (IO) Virtualization (IOV) system provides for sharing of computer peripheral devices between multiple host computers by presenting a single device multiple times to numerous host systems. The IOV system, in coupling or connecting multiple host computers and multiple IO devices, provides IO virtualization and host-to-host communication services to the host computers. The system comprises device interfaces coupled to IO devices, and host interfaces coupled to each of a number of host computers. The IO devices are initialized in a first domain. Each host interface exposes functions of the independent IO devices to the host computer to which it is coupled. Each host computer accesses functions from a host domain that is an independent domain specific to the host computer performing the access. The first domain is different from the host domain.

Claims

1. A system comprising:

a management central processor unit (MCPU) coupled to transport fabric;

a plurality of device interfaces coupled to the transport fabric and to a plurality of independent input/output (IO) devices, the plurality of independent IO devices initialized in an MCPU domain; and

a plurality of host interfaces coupled to the transport fabric, wherein each host interface couples to a host computer of a plurality of independent host computers and exposes functions of the plurality of independent IO devices to the host computer, each host computer of the plurality of independent host computers accessing the functions from a host domain that is an independent domain specific to the host computer performing the access, wherein the MCPU domain is different from the host domain.

2. The system of claim 1, wherein the plurality of independent IO devices comprise a plurality of endpoint functions (EPFs) shared across the plurality of independent host computers.

3. The system of claim 2, the MCPU controlling transfer of data between the plurality of independent host computers and the plurality of independent IO devices by managing source addresses of a host interface proxy of each host interface and destination addresses of each EPF of each IO device.

4. The system of claim 3, comprising receiving a data packet at a host interface, the data packet including a data payload.

5. The system of claim 4, comprising identifying a source address of the host interface proxy of the host interface.

6. The system of claim 5, comprising identifying a destination address of a device interface corresponding to an EPF to which the data packet will be routed.

7. The system of claim 6, wherein the destination address of the device interface is an Ethernet destination address.

8. The system of claim 6, comprising identifying an EPF identifier corresponding to the EPF.

9. The system of claim 8, comprising generating a data capsule by adding a header to the data payload of the data packet, the header comprising the EPF identifier.

10. The system of claim 9, comprising, using a combination of the destination address and the header, transferring the data capsule to the EPF.

11. The system of claim 9, comprising, using the destination address, transferring the data capsule to the device interface via the transport fabric.

12. The system of claim 9, comprising, using the header, transferring the data capsule to the EPF.

13. The system of claim 2, the MCPU controlling transfer of data between the plurality of independent host computers and the plurality of independent IO devices by managing source addresses of an EPF of each IO device and destination addresses of a host interface proxy of each host interface.

14. The system of claim 13, comprising receiving a data packet at a device interface of an IO device that includes an EPF, the data packet including a data payload, wherein the data is addressed to a host computer.

15. The system of claim 14, comprising identifying a source address of the device interface corresponding to the EPF.

16. The system of claim 15, comprising identifying a destination address of a host interface corresponding to a host interface proxy of a host computer to which the data packet will be routed.

17. The system of claim 16, wherein the destination address of the host interface is an Ethernet destination address.

18. The system of claim 16, comprising identifying a proxy identifier corresponding to the host interface proxy.

19. The system of claim 18, comprising generating a data capsule by adding a header to the data payload of the data packet, the header comprising the proxy identifier.

20. The system of claim 19, comprising, using a combination of the destination address and the header, transferring the data capsule to the host computer.

21. The system of claim 19, comprising, using the destination address, transferring the data capsule to the host interface via the transport fabric.

22. The system of claim 19, comprising, using the header, transferring the data capsule to the host interface proxy.

23. The system of claim 2, each host interface comprising a proxy corresponding to each EPF reserved for use by a corresponding host computer corresponding to the host interface.

24. The system of claim 23, comprising translating parameters between the MCPU domain and the host domain.

25. The system of claim 24, wherein the parameter is a sender identification.

26. The system of claim 25, wherein the MCPU discovers and enumerates the plurality of independent IO devices in the MCPU domain.

27. The system of claim 26, comprising assigning to each EPF of the plurality of EPFs a first bus-device-function (BDF) number in the MCPU domain, wherein the first BDF number is unique to the MCPU domain.

28. The system of claim 27, wherein the corresponding host computer discovers and enumerates the proxy in the host domain using a second BDF number, wherein the second BDF number is unique to the host domain and is different than the first BDF number.

29. The system of claim 28, comprising translating the BDF number between the MCPU domain and the host domain during accesses to and from the EPF.

30. The system of claim 29, comprising translating the BDF number in the host interface.

31. The system of claim 29, comprising translating the BDF number in the device interface.

32. The system of claim 29, comprising translating the BDF number of the MCPU domain to a BDF number of the host domain during accesses to the EPF.

33. The system of claim 29, comprising translating the BDF number of the host domain to a BDF number of the MCPU domain during accesses from the EPF.

34. The system of claim 24, wherein the parameter is a memory address.

35. The system of claim 34, comprising assigning to each EPF of the plurality of EPFs a first memory range in the MCPU domain, wherein the first memory range is unique to the MCPU domain.

36. The system of claim 35, comprising assigning to each proxy a second memory range in the host domain, wherein the second memory range is unique to the host domain.

37. The system of claim 36, comprising determining a memory range identifier that identifies the second memory range corresponding to a packet received at a proxy of a receiving host interface, wherein the packet is an address-routed packet.

38. The system of claim 37, comprising transferring the memory range identifier to a destination EPF along with the packet.

39. The system of claim 38, comprising, using the memory range identifier, identifying the first memory range in the MCPU domain that corresponds to the second memory range.

40. The system of claim 39, comprising translating the second memory range to the first memory range.

41. The system of claim 40, wherein the translating comprises replacing a plurality of upper bits in the second memory range with corresponding bits from the first memory range.

42. The system of claim 41, wherein the first memory range and the second memory range are determined by base address registers.

43. The system of claim 1, wherein the MCPU domain is a Peripheral Component Interconnect Express (PCIe) domain of the MCPU.

44. The system of claim 1, wherein the host domain is a Peripheral Component Interconnect Express (PCIe) domain of at least one of the plurality of independent host computers.

45. The system of claim 1, wherein the device interfaces and the host interfaces comprise at least one of an encapsulation protocol and a decapsulation protocol for Peripheral Component Interconnect Express (PCIe) traffic to traverse the transport fabric.

46. The system of claim 1, wherein the MCPU discovers and initializes the plurality of independent IO devices.

47. The system of claim 46, wherein the MCPU runs a privileged driver for each IO device of the plurality of independent IO devices, wherein the privileged driver controls the IO device.

48. The system of claim 1, wherein the plurality of independent IO devices include an IO device with one endpoint function (EPF), wherein the MCPU assigns the IO device with one EPF to a single host computer, wherein the single host computer has privileged control over the EPF.

49. The system of claim 1, wherein the plurality of independent IO devices include an IO device with a plurality of endpoint functions (EPFs), wherein the IO device with the plurality of EPFs is shared across a set of host computers of the plurality of independent host computers.

50. The system of claim 1, wherein each host interface comprises a proxy including configuration data.

51. The system of claim 50, wherein the configuration data comprises a bus-device-function (BDF) number in the host domain.

52. The system of claim 50, wherein the configuration data comprises a memory range identifier.

53. The system of claim 50, wherein the configuration data comprises an endpoint function (EPF) identifier of an EPF of at least one IO device reserved for use by the corresponding host computer.

54. The system of claim 50, wherein the configuration data comprises a destination address of a device interface corresponding to an endpoint function (EPF) of at least one IO device reserved for use by the corresponding host computer.

55. The system of claim 50, wherein configuration accesses to the proxy are directed to the MCPU, wherein the MCPU controls configuration of the EPFs and updates the host computer with current configuration status.

56. The system of claim 1, wherein the plurality of independent IO devices are coupled to the transport fabric using a Peripheral Component Interconnect Express (PCIe) interface.

57. A system comprising:

a plurality of device interfaces coupled to transport fabric and to a plurality of independent input/output (IO) devices that are initialized in a first domain; and

a plurality of host interfaces coupled to the transport fabric, wherein each host interface couples to a host computer of a plurality of independent host computers and exposes functions of the plurality of independent IO devices to the host computer, each host computer of the plurality of independent host computers accessing the functions from a host domain that is an independent domain specific to the host computer performing the access, wherein the first domain is different from the host domain.

58. A method comprising:

coupling a plurality of device interfaces to the transport fabric and to a plurality of independent input/output (IO) devices that are initialized in a first domain;

coupling a plurality of host interfaces to the transport fabric by coupling each host interface to a host computer of a plurality of independent host computers; and

exposing functions of the plurality of independent IO devices to the host computer, wherein the exposing comprises each host computer of the plurality of independent host computers accessing the functions from a host domain that is an independent domain specific to the host computer performing the access, wherein the first domain is different from the host domain.

59. The method of claim 58, comprising sharing a plurality of endpoint functions (EPFs) of the plurality of independent IO devices across the plurality of independent host computers.

60. The method of claim 59, comprising controlling transfer of data between the plurality of independent host computers and the plurality of independent IO devices by managing source addresses of a host interface proxy of each host interface and destination addresses of each EPF of each IO device.

61. The method of claim 60, comprising receiving a data packet at a host interface, the data packet including a data payload.

62. The method of claim 61, comprising identifying a source address of the host interface proxy of the host interface.

63. The method of claim 62, comprising identifying a destination address of a device interface corresponding to an EPF to which the data packet will be routed.

64. The method of claim 63, wherein the destination address of the device interface is an Ethernet destination address.

65. The method of claim 63, comprising identifying an EPF identifier corresponding to the EPF.

66. The method of claim 65, comprising generating a data capsule by adding a header to the data payload of the data packet, the header comprising the EPF identifier.

67. The method of claim 66, comprising, using a combination of the destination address and the header, transferring the data capsule to the EPF.

68. The method of claim 66, comprising, using the destination address, transferring the data capsule to the device interface via the transport fabric.

69. The method of claim 66, comprising, using the header, transferring the data capsule to the EPF.

70. The method of claim 59, comprising controlling transfer of data between the plurality of independent host computers and the plurality of independent IO devices by managing source addresses of an EPF of each IO device and destination addresses of a host interface proxy of each host interface.

71. The method of claim 70, comprising receiving a data packet at a device interface of an IO device that includes an EPF, the data packet including a data payload, wherein the data is addressed to a host computer.

72. The method of claim 71, comprising identifying a source address of the device interface corresponding to the EPF.

73. The method of claim 72, comprising identifying a destination address of a host interface corresponding to a host interface proxy of a host computer to which the data packet will be routed.

74. The method of claim 73, wherein the destination address of the host interface is an Ethernet destination address.

75. The method of claim 73, comprising identifying a proxy identifier corresponding to the host interface proxy.

76. The method of claim 75, comprising generating a data capsule by adding a header to the data payload of the data packet, the header comprising the proxy identifier.

77. The method of claim 76, comprising, using a combination of the destination address and the header, transferring the data capsule to the host computer.

78. The method of claim 76, comprising, using the destination address, transferring the data capsule to the host interface.

79. The method of claim 76, comprising, using the header, transferring the data capsule to the host interface proxy.

80. The method of claim 59, each host interface comprising a proxy corresponding to each EPF reserved for use by a corresponding host computer corresponding to the host interface.

81. The method of claim 80, comprising translating parameters between the first domain and the host domain.

82. The method of claim 81, wherein the parameter is a sender identification.

83. The method of claim 82, comprising discovering and enumerating the plurality of independent IO devices in the first domain.

84. The method of claim 83, comprising assigning to each EPF of the plurality of EPFs a first bus-device-function (BDF) number in the first domain, wherein the first BDF number is unique to the first domain.

85. The method of claim 84, wherein the corresponding host computer discovers and enumerates the proxy in the host domain using a second BDF number, wherein the second BDF number is unique to the host domain and is different than the first BDF number.

86. The method of claim 85, comprising translating the BDF number between the first domain and the host domain during accesses to and from the EPF.

87. The method of claim 86, comprising translating the BDF number in the host interface.

88. The method of claim 86, comprising translating the BDF number in the device interface.

89. The method of claim 86, comprising translating the BDF number of the first domain to a BDF number of the host domain during accesses to the EPF.

90. The method of claim 86, comprising translating the BDF number of the host domain to a BDF number of the first domain during accesses from the EPF.

91. The method of claim 81, wherein the parameter is a memory address.

92. The method of claim 91, comprising assigning to each EPF of the plurality of EPFs a first memory range in the first domain, wherein the first memory range is unique to the first domain.

93. The method of claim 92, comprising assigning to each proxy a second memory range in the host domain, wherein the second memory range is unique to the host domain.

94. The method of claim 93, comprising determining a memory range identifier that identifies the second memory range corresponding to a packet received at a proxy of a receiving host interface, wherein the packet is an address-routed packet.

95. The method of claim 94, comprising transferring the memory range identifier to a destination EPF along with the packet.

96. The method of claim 95, comprising, using the memory range identifier, identifying the first memory range in the first domain that corresponds to the second memory range.

97. The method of claim 96, comprising translating the second memory range to the first memory range.

98. The method of claim 97, wherein the translating comprises replacing a plurality of upper bits in the second memory range with corresponding bits from the first memory range.

99. The method of claim 98, wherein the first memory range and the second memory range are determined by base address registers.

100. The method of claim 58, wherein the first domain is a Peripheral Component Interconnect Express (PCIe) domain.

101. The method of claim 58, wherein the host domain is a Peripheral Component Interconnect Express (PCIe) domain of at least one of the plurality of independent host computers.

102. The method of claim 58, wherein the device interfaces and the host interfaces comprise at least one of an encapsulation protocol and a decapsulation protocol for Peripheral Component Interconnect Express (PCIe) traffic.

103. The method of claim 58, comprising discovering and initializing the plurality of independent IO devices.

104. The method of claim 103, comprising running a privileged driver for each IO device of the plurality of independent IO devices, wherein the privileged driver controls the IO device.

105. The method of claim 58, wherein the plurality of independent IO devices include an IO device with one endpoint function (EPF), comprising assigning the IO device with one EPF to a single host computer, wherein the single host computer has privileged control over the EPF.

106. The method of claim 58, wherein the plurality of independent IO devices include an IO device with a plurality of endpoint functions (EPFs), wherein the IO device with the plurality of EPFs is shared across a set of host computers of the plurality of independent host computers.

107. The method of claim 58, wherein each host interface comprises a proxy including configuration data.

108. The method of claim 107, wherein the configuration data comprises a bus-device-function (BDF) number in the host domain.

109. The method of claim 107, wherein the configuration data comprises a memory range identifier.

110. The method of claim 107, wherein the configuration data comprises an endpoint function (EPF) identifier of an EPF of at least one IO device reserved for use by the corresponding host computer.

111. The method of claim 107, wherein the configuration data comprises a destination address of a device interface corresponding to an endpoint function (EPF) of at least one IO device reserved for use by the corresponding host computer.

112. The method of claim 107, comprising controlling configuration of the EPFs and updating the host computer with current configuration status.