Liquid submerged, horizontal computer server rack and systems and method of cooling such a server rack

• US 10,123,463 B2
• Filed: 08/10/2009
• Issued: 11/06/2018
• Est. Priority Date: 08/11/2008
• Status: Active Grant

• Current Assignee: Green Revolution Cooling, Inc.
• Original Assignee: Green Revolution Cooling, Inc.
• Inventors: Best, Christiaan Scott, Garnett, Mark
• Primary Examiner(s): Zerphey, Christopher R
• Application Number:
  US13/057,881
• Publication Number:
  US 20110132579A1
• Time in Patent Office: 3,375 Days
• Field of Search: 622592, 62263, 62434, 62435, 361699, 16510433
• CPC Class Codes: F28D 15/00 Heat-exchange apparatus with the intermediate heat-transfer medium in closed tubes passing into or through the conduit walls ; Heat-exchange apparatus employing intermediate heat-transfer medium or bodies F28D17/00, F28D19/00, F28D20/00 take precedence
• CPC Class Codes: G06F 1/20 Cooling means
• CPC Class Codes: G06F 1/206 comprising thermal management
• CPC Class Codes: G06F 2200/201 Cooling arrangements using cooling fluid
• CPC Class Codes: H01L 2924/00 Indexing scheme for arrangements or methods for connecting or disconnecting semiconductor or solid-state bodies as covered by H01L24/00
• CPC Class Codes: H01L 2924/0002 Not covered by any one of groups H01L24/00, H01L24/00 and H01L2224/00
• CPC Class Codes: H05K 7/20 Modifications to facilitate cooling, ventilating, or heating
• CPC Class Codes: H05K 7/20236 by immersion
• CPC Class Codes: H05K 7/20281 Thermal management, e.g. liquid flow control
• CPC Class Codes: H05K 7/203 by immersion
• CPC Class Codes: H05K 7/20327 Accessories for moving fluid, for connecting fluid conduits, for distributing fluid or for preventing leakage, e.g. pumps, tanks or manifolds
• CPC Class Codes: H05K 7/20381 Thermal management, e.g. evaporation control
• CPC Class Codes: H05K 7/20763 Liquid cooling without phase change
• CPC Class Codes: H05K 7/20772 within server blades for removing heat from heat source
• CPC Class Codes: H05K 7/20781 within cabinets for removing heat from server blades
• CPC Class Codes: H05K 7/2079 within rooms for removing heat from cabinets
• CPC Class Codes: H05K 7/20827 within rooms for removing heat from cabinets, e.g. air conditioning devices
• CPC Class Codes: H05K 7/20836 Thermal management, e.g. server temperature control
• CPC Class Codes: Y10T 29/4973 Replacing of defective part

Abstract

Apparatus, systems, and methods for efficiently cooling computing devices having heat-generating electronic components, such as, for example, independently operable servers, immersed in a dielectric liquid coolant in a tank.

Claims

1. An apparatus for cooling a plurality of rack-mountable servers containing heat generating electronic components, the cooling apparatus comprising:

at least one tank defining an open interior volume and having a coolant inlet for receiving a dielectric liquid coolant within the open interior volume and having a coolant outlet for allowing the dielectric liquid coolant to flow from the open interior volume, the coolant inlet and the coolant outlet being fluidly coupled to each other, wherein the at least one tank comprises an open or openable top sized to receive at least one row comprising a plurality of rack-mountable servers, each of at least two of the plurality of rack-mountable servers in the at least one row comprising a motherboard and heat producing components mounted to the motherboard;

one or more mounting members positioned within the interior volume and configured to mountably receive the plurality of rack-mountable servers within the interior volume, wherein the mounting members are configured to hold each of at least two of the rack-mountable servers in the at least one row in a horizontally stacked relationship with one another, with the rack-mountable servers in a vertical orientation such that one of the two smallest sides of the rack-mountable server faces upward and the motherboard of the rack-mountable server is vertically oriented, wherein at least two of the rack-mountable servers are independently operable from one another, wherein each of at least two of the rack-mountable servers is configured to connect to a computer network;

a volume of dielectric coolant at least partially held in the at least one tank; and

a multi-circuit fluid cooling apparatus,

wherein, when the vertically oriented rack-mountable servers are mountably received in at least one row in the at least one tank, at least a portion of at least two of the vertically oriented rack-mountable servers being mountably received are held in the at least one tank such that the vertically oriented rack-mountable servers are commonly at least partially submerged in the volume of the dielectric liquid coolant in the at least one tank for cooling the vertically oriented rack-mountable servers,

wherein, when the vertically oriented rack-mountable servers are mountably received in the at least one tank and commonly at least partially submerged in the volume of dielectric liquid coolant in the at least one tank, the vertically oriented rack-mountable servers are thermally coupled to a remote heat exchanger via the external fluid circuit, wherein the multi-circuit fluid cooling apparatus comprises;

a dielectric liquid cooling apparatus;

a secondary cooling apparatus comprising the remote heat exchanger; and

a liquid-to-liquid or liquid-to-refrigerant heat exchanger;

wherein the dielectric liquid cooling apparatus includes one or more pumps, and a plurality of fluid velocity augmentation devices in the volume of dielectric liquid coolant in the at least one tank, wherein at least one of the pumps, the coolant inlet, the coolant outlet, and the mounting members are configured such that the at least one pump moves at least a portion of the dielectric liquid coolant vertically across heat producing components on the vertically oriented motherboard of one of the vertically oriented rack-mountable servers in parallel flow with dielectric liquid coolant moved vertically across heat producing components on the vertically oriented motherboard of at least one other of the vertically oriented rack-mountable servers,

wherein each of at least two of the fluid velocity augmentation devices is configured to increase upward fluid velocity of flow of dielectric liquid coolant through the vertically oriented rack-mountable servers or upwardly between adjacent ones of the vertically oriented rack-mountable servers,

wherein the volume of dielectric liquid coolant comprises at least one passage in the tank that is outside of the vertically oriented rack-mountable servers, wherein, when the at least one pump is operated to move the dielectric liquid coolant vertically across the heat producing components on the vertically oriented motherboards, a circuit is formed in which a first portion of dielectric liquid coolant is moved vertically upward across the heat producing components on the vertically oriented motherboards and then downward outside of the rack mountable servers in the at least one passage, while a second portion of the dielectric liquid coolant flows out of the tank and through the liquid-to-liquid or liquid-to-refrigerant heat exchanger,

wherein the liquid-to-liquid or liquid-to-refrigerant heat exchanger is configured to transfer heat from the second portion of dielectric liquid coolant in the circuit to liquid or refrigerant moving in the secondary cooling apparatus,

wherein the secondary cooling apparatus is configured to reject, by way of the remote heat exchanger, at least part of the heat transferred from the second portion of the dielectric liquid coolant,

wherein the vertically oriented rack-mountable servers are mountable in the at least one tank such that, when the vertically oriented rack-mountable servers are commonly at least partially submerged in the volume of dielectric liquid coolant in the at least one tank and the dielectric liquid coolant is moved within the at least one tank, at least one of the vertically oriented rack-mountable servers is independently vertically removable from the volume of dielectric liquid coolant in which the at least two vertically oriented rack-mountable servers are commonly at least partially submerged and from the at least one tank through the top of the tank while at least two of the other vertically oriented rack-mountable servers in the at least one row remain commonly at least partially submerged in the volume of dielectric liquid coolant in the at least one tank and the dielectric liquid coolant in the at least one tank continues to flow across heat producing electronic components of the at least two other vertically oriented rack-mountable servers remaining at least partially commonly submerged in the at least one row.

2. The apparatus of claim 1, wherein:

the one or more mounting members are configured to mountably receive the vertically oriented rack-mountable servers above the bottom of the at least one tank to form a volume between each respective vertically oriented rack-mountable server and the at least one tank to permit the flow of dielectric liquid coolant through the plurality of vertically oriented rack-mountable servers.

3. The apparatus of claim 1, wherein:

the vertically oriented rack-mountable servers have a plurality of rack ears;

the at least one tank comprises a tank which is sized to receive at least one row of the vertically oriented rack-mountable servers; and

the one or more mounting members comprises a plurality of mounting members fixedly attached to the at least one tank and configured to mountably receive the rack ears of the plurality of vertically oriented rack-mountable servers in the at least one row of vertically oriented rack-mountable servers to permit installation and removal of the at least one vertically oriented rack-mountable server without affecting the operational status of the other vertically oriented rack-mountable servers remaining within the at least one tank.

4. The apparatus of claim 1, further comprising:

at least one cable tray to receive and support signal and control network cabling to and from the plurality of vertically oriented rack-mountable servers; and

at least one power distribution unit to receive and support power distribution cables for distributing electrical power to the plurality of vertically oriented rack-mountable servers;

wherein the at least one cable tray and the at least one power distribution unit are mounted in proximity to the open top of the at least one tank so that the at least one vertically oriented rack-mountable server is independently removable from the tank through the open top of the tank without the need to remove the other vertically oriented rack-mountable servers from the tank or disturb the location or operational status of the other vertically oriented rack-mountable servers remaining within the tank.

5. The apparatus of claim 1, wherein at least one of the vertically oriented rack-mountable servers is vertically removable from the at least one tank without the need to disturb the operational status of the other vertically oriented rack-mountable servers remaining within the at least one tank.

6. The apparatus of claim 1, wherein at least one of the vertically oriented rack-mountable servers is vertically removable from the at least one tank without the need for horizontal detachment of the at least one rack-mountable server.

7. The apparatus of claim 1, wherein at least one of the vertically oriented rack-mountable servers comprises an enclosure, wherein flow of the dielectric liquid coolant around the vertically oriented rack-mountable servers is restricted such that flow through the enclosure of the at least one vertically oriented rack-mountable server is enhanced.

8. The apparatus of claim 1, wherein one or more of the vertically oriented rack-mountable servers are not attached to a lid for the apparatus.

9. The apparatus of claim 1, wherein the one or more mounting members comprise:

a first set of one or more mounting members coupled in a fixed relationship to the tank; and

a second set of one or more mounting members coupled in a fixed relationship to the tank,

wherein the first set of one or more mounting members and the second set of mounting members are spaced apart from one another to form at least one opening for receiving a row of two or more of the vertically oriented rack-mountable servers in a vertical orientation within the interior volume, wherein the at least one mounting member in the first set of mounting members supports at least one vertically oriented rack-mountable server of the vertically oriented rack-mountable servers on one side of the row and wherein at least one mounting member in the second set of mounting members supports the at least one vertically oriented rack-mountable server on the other side of the row.

10. The apparatus of claim 1, wherein the plurality of fluid velocity augmentation devices comprise a plurality of nozzles configured to direct dielectric liquid coolant from the coolant inlet piping toward a desired entry point into the vertically oriented rack-mounted servers to augment the fluid velocity of the liquid coolant through the vertically oriented rack-mounted servers.

11. The apparatus of claim 10, further comprising a line from the coolant inlet, wherein the plurality of nozzles are mounted on the line, wherein each of at least two of the plurality of nozzles are configured to direct dielectric liquid coolant toward a desired entry point into the vertically oriented rack-mounted servers to augment the fluid velocity of the liquid coolant through the vertically oriented rack-mounted servers.

12. The apparatus of claim 1, wherein at least two of the fluid velocity augmentation devices are mounted in the volume of dielectric liquid coolant between the at least one row of vertically oriented rack-mountable servers and bottom of the at least one tank such that the fluid velocity augmentation devices increase upward flow of the dielectric liquid coolant through or between the plurality of vertically oriented rack-mountable servers, wherein each of at least two of the fluid velocity augmentation devices is configured to increase upward fluid velocity of flow of dielectric liquid coolant in a different part of the volume of dielectric liquid coolant in which the at least two vertically oriented rack mountable servers are commonly at least partially submerged and upwardly through different ones of the vertically oriented rack-mountable servers or upwardly through a gap between different adjacent pairs of the fluid velocity augmentation devices rack-mountable servers.

13. The apparatus of claim 1, wherein at least one of the one or more pumps is configured to move a portion of the dielectric liquid coolant across heat producing electronic components of the other vertically oriented rack-mountable servers while the at least one of the plurality of vertically oriented rack-mountable servers is removed from the volume of dielectric liquid coolant.

14. The apparatus of claim 1, wherein, when the vertically oriented rack-mountable servers are mountably received in the at least one tank and the dielectric liquid coolant is moved within the at least one tank, heated dielectric liquid coolant at the surface of the volume of dielectric liquid coolant is moved crossways relative to the direction of the row of vertically oriented rack-mountable servers.

15. The apparatus of claim 1, further comprising a controller configured to:

monitor one or more operational parameters of cooling of the vertically oriented rack-mountable servers;

determine, based at least in part on at least one of the monitored operational parameters, an elevated temperature of dielectric liquid coolant exiting the vertically oriented rack-mountable servers, wherein the elevated temperature is selected to sufficiently cool each of the vertically oriented rack-mountable servers while reducing the total amount of energy need to cool the vertically oriented rack-mountable servers; and

maintain dielectric liquid coolant exiting the vertically oriented rack-mountable servers at the selected elevated temperature.

16. The apparatus of claim 15, wherein the controller is configured to operate at least one of the one or more pumps to maintain dielectric liquid coolant exiting the vertically oriented rack-mountable servers at a temperature that is significantly higher than comfortable room temperature and lower than the maximum permissible temperature of the most sensitive heat generating electronic component of the vertically oriented rack-mountable servers.

17. The apparatus of claim 1, wherein the liquid-to-liquid heat exchanger or refrigerant-to-liquid heat exchanger is located on the at least one tank.

27. The apparatus of claim 1, further comprising:

a controller separate from the vertically oriented rack-mountable servers for controlling the cooling apparatus, wherein said controller provides instructions for periodically monitoring the temperature of the liquid coolant at at least one location within the first fluid circuit; and

in response to the periodic determination of the temperature, adjusting the amount of heat to be rejected through the remote heat exchanger such that the dielectric liquid coolant is maintained at an elevated temperature in at least a portion of the first fluid circuit between the exits of the vertically oriented rack-mountable servers and the coolant outlet.

28. The apparatus of claim 27, wherein the controller provides further instructions for periodically:

determining the optimum elevated temperature of the heated dielectric liquid coolant as it exits the plurality of vertically oriented rack-mountable servers in order for the dielectric liquid coolant to sufficiently cool the vertically oriented rack-mountable servers while reducing the amount of energy consumed to sufficiently cool each vertically oriented respective server;

determining the amount of energy needed to reject the absorbed heat for cooling the vertically oriented rack-mountable servers; and

in response to the periodic determinations of the optimum elevated temperature and the amount of energy needed to reject the absorbed heat, adjusting the amount of heat to be rejected through the remote heat exchanger in order that the dielectric liquid coolant is maintained at approximately the optimum elevated temperature in the at least a portion of the circuit between the exits of the vertically oriented rack-mountable servers and the coolant outlet.

29. The apparatus of claim 27, wherein the elevated temperature is a temperature in the range of 90 degrees F. and 130 degrees F.

30. The apparatus of claim 27, wherein the elevated temperature is a temperature in the range of 100 degrees F. and 110 degrees F.

31. The apparatus of claim 1,

wherein the circuit includes heated dielectric coolant that has exited the vertically oriented rack-mountable servers at the top of the at least one row of vertically oriented rack-mountable servers,

wherein one part of the heated dielectric liquid coolant that has exited the vertically oriented rack-mountable servers at the top of the at least one row of vertically oriented rack-mountable servers recirculates downward in the at least one passage to the bottom of the at least one tank,

wherein another part of the heated dielectric liquid coolant that has exited the vertically oriented rack-mountable servers at the top of the at least one row of vertically oriented rack-mountable servers exits the at least one tank in the second portion of dielectric liquid coolant.

32. The apparatus of claim 1,

wherein heated dielectric liquid coolant exiting the rack-mountable servers collects in a common manifold area above the top of the vertically oriented rack-mountable servers in the at least one row,

wherein one part of the heated dielectric liquid coolant that has collected in the common manifold area recirculates downward in at least one passage to the bottom of the at least one tank, and

wherein another part of the heat heated dielectric coolant that has collected in the common manifold area exits the at least one tank in the second portion of dielectric liquid coolant.

33. The apparatus of claim 1,

wherein the cooling outlet is proximate to surface of the volume of dielectric liquid coolant in the at least one tank, and

wherein at least a portion of the second portion of dielectric liquid coolant exiting the at least one tank is drawn from heated dielectric liquid coolant proximate to the surface of the volume of dielectric liquid coolant.

18. An apparatus for cooling a plurality of rack-mountable servers containing heat generating electronic components in a server room, the cooling apparatus comprising:

at least one tank defining an open interior volume and having a coolant inlet for receiving a dielectric liquid coolant within the open interior volume and having a coolant outlet for allowing the coolant to flow from the open interior volume, the coolant inlet and the coolant outlet being fluidly coupled to each other, each of at least two of the plurality of rack-mountable servers in the at least one row comprising a motherboard and heat producing components mounted to the motherboard;

one or more mounting members positioned within the interior volume and configured to mountably receive the plurality of rack-mountable servers within the interior volume, wherein the mounting members are configured to hold each of at least two of the rack-mountable servers in the at least one row in a horizontally stacked relationship with one another, with the rack-mountable servers in a vertical orientation such that one of the two smallest sides of the rack-mountable server faces upward and the motherboard of the rack-mountable server is vertically oriented, wherein at least two of the rack-mountable servers are independently operable from one another, wherein each of at least two of the rack-mountable servers is configured to connect to a computer network,

a volume of dielectric coolant at least partially held in the at least one tank; and

a multi-circuit fluid cooling apparatus,

wherein, when the vertically oriented rack-mountable servers are mountably received in at least one row in the at least one tank, at least a portion of at least two of the vertically oriented rack-mountable servers being mountably received are held in the at least one tank such that the vertically oriented rack-mountable servers are commonly submerged in the volume of the dielectric liquid coolant in the at least one tank for cooling the vertically oriented rack-mountable servers,

wherein, when vertically oriented rack-mountable servers are mountably received in the at least one tank and commonly at least partially submerged in the volume of dielectric liquid coolant in the at least one tank, the vertically oriented rack-mountable servers are thermally coupled to a remote heat exchanger outside of a server room in which the at least one tank is located via the multi-circuit fluid cooling apparatus, wherein the multi-circuit fluid cooling apparatus comprises;

a dielectric liquid cooling apparatus;

a secondary cooling apparatus comprising a remote heat exchanger; and

a liquid-to-liquid or liquid-to-refrigerant heat exchanger;

wherein the dielectric liquid cooling apparatus includes one or more pumps, and a plurality of fluid velocity augmentation devices in the volume of dielectric liquid coolant in the at least one tank, wherein at least one of the pumps, the coolant inlet, the coolant outlet, and the mounting members are configured such that the at least one pump moves at least a portion of the dielectric liquid coolant vertically across heat producing components on the vertically oriented motherboard of one of the vertically oriented rack-mountable servers in parallel flow with dielectric liquid coolant moved vertically across heat producing components on the vertically oriented motherboard of at least one other of the vertically oriented rack-mountable servers; and

wherein each of at least two of the fluid velocity augmentation devices is configured to increase upward fluid velocity of flow of dielectric liquid coolant through the vertically oriented rack-mountable servers or upwardly between adjacent ones of the vertically oriented rack-mountable servers,

wherein the volume of dielectric liquid coolant comprises at least one passage in the tank that is outside of the vertically oriented rack-mountable servers, wherein, when the at least one pump is operated to move the dielectric liquid coolant vertically across the heat producing components on the vertically oriented motherboards, a circuit is formed in which a first portion of dielectric liquid coolant is moved vertically upward across the heat producing components on the vertically oriented motherboards and then downward outside of the rack mountable servers in the at least one passage, while a second portion of the dielectric liquid coolant flows out of the tank and through the liquid-to-liquid or liquid-to-refrigerant heat exchanger,

wherein the liquid-to-liquid or liquid-to-refrigerant heat exchanger is configured to transfer heat from the second portion of dielectric liquid coolant in the circuit to liquid or refrigerant moving in the secondary cooling apparatus,

wherein the secondary cooling apparatus is configured to reject, by way of the remote heat exchanger, at least part of the heat transferred from the second portion of the dielectric liquid coolant.

19. The apparatus of claim 18, wherein, when the plurality of vertically oriented rack-mountable servers are mountably received, each respective rack-mountable server is completely submerged within the dielectric liquid coolant such that a volume of dielectric liquid coolant collects in a common manifold area above the plurality of rack-mountable servers to improve the circulation of the liquid coolant through or between the plurality of vertically oriented rack-mountable servers, thereby enhancing the cooling of the vertically oriented rack mountable servers.

20. The apparatus of claim 18, wherein the at least one tank comprises an open or openable top, wherein the vertically oriented rack-mountable servers are mountable such that at least one of the vertically oriented rack-mountable servers can be removed from the at least one tank through the open or openable top while at least two of the other vertically oriented rack-mountable servers remain commonly at least partially submerged in the dielectric liquid coolant and dielectric liquid coolant continues to flow across the at least two other vertically oriented rack-mountable servers.

21. The apparatus of claim 18, further comprising a controller configured to maintain dielectric liquid coolant exiting the vertically oriented rack-mountable servers at a temperature that is significantly higher than comfortable room temperature and lower than the maximum permissible temperature of the most sensitive heat generating electronic component of the vertically oriented rack-mountable servers.

22. An apparatus for cooling a plurality of rack-mountable servers containing heat generating electronic components in a server room, the cooling apparatus comprising:

at least one tank defining an open interior volume and having a coolant inlet for receiving a dielectric liquid coolant within the open interior volume and having a coolant outlet for allowing the coolant to flow from the open interior volume, the coolant inlet and the coolant outlet being fluidly coupled to each other, wherein the at least one tank comprises an open or openable top sized to receive at least one row comprising a plurality of rack-mountable servers, each of at least two of the plurality of rack-mountable servers in the at least one row comprising a motherboard and heat producing components mounted to the motherboard;

one or more mounting members positioned within the interior volume and configured to mountably receive the plurality of rack-mountable servers within the interior volume, wherein the mounting members are configured to hold each of at least two of the rack-mountable servers in the at least one row in a horizontally stacked relationship with one another, with the rack-mountable servers in a vertical orientation such that one of the two smallest sides of the rack-mountable server faces upward and the motherboard of the rack-mountable server is vertically oriented, wherein at least two of the rack-mountable servers are independently operable from one another, wherein each of at least two of the rack-mountable servers is configured to connect to a computer network,

a volume of dielectric coolant at least partially held in the at least one tank; and

a multi-circuit fluid cooling apparatus,

wherein, when the vertically oriented rack-mountable servers are mountably received in at least one row in the at least one tank, at least a portion of at least two of the vertically oriented rack-mountable servers being mountably received is held in the at least one tank such that at least two of the vertically oriented rack-mountable servers can be commonly at least partially submerged in the volume of the dielectric liquid coolant in the at least one tank for cooling the vertically oriented rack-mountable servers,

wherein, when the vertically oriented rack-mountable servers are mountably received in the at least one tank and commonly at least partially submerged in the volume of dielectric liquid coolant in the at least one tank, the vertically oriented rack-mountable servers are thermally coupled to a remote heat exchanger outside of the server room via the multi-circuit fluid cooling apparatus, wherein the multi-circuit fluid cooling apparatus comprises;

a dielectric liquid cooling apparatus; and

a secondary cooling apparatus comprising a remote heat exchanger,

wherein the dielectric liquid cooling apparatus includes one or more pumps, and a plurality of fluid velocity augmentation devices in the volume of dielectric liquid coolant in the at least one tank, wherein at least one of the pumps, the coolant inlet, the coolant outlet, and the mounting members are configured such that the at least one pump moves at least a portion of the dielectric liquid coolant in the volume of dielectric liquid coolant vertically across heat producing components on the vertically oriented motherboard of one of the vertically oriented rack-mountable servers in parallel flow with dielectric liquid coolant moved vertically across heat producing components on the vertically oriented motherboard of at least one other of the vertically oriented rack-mountable servers,

wherein the vertically oriented rack-mountable servers are mountable in the at least one tank such that, when the vertically oriented rack-mountable servers are commonly at least partially submerged in the volume of dielectric liquid coolant in the at least one tank and the dielectric liquid coolant is moved within the at least one tank, at least one of the vertically oriented rack-mountable servers is independently vertically removable from the volume of dielectric liquid coolant in which the at least two vertically oriented rack-mountable servers are commonly at least partially submerged and from the at least one tank through the top of the tank while at least two of the other vertically oriented rack-mountable servers in the at least one row remain commonly at least partially submerged in the volume of dielectric liquid coolant in the at least one tank and the dielectric liquid coolant in the at least one tank continues to flow across heat producing electronic components of the at least two other rack-mountable servers remaining at least partially commonly submerged in the at least one row,

wherein each of at least two of the fluid velocity augmentation devices is configured to increase upward fluid velocity of flow of dielectric liquid coolant through the vertically oriented rack-mountable servers or upwardly between adjacent ones of the vertically oriented rack-mountable servers,

wherein the volume of dielectric liquid coolant comprises at least one passage in the tank that is outside of the vertically oriented rack-mountable servers, wherein, when the at least one pump is operated to move the dielectric liquid coolant vertically across the heat producing components on the vertically oriented motherboards, a circuit is formed in which a first portion of dielectric liquid coolant is moved vertically upward across the heat producing components on the vertically oriented motherboards and then downward outside of the rack mountable servers in the at least one passage, while a second portion of the dielectric liquid coolant flows out of the tank,

wherein the secondary cooling apparatus is configured to reject, by way of the remote heat exchanger, heat transferred from the second portion of the dielectric liquid coolant.

23. The apparatus of claim 22, wherein, when the plurality of rack-mountable servers are mountably received, at least one of the vertically oriented rack-mountable servers is completely submerged within the dielectric liquid coolant such that a volume of dielectric liquid coolant collects in a common manifold area above the at least one vertically oriented rack-mountable server to improve the circulation of the liquid coolant through the vertically oriented rack-mountable servers, thereby enhancing the cooling of the vertically oriented rack-mountable server.

24. The apparatus of claim 22, wherein the at least one tank comprises an open or openable top, wherein the vertically oriented rack-mountable servers are mountable such that at least one of the vertically oriented rack-mountable servers can be removed from the tank through the open or openable top while at least two of the other vertically oriented rack-mountable servers remain commonly at least partially submersed in the dielectric liquid coolant and dielectric liquid coolant continues to flow across the at least two other vertically oriented rack-mountable servers.

25. An apparatus for cooling a plurality of rack-mountable servers containing heat generating electronic components in a server room, the cooling apparatus comprising:

at least one tank defining an open interior volume and having a coolant inlet for receiving a dielectric liquid coolant within the open interior volume and having a coolant outlet for allowing the dielectric liquid coolant to flow from the open interior volume, the coolant inlet and the coolant outlet being fluidly coupled to each other, wherein the at least one tank comprises an open or openable top sized to receive at least one row comprising a plurality of rack-mountable servers, each of at least two of the plurality of rack-mountable servers in the at least one row comprising a motherboard and heat producing components mounted to the motherboard;

one or more mounting members positioned within the interior volume and configured to mountably receive the plurality of rack-mountable servers within the interior volume, wherein the mounting members are configured to hold each of at least two of the rack-mountable servers in the at least one row in a horizontally stacked relationship with one another, with the rack-mountable servers in a vertical orientation such that one of the two smallest sides of the rack-mountable server faces upward and the motherboard of the rack-mountable server is vertically oriented, wherein at least two of the vertically oriented rack-mountable servers are independently operable from one another, wherein each of at least two of the vertically oriented rack-mountable servers is configured to connect to a computer network,

a volume of dielectric coolant at least partially held in the at least one tank; and

a multi-circuit fluid cooling apparatus,

wherein, when the vertically oriented rack-mountable servers are mountably received in at least one row in the at least one tank, at least a portion of at least two of the vertically oriented rack-mountable servers being mountably received is held in the at least one tank such that at least two of the vertically oriented rack-mountable servers can be commonly at least partially submerged in a volume of the dielectric liquid coolant in the at least one tank for cooling the vertically oriented rack-mountable servers,

wherein, when the vertically oriented rack-mountable servers are mountably received in the at least one tank and commonly at least partially submerged in the volume of dielectric liquid coolant in the at least one tank, the plurality of vertically oriented rack-mountable servers are thermally coupled to a remote heat exchanger outside of the server room via the multi-circuit fluid cooling apparatus, wherein the multiple-circuit fluid cooling apparatus comprises;

a dielectric liquid cooling apparatus;

a secondary cooling apparatus comprising a remote heat exchanger; and

a liquid-to-liquid or liquid-to-refrigerant heat exchanger;

wherein the dielectric liquid cooling apparatus including one or more pumps, and a plurality of fluid velocity augmentation devices in the volume of dielectric liquid coolant in the at least one tank, wherein at least one of the pumps, the coolant inlet, the coolant outlet, and the mounting members are configured such that the at least one pump moves at least a portion of the dielectric liquid coolant in the volume of dielectric liquid coolant vertically across heat producing components on the vertically oriented motherboard of one of the vertically oriented rack-mountable servers in parallel flow with dielectric liquid coolant moved vertically across heat producing components on the vertically oriented motherboard of at least one other of the vertically oriented rack-mountable servers,

wherein the vertically oriented rack-mountable servers are mountable in the at least one tank such that, when the vertically oriented rack-mountable servers are commonly at least partially submerged in the volume of dielectric liquid coolant in the at least one tank and the dielectric liquid coolant is moved within the at least one tank, at least one of the vertically oriented rack-mountable servers is independently vertically removable from the volume of dielectric liquid coolant in which the at least two vertically oriented rack-mountable servers are commonly at least partially submerged and from the at least one tank through the top of the tank while at least two of the other vertically oriented rack-mountable servers in the at least one row remain commonly at least partially submerged in the volume of dielectric liquid coolant in the at least one tank and the dielectric liquid coolant in the at least one tank continues to flow across heat producing electronic components of the at least two other vertically oriented rack-mountable servers remaining at least partially commonly submerged in the at least one row,

wherein the one or more mounting members are configured to mountably receive the plurality of vertically oriented rack-mountable servers above the bottom of the at least one tank to form a volume between the plurality of vertically oriented rack-mountable servers and the bottom of the at least one tank in which the dielectric liquid coolant can collect to permit the flow of dielectric liquid coolant through the plurality of vertically oriented rack-mountable servers,

wherein the volume of dielectric liquid coolant comprises at least one passage in the tank that is outside of the vertically oriented rack-mountable servers, wherein, when the at least one pump is operated to move the dielectric liquid coolant vertically across the heat producing components on the vertically oriented motherboards, a circuit is formed in which a first portion of dielectric liquid coolant is moved vertically upward across the heat producing components on the vertically oriented motherboards and then downward outside of the rack mountable servers in the at least one passage, while a second portion of the dielectric liquid coolant flows out of the tank and through the liquid-to-liquid or liquid-to-refrigerant heat exchanger,

wherein the liquid-to-liquid or liquid-to-refrigerant heat exchanger is configured to transfer heat from the second portion of dielectric liquid coolant in the circuit to liquid or refrigerant moving in the secondary cooling apparatus,

wherein the secondary cooling apparatus is configured to reject, by way of the remote heat exchanger, at least part of the heat transferred from the second portion of the dielectric liquid coolant

wherein, when the plurality of rack-mountable servers are mountably received, the plurality of rack-mountable servers are completely submerged within the dielectric liquid coolant such that a volume of dielectric liquid coolant collects in a common manifold area above the plurality of rack-mountable servers,

wherein one part of the heated dielectric liquid coolant that has collected in the common manifold area recirculates downward in at least one passage to the bottom of the at least one tank, and

wherein another part of the heat heated dielectric coolant that has collected in the common manifold area exits the at least one tank through the coolant outlet in the second portion of dielectric liquid coolant.

26. The apparatus of claim 25, wherein the dielectric liquid coolant flows from the volume between the plurality of vertically oriented rack-mountable servers and the at least one tank to the common manifold area above the plurality of vertically oriented rack-mountable servers.