Commmonly submersed servers with velocity augmentation and partial recirculation in tank

• US 9,992,914 B2
• Filed: 03/24/2015
• Issued: 06/05/2018
• Est. Priority Date: 08/11/2008
• Status: Active Grant

• Current Assignee: Green Revolution Cooling, Inc.
• Original Assignee: Green Revolution Cooling, Inc.
• Inventors: Best, Christiaan, Garnett, Mark
• Primary Examiner(s): Zerphey, Christopher R
• Application Number:
  US14/667,091
• Publication Number:
  US 20150195953A1
• Time in Patent Office: 1,169 Days
• CPC Class Codes: F28D 15/00 Heat-exchange apparatus with the intermediate heat-transfer medium in closed tubes passing into or through the conduit walls ; Heat-exchange apparatus employing intermediate heat-transfer medium or bodies F28D17/00, F28D19/00, F28D20/00 take precedence
• CPC Class Codes: G06F 1/20 Cooling means
• CPC Class Codes: G06F 1/206 comprising thermal management
• CPC Class Codes: G06F 2200/201 Cooling arrangements using cooling fluid
• CPC Class Codes: H01L 2924/00 Indexing scheme for arrangements or methods for connecting or disconnecting semiconductor or solid-state bodies as covered by H01L24/00
• CPC Class Codes: H01L 2924/0002 Not covered by any one of groups H01L24/00, H01L24/00 and H01L2224/00
• CPC Class Codes: H05K 7/20 Modifications to facilitate cooling, ventilating, or heating
• CPC Class Codes: H05K 7/20236 by immersion
• CPC Class Codes: H05K 7/20281 Thermal management, e.g. liquid flow control
• CPC Class Codes: H05K 7/203 by immersion
• CPC Class Codes: H05K 7/20327 Accessories for moving fluid, for connecting fluid conduits, for distributing fluid or for preventing leakage, e.g. pumps, tanks or manifolds
• CPC Class Codes: H05K 7/20381 Thermal management, e.g. evaporation control
• CPC Class Codes: H05K 7/20763 Liquid cooling without phase change
• CPC Class Codes: H05K 7/20772 within server blades for removing heat from heat source
• CPC Class Codes: H05K 7/20781 within cabinets for removing heat from server blades
• CPC Class Codes: H05K 7/2079 within rooms for removing heat from cabinets
• CPC Class Codes: H05K 7/20827 within rooms for removing heat from cabinets, e.g. air conditioning devices
• CPC Class Codes: H05K 7/20836 Thermal management, e.g. server temperature control
• CPC Class Codes: Y10T 29/4973 Replacing of defective part

Abstract

An apparatus for holding and cooling rack-mountable servers having heat-producing electronic components includes a tank, a volume of dielectric liquid coolant in the tank, mounting members that hold rack-mountable servers in the tank, one or more pumps, fluid velocity augmentations devices, and a liquid-to-liquid or liquid-to-refrigerant heat exchanger outside of the tank. The mounting members hold the rack-mountable servers in a vertical orientation within the tank such that the servers are commonly submersed in the volume of the dielectric liquid coolant. When a pump is operated to move the dielectric liquid coolant vertically across the heat producing components of the rack-mountable servers, a circuit is formed in which a first portion of heated dielectric liquid coolant is moved vertically upward across the heat producing components and then downward outside of the rack mountable servers in a passage external to the rack-mountable servers, while a second portion of heated dielectric liquid coolant flows out of the tank and through the liquid-to-liquid or liquid-to-refrigerant heat exchanger.

Claims

1. An apparatus for holding and cooling rack-mountable servers having heat-producing electronic components, comprising:

a tank defining an open interior volume and having a coolant inlet for receiving a dielectric liquid coolant within the open interior volume and having a coolant outlet for allowing the coolant to flow from the open interior volume, the coolant inlet and the coolant outlet being fluidly coupled to each other;

a volume of dielectric liquid coolant in the tank;

one or more mounting members positioned within the interior volume and configured to hold a plurality of rack-mountable servers in a horizontally stacked relationship with one another, with the rack-mountable servers in a vertical orientation within the interior volume such that the plurality of servers are commonly at least partially submersed in the volume of the dielectric liquid coolant;

one or more pumps configured to move at least portion of the dielectric liquid coolant in the tank;

a controller configured to maintain a temperature of the dielectric liquid coolant in the range of about 90°

F. to about 130°

F.;

a plurality of fluid velocity augmentation devices; and

a liquid-to-liquid or liquid-to-refrigerant heat exchanger outside of the tank, wherein the volume of dielectric liquid coolant is thermally coupled to a distal location via the liquid-to-liquid or liquid-to-refrigerant heat exchanger,

wherein the coolant inlet, the coolant outlet, and the mounting members are configured such that the one or more pumps are operable to move at least a portion of the dielectric liquid coolant vertically across heat producing components of one of the rack-mountable servers in parallel flow with dielectric liquid coolant moved vertically across heat producing components of at least one other of the rack-mountable servers,

wherein each of the plurality of fluid velocity augmentation devices is configured to increase the velocity of flow of the dielectric liquid coolant through at least one of the rack-mountable servers or between adjacent ones of the rack-mountable servers,

wherein at least one passage is defined in the tank that is outside of the vertically oriented rack-mountable servers, wherein at least one of the passages is in fluid connection with one or more of the fluid velocity augmentation devices, and wherein a circuit is formed in which a first portion of heated dielectric liquid coolant is moved vertically by the one or more fluid velocity augmentation devices across the vertically oriented rack-mountable servers, wherein the first portion of the dielectric liquid coolant is split into a second portion and a third portion after passing through the vertically oriented rack-mountable servers, wherein the second portion of the dielectric liquid coolant flows out of the tank and through the liquid-to-liquid or liquid-to-refrigerant heat exchanger such that heat is rejected to the distal location, and wherein the third portion flows through the at least one passage back to the one or more fluid velocity augmentation devices;

wherein the one or more mounting members hold the rack-mountable servers such that at least one of the rack-mountable servers is independently removable, through an opening in the top of the tank, from the volume of dielectric liquid coolant while at least two other rack-mountable servers remain commonly submersed in the volume of dielectric liquid coolant in the tank and sufficiently submersed to maintain cooling by the flow of the dielectric liquid coolant in the tank across heat producing electronic components of the at least two other rack-mountable servers, wherein any one or more of the servers can be removed while at least one other of the servers remains operating;

wherein at least one of the one or more mounting members is capable of mounting a first rack-mountable server, a second rack-mountable server, a third rack-mountable server and a fourth rack-mountable server such that;

a first gap exists between the first rack-mountable server and the second rack-mountable server;

a second gap exists between the second rack-mountable server and the third rack-mountable server and a third gap exists between the third rack-mountable server and the fourth rack-mountable server;

wherein dielectric liquid coolant is introduced into the tank through coolant inlet piping coupled to the plurality of fluid augmentation devices, wherein a first fluid augmentation device is positioned such that the fluid augmentation device directs the introduced dielectric liquid coolant into the first gap, wherein a second fluid augmentation device is positioned such that the fluid augmentation device directs the introduced dielectric liquid coolant into the second gap, and wherein a third fluid augmentation device is positioned such that the fluid augmentation device directs the introduced dielectric liquid coolant into the third gap.

2. The apparatus of claim 1, further comprising a remote heat exchanger at a distal location from the tank, wherein the remote heat exchanger is fluidly coupled to the liquid-to-liquid or liquid-to-refrigerant heat exchanger.

3. The apparatus of claim 2, wherein the distal location is outside of a server room in which the tank is located.

4. The apparatus of claim 1, wherein at least one of the pumps, the mounting members, the coolant inlet, and the coolant outlet are configured such that, when the at least one pump is operated, the at least one pump moves at least a portion of the dielectric liquid coolant vertically across heat producing components of at least one of the rack-mountable servers in parallel flow with dielectric liquid coolant moved vertically across the heat producing components.

5. The apparatus of claim 1, wherein the rack-mountable servers are mounted above the bottom of the tank, wherein the dielectric liquid coolant that is moved vertically across heat producing components of the rack-mountable servers is pumped from a volume of dielectric liquid coolant between the rack-mountable servers and the bottom of the tank.

6. The apparatus of claim 1, wherein at least the fluid velocity augmentation devices, the mounting members, and the at least one passage are configured such that at least part of the first portion of the heated dielectric liquid coolant in the circuit is moved horizontally after exiting out the top of the rack-mountable servers, then downward outside of the rack-mountable servers in the at least one passage, then horizontally across the bottom of the tank, then vertically upward across the heat producing components.

7. The apparatus of claim 1, wherein the one or more mounting members comprise:

a first set of one or more mounting members coupled in a fixed relationship to the tank; and

a second set of one or more mounting members coupled in a fixed relationship to the tank,

wherein the first set of one or more mounting members and the second set of mounting members are spaced apart from one another to form at least one opening for mounting a row of two or more of the rack-mountable servers in a vertical orientation within the interior volume so as to permit vertical installation and removal at least one of the rack-mountable server without removing or affecting the operational status of at least one other of the rack-mountable servers.

8. The apparatus of claim 7, wherein each of the rack-mountable servers has a pair of rack ears on opposing sides of the rack-mountable server, and wherein the first set and second set of mounting members are configured to receive and support the respective rack ears of each of the plurality of rack-mountable servers.

9. The apparatus of claim 1, further comprising:

one or more sensors configured to measure a temperature in at least one location in at least one of the rack-mountable servers or dielectric liquid coolant in the tank, wherein the one or more sensors are coupled to the controller;

wherein the controller is configured to control at least one characteristic of the apparatus based at least in part on at least one temperature measured using at least one sensor of the one or more sensors.

10. The apparatus of claim 1, wherein the controller is configured to maintain a temperature of the dielectric liquid coolant in the range of 100 degrees F. and 110 degrees F.

11. The apparatus of claim 1, wherein the controller is configured to adjust, based at least in part on the measured temperature, flow of the dielectric liquid coolant in the tank.

12. The apparatus of claim 9, wherein the controller is configured to adjust, based at least in part on the measured temperature, flow of a liquid coolant in the secondary liquid cooling circuit that receives heat from the dielectric liquid coolant.

13. The apparatus of claim 9, wherein the controller provides further instructions for periodically:

determining the optimum elevated temperature of the heated dielectric liquid coolant as it exits the plurality of rack-mountable servers in order for the dielectric liquid coolant to sufficiently cool the plurality of rack-mountable servers while reducing the amount of energy consumed to sufficiently cool each respective server;

determining the amount of energy needed to reject the absorbed heat for cooling the plurality of rack-mountable servers; and

in response to the periodic determinations of the optimum elevated temperature and the amount of energy needed to reject the absorbed heat, adjusting the amount of heat to be rejected through at least one remote heat exchanger in order that the dielectric liquid coolant is maintained at approximately the optimum elevated temperature in the at least a portion of a fluid circuit between the exits of the rack-mountable servers and the coolant outlet.

14. The apparatus of claim 1, further comprising:

at least one cable tray to receive and support signal and control network cabling to and from the plurality of rack-mountable servers; and

at least one power distribution unit to receive and support power distribution cables for distributing electrical power to the plurality of rack-mountable servers,

wherein the at least one cable tray and the at least one power distribution unit are mounted in proximity to an openable top of the tank so that the at least one rack-mountable server is independently removable from the tank through the open top of the tank without the need to remove the other rack-mountable servers from the tank or disturb the location or operational status of the other rack-mountable servers remaining within the tank.

15. An apparatus for holding and cooling rack-mountable servers having heat-producing electronic components, comprising:

a tank defining an open interior volume and having a coolant inlet for receiving a dielectric liquid coolant within the open interior volume and having a coolant outlet for allowing the coolant to flow from the open interior volume, the coolant inlet and the coolant outlet being fluidly coupled to each other;

a volume of dielectric liquid coolant in the tank;

one or more mounting members positioned within the interior volume and configured to hold a plurality of rack-mountable servers in a horizontally stacked relationship with one another, with the rack-mountable servers in a vertical orientation within the interior volume such that the plurality of servers are commonly at least partially submersed in the volume of the dielectric liquid coolant;

one or more pumps configured to move at least portion of the dielectric liquid coolant in the tank;

a controller configured to maintain a temperature of the dielectric liquid coolant in the range of about 90°

F. to about 130°

F.; and

a liquid-to-liquid or liquid-to-refrigerant heat exchanger outside of the tank,

wherein the volume of dielectric liquid coolant is thermally coupled to a distal location via the liquid-to-liquid or liquid-to-refrigerant heat exchanger,

wherein at least one of the pumps, the coolant inlet, the coolant outlet, and the mounting members are configured such that the at least one pump is operable to move at least a portion of the dielectric liquid coolant vertically across heat producing components of the rack-mountable servers,

wherein the one or more mounting members hold the rack-mountable servers such that at least one of the rack-mountable servers is independently removable, through an opening in the top of the tank, from the volume of dielectric liquid coolant while at least two other rack-mountable servers remain commonly submersed in the volume of dielectric liquid coolant in the tank and sufficiently submersed to maintain cooling by the flow of the dielectric liquid coolant in the tank across heat producing electronic components of the at least two other rack-mountable servers, wherein any one or more of the servers can be removed while at least one other of the servers remains operating.

16. The apparatus of claim 15, further comprising one or more fluid velocity augmentations devices, wherein each of at least one of the fluid velocity augmentation devices is configured to increase the velocity of flow of the dielectric liquid coolant through at least one of the rack-mountable servers or between adjacent ones of the rack-mountable servers.

17. The apparatus of claim 15, wherein the volume of dielectric liquid coolant comprises at least one passage in the tank that is outside of the rack-mountable servers, wherein, when the at least one pump is operated to move the dielectric liquid coolant vertically across the heat producing components, a circuit is formed in which a first portion of heated dielectric liquid coolant is moved upward across the heat producing components and then outside of the rack mountable servers in the at least one passage, while a second portion of heated dielectric liquid coolant flows out of the tank and through the liquid-to-liquid or liquid-to-refrigerant heat exchanger such that heat is rejected to the distal location.

18. The apparatus of claim 17, wherein at least one of the pumps, the mounting members, and the at least one passage are configured such that at least part of the first portion of the heated dielectric liquid coolant in the circuit is moved horizontally after exiting the rack-mountable servers, then outside of the rack-mountable servers in the at least one passage, then horizontally across the tank, then vertically across the heat producing components.

19. The apparatus of claim 15, wherein at least one of the pumps and the mounting members are configured such that a portion of the heated dielectric liquid coolant is moved horizontally at the surface of the volume of dielectric liquid coolant after exiting out the top of the rack-mountable servers.

20. The apparatus of claim 15, wherein the tank, at least one of the pumps, the coolant inlet, the coolant outlet, and the mounting members are configured such that the at least one pump moves at least a portion of the dielectric liquid coolant vertically across heat producing components of one of the rack-mountable servers in parallel flow with dielectric liquid coolant moved vertically across heat producing components of at least one other of the rack-mountable servers and such that dielectric liquid coolant that has been heated by heat producing components in multiple ones of the rack-mountable servers at least partially collects in a manifold area proximate to the surface of the volume of dielectric liquid.

21. The apparatus of claim 15, wherein the one or more mounting members comprise:

a first set of one or more mounting members coupled in a fixed relationship to the tank; and

a second set of one or more mounting members coupled in a fixed relationship to the tank, wherein the first set of one or more mounting members and the second set of mounting members are spaced apart from one another to form at least one opening for mounting a row of two or more of the rack-mountable servers in a vertical orientation within the interior volume so as to permit vertical installation and removal at least one of the rack-mountable server without removing or affecting the operational status of at least one other of the rack-mountable server.

22. A method of providing computing resources, comprising:

introducing a dielectric liquid coolant into a tank at least partially enclosing a plurality of rack-mountable servers in a vertical orientation in a horizontally stacked relationship, and wherein the tank holds a volume of dielectric liquid coolant in which at least two of the rack-mountable servers are commonly at least partially submersed in the volume of dielectric liquid coolant;

heating the dielectric coolant to a temperature in the range of about 90°

F. to about 130°

F.;

pumping at least a portion of the dielectric liquid coolant such that dielectric liquid coolant flows vertically across heat producing components in at least one of the rack-mountable servers in the tank and in parallel flow across heat producing components in at least two of the rack-mountable servers commonly submersed in the volume of dielectric liquid coolant, wherein pumping the at least a portion of the dielectric liquid coolant comprises;

circulating a first portion of the volume of dielectric liquid coolant inside the tank such that dielectric liquid coolant is moved vertically across the heat producing components and then outside of the rack-mountable servers in at least one passage outside the rack-mountable servers; and

moving, while the first portion of dielectric liquid coolant circulates in the tank, a second portion of the dielectric liquid coolant out of the tank and through a liquid-to-liquid or liquid-to-refrigerant heat exchanger such that heat from the heat producing components in the rack-mountable servers is rejected to a distal location via the liquid-to-liquid or liquid-to-refrigerant heat exchanger; and

vertically removing, through an openable top of the tank, at least one of the rack-mounted servers from the volume of dielectric liquid coolant while at least one other of the rack-mounted servers remains operating and commonly at least partially submersed in the volume of dielectric liquid coolant and while a portion of the dielectric liquid coolant is moved over heat producing components of at least one of the rack-mountable servers that remain operating and submersed in the volume of the dielectric liquid coolant.

23. The method of claim 22, further comprising augmenting velocity of dielectric liquid coolant moving across heat producing components using at least one fluid velocity augmentation device.

24. The method of claim 22, further comprising:

measuring at least one temperature in at least one location in at least one of the rack-mountable servers or dielectric liquid coolant in the tank; and

maintaining an elevated temperature of dielectric liquid coolant in at least one location in the tank based at least in part on at least one measured temperature.

25. The method of claim 22, further comprising replacing or reinstalling, while at least one other rack-mounted server remain commonly at least partially submersed in the volume of dielectric liquid coolant and in operation, at least one of the removed rack-mountable servers.