Cooling electronic devices in a data center

• US 10,645,847 B2
• Filed: 04/20/2018
• Issued: 05/05/2020
• Est. Priority Date: 04/20/2018
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Edwards, William, Iyengar, Madhusudan Krishnan, Rajan, Sundar, Padilla, Jorge, Jouppi, Norman Paul
• Primary Examiner(s): Smith, Courtney L
• Application Number:
  US15/958,567
• Publication Number:
  US 20190327860A1
• Time in Patent Office: 746 Days
• Field of Search: None
• CPC Class Codes: G06F 1/20 Cooling means
• CPC Class Codes: G06F 2200/201 Cooling arrangements using cooling fluid
• CPC Class Codes: H05K 7/20772 within server blades for removing heat from heat source
• CPC Class Codes: H05K 7/20809 within server blades for removing heat from heat source
• CPC Class Codes: H05K 7/20836 Thermal management, e.g. server temperature control

Abstract

A data center cooling system includes a server rack sub-assembly that includes a motherboard mounted on a support member and a heat generating electronic devices mounted on the a motherboard; a cold plate positioned in thermal communication with at least a portion of the heat generating electronic devices, the cold plate configured to receive a flow of a cooling liquid circulated through a supply conduit fluidly coupled to a liquid inlet of the cold plate, through the cold plate, and through a return conduit fluidly coupled to a liquid outlet of the cold plate; and a modulating control valve attached to either of the motherboard or the support member and positioned in either of the supply conduit or the return conduit, the modulating control valve configured to adjust a rate of the flow of the cooling liquid based on an operating condition of at least one of the heat generating electronic devices.

Claims

1. A data center cooling system, comprising:

a server rack sub-assembly that comprises at least one motherboard mounted on a support member and a plurality of heat generating electronic devices mounted on the at least one motherboard;

at least one cold plate mounted on top of and positioned in conductive thermal contact with at least a portion of the plurality of heat generating electronic devices, the cold plate configured to receive a flow of a cooling liquid circulated through a supply conduit fluidly coupled to a liquid inlet of the at least one cold plate, through the at least one cold plate, and through a return conduit fluidly coupled to a liquid outlet of the at least one cold plate; and

a modulating control valve attached to the server rack sub-assembly, the modulating control valve disposed on top of the support member and positioned in either of the supply conduit or the return conduit, the modulating control valve configured to adjust a rate of the flow of the cooling liquid in response to receiving a command from a controller, the command configured to adjust the valve based at least in part on an operating condition of at least one of the plurality of heat generating electronic devices.

2. The data center cooling system of claim 1, wherein the at least one cold plate comprises a plurality of cold plates, each of the plurality of cold plates mounted on top of and positioned in conductive thermal contact with a respective heat generating electronic device of the plurality of heat generating electronic devices.

3. The data center cooling system of claim 2, wherein the supply conduit is directly coupled to each liquid inlet of the plurality of cold plates, and the return conduit is directly coupled to each liquid outlet of the plurality of cold plates.

4. The data center cooling system of claim 2, wherein the supply conduit is directly coupled to the liquid inlet of one of the plurality of cold plates, and the return conduit is directly coupled to the liquid outlet of one of the plurality of cold plates.

5. The data center cooling system of claim 2, wherein the supply conduit is directly coupled to the liquid inlets of two or more of the plurality of cold plates, and the return conduit is directly coupled to the liquid outlets of two or more of the plurality of cold plates.

6. The data center cooling system of claim 1, further comprising a controller communicably coupled to the modulating control valve.

7. The data center cooling system of claim 6, wherein the controller is attached to the server rack sub-assembly.

8. The data center cooling system of claim 6, wherein the controller is configured to perform operations comprising:

determining a value of the operating condition of the at least one heat generating electronic device; and

adjusting the modulating control valve to open or close based on the determined value.

9. The data center cooling system of claim 8, wherein determining the value of the operating condition of the at least one heat generating electronic device comprises:

determining an operating temperature for each of the plurality heat generating electronic devices;

determining a maximum operating temperature from the determined operating temperatures; and

calculating a thermal margin between the maximum operating temperature and at least one of the operating temperatures.

10. The data center cooling system of claim 9, wherein adjusting the modulating control valve to open or close based on the determined value comprises adjusting the modulating control valve to open based on the calculated thermal margin greater than a threshold thermal margin value.

11. The data center cooling system of claim 8, wherein determining the value of the operating condition of the at least one heat generating electronic device comprises:

determining a supply cooling liquid temperature;

determining a return cooling liquid temperature; and

calculating a temperature difference between the supply cooling liquid temperature and the return cooling liquid temperature.

12. The data center cooling system of claim 11, wherein adjusting the modulating control valve to open or close based on the determined value comprises adjusting the modulating control valve to open based on the calculated temperature difference greater than a threshold temperature difference value.

13. The data center cooling system of claim 11, wherein determining a supply cooling liquid temperature comprises receiving a measured temperature value from an inlet temperature sensor in thermal communication with the flow of the cooling liquid through the supply conduit, and determining a return cooling liquid temperature comprises receiving a measured temperature value from an outlet temperature sensor in thermal communication with the flow of the cooling liquid through the return conduit.

14. A method for cooling heat generating devices in a data center, comprising:

circulating a flow of a cooling liquid to a server rack sub-assembly that comprises at least one motherboard mounted on a support member and a plurality of heat generating electronic devices mounted on the at least one motherboard;

circulating the flow of the cooling liquid through a supply conduit to a liquid inlet of at least one cold plate mounted on top of and positioned in conductive thermal contact with at least a portion of the plurality of heat generating electronic devices;

circulating the flow of the cooling liquid through the at least one cold plate to receive heat into the cooling liquid from the portion of the plurality of heat generating electronic devices;

circulating the flow of the heated cooling liquid from a liquid outlet of the at least one cold plate through a return conduit; and

controlling a modulating control valve attached to the server rack sub-assembly and disposed on top of the support member to adjust a rate of the flow of the cooling liquid in response to receiving a command from a controller, the command configured to adjust the valve based at least in part on an operating condition of at least one of the plurality of heat generating electronic devices.

15. The method of claim 14, wherein the at least one cold plate comprises a plurality of cold plates, each of the plurality of cold plates mounted on top of and positioned in conductive thermal contact with a respective heat generating electronic device of the plurality of heat generating electronic devices.

16. The method of claim 15, further comprising:

circulating the flow of cooling liquid directly from the supply conduit to each liquid inlet of the plurality of cold plates; and

circulating the flow of the heated cooling liquid directly from each liquid outlet of the plurality of cold plates to the return conduit.

17. The method of claim 15, further comprising:

circulating the flow of cooling liquid directly from the supply conduit to a liquid inlet of one of the plurality of cold plates;

heating the cooling liquid in the one of the plurality of cold plates;

circulating the flow of the heated cooling liquid from a liquid outlet of the one of the plurality of cold plates to a liquid inlet of another of the plurality of cold plates;

further heating the heated cooling liquid in the another of the plurality of cold plates; and

circulating the flow of the further heated cooling liquid directly from a liquid outlet of the another of the plurality of cold plates to the return conduit.

18. The method of claim 15, further comprising:

circulating the flow of cooling liquid directly from the supply conduit to liquid inlets of a first portion of the plurality of cold plates;

heating the cooling liquid in the portion of the plurality of cold plates;

circulating the flow of the heated cooling liquid from liquid outlets of the first portion of the plurality of cold plates to liquid inlets of a second portion of the plurality of cold plates;

further heating the heated cooling liquid in the second portion of the plurality of cold plates; and

circulating the flow of the further heated cooling liquid directly from liquid outlets of the second portion of the plurality of cold plates to the return conduit.

19. The method of claim 14, further comprising:

determining a value of the operating condition of the at least one heat generating electronic device; and

adjusting the modulating control valve to open or close based on the determined value.

20. The method of claim 19, wherein determining the value of the operating condition of the at least one heat generating electronic device comprises:

determining an operating temperature for each of the plurality heat generating electronic devices;

determining a maximum operating temperature from the determined operating temperatures; and

calculating a thermal margin between the maximum operating temperature and at least one of the operating temperatures.

21. The method of claim 20, wherein adjusting the modulating control valve to open or close based on the determined value comprises adjusting the modulating control valve to open based on the calculated thermal margin greater than a threshold thermal margin value.

22. The method of claim 19, wherein determining the value of the operating condition of the at least one heat generating electronic device comprises:

determining a supply cooling liquid temperature;

determining a return cooling liquid temperature; and

calculating a temperature difference between the supply cooling liquid temperature and the return cooling liquid temperature.

23. The method of claim 22, wherein adjusting the modulating control valve to open or close based on the determined value comprises adjusting the modulating control valve to open based on the calculated temperature difference greater than a threshold temperature difference value.

24. The method of claim 22, wherein determining a supply cooling liquid temperature comprises receiving a measured temperature value from an inlet temperature sensor in thermal communication with the flow of the cooling liquid through the supply conduit, and determining a return cooling liquid temperature comprises receiving a measured temperature value from an outlet temperature sensor in thermal communication with the flow of the cooling liquid through the return conduit.

25. A data center cooling system, comprising:

a server rack sub-assembly that comprises at least one motherboard mounted on a support member and a plurality of heat generating electronic devices mounted on the at least one motherboard;

at least one cold plate positioned in thermal communication with at least a portion of the plurality of heat generating electronic devices, the cold plate configured to receive a flow of a cooling liquid circulated through a supply conduit fluidly coupled to a liquid inlet of the at least one cold plate, through the at least one cold plate, and through a return conduit fluidly coupled to a liquid outlet of the at least one cold plate;

a modulating control valve attached to the server rack sub-assembly and positioned in either of the supply conduit or the return conduit, the modulating control valve configured to adjust a rate of the flow of the cooling liquid based at least in part on an operating condition of at least one of the plurality of heat generating electronic devices; and

a controller communicably coupled to the modulating control valve, the controller configured to perform operations comprising;

determining a value of the operating condition of the at least one heat generating electronic device, wherein the determining comprises;

determining an operating temperature for each of the plurality heat generating electronic devices,

determining a maximum operating temperature from the determined operating temperatures, and

calculating a thermal margin between the maximum operating temperature and at least one of the operating temperatures; and

adjusting the modulating control valve to open or close based on the determined value.

26. The data center cooling system of claim 25, wherein the at least one cold plate comprises a plurality of cold plates, each of the plurality of cold plates mounted to a respective heat generating electronic device of the plurality of heat generating electronic devices.

27. The data center cooling system of claim 26, wherein the supply conduit is directly coupled to each liquid inlet of the plurality of cold plates, and the return conduit is directly coupled to each liquid outlet of the plurality of cold plates.

28. The data center cooling system of claim 26, wherein the supply conduit is directly coupled to a liquid inlet of one of the plurality of cold plates, and the return conduit is directly coupled to a liquid outlet of one of the plurality of cold plates.

29. The data center cooling system of claim 26, wherein the supply conduit is directly coupled to liquid inlets of a portion of the plurality of cold plates, and the return conduit is directly coupled to liquid outlets of a portion of the plurality of cold plates.

30. A method for cooling heat generating devices in a data center, comprising:

circulating a flow of a cooling liquid to a server rack sub-assembly that comprises at least one motherboard mounted on a support member and a plurality of heat generating electronic devices mounted on the at least one motherboard;

circulating the flow of the cooling liquid through a supply conduit to a liquid inlet of at least one cold plate positioned in thermal communication with at least a portion of the plurality of heat generating electronic devices;

circulating the flow of the cooling liquid through the at least one cold plate to receive heat into the cooling liquid from the portion of the plurality of heat generating electronic devices;

circulating the flow of the heated cooling liquid from a liquid outlet of the at least one cold plate through a return conduit; and

controlling a modulating control valve attached to the server rack sub-assembly to adjust a rate of the flow of the cooling liquid based at least in part on an operating condition of at least one of the plurality of heat generating electronic devices;

determining a value of the operating condition of the at least one heat generating electronic device, the determining comprising;

determining an operating temperature for each of the plurality heat generating electronic devices,

determining a maximum operating temperature from the determined operating temperatures, and

calculating a thermal margin between the maximum operating temperature and at least one of the operating temperatures; and

adjusting the modulating control valve to open or close based on the determined value.

31. The method of claim 30, wherein the at least one cold plate comprises a plurality of cold plates, each of the plurality of cold plates mounted to a respective heat generating electronic device of the plurality of heat generating electronic devices.

32. The method of claim 31, further comprising:

circulating the flow of cooling liquid directly from the supply conduit to each liquid inlet of the plurality of cold plates; and

circulating the flow of the heated cooling liquid directly from each liquid outlet of the plurality of cold plates to the return conduit.

33. The method of claim 31, further comprising:

circulating the flow of cooling liquid directly from the supply conduit to a liquid inlet of one of the plurality of cold plates;

heating the cooling liquid in the one of the plurality of cold plates;

circulating the flow of the heated cooling liquid from a liquid outlet of the one of the plurality of cold plates to a liquid inlet of another of the plurality of cold plates;

further heating the heated cooling liquid in the another of the plurality of cold plates; and

circulating the flow of the further heated cooling liquid directly from a liquid outlet of the another of the plurality of cold plates to the return conduit.

34. The method of claim 31, further comprising:

circulating the flow of cooling liquid directly from the supply conduit to liquid inlets of a first portion of the plurality of cold plates;

heating the cooling liquid in the portion of the plurality of cold plates;

circulating the flow of the heated cooling liquid from liquid outlets of the first portion of the plurality of cold plates to liquid inlets of a second portion of the plurality of cold plates;

further heating the heated cooling liquid in the second portion of the plurality of cold plates; and

circulating the flow of the further heated cooling liquid directly from liquid outlets of the second portion of the plurality of cold plates to the return conduit.