Methods and apparatus for maintaining remote cluster state information

• US 7,346,744 B1
• Filed: 05/09/2003
• Issued: 03/18/2008
• Est. Priority Date: 11/04/2002
• Status: Active Grant

• Current Assignee: Sanmina-SCI Corporation
• Original Assignee: Newisys, Inc. (Sanmina-SCI Corporation)
• Inventors: Glasco, David Brian
• Primary Examiner(s): Sparks, Donald
• Assistant Examiner(s): Diller, Jesse
• Application Number:
  US10/435,072
• Time in Patent Office: 1,775 Days
• Field of Search: 711/141
• US Class Current: 711/141
• CPC Class Codes: G06F 12/0817 using directory methods

Abstract

According to the present invention, methods and apparatus are provided for increasing the efficiency of data access in a multiple processor, multiple cluster system. Mechanisms for improving the accuracy of information available to a cache coherence controller are provided in order to allow the cache coherence controller to reduce the number of transactions in a multiple cluster system. Non-change probes and augmented non-change probe responses are provided to acquire state information in remote clusters without affecting the state of the probed memory line. Augmented probe responses associated with shared and invalidating probes are provided to update state information in a coherence directory during read and read/write probe requests.

Claims

1. A computer system, comprising:

a home cluster including a first plurality of processors and a home cache coherence controller associated with a coherence directory, the coherence directory having state information corresponding to home cluster memory lines cached remotely;

a remote cluster including a second plurality of processors and a remote cache coherence controller, the remote cache coherence controller configured to receive a non-change probe associated with a memory line from the home cluster and probe remote cluster nodes to obtain cache state information for the memory line, wherein the non-change probe is initiated by the home cluster cache coherence controller.

2. The computer system of claim 1, wherein the non-change probe is initiated by the home cluster cache coherence controller periodically to update the coherence directory.

3. The computer system of claim 1, wherein the non-change probe is initiated by the home cluster cache coherence controller when an occupancy vector in the coherence directory is above a predetermined size.

4. The computer system of claim 1, wherein the non-change probe is initiated by the home cluster cache coherence controller when a read request that does not require probing is received by home cache coherence controller.

5. The computer system of claim 1, wherein cache state information is included in an augmented probe response.

6. The computer system of claim 1, wherein cache state information includes an owned/modified indicator.

7. The computer system of claim 1, wherein cache state information includes a shared/exclusive indicator.

8. The computer system of claim 1, wherein cache state information includes a cached indicator.

9. The computer system of claim 1, wherein the first plurality of processors and the home cache coherence controller are interconnected in a point-to-point architecture.

10. The computer system of claim 1, wherein the second plurality of processors and the remote cache coherence controller are interconnected in a point-to-point architecture.

11. The computer system of claim 1, wherein the remote cluster cache coherence controller is operable to send the cache state information to the home cluster cache coherence controller.

12. The computer system of claim 11, wherein home cluster cache coherence controller is operable to update the coherence directory using the received cache state information.

13. The computer system of claim 12, wherein updating the coherence directory comprises updating the occupancy vector.

14. The computer system of claim 13, wherein the occupancy vector is set to include the remote cluster if the cache state information indicates that the memory line is cached.

15. The computer system of claim 13, wherein the occupancy vector is set to exclude the remote cluster if the cache state information indicates that the memory line is not cached.

16. The computer system of claim 11, wherein updating the coherence directory comprises updating the dirty data owner information field.

17. The computer system of claim 11, wherein the state of the memory line in the coherence directory is updated to invalid if the cache state information indicates that the memory line is not cached.

18. The computer system of claim 17, wherein the state of the memory line in the coherence directory is updated to shared if the cache state information indicates that the memory line in the remote cluster is shared and cached.

19. A method for probing nodes, the method comprising:

generating a non-change probe associated with a memory line at a home cluster having a coherence directory;

forwarding the non-change probe to a remote cluster, wherein the remote cluster is configured to receive the non-change probe and probe remote cluster nodes to obtain cache state information associated with the memory line, wherein the non-change probe is initiated by a home cluster cache coherence controller.

20. The method of claim 19, wherein the non-change probe is initiated by the home cluster cache coherence controller periodically to update the coherence directory.

21. The method of claim 19, wherein the non-change probe is initiated by the home cluster cache coherence controller when an occupancy vector in the coherence directory is above a predetermined size.

22. The method of claim 19, wherein the non-change probe is initiated by the home cluster cache coherence controller when a read request that does not require probing is received by home cache coherence controller.

23. The method of claim 19, wherein cache state information is included in an augmented probe response.

24. The method of claim 19, wherein cache state information includes an owned/modified indicator.

25. The method of claim 19, wherein cache state information includes a shared/exclusive indicator.

26. The method of claim 19, wherein cache state information includes a cached indicator.

27. The method of claim 19, wherein the first plurality of processors and the home cache coherence controller are interconnected in a point-to-point architecture.

28. The method of claim 19, wherein the second plurality of processors and the remote cache coherence controller are interconnected in a point-to-point architecture.

29. A coherence controller, comprising:

means for generating a non-change probe associated with a memory line at a home cluster;

means for forwarding the non-change probe to a remote cluster, wherein the remote cluster is configured to receive the non-change probe and probe remote cluster nodes to obtain cache state information associated with the memory line, wherein the non-change probe is initiated periodically to update the coherence directory.

30. The coherence controller of claim 29, wherein the non-change probe is initiated when an occupancy vector in the coherence directory is above a predetermined size.

31. The coherence controller of claim 29, wherein the non-change probe is initiated when a read request that does not require probing is received by home cache coherence controller.

32. The coherence controller of claim 29, wherein cache state information is included in an augmented probe response.

33. The coherence controller of claim 29, wherein cache state information includes an owned/modified indicator.

34. The coherence controller of claim 29, wherein cache state information includes a shared/exclusive indicator.

35. The coherence controller of claim 29, wherein cache state information includes a cached indicator.

36. A computer system, comprising:

a home cluster including a first plurality of processors and a home cache coherence controller associated with a coherence directory, the coherence directory having state information corresponding to home cluster memory lines cached remotely, wherein the home cache coherence controller receives a probe associated with a memory line from a home cluster processor;

a remote cluster including a second plurality of processors and a remote cache coherence controller, the remote cache coherence controller configured to receive a probe from the home cluster cache coherence controller and probe remote cluster nodes to obtain an augmented probe response for the memory line, wherein cache state information includes a shared/exclusive indicator.

37. The computer system of claim 36, wherein the augmented probe response comprises cache state information and probe response information.

38. The computer system of claim 37, wherein the probe from the home cluster is a shared probe associated with a read request.

39. The computer system of claim 37, wherein the probe from the home cluster is an invalidating probe associated with a read/write request.

40. The computer system of claim 37, wherein cache state information includes an owned/modified indicator.

41. The computer system of claim 37, wherein cache state information includes a cached indicator.

42. The computer system of claim 37, wherein the first plurality of processors and the home cache coherence controller are interconnected in a point-to-point architecture.

43. The computer system of claim 37, wherein the second plurality of processors and the remote cache coherence controller are interconnected in a point-to-point architecture.

44. The computer system of claim 37, wherein the remote cluster cache coherence controller is operable to send the cache state information to the home cluster cache coherence controller.

45. The computer system of claim 44, wherein home cluster cache coherence controller is operable to update the coherence directory using the received cache state information.

46. The computer system of claim 45, wherein updating the coherence directory comprises updating the occupancy vector.

47. The computer system of claim 46, wherein the occupancy vector is set to include the remote cluster if the cache state information indicates that the memory line is cached.

48. The computer system of claim 46, wherein the occupancy vector is set to exclude the remote cluster if the cache state information indicates that the memory line is not cached.

49. The computer system of claim 45, wherein updating the coherence directory comprises updating the dirty data owner information field.

50. The computer system of claim 45, wherein the state of the memory line in the coherence directory is updated to invalid if the cache state information indicates that the memory line is not remotely cached.

51. The computer system of claim 45, wherein the state of the memory line in the coherence directory is updated to shared if the cache state information indicates that the memory line in the remote cluster is shared and cached.

52. A method for probing cache lines, the method comprising:

receiving a probe request associated with a memory line from a local processor;

forwarding a probe associated with the probe request to a remote cluster of processors, wherein the remote cluster is configured to provide an augmented probe response, the augmented probe response having cache state information and probe response information for the memory line, wherein cache state information includes a shared/exclusive indicator.

53. The method of claim 52, wherein the probe is a shared probe associated with a read request.

54. The method of claim 52, wherein the probe is an invalidating probe associated with a read/write request.

55. The method of claim 52, wherein cache state information includes an owned/modified indicator.

56. The method of claim 52, wherein cache state information includes a cached indicator.

57. The method of claim 52, wherein the request cluster of processors are interconnected in a point-to-point architecture.

58. The method of claim 52, wherein the remote cluster of processors are interconnected in a point-to-point architecture.

59. A controller, the controller comprising:

means for receiving a probe request associated with a memory line from a local processor;

means for forwarding a probe associated with the probe request to a remote cluster of processors, wherein the remote cluster is configured to provide an augmented probe response, the augmented probe response having cache state information and probe response information for the memory line, wherein cache state information includes a shared/exclusive indicator.

60. A computer system, comprising:

a request cluster including a first plurality of processors and a request cache coherence controller, wherein the request cache coherence controller is configured to receive a request associated with a memory line from a request cluster processor and forward the request;

a home cluster including a second plurality of processors and a home cache coherence controller associated with a coherence directory, the coherence directory having state information corresponding to home cluster memory lines cached remotely, wherein the home cache coherence controller receives the forwarded request from the request cluster;

a remote cluster including a third plurality of processors and a remote cache coherence controller, the remote cache coherence controller configured to receive a probe associated with the memory line from the home cluster cache coherence controller and probe remote cluster nodes to obtain an augmented probe response for the memory line having cache state information and probe response information for the memory line, wherein cache state information includes a shared/exclusive indicator.

61. The computer system of claim 60, wherein the augmented probe response comprises cache state information and probe response information for the memory line.

62. The computer system of claim 61, wherein the probe from the home cluster is a shared probe associated with a read request.

63. The computer system of claim 61, wherein the probe from the home cluster is an invalidating probe associated with a read/write request.

64. The computer system of claim 61, wherein cache state information includes an owned/modified indicator.

65. The computer system of claim 61, wherein cache state information includes a cached indicator.

66. The computer system of claim 61, wherein the first plurality of processors and the request cache coherence controller are interconnected in a point-to-point architecture.

67. The computer system of claim 61, wherein the second plurality of processors and the home cache coherence controller are interconnected in a point-to-point architecture.

68. The computer system of claim 61, wherein the third plurality of processors and the remote cache coherence controller are interconnected in a point-to-point architecture.

69. The computer system of claim 61, wherein the remote cluster cache coherence controller is operable to send the cache state information to the home cluster cache coherence controller.

70. The computer system of claim 69, wherein the home cluster cache coherence controller is operable to update the coherence directory using the received cache state information.

71. The computer system of claim 70, wherein updating the coherence directory comprises updating the occupancy vector.

72. The computer system of claim 71, wherein the occupancy vector is set to include the request cluster.

73. The computer system of claim 71, wherein the occupancy vector is set to include the remote cluster if the cache state information indicates that the memory line is cached.

74. The computer system of claim 71, wherein the occupancy vector is set to exclude the remote cluster if the cache state information indicates that the memory line is not cached.

75. The computer system of claim 70, wherein updating the coherence directory comprises updating the dirty data owner information field.

76. The computer system of claim 69, wherein the state of the memory line in the coherence directory is updated to shared if the cache state information indicates that the memory line in the remote cluster is shared and cached.

78. The computer system of claim 60, wherein the augmented probe response comprises cache state information and probe response information for the memory line.

79. The method of claim 78, wherein the probe is a shared probe associated with a read request.

80. The method of claim 78, wherein the probe is an invalidating probe associated with a read/write request.

81. The method of claim 78, wherein cache state information includes an owned/modified indicator.

82. The method of claim 78, wherein cache state information includes a cached indicator.

83. The method of claim 78, wherein the request cluster of processors are interconnected in a point-to-point architecture.

84. The method of claim 78, wherein the remote cluster of processors are interconnected in a point-to-point architecture.

77. A method for probing cache lines, the method comprising:

receiving a probe request associated with a memory line from a request cluster of processors;

accessing a coherence directory to identify a remote cluster for probing;

forwarding a probe associated with the probe request to the remote cluster of processors, wherein the remote cluster is configured to provide an augmented probe response, having cache state information and probe response information for the memory line, wherein cache state information includes a shared/exclusive indicator.

85. A cache coherence controller, comprising:

means for receiving a probe request associated with a memory line from a request cluster of processors;

means for accessing a coherence directory to identify a remote cluster for probing;

means for forwarding a probe associated with the probe request to the remote cluster of processors, wherein the remote cluster is configured to provide an augmented probe response, having cache state information and probe response information for the memory line, wherein cache state information includes a shared/exclusive indicator.