System and method for checking data accumulators for consistency

• US 7,221,650 B1
• Filed: 12/23/2002
• Issued: 05/22/2007
• Est. Priority Date: 12/23/2002
• Status: Expired due to Fees

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Cooper, Jeffrey L., Benz, Henry Charles
• Primary Examiner(s): Nguyen; Chau
• Assistant Examiner(s): Juntima; Nittaya
• Application Number:
  US10/328,445
• Time in Patent Office: 1,611 Days
• Field of Search: 370229-231, 370/235, 370/252, 370/394, 370/412, 370/413, 370/449, 370/503, 370/414, 370/395.1, 370/236
• US Class Current: 370/236
• CPC Class Codes: H04L 49/254 Centralised controller, i.e. arbitration or scheduling
• CPC Class Codes: H04L 49/508 Head of Line Blocking Avoidance
• CPC Class Codes: H04L 49/555 Error detection

Abstract

A system and method checks whether messages exchanged between first and second modules are being lost or gained. The first module has a request counter and a capture register. The second module has a request accumulator and a capture register. As the first module issues and receives messages, it increments and decrements its request counter. As the second module receives and issues messages, it increments and decrements its request accumulator. To check for lost or gained messages, the first module copies the current value of its request counter into its capture register, and issues a marker to the second module. The first module decrements its capture register in response to receiving post-marker messages, but does not increment its capture register. Upon receipt of the marker, the second module copies the current value of its request accumulator into its capture register, and returns the marker to the first module. When the first module receives the marker, it stops decrementing its capture register. The two capture registers should then contain the same values provided that no messages have been lost or gained.

Claims

1. In a system in which a first module issues a plurality of first messages to a second module that responds by issuing a plurality of second messages back to the first module, a method of checking whether one or more of the first or second messages have been lost or gained, the method comprising the steps of:

providing a request counter at the first module;

incrementing the request counter in response to each first message issued by the first module;

decrementing the request counter in response to each second message received at the first module;

providing a capture register at the first module;

providing a request accumulator at the second module;

incrementing the request accumulator in response to each first message received at the second module;

decrementing the request accumulator in response to each second message issued by the second module;

providing a capture register at the second module;

issuing a marker message from the first module to the second module;

storing the current value of the request counter in the capture register of the first module upon issuance of the marker message;

upon receiving the marker message at the second module, copying the current value of the request accumulator into the capture register of the second module;

returning the marker message back to the first module;

decrementing the capture register of the first module in response to each second message received at the first module from the time the marker message is issued until the marker message is returned to the first module; and

comparing the contents of the capture registers at the first and second modules.

2. The method of claim 1 further comprising the step of concluding that no first or second messages have been lost or gained if the compared contents of the two capture registers are the same.

3. The method of claim 1 wherein the marker message issued by the first module is part of a selected first message.

4. The method of claim 1 wherein the marker message returned by the second module is part of a selected second message.

5. The method of claim 1 wherein

the first module is an input/output card (IOC) configured to send and receive messages to and from a computer network;

the second module is a switch fabric card (SFC) configured to switch messages from a source IOC to a destination IOC;

the first message is a request seeking authorization to send a received network message from a source IOC to a selected destination IOC via the SFC; and

the second message is a grant providing authorization to the source IOC to send the received network message to the destination IOC via the SFC.

6. The method of claim 5 wherein the network switch is configured to operate in accordance with the InfiniBand Architecture (IBA) specification standard.

7. The method of claim 1 wherein the first and second modules are interconnected by a high-speed serial control link.

8. The method of claim 7 wherein

the first and second modules operate on parallel data streams;

and

a plurality of serializer/deserializer (SERDES) devices are utilized to convert parallel data to serial data for transmission across the serial link and to recover parallel data from serial data received from the serial link.

9. The method of claim 1 wherein the first and second modules operate in different clock domains with respect to each other.

10. The method of claim 1 wherein, at any given time, there are one or more first messages issued by the first module but not yet received by the second module and one or more second messages issued by the second module but not yet received by the first module.

11. In a network switch having first and second entities configured to exchange messages, a system adapted to check whether one or more messages have been lost or gained, the system comprising:

a request counter at the first entity, the request counter configured to track the number of messages issued by the first entity;

a first capture register associated with the request counter;

a first logic circuit operably coupled to the request counter and the first capture register;

a request accumulator at the second entity, the request accumulator configured to track the number of messages received by the second entity;

a second capture register associated with the request accumulator;

a second logic circuit operably coupled to the request accumulator and the second capture register;

wherein,

the first logic circuit increments the request counter in response to each message issued to the second entity, decrements the request counter in response to each message received from the second entity, issues a marker message to the second entity and copies a current value of the request counter into the first capture register,

the second logic circuit increments the request accumulator in response to each message received from the first entity, decrements the request accumulator in response to each message issued to the first entity and, in response to receiving the marker message, copies a current value of the request accumulator into the second capture register and returns the marker message to the first entity,

the first logic circuit decrements the first capture register in response to each message received between issuance of the marker message and its return, and

a comparison of the first and second capture registers identifies whether one or more messages have been lost or gained.

12. The system of claim 11 wherein

the network switch has a plurality of input/output cards (IOCs),

the first entity is one of the IOCs,

the second entity is a switch fabric card (SFC), and

the IOC and SFC are each disposed on different modules.

13. The system of claim 11 wherein the messages issued by the IOC are requests to access one or more IOCs and the messages issued by the SFC are grants permitting the IOC to access one or more selected IOCs.

14. The system of claim 13 wherein the SFC has a set of request accumulators for each IOC that may be requested and, within each set of request accumulators, one request accumulator is assigned to each IOC.

15. The system of claim 14 further comprising at least one second capture register for each set of request accumulators.

16. The system of claim 15 wherein a single marker message checks all of the request counters of a given IOC and all of the request accumulators assigned to the given IOC.

17. The system of claim 11 wherein at any given time, one or more first and second messages are in-flight between the first and second entities.

18. In a system in which a first entity having a request counter and a capture register issues a plurality of first messages to a second entity having a request accumulator and a capture register that responds by issuing a plurality of second messages back to the first entity, a method of checking whether one or more of the first or second messages have been lost or gained, the method comprising the steps of:

incrementing the request counter in response to each first message issued by the first entity;

decrementing the request counter in response to each second message received at the first entity;

incrementing the request accumulator in response to each first message received at the second entity;

decrementing the request accumulator in response to each second message issued by the second entity;

issuing a marker message from the first entity to the second entity;

storing the current value of the request counter in the capture register of the first entity upon issuance of the marker message;

upon receiving the marker message at the second entity, copying the current value of the request accumulator into the capture register of the second entity;

returning the marker message back to the first entity;

decrementing the capture register of the first entity in response to each second message received at the first entity from the time the marker message is issued until the marker message is returned to the first entity; and

comparing the contents of the capture registers at the first and second entities.

19. The method of claim 18 wherein at any given time, one or more first and second messages are in-flight between the first and second entities.

20. The method of claim 18 wherein the first and second entities are separate modules interconnected by one or more control links configured to carry the first and second messages between the first and second entities.

21. A network switch for receiving and forwarding packets with a computer network, the network switch comprising:

a plurality of input/output card (IOC) modules configured to send and receive packets to and from the computer network; and

a switch fabric cards (SFC) in communicating relationship with the IOCs, the SFC configured to switch packets received from the computer network at a source IOC to a destination IOC for transmission back into the computer network, wherein

at least one of the IOCs has

IOC logic configured to issue a request to the SFC for sending a received packet to a destination IOC,

a request counter configured to track the number of requests issued by the at least one IOC, and

a first capture register associated with the request counter, the SFC has

SFC logic configured to respond to the requests by issuing a grant to the at least one IOC providing authorization to send the received packet to the destination IOC,

a request accumulator configured to track the number of requests received from the at least one IOC, and

a second capture register associated with the request accumulator,

the IOC logic increments the request counter in response to each request issued to the SFC, decrements the request counter in response to each grant received from the SFC, issues a marker message to the SFC and copies a current value of the request counter into the first capture register, and

the SFC logic increments the request accumulator in response to each request received from the at least one IOC, decrements the request accumulator in response to each grant issued to the at least one IOC and, in response to receiving the marker message, copies a current value of the request accumulator into the second capture register and returns the marker message to the at least one IOC,

the IOC logic circuit decrements the first capture register in response to each grant received between issuance of the marker message and its return, and

a comparison of the first and second capture registers identifies whether one or more requests or grants have been lost or gained.

22. The network switch of claim 21 wherein the SFC implements a cross-bar switch fabric configured to switch packets in terms of fixed size cells.

23. The network switch of claim 22 wherein

the at least one IOC has a separate request counter and companion capture register for each potential destination IOC, and

the SFC has a set of request accumulators for each potential destination IOC and, within each set of request accumulators, one request accumulator is assigned to each IOC.