Methods and apparatus for performing fault isolation and failure analysis in link-connected systems

• US 5,157,667 A
• Filed: 04/30/1990
• Issued: 10/20/1992
• Est. Priority Date: 04/30/1990
• Status: Expired due to Term

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: International Business Machines Corporation
• Inventors: Garrigan, Albert W., Spencer, David E., Taylor, Jordan M., Moffitt, Gerald T., Carusone, Anthony Jr., Hunsinger, Wayne
• Primary Examiner(s): Beausoliel, Robert W.
• Assistant Examiner(s): HUA, LY
• Application Number:
  US07/516,387
• Time in Patent Office: 904 Days
• Field of Search: 371/29.1, 371/16.5, 340/825.1, 370/16, 455/8
• US Class Current: 714/45
• CPC Class Codes: G06F 11/16 Error detection or correction of the data by redundancy in hardware
• CPC Class Codes: G06F 11/2205 using arrangements specific to the hardware being tested

Abstract

The invention relates to methods and apparatus for isolating faults in link-connected systems utilizing fault reports generated from within the system itself. The reports are transmitted to a central location, preferably during a predetermined time period, and are used to create a single error message identifying the probable nature and location of the fault. A preferred embodiment of the invention does not require either the construction or maintenance of systemwide configuration tables, commonly used performing fault location and analysis. Instead, each unit of a pair of link coupled units, initially or on reconnection, interrogates a link adapter at the other end of the link for an identifier that identifies both the remote unit and the remote link adapter. This "nearest neighbor" information is stored locally at each unit, and is transmitted to the central location when an error is detected. The nearest neighbor information, rather than information from a configuration table, may be used to combine multiple records relating to a fault event, locate the fault and diagnose its cause. Additionally, a preferred embodiment of the invention provides a plurality of reporting paths for each unit in the system, so that the failure of a given unit or link does not prevent error information from being communicated to the central location.

Claims

1. Apparatus for isolating and analyzing faults in a link-connected system, utilizing fault reports generated from within the system itself, wherein said system comprises a plurality of units interconnected by a plurality of links, and further wherein said units are coupled to a central reporting location, comprising:

(a) means for detecting faults as they occur in the system;

(b) means for transmitting reports of detected faults to said central location, said transmitting means providing, as part of at least one of said fault reports, nearest neighbor information;

(c) means for associating fault reports transmitted to said central location utilizing said nearest neighbor information; and

(d) means for generating a single error message from associated fault reports.

2. Apparatus as set forth in claim 1 wherein said transmitting means further comprises means for generating at least one fault report for each detected fault, wherein each report includes at least an identification of the unit reporting the fault, an identification of the link on which a fault was detected by the reporting unit, and an indication of any detected fault symptoms.

3. Apparatus as set forth in claim 2 wherein said means for generating a single error message from associated fault reports is operative to both isolate and diagnose the cause of a fault based at least in part on said indication of detected fault symptoms.

4. Apparatus as set forth in claim 1 wherein said means for associating detected faults is operative to collect fault reports during a predetermined time period.

5. Apparatus as set forth in claim 4 wherein said means for associating detected faults is further operative to perform fault report association on those reports collected during said predetermined time period.

6. Apparatus as set forth in claim 4 wherein said predetermined time period commences with the receipt and collection of a fault report at said central location.

7. Apparatus as set forth in claim 1 wherein said nearest neighbor information further comprises at least one pair of link adapter identifiers (LAIDs), wherein the first identifier in each LAID pair identifies both the transmitting unit and link adapter on the transmitting unit that is coupled to a given link on which a fault is detected, and further wherein the second identifier in each LAID pair identifies the unit and link adapter attached to the opposite end of said given link.

8. Apparatus as set forth in claim 1 wherein said transmitting means includes at least one of the link-coupled units of said system.

9. Apparatus as set forth in claim 1 wherein said transmitting means includes at least a pair of link-coupled units of said system.

10. Apparatus as set forth in claim 1 wherein said transmitting means further comprises a set of predefined primary or alternative reporting paths for each unit in said system.

11. Apparatus as in claim 1 in which each of said units comprises means responsive to the loss of an incoming signal on a link for transmitting a predetermined outgoing signal on said link.

12. Apparatus as in claim 11 in which said message generating means, in response to the receipt of a first fault report from a unit at one end of a link indicating the loss of an incoming signal on said link and a second fault report from a unit at the other end of said link indicating the presence of said predetermined signal on said link, generates an error message indicating the receiver at said one end of said link or the driver at said other end of said link as a probable fault source.

13. Apparatus as in claim 1 in which said links are fiber optic links, each of said units comprising means responsive to the loss of incoming light on a link for generating a predetermined outgoing sequence on said link.

14. Apparatus as set forth in claim 1 wherein said means for associating fault reports comprises:

(a) means for collecting fault reports during a predetermined time period; and

(b) means for associating fault reports collected during said predetermined time period utilizing said nearest neighbor information.

15. Apparatus for isolating and analyzing faults in a link-connected system, utilizing fault reports generated from within the system itself, wherein said system comprises a plurality of units interconnected by a plurality of links, and further wherein said units are coupled to a central reporting location, comprising:

(a) a plurality of link adapters associated with said plurality of units for attaching said units to said plurality of links, each link adapter having a unique link adapter identifier (LAID) assigned thereto;

(b) means for locally storing at each unit the LAID of each adapter at a given unit that is coupled to a link, along with the LAID of each remote adapter coupled to said given unit, so as to form locally stored LAID pairs of nearest neighbor information for each link-coupled link adapter of each unit;

(c) means for transmitting fault reports containing said nearest neighbor information, along with any detected fault symptoms, from each unit observing a fault to said central location;

(d) means for storing said fault reports at said central location;

(e) means for associating said stored fault reports utilizing said nearest neighbor information; and

(f) means for generating a single error message from associated fault reports.

16. A method of isolating faults in a link-connected system comprising the steps of:

(a) detecting faults as they occur in said system;

(b) transmitting reports of the detected faults to a reporting location, at least one of said reports including nearest neighbor information relative to the transmitting unit;

(c) associating those fault reports transmitted to said location utilizing said nearest neighbor information to determine which reports are related to a particular error event;

(d) generating a single error message from the associated fault reports relating to said particular error event, wherein said message includes fault isolation and diagnostic information.

17. A method as set forth in claim 16 wherein said step of associating is performed within a predetermined time period.

18. A method as in claim 16 in which said step of associating further comprises the steps of:

(a) beginning a data-gathering interval in response to the receipt of a fault report at said reporting location;

(b) ending said data-gathering interval upon the lapse of a predetermined time period following the beginning of said interval; and

(c) associating those reports received at said location within said interval.

19. A method as in claim 18 in which an interval is started upon the receipt of each report.

20. A method as set forth in claim 16 wherein said step of transmitting is performed by each unit in said system that detects a fault, and further wherein each fault report includes at least an identification of the unit transmitting the fault report, an identification of the link on which a fault was detected by the transmitting unit, and an indication of any detected fault symptoms.

21. A method as set forth in claim 16 wherein said step of transmitting further comprises the step of attempting to transmit each report over a predefined primary or alternative reporting path to said reporting location.

22. A method of isolating faults in a network of units interconnected by links, each link coupling a pair of units via respective link adapters associated with said units, each of said link adapters having a unique identifier distinguishing it from the other link adapters of said network, said method comprising the steps of:

(a) transmitting a fault report to a reporting location in response to the detection of a fault by a link adapter, said fault report including at least any detected symptoms of said fault, the identifier of the link adapter detecting said fault, and nearest neighbor information in the form of the identifier of the link adapter at the opposite end of the link to which the link adapter detecting said fault is attached;

(b) associating fault reports transmitted to said reporting location utilizing said nearest neighbor information; and

(c) generating a single error message from said associated fault reports.

23. A method as set forth in claim 22, further comprising the step of transmitting the identifier of each of said link adapters to the link adapter at the other end of each link coupling a pair of units.

24. A method as set forth in claim 22 wherein said identifier has a first portion identifying the unit and a second portion identifying the link adapter of that unit.

25. A method as in claim 22 in which said step of generating further comprises the steps of:

(a) beginning a data-gathering interval in response to the receipt of a fault report at said reporting location;

(b) ending said data-gathering interval upon the lapse of a predetermined time period following the beginning of said interval; and

(c) generating said message from the reports received within said interval.

26. Apparatus for isolating and analyzing faults in a link-connected system having a plurality of units coupled to a central reporting location and links coupling respective pairs of said units, said units having respective unit identifiers associated therewith, said apparatus comprising:

means for locally storing at each unit the unit identifier associated with said unit, along with the unit identifier of each unit directly attached to said unit by a link, so as to provide for each such link nearest neighbor information comprising a locally stored pair of unit identifiers identifying said link;

means responsive to the detection of a fault on a link by a unit attached to said link for transmitting a fault report containing said nearest neighbor information to said reporting location; and

means for associating fault reports transmitted to said reporting location utilizing said nearest neighbor information.

27. Apparatus as in claim 26 in which said units are attached to said links via respective interfaces having interface identifiers associated therewith, each of said units locally storing the interface identifiers associated with the interfaces of said unit, along with the interface identifiers of each interface directly attached to said unit by a link, so as to provide for each such link nearest neighbor information comprising a locally stored pair of interface identifiers identifying said link.