Persistent context-based behavior injection or testing of a computing system

• US 6,976,189 B1
• Filed: 03/22/2002
• Issued: 12/13/2005
• Est. Priority Date: 03/22/2002
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: Network Appliance Incorporated (NetApp, Inc.)
• Inventors: Schoenthal, Scott, Viswanathan, Srinivasan
• Primary Examiner(s): Iqbal, Nadeem
• Assistant Examiner(s): Bonura, Timothy M.
• Application Number:
  US10/105,060
• Time in Patent Office: 1,362 Days
• Field of Search: 714/35, 714/41
• US Class Current: 714/41
• CPC Class Codes: G06F 11/3672 Test management

Abstract

The invention provides a method and system for persistent context-based behavior injection in a computing system, such as in a redundant storage system or another system having a layered or modular architecture. Behaviors that are injected can be specified to have triggering conditions, such that the behavior is not injected unless the conditions are true. Triggering conditions may include a selected ordering of conditions and a selected context for each behavior. In a system having a layered architecture, behavior injection might be used to evaluate correct responses in the face of cascaded errors in a specific context or thread, other errors that are related by context, concurrent errors, or multiple errors. Behavior injection uses non-volatile memory to preserve persistence of filter context information across possible system errors, for reporting of the results of behavior injection, and to preserve information across recovery from system errors. Multiple behavior injection threads are also provided. Behavior injection can also be performed in a logically distributed system or from a logically remote system.

Claims

1. A method, including steps of performing a set of program instructions using a computing device, said program instructions including one or more breakout lines;

maintaining a set of context information regarding one or more operations in progress by said computing device;

injecting a behavior in response to at least one of said breakout lines, conditionally in response to at least a portion of said context information;

wherein said instructions include a plurality of independent software components; and

said context information is preserved across transfers of control within computing device between instructions for different ones of said software components.

2. A method as in claim 1, including steps of reporting a result of said steps of injecting.

3. A method as in claim 1, including steps of reporting a result of said steps of injecting;

and

altering said program instructions in response to a result of said steps of reporting, and producing an altered set of said program instructions which complies with developer expectations with regard to said injected behaviors.

4. A method as in claim 1, wherein a result of said injecting steps includes a simulation of a set of cascaded errors.

5. A method as in claim 1, wherein a result of said injecting steps includes a simulation of a set of concurrent errors in response to multiple requests received by said computing device.

6. A method as in claim 1, wherein a result of said injecting steps includes a simulation of a correct behavior.

7. A method as in claim 1, wherein a result of said injecting steps includes at least one of:

altering a system variable, causing a system panic, causing a system reboot, modifying contents of system memory.

8. A method as in claim 1, wherein

said computing device includes a redundant data storage system;

and

said context information includes information regarding a file server request.

9. A method as in claim 1, wherein said context information includes at least one of:

a request received by said computing device;

a set of behaviors already injected in response to at least one said breakout line.

10. A method as in claim 1, wherein

said computing device performs multiple threads of said instructions concurrently;

and

said context information for a first said thread is responsive to at least one operation in progress in a second said thread.

11. A method as in claim 1, wherein

said set of program instructions are maintained on a first said computing device;

and

said steps of injecting are performed at least in part at a second said computing device different from said first computing device.

12. A method as in claim 1, wherein said steps of injecting are responsive to one or more behavior injection filters, each said behavior injection filter including at least one associated condition and at least one associated behavior for injection.

13. A method as in claim 1, including a non-volatile memory, said nonvolatile memory including sufficient information to preserve persistence of said context information across a system reboot.

14. A method as in claim 13, wherein said injected behavior includes a system reboot.

15. A method as in claim 1, wherein said context information is updated in response to said transfers of control.

16. A method as in claim 1, wherein

said software components are disposed in an architecture having a first and second layer;

said context information is incremented in response to transfers of control from said first to said second layer; and

said context information is decremented in response to transfers of control from said second to said first layer.

17. A method, including steps of

performing a set of program instructions using a computing device, said program instructions including one or more breakout lines;

maintaining a set of context information regarding one or more operations in progress by said computing device;

injecting a behavior in response to at least one of said breakout lines conditionally in response to at least a portion of said context information;

wherein

said steps of injecting are responsive to a plurality of behavior injection filters, each said behavior injection filter including at least one associated condition and at least one associated behavior for injection; and

said context information includes a set of trigger information for a first one of said behavior injection filters, said trigger information indicative of whether said first behavior injection filter has been triggered within said context;

and including steps of

injecting a second behavior in response to said trigger information and in response to a change in said context information.

18. A method, including steps of

performing a set of program instructions using a computing device said program instructions including one or more breakout lines;

maintaining a set of context information regarding one or more operations in progress by said computing device;

injecting a behavior in response to at least one of said breakout lines conditionally in response to at least a portion of said context information;

wherein

said steps of injecting are responsive to a plurality of behavior injection filters, each said behavior injection filter including at least one associated condition and at least one associated behavior for injection;

said context information includes a set of trigger information for a first one of said behavior injection filters, said trigger information indicative of whether said first behavior injection filter has been triggered within said context;

said sequence of instructions includes one or more scope update lines, said scope update lines being indicative of a change in said context information;

and including steps of injecting a second behavior in response to said trigger information and in response to at least one of said scope update lines.

19. Apparatus including

a computing device having a set of program instructions, said program instructions including one or more breakout lines;

a storage medium accessible by said computing device, said storage medium having a set of context information regarding one or more operations in progress by said computing device;

wherein said computing device includes an instruction responsive to at least one of said breakout lines, said instruction causing a behavior to be injected in response to said one breakout line, conditionally in response to at least a portion of said context information;

wherein

said instructions include a plurality of independent software components; and

said computing device includes an instruction preserving said context information across transfers of control within said computing device between instructions for different ones of said software components.

20. Apparatus as in claim 19, wherein said storage medium includes a result of said steps of injecting.

21. Apparatus as in claim 19, wherein

said computing device includes a redundant data storage system;

and

said context information includes information regarding a file server request.

22. Apparatus as in claim 19, wherein said context information includes at least one of:

a request received by said computing device;

a set of behaviors already injected in response to at least one said breakout line.

23. Apparatus as in claim 19, wherein

said computing device includes multiple concurrent threads of said instructions;

and

said context information for a first said thread includes information regarding at least one operation in progress in a second said thread.

24. Apparatus as in claim 19, wherein

said set of program instructions are maintained on a first said computing device;

and

said instruction causing a behavior to be injected causes at least one additional instruction to be performed at least in part at a second said computing device different from said first computing device.

25. Apparatus as in claim 19, including one or more behavior injection filters, each said behavior injection filter including at least one associated condition and at least one associated behavior for injection.

26. Apparatus as in claim 19, including a non-volatile memory, said nonvolatile memory including sufficient information to preserve persistence of said context information across a system reboot.

27. Apparatus as in claim 19, wherein said computing device includes an instruction updating said context information in response to said transfers of control.

28. Apparatus as in claim 19, wherein

said software components are disposed in an architecture having a first and second layer;

said computing device includes an instruction incrementing said context information in response to transfers of control from said first to said second layer; and

said computing device includes an instruction decrementing said context information in response to transfers of control from said second to said first layer.

29. Apparatus, including

a computing device having a set of program instructions, said program instructions including one or more breakout lines;

a storage medium accessible by said computing device said storage medium having a set of context information regarding one or more operations in progress by said computing device;

wherein said computing device includes an instruction responsive to at least one of said breakout lines, said instruction causing a behavior to be injected in response to said one breakout line, conditionally in response to at least a portion of said context information;

and including a plurality of behavior injection filters, each said behavior injection filter including at least one associated condition and at least one associated behavior for injection;

wherein

said context information includes a set of trigger information for a first one of said behavior injection filters, said trigger information indicative of whether said first behavior injection filter has been triggered within said context; and

said computing device includes an instruction responsive to said trigger information, said instruction causing a second behavior to be injected in response to a change in said context information.

30. Apparatus including

a computing device having a set of program instructions, said program instructions including one or more breakout lines;

a storage medium accessible by said computing device, said storage medium having a set of context information regarding one or more operations in progress by said computing device;

wherein said computing device includes an instruction responsive to at least one of said breakout lines, said instruction causing a behavior to be injected in response to said one breakout line, conditionally in response to at least a portion of said context information;

and including a plurality of behavior injection filters, each said behavior injection filter including at least one associated condition and at least one associated behavior for injection;

wherein

said context information includes a set of trigger information for a first one of said behavior injection filters, said trigger information indicative of whether said first behavior injection filter has been triggered within said context;

said sequence of instructions includes one or more scope update lines, said scope update lines being indicative of a change in said context information; and

said computing device includes an instruction responsive to said trigger information, said instruction causing a second behavior to be injected in response to at least one of said scope update lines.