Software test environment for regression testing ground combat vehicle software

• US 7,441,236 B2
• Filed: 10/27/2005
• Issued: 10/21/2008
• Est. Priority Date: 10/27/2004
• Status: Expired due to Fees

• Current Assignee: BAE Systems Land and Armaments LP (BAE Systems Plc)
• Original Assignee: BAE Systems Land and Armaments LP (BAE Systems Plc)
• Inventors: Ly, Kien, Ly, Kiu, Phung, Vincent, Hamamoto, Brian, Chon, Kumil
• Primary Examiner(s): VU, TUAN A
• Application Number:
  US11/260,508
• Publication Number:
  US 20060212540A1
• Time in Patent Office: 1,090 Days
• Field of Search: 717100-104, 717/135, 709/217, 709/223, 719/313, 702/188, 702/121, 714/724, 714/703, 705/7, 705/9, 703/6, 703/20, 703/23
• US Class Current: 717/135
• CPC Class Codes: G06F 11/261 by simulating additional hardware, e.g. fault simulation
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks

Abstract

A STE for automated testing of ground combat vehicle software application to validate vehicle software logic provides a system and method to conduct interactive (manual) testing of the vehicle software while under development, record information related to the interactive testing activities, including but not limited to tester inputs and expected outcomes, and perform automated testing of the combat vehicle software application using the recorded information. Preferably, reconfiguration of the STE to support changes that arise due to the evolution of the combat vehicle software application system and the subsystems under control of the evolving software system is provided.

Claims

1. A verification system for combat vehicle software application comprising:

A first processor communicatively coupled to at least one combat vehicle software application (CVSA) and a second operating system environment;

a second processor communicatively coupled to a set of extensible simulation primitives and a second operating system environment;

a CVSA configured for operation on the first processor under the first operating system environment, the CVSA communicatively coupled to and configured for communication with a plurality of combat vehicle subsystems (CVSUB)s by exchanging a plurality of messages according to a first predefined protocol, each CVSUB characterized by at least one expected behavior triggered responsive to an exchange of at least one of the plurality of messages with the CVSA, wherein the CVSA is operative to generate at least a first event representing a resultant of one or more interdependent expected behaviors of a first plurality of CVSUBs triggered responsive to the exchange, between the CVSA and the first plurality of CVSUBs, of a first ordered sequence of messages from among the plurality of messages;

a set of extensible simulation primitives comprising computer executable instructions suitable for execution on the second processor under the second operating system environment, the set of extensible simulation primitives designed so that at least one subset of extensible simulation primitives is capable of cooperating to simulate the at least one expected behavior of a selected CVSUB from among the plurality of CVSUBS;

a plurality of simulated subsystems (SIMSUB)s corresponding to the plurality of CVSUIBs, each SIMSUB comprising the at least one subset of extensible simulation primitives capable of cooperating to simulate the at least one expected behavior of the CVSUB that corresponds to the SIMSUB by exchanging the at least one of the plurality of messages with the CVSA sufficient to trigger the at least one expected behavior of the CVSUB;

a graphical user interface (GUI) comprising at least one GUI panel corresponding to each of the SIMSUBs, each GUI panel associated with a selected SIMSUB adapted to enable a user to inject input to and receive output from the selected SIMSUB wherein the input and the output affect the generation of the first event;

a record-replay module executing on the second processor and communicatively coupled to the first plurality of SIMSUBs to selectively store the injected input and the received output from each GUI panel associated with the first plurality of SIMSUBs involved in the exchange of the first ordered sequence of messages with the CVSA in a test vector file, and responsive to user input, replay the test vector file to recreate the exchange of the first ordered sequence of messages between the first plurality of SIMSUBs and an altered CVSA, the record-play module comprising;

a test vector file accessible using the second processor, the test vector file configured to store the injected input and the received output from each GUI panel associated with the first plurality of SIMSUBs involved in the exchange of the first ordered sequence of messages with the CVSA to affect generation of the first event by the CVSA; and

seven components, wherein each component includes computer executable code defining a logic structure, and wherein;

the first component is configured with indicia to indicate and confirm activation of a RECORD mode and to open the test vector file responsive to the RECORD mode being activated;

the second component is configured with storage logic to cause storage of the injected input and the received output in the test vector file;

the third component is configured with indicators to indicate and confirm activation of a REPLAY mode;

the fourth component is provided with an array, the fourth component operative to retrieve the test vector file in which the second component has stored the injected input and transfer the received output as entries in the array before transferring the array to the fifth component;

the fifth component configured with control logic adapted to send the received output received from the fourth component to a comparator module;

the sixth component configured with analysis logic adapted to compare the entries in the array received from the fourth component with the responses received from the SSM to determine a whether a disagreement indicative of a regression failure exists, generate pass/fail results for the test file based on the existence of a disagreement, and store the entries of the array; and

the seventh component configured with report logic adapted to generate a report using the pass/fail results from the sixth component; and

a comparator module communicating with the record-replay module and the GUI, the comparator module operative to capture and dynamically compare the received output from the attempted regeneration of the first event using the altered CVSA to the received output from the generation of the first event by the unaltered CVSA and output indicia indicative of each disagreement between the two;

wherein a user interfacing with the GUI selectively controls the operation of the verification system in a first operating system environment to selectively perform the regression testing of the CVSA operating in a second operating system environment by simulating external CVSUBs and selectively recording and replaying simulated subsystem commands and subsystem responses to generate a pass/fail report from the comparator module to indicate the results of the regression test on the CVSA.

2. The system of claim 1 wherein the set of extensible simulation primitives comprises at least one command primitive, at least one behavior primitive and at least one response primitive wherein the at least one command primitive is configured to receive a command signal from the CVSA, the at least one behavior primitive is configured to simulate the at least one expected behavior of the CVSUB responsive to the at least one command signal and the at least one response primitive is configured to output a response signal generated responsive to the at least one expected behavior.

3. The system of claim 2 further including a bi-directional data bus to communicatively couple the CVSA with the plurality of SIMSUBs for transferring the plurality of messages exchanged between the CVSA and the SIMSUBs.

4. The system of claim 3 wherein each message in the plurality of messages communicated over the bi-directional bus is a TCP/IP packet.

5. The system of claim 4 wherein the set of extensible simulation primitives further includes a translator primitive, the translator primitive interoperable with the CVSA and the SIMSUBs to translate messages between MIL-STD-1553 and TCP/IP wherein the TCP/IP protocol is utilized during transfer of messages over the bi-directional bus.

6. The system of claim 4 further comprising an interface control document (ICD) having information to enable translation of the command signal and the response signal from the MIL-STD-1553 to the TCP/IP communication protocols for transmission on the bi-directional data bus and from the TCP/IP to the MIL-STD-1553 protocol for processing using the first and the second processor.

7. The system of claim 1 wherein the computer executable instructions are coded using JAVA.

8. The system of claim 1 wherein the first predefined protocol is MIL-STD-1553.

9. The system of claim 1 further comprising at least one table based description specifying a relationship between the command signal from the CVSA, the at least one expected behavior of the CVSUB responsive to the command signal and the response signal generated responsive to the at least one expected behavior.

10. The system of claim 1 wherein the at least one expected behavior simulated by the behavior primitive is selected from the group consisting of PASS-THROUGH, INCREMENT, TOGGLE, and GUI-INPUT.

11. The system of claim 1 wherein the CVSUBs are selected from the group consisting of an Improved Bradley Acquisition Subsystem (IBAS), a Gunner'"'"'s Sight Control Panel (GSCP) and a Gunner'"'"'s Hand Station (GHS).

12. The system of claim 1 wherein the disagreement between the received output from the attempted regeneration of the first event using the altered CVSA and the received output from the generation of the first event by the unaltered CVSA includes a time delay between an initiation of the command signal and the receipt of the respective output from the attempted regeneration of the first event using the altered CVSA and from the generation of the first event by the unaltered CVSA.

13. A software test environment (STE) for conducting a regression test of a combat vehicle software application (CVSA) running on a first processor under a first operating system environment, the CVSA configured to interact with combat vehicle software subsystems applications (CVSUBAs) by encoded messages communicated over a data bus to generate a set of events, each event resulting from an interaction of one or more CVSUBAs with the CVSA, the STE comprising:

a first processor communicatively coupled to a CVSA and a first operating system environment;

a second processor communicatively coupled to one or more simulated system modules (SSMs);

a simulated subsystem module (SSM) having computer readable instructions defining a separate logic structure for each of the CVSUBAs, the logic structures configured for operation on the second processor under a second operating system environment wherein each separate logic structure simulates at least one operation of at least one of the CVSUBAs, wherein a first event results from one or more simulated operations representing the interaction of the at least one CVSUBA or a combination of CVSUBAs with the CVSA using a first set of electronic message interchanges;

a bi-directional communication link communicatively coupled to the first and second processors to allow an exchange of electronic messages between the CVSA and the SSM using a predetermined communications protocol;

a translator module coupled intermediate the bi-directional communication link and each of the first and second processors, the translator module configured to provide a communication interface between the CVSA and the SSM;

a graphical user interface (GUI) coupled to the SSM and having a plurality of control panels for data input so as to initiate and to continue the generation of the first event;

a record-replay module communicatively coupled to the SSM to selectively store the electronic message interchanges associated with the generation of the first event in a first test file and responsive to user input, replay the first test file to selectively supply at least one of the first set of electronic message interchanges stored therein to the SSM, in lieu of the data input, to attempt to initiate and automatically continue the generation of the first event, the record-replay module effective to dynamically compare the resulting second set of electronic message interactions with the first set of electronic message interactions to determine a disagreement between them wherein the disagreement is indicative of the CVSA failing the regression test, the record-play module comprising;

a first test file comprising the electronic message interchanges that include subsystem commands and subsystem responses; and

seven components, wherein each component includes computer executable code defining a logic structure, and wherein;

the first component is configured with indicia to indicate and confirm activation of a RECORD mode and to open the at least one test file responsive to the RECORD mode being activated;

the second component is configured with storage logic to cause storage of the subsystem commands and subsystem responses in the test file;

the third component is configured with indicators to indicate and confirm activation of a REPLAY mode;

the fourth component is provided with an array, the fourth component operative to retrieve the test file in which the second component has stored the simulated subsystem commands and subsystem responses and transfer the stored subsystem commands and subsystem responses as entries in the array before transferring the array to the fifth and the sixth component;

the fifth component configured with control logic adapted to send the stored subsystem commands and subsystem responses received from the fourth component to the SSM;

the sixth component configured with analysis logic adapted to compare the entries in the array received from the fourth component with the responses received from the SSM to determine a whether a disagreement indicative of a regression failure exists, generate pass/fail results for the test file based on the existence of a disagreement, and store the entries of the array; and

the seventh component configured with report logic adapted to generate a report using the pass/fail results from the sixth component; and

wherein a user interfacing with the GUI selectively controls the operation of the STE in a first operating system environment to selectively perform the regression testing of the CVSA operating in a second operating system environment by simulating external combat vehicle subsystems and selectively recording and replaying simulated subsystem commands and subsystem responses to generate a pass/fail report to indicate the results of the regression test on the CVSA.

14. The software test environment (STE) of claim 13 wherein the simulated subsystem module (SSM) comprises:

a first component, a second component and a third component, wherein the first component is communicatively coupled to the second component and the second component is communicatively coupled to the third component, wherein;

the first component is placed in data communication with the translator module and operative to send and receive MIL-STD-1553 data to and from the translator communication interface between the CVSA and the SSM;

the second component is placed in data communication with the record-replay module and provided with an interface control document (ICD) associated with each CVSUBA for encoding and decoding MIL-STD-1553 data, the second component operative to send encoded MIL-STD-1553 data to the record-replay module for recording when a RECORD mode is activated and to receive a record-replay input from the record-replay module for encoding into MIL-STD-1553 data before forwarding the record-replay input to the first component; and

the third component is placed in data communication with the GUI, and operative to determine a response to a command received from the CVSA based on the ICD and to send the response to the GUI and optionally to the second component for encoding into MIL-STD-1553 data, the third component also configured to receive a gui-input from the GUI and the record-replay input from the record-replay module, determine responses to each of the gui-input and the record-replay input and send the responses to the second component for encoding.

15. The software test environment of claim 13 wherein the GUI comprises first, second, third and fourth components with the first and second components placed in data communication with a tester module, the second component placed in data communication with the third and fourth component, the third and the fourth components placed in data communication with the simulated subsystem (SSM);

wherein;

the first component is provided with a first GUI panel for test mode selection from the tester module, the GUI panel provided with display of icons to allow user interaction, the icons selected from the group consisting of toggle switches, radio buttons, user text fields, checkboxes, LEDs and labels;

the second component is provided with a plurality of second GUI panels to allow user selection of a user selected simulated subsystem and to generate at least one stimulus;

the third component is provided with a control panel corresponding to the user selected simulated subsystem, the control panel configured to present GUI objects for user-selection, selected GUI objects translated by the third component into messages and transmitted to the user selected simulated subsystem; and

the fourth component is provided with a display panel to display a status of the user selected simulated subsystem by processing the at least one stimulus together with a response generated by the selected simulated subsystem responsive to the at least one stimulus, the display panel having buttons, labels and LEDs to display the status.

16. The system of claim 13 wherein the predetermined communications protocol is TCP/IP.

17. The system of claim 13 further including a tester module having a first component and a second component, the first component communicatively coupled to the GUI and operative to select objects for display on the GUI;

and the second component communicatively coupled to the GUI and the record-replay module, the second component operative to select a test mode from the group consisting of the RECORD mode and the REPLAY mode and send the selected test mode to the GUI and the record-replay module.

18. A system for regression testing combat vehicle software application configured for cooperative operation with at least one combat vehicle subsystem, the system comprising:

a first processor operating under the control of a first operating system;

at least one second processor operating under the control of a second operating system;

at least one data store communicatively coupled to at least one of the first and the second processors;

a combat vehicle software application (CVSA) configured to execute on the first processor, the CVSA operative to receive regression testing related information defining at least one stimulus and to selectively generate regression testing related information defining at least one signal command responsive to the at least one stimulus; and

at least one reconfigurable logic module including instructions executable on the second processor, the instructions operative to generate regression testing related information defining a signal response responsive to the at least one signal command and regression testing related information defining a time for response, the signal response indicative of a predefined behavior of the at least one combat vehicle subsystem in response to the at least one signal command, the time for response indicative of an anomalous delay between the at least one stimulus and the signal response;

a bi-directional communication link operative to communicatively couple the first and the second processors for transmission of regression testing related information between the combat vehicle software and the at least one reconfigurable logic module;

a translator module configured to interoperate with the bi-directional communication link and the combat vehicle software at the first processor and with the bi-directional communication link and the at least one reconfigurable logic module at the second processor, the translator module operative to reformat regression testing related information for each passage of the regression testing related information to and from the bi-directional communication link;

an input device configured to receive user input and to generate regression testing related information defining an input signal; and

a record-replay module in data communication with the at least one reconfigurable logic module and the at least one data store, the record-replay module comprising;

at least one test file that receives the input signal from the input device, including the subsystem commands and subsystem responses; and

seven components, wherein each component includes computer executable code defining a logic structure, and wherein;

the first component is configured with indicia to indicate and confirm activation of a RECORD mode and to open the at least one test file responsive to the RECORD mode being activated;

the second component is configured with storage logic to cause storage of the subsystem commands and subsystem responses in the at least one test file;

the third component is configured with indicators to indicate and confirm activation of a REPLAY mode;

the fourth component is provided with an array, the fourth component operative to retrieve the at least one test file in which the second component has stored the simulated subsystem commands and subsystem responses and transfer the stored subsystem commands and subsystem responses as entries in the array before transferring the array to the fifth and the sixth component;

the fifth component configured with control logic adapted to send the stored subsystem commands and subsystem responses received from the fourth component to the SSM;

the sixth component configured with analysis logic adapted to compare the entries in the array received from the fourth component with the responses received from the SSM to determine whether a disagreement indicative of a regression failure exists, generate pass/fail results for the test file based on the existence of a disagreement, and store the entries of the array; and

the seventh component configured with report logic adapted to generate a report using the pass/fail results from the sixth component;

such that the record-replay module is operative to record the regression testing related information corresponding to each test case wherein each test case is characterized by the regression testing related information flowing through the system beginning with the regression testing related information defining at least one stimulus and ending with the regression testing related information defining the at least one signal response wherein a difference between regression testing related information corresponding to two test cases is indicative of a regression of the system.