Software test environment for regression testing ground combat vehicle software

• US 20060212540A1
• Filed: 10/27/2005
• Published: 09/21/2006
• Est. Priority Date: 10/27/2004
• Status: Active Grant

• Current Assignee: BAE Systems Land & Armaments Incorporated (BAE Systems Plc)
• Original Assignee: BAE Systems Land & Armaments Incorporated (BAE Systems Plc)
• Inventors: Ly, Kien, Ly, Kiu, Phung, Vincent, Hamamoto, Brian, Chon, Kumil
• Granted Patent: US 7,441,236 B2
• Time in Patent Office: Days
• US Class Current: 709/218
• CPC Class Codes: G06F 11/261 by simulating additional hardware, e.g. fault simulation
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks

Abstract

A STE for automated testing of ground combat vehicle software application to validate vehicle software logic provides a system and method to conduct interactive (manual) testing of the vehicle software while under development, record information related to the interactive testing activities, including but not limited to tester inputs and expected outcomes, and perform automated testing of the combat vehicle software application using the recorded information. Preferably, reconfiguration of the STE to support changes that arise due to the evolution of the combat vehicle software application system and the subsystems under control of the evolving software system is provided.

Claims

1. A verification system for combat vehicle software application comprising:

at least one combat vehicle software application (CVSA) configured for operation on a first processor under a first operating system environment, the CVSA communicatively coupled to and configured for communication with a plurality of combat vehicle subsystems (CVSUB)s by exchanging a plurality of messages according to a first predefined protocol, each CVSUB characterized by at least one expected behavior triggered responsive to an exchange of at least one of the plurality of messages with the CVSA, wherein the CVSA is operative to generate at least a first event representing a resultant of one or more interdependent expected behaviors of a first plurality of CVSUIBs triggered responsive to the exchange, between the CVSA and the first plurality of CVSUBs, of a first ordered sequence of messages from among the plurality of messages;

a set of extensible simulation primitives comprising computer executable instructions suitable for execution on a second processor under a second operating system environment, the set of extensible simulation primitives designed so that at least one subset of extensible simulation primitives is capable of cooperating to simulate the at least one expected behavior of a selected CVSUB from among the plurality of CVSUBS;

a plurality of simulated subsystems (SIMSUB)s corresponding to the plurality of CVSUBs, each SIMSUB comprising the at least one subset of extensible simulation primitives capable of cooperating to simulate the at least one expected behavior of the CVSUB that corresponds to the SIMSUB by exchanging the at least one of the plurality of messages with the CVSA sufficient to trigger the at least one expected behavior of the CVSUB;

a graphical user interface (GUI) comprising at least one GUI panel corresponding to each of the SIMSUBs, each GUI panel associated with a selected SIMSUB adapted to enable a user to inject input to and receive output from the selected SIMSUB wherein the input and the output affect the generation of the first event;

a test vector file accessible using the second processor, the test vector file configured to store the injected input and the received output from each GUI panel associated with the first plurality of SIMSUBs involved in the exchange of the first ordered sequence of messages with the CVSA to affect generation of the first event by the CVSA;

a replay module executing on the second processor, the replay module configured to attempt regeneration of the first event by replaying the stored injected input from the data structure to recreate the exchange of the first ordered sequence of messages between the first plurality of SIMSUBs and an altered CVSA; and

a comparator module communicating with the test vector file, the replay module and the GUI, the comparator module operative to capture and dynamically compare the received output from the attempted regeneration of the first event using the altered CVSA to the received output from the generation of the first event by the unaltered CVSA and output indicia indicative of each disagreement between the two.

2. The system of claim 1 wherein the set of extensible simulation primitives comprises at least one command primitive, at least one behavior primitive and at least one response primitive wherein the at least one command primitive is configured to receive a command signal from the CVSA, the at least one behavior primitive is configured to simulate the at least one expected behavior of the CVSUB responsive to the at least one command signal and the at least one response primitive is configured to output a response signal generated responsive to the at least one expected behavior.

3. The system of claim 1 wherein the computer executable instructions are coded using JAVA.

4. The system of claim 1 wherein the first predefined protocol is MIL-STD-1553.

5. The system of claim 2 further including a bi-directional data bus to communicatively couple the CVSA with the plurality of SIMSUBs for transferring the plurality of messages exchanged between the CVSA and the SIMSUBs.

6. The system of claim 5 wherein each message in the plurality of messages communicated over the bi-directional bus is a TCP/IP packet.

7. The system of claim 6 wherein the set of extensible simulation primitives further includes a translator primitive, the translator primitive interoperable with the CVSA and the SIMSUBs to translate messages between MIL-STD-1553 and TCP/IP wherein the TCP/IP protocol is utilized during transfer of messages over the bi-directional bus.

8. The system of claim 6 further comprising an interface control document (ICD) having information to enable translation of the command signal and the response signal from the MIL-STD-1553 to the TCP/IP communication protocols for transmission on the bi-directional data bus and from the TCP/IP to the MIL-STD-1553 protocol for processing using the first and the second processor.

9. The system of claim 1 further comprising at least one table based description specifying a relationship between the command signal from the CVSA, the at least one expected behavior of the CVSUB responsive to the command signal and the response signal generated responsive to the at least one expected behavior;

10. The system of claim 1 wherein the at least one expected behavior simulated by the behavior primitive is selected from the group consisting of PASS-THROUGH, INCREMENT, TOGGLE, and GUI-INPUT.

11. The system of claim 1 wherein the CVSUBs are selected from the group consisting of an Improved Bradley Acquisition Subsystem (IBAS), a Gunner'"'"'s Sight Control Panel (GSCP) and a Gunner'"'"'s Hand Station (GHS).

12. The system of claim I wherein the disagreement between the received output from the attempted regeneration of the first event using the altered CVSA and the received output from the generation of the first event by the unaltered CVSA includes a time delay between an initiation of the command signal and the receipt of the respective output from the attempted regeneration of the first event using the altered CVSA and from the generation of the first event by the unaltered CVSA.

13. A computer-implemented method for determining an effect of changed code corresponding to a second version of a combat vehicle software application (CVSA) relative to a first version of the CVSA, the CVSA executing in a first operating system environment on a first processor and configured to interact with a plurality of external combat vehicle subsystems, the method comprising the steps of:

a) providing a plurality of separate combat vehicle subsystem simulators (CVSUIBS) executing in a second operating system environment on a second processor wherein some of the CVSUBS are different from others in the plurality and wherein each of the CVSUBS is equipped with a behavioral model to simulate an operation of one of the plurality of external combat vehicle subsystems by receiving command information, generated by the CVSA responsive to receiving stimuli information, and transmitting response information produced by applying the received command information to the behavioral model;

b) communicatively coupling the CVSA to the CVSUBS by using a bi-directional bus to allow each of the CVSUBS to communicate with the CVSA in accordance with a predetermined communications protocol;

c) supplying selected stimuli information to the CVSA to cause the CVSA to selectively interact with one or more CVSUBS and generate a simulation scenario from the response information output at the selected one or more CVSUBS;

d) collecting and storing the stimulus, command and response information produced during the generation of a first simulation scenario using the first version of the CVSA code and during the generation of the first simulation scenario using the second version of the CVSA code; and

e) analyzing the collected and stored information to determine the effect of changed code corresponding to the second version of the CVSA relative to the first version of the CVSA code.

14. The method of claim 13 wherein the collecting and storing step includes:

collecting the information sequentially according to a time at which the information is generated to create a first and second time ordered sequence of information corresponding to each of the first and second versions of the CVSA code respectively; and

storing each of the time ordered sequences of information in a first and second test vector file respectively.

15. The method of claim 14 wherein the analyzing step comprises the step of

comparing the first time ordered sequence of information stored in the first test vector file with the second time ordered sequence of information stored in the second test vector file;

tabulating a first set comprising differences between an ordering of a first piece of information in the first time ordered sequence and the ordering of the corresponding first piece of information in the second time ordered sequence; and

tabulating a second set comprising deviations in an elapsed time between generation of a first piece of information and a second piece of information in the first time ordered sequence and generation of the corresponding first and second pieces of information in the second time ordered sequence;

wherein each of the first and second sets is indicative of the effect of changed code in the second version of the CVSA code relative to the first version of the CVSA code.

16. The method of claim 13 wherein each of the command information, stimulus information and response information is in MIL-STD-1553 format that specifies a remote terminal number, a sub-address, a word count and data words.

17. The method of claim 16 wherein the communicatively coupling step further includes the steps of:

providing a communication interface interposed between the bi-directional bus and the CVSA and the bi-directional bus and the CVSUBS; and

using the communication interface to translate MIL-STD-1553 formatted information into a predetermined communications protocol and the predetermined communications protocol into MIL-STD-1553 formatted information before transmission to and after receipt from the bi-directional bus.

18. The method of claim 17 wherein the predetermined communications protocol is TCP/IP wherein each TCP/IP packet is characterized by a predefined header.

19. The m of claim 16 wherein the step of providing the plurality of separate combat vehicle subsystem simulators (CVSUBS) comprises the step of:

providing an interface control document (ICD) to decode MIL-STD-1553 data to obtain decoded MIL-STD-1553 data and encode MIL-STD-1553 data to obtain encoded MIL-STD-1553 data.

20. The method of claim 13 wherein the supplying selected stimuli information to the CVSA step comprises:

providing a graphical user interface (GUI) communicatively coupled to the CVSUBS for receiving user input;

translating the user input into a stimulus information;

formatting the stimulus information into a predetermined communications protocol for transmission over the bi-directional bus; and

transmitting the stimulus information to the CVSA over the bidirectional bus.

21. A software test environment (STE) for conducting a regression test of a combat vehicle software application (CVSA) running on a first processor under a first operating system environment, the CVSA configured to interact with combat vehicle software subsystems applications (CVSUBAs) by encoded messages communicated over a data bus to generate a set of events, each event resulting from an interaction of one or more CVSUBAs with the CVSA, the STE comprising:

a simulated subsystem module (SSM) having computer readable instructions defining a separate logic structure for each of the CVSUBAs, the logic structures configured for operation on a second processor under a second operating system environment wherein each separate logic structure simulates at least one operation of one of the CVSUBAs wherein a first event results from one or more simulated operations representing the interaction of the at least one CVSUBA or a combination of CVSUBAs with the CVSA using a first set of electronic message interchanges;

a bi-directional communication link communicatively coupled to the first and second processors to allow an of exchange electronic messages between the CVSA and the SSM using a predetermined communications protocol;

a translator module coupled intermediate the bi-directional communication link and each of the first and second processors, the translator module configured to provide a communication interface between the CVSA and the SSM;

a graphical user interface (GUI) coupled to the SSM and having a plurality of control panels for data input so as to initiate and to continue the generation of the first event;

a record-replay module coupled to the SSM to selectively store the electronic messages interactions, associated with the generation of the first event, in a first test file and responsive to user input, replay the first test file to selectively supply at least one of the first set of electronic message interactions stored therein to the SSM, in lieu of the data input, to attempt to initiate and automatically continue the generation of the first event, the record-replay module effective to dynamically compare the resulting second set of electronic message interactions with the first set of electronic message interactions to determine a disagreement between them wherein the disagreement is indicative of the CVSA failing the regression test.

22. The software test environment (STE) of claim 21 wherein the simulated subsystem module (SSM) comprises:

a first component, a second component and a third component, wherein the first component is communicatively coupled to the second component and the second component is communicatively coupled to the third component, wherein;

the first component is placed in data communication with the translator module and operative to send and receive MIL-STD-1553 data to and from the translator communication interface between the CVSA and the SSM;

the second component is placed in data communication with the record-replay module and provided with an interface control document (ICD) associated with each CVSUBA for encoding and decoding MIL-STD-1553 data, the second component operative to send encoded MIL-STD-1553 data to the record-replay module for recording when a RECORD mode is activated and to receive a record-replay input from the record-replay module for encoding into MIL-STD-1553 data before forwarding the record-replay input to the first component; and

the third component is placed in data communication with the GUI, and operative to determine a response to a command received from the CVSA based on the ICD and to send the response to the GUI and optionally to the second component for encoding into MIL-STD-1553 data, the third component also configured to receive a gui-input from the GUI and the record-replay input from the record-replay module, determine responses to each of the gui-input and the record-replay input and send the responses to the second component for encoding.

23. The software test environment of claim 21 wherein the GUI comprises first, second, third and fourth components with the first and second components placed in data communication with a tester module, the second component placed in data communication with the third and fourth component, the third and the fourth components placed in data communication with the simulated subsystem (SSM);

wherein;

the first component is provided with a first GUI panel for test mode selection from the tester module, the GUI panel provided with display of icons to allow user interaction, the icons selected from the group consisting of toggle switches, radio buttons, user text fields, checkboxes, LEDs and labels;

the second component is provided with a plurality of second GUI panels to allow user selection of a user selected simulated subsystem and to generate at least one stimulus;

the third component is provided with a control panel corresponding to the user selected simulated subsystem, the control panel configured to present GUI objects for user-selection, selected GUI objects translated by the third component into messages and transmitted to the user selected simulated subsystem; and

the fourth component is provided with a display panel to display a status of the user selected simulated subsystem by processing the at least one stimulus together with a response generated by the selected simulated subsystem responsive to the at least one stimulus, the display panel having buttons, labels and LEDs to display the status.

24. The software test environment of claim 21 wherein the record-replay module is provided with a test case directory for storing at least one test case file and optionally a batch test file;

and

seven components wherein each component includes computer executable code defining a logic structure;

wherein;

the first and third components are communicatively coupled to the tester module;

the first component is communicatively coupled to the second component with the second component placed in data communication with a simulated subsystem;

the third component is communicatively coupled to the fourth component with the fourth component communicatively coupled to the fifth component, the fifth component being placed in data communication with the simulated subsystem;

the sixth component is communicatively coupled to the simulated subsystem for receiving responses and to the seventh component;

wherein

the first component is configured with indicia to indicate and confirm activation of a RECORD mode and to open the at least one test case file responsive to the RECORD mode being activated;

the second component is configured with storage logic to cause storage of the simulated subsystem commands and responses in the test file;

the third component is configured with indicators to indicate and confirm activation of the REPLAY mode;

the fourth component is provided with an array, the fourth component operative to retrieve the test file in which the second component has stored the simulated subsystem commands and responses and transfer the stored commands and responses as entries in the array before transferring the array to the fifth and the sixth component;

the fifth component configured with control logic adapted to send the stored commands and responses received from the fourth component to the simulated subsystem;

the sixth component configured with analysis logic adapted to compare the entries in the array received from the fourth component with the responses received from the simulated subsystem and generate pass/fail results for the test file storing the entries of the array; and

the seventh component configured with report logic adapted to generate a report using the pass/fail results from the sixth component.

25. The system of claim 21 wherein the predetermined communications protocol is TCP/IP.

26. The system of claim 21 further including a tester module having a first component and a second component, the first component communicatively coupled to the GUI and operative to select objects for display on the GUI;

and the second component communicatively coupled to the GUI and the record-replay module, the second component operative to select a test mode from the group consisting of the RECORD mode and the REPLAY mode and send the selected test mode to the GUI and the record-replay module.

27. A method to verify a combat vehicle software application used to control the operation of combat vehicle subsystems wherein the combat vehicle software applications communicate with the combat vehicle subsystems over a communication bus using a MIL-STD-1553 protocol, the method comprising the steps of:

a) executing a combat vehicle software application (CVSA) on a first processor under a first operating system;

b) providing a table-based behavioral description of each combat vehicle subsystem (CVSUB) for access thru a computer code executing on a second processor under a second operating system, the table-based behavioral description defining a relationship between at least one command signal from the CVSA, at least one behavior associated with at least one combat vehicle subsystem (CVSUB) and at least one response signal output by the CVSUB based upon the at least one behavior;

c) creating a set of simulation primitives, each simulation primitive comprising a set of computer instructions executable on the second processor and configured to accept input, perform a basic operation based upon the input and optionally generate an output;

d) configure a selected subset of the set of simulation primitives corresponding to each selected CVSUB to create a simulated subsystem (SIMSUB) representing a behavioral model of the selected CVSUB, the subset of simulation primitives communicatively coupled through their inputs and outputs for cooperative operation wherein each subset of simulation primitives presents a common input interface and a common output interface, the SIMSUB configured to receive the at least one command signal and at least one user data input at the common input interface, retrieve the table-based behavioral description related to the at least one command signal, perform basic operations resulting in the at least one behavior associated with at least one combat vehicle subsystem (CVSUB) and output the at least one response associated with the at least one behavior of the CVSUB;

e) providing a graphical user interface (GUI) comprising one or more GUI panels, each SIMSUB associated with at least one GUI panel, each GUI panel having computer executable instructions to generate a user-manipulable display coupled to the second processor for enabling user data input to and echoing information from the SIMSUB corresponding to the GUI panel;

f) calling up the GUI panel associated with a selected SIMSUB and inputting the at least one user data input using the GUI panel, the SIMSUB configured to generate a response signal representing a stimulus for transmission to the CVSA responsive to the user data input, the stimulus effective to trigger a command signal by the CVSA;

g) sequentially recording the user data input at the GUI panel and the information from the SIMNSUB echoed at GUI panel responsive to the user data input, the user data input and the information echoed at the GUI panel affecting generation of a first event by the CVSA;

h) replaying the recorded user data input to the GUI panel after an alteration to the CVSA to attempt regeneration of the first event by the CVSA;

i) dynamically determining each agreement and each disagreement between the information from the SIMSUB echoed at the GUI panel before and after the alteration; and

j) outputting a pass and a fail corresponding to each agreement and each disagreement respectively.

28. The method of claim 27 further including providing a bi-directional data bus for data communication between the CVSA and each SIMSUB.

29. The method of claim 28 wherein the data communication between the CVSA and the SIMSUB is by transferring TCP/IP packets.

30. The method of claim 28 further including the step of translating communications between the CVSA and each SIMSUB to and from MIL-STD-1553 protocol and the TCP/IP protocol wherein the TCP/IP protocol is used for communications over the bi-directional data bus and the MIL-STD-1553 protocol is used during processing by the CVSA and each SIMSUB.

31. The method of claim 27 wherein the table-based behavioral description is provided using an interface control document (ICD).

32. The method of claim 27 wherein the at least one expected behavior is selected from the group consisting of PASS-THROUGH, INCREMENT, TOGGLE, and GUI-INPUT.

33. The method of claim 27 wherein the CVSUBs are selected from the group consisting of an Improved Bradley Acquisition Subsystem (IBAS), a Gunner'"'"'s Sight Control Panel (GSCP) and a Gunner'"'"'s Hand Station (GHS).

34. The method of claim 27 wherein the first event is selected from the group consisting of selecting a last target range, activating a laser range finder for a return range of the selected last target and adjusting a sight video gain level.

35. A system for testing a combat vehicle software application the system comprising:

a combat vehicle software (CVS) logic structure for execution in a first operating system environment on a first processor, the logic structure configured to receive a plurality of signal inputs;

a stimuli selection logic to select at least one stimulus to apply to the CVS logic structure from among the plurality of signal inputs;

at least one signal command output by the CVS logic structure responsive to the at least one stimulus;

a plurality of reconfigurable vehicle software subsystem (VSS) logic structures for execution in a second operating system environment on a second processor, the VSS logic structures configured to respond deterministically to the at least one signal command, at least one of the plurality of reconfigurable vehicle software subsystem (VSS) logic structures configured for application to the at least one signal command output corresponding to the at least one stimulus;

at least one event output resulting from the application of the at least one VSS logic structure to the at least one signal command output corresponding to the at least one stimulus;

a time delay associated with the at least one event output, the time delay representing the time elapsed between the application of the at least one stimulus and the occurrence of the at least one event output;

a bus interface communicatively coupling the first processor to the second processor, the bus interface adapted to transmit the selected at least one stimulus for receipt by the CVS logic structure and the at least one signal command for receipt by the VSS logic structure;

a test vector to store the selected at least one stimulus and information resulting from the application of the selected at least one stimulus to the CVS logic structure including the at least one signal command, the at least one event and the time delay; and

a comparator for evaluating successive test vectors generated by the CVS logic structure to determine whether the CVS logic structure is operating properly.

36. A system for regression testing combat vehicle software application configured for cooperative operation with at least one combat vehicle subsystem, the system comprising:

a first processor operating under the control of a first operating system;

at least one second processor operating under the control of a second operating system;

at least one data store communicatively coupled to at least one of the first and the second processors;

a combat vehicle software application configured to execute on the first processor, the combat vehicle software operative to receive regression testing related information defining at least one stimulus and to selectively generate regression testing related information defining at least one signal command responsive to the at least one stimulus; and

at least one reconfigurable logic module including instructions executable on the second processor, the instructions operative to generate regression testing related information defining a signal response responsive to the at least one signal command and regression testing related information defining a time for response, the signal response indicative of a predefined behavior of the at least one combat vehicle subsystem in response to the at least one signal command, the time for response indicative of an anomalous delay between the at least one stimulus and the signal response;

a bi-directional communication link operative to communicatively couple the first and the second processors for transmission of regression testing related information between the combat vehicle software and the at least one reconfigurable logic module;

a translator module configured to interoperate with the bi-directional communication link and the combat vehicle software at the first processor and with the bi-directional communication link and the at least one reconfigurable logic module at the second processor, the translator module operative to reformat regression testing related information for each passage of the regression testing related information to and from the bi-directional communication link;

an input device configured to receive user input and to generate regression testing related information defining an input signal; and

a record-replay module in data communication with the at least one reconfigurable logic module and the at least one data store, the record-replay module operative to record the regression testing related information corresponding to each test case wherein each test case is characterized by the regression testing related information flowing through the system beginning the regression testing related information defining at least one stimulus and ending with the regression testing related information defining the at least one signal response wherein a difference between regression testing related information corresponding to two test cases is indicative of a regression of the system.