Method and apparatus for software testing using a differential testing technique to test compilers
First Claim
1. A method executed in a computer system for testing machine executable programs, the method comprising the steps of:
- generating a test program;
producing, using said test program and a first language processor, a first corresponding machine executable;
producing, using said first corresponding machine executable, a first output;
producing, using said test program and a second language processor, a second corresponding machine executable;
producing, using said second corresponding machine executable, a second output;
determining if there is a difference between said first and said second outputs; and
detecting a test failure if there is a difference between said first and second outputs and said test program conforms to a runtime constraint included in a standard specifying how a machine executable produced using said test program behaves if said test program conforms to constraints included in said standard, said constraints including said runtime constraint, said test failure indicating that one of said first or second language processors is not in compliance with said standard.
4 Assignments
0 Petitions
Accused Products
Abstract
Techniques used in testing software are described. A test generator produces a source program used to test two or more compilers using a differential testing technique. The test generator includes a grammar with added semantic constraints to minimize the generation of non-conforming source programs. The source program is a conforming source program conforming to constraints included in a programming language standard. By using properties of a conforming source code, a differential testing technique is described in which a test failure indicates that one or more of the compilers is not processing the source program correctly in accordance with the programming language standard. If a test failure is detected, the source program causing the test failure is reduced using various reduction and simplification techniques.
-
Citations
30 Claims
-
1. A method executed in a computer system for testing machine executable programs, the method comprising the steps of:
-
generating a test program; producing, using said test program and a first language processor, a first corresponding machine executable; producing, using said first corresponding machine executable, a first output; producing, using said test program and a second language processor, a second corresponding machine executable; producing, using said second corresponding machine executable, a second output; determining if there is a difference between said first and said second outputs; and detecting a test failure if there is a difference between said first and second outputs and said test program conforms to a runtime constraint included in a standard specifying how a machine executable produced using said test program behaves if said test program conforms to constraints included in said standard, said constraints including said runtime constraint, said test failure indicating that one of said first or second language processors is not in compliance with said standard. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)
-
-
12. A method of generating a source program including statements of a programming language, said method comprising the steps of:
-
expanding a nonterminal symbol of a grammar in accordance with grammar rules included in said grammar by selecting one or more of said grammar rules, said grammar rules defining said programming language and describing statements of said programming language wherein said statements are in conformance with a portion of a predetermined number of constraints, said grammar including one or more nonterminal symbols and one or more terminal symbols, each of said grammar rules specifying a mapping for one of said nonterminal symbols to one or more of said nonterminal and said terminal symbols; dynamically modifying one of said grammar rules in response to one of said grammar rules selected in said expanding step; dynamically adding a new rule to said grammar rules in response to one of said grammar rules selected in said expanding step; repeating said expanding step until all nonterminal symbols have been expanded producing a derived string of terminal symbols; and producing said source program using said derived string. - View Dependent Claims (13, 14, 15, 16, 17, 18)
-
-
19. A test generator comprising:
-
a grammar including; grammar rules defining a language and describing statements of said language wherein said statements conform to a portion of predetermined constraints, said predetermined constraints including syntactic and semantic constraints, one or more nonterminal symbols, said grammar rules providing a definition for a portion of said nonterminal symbols, and one or more terminal symbols used to define said nonterminal symbols; means, coupled to said grammar, for expanding a nonterminal symbol to derive a set of one or more terminal symbols by selecting one or more of said grammar rules; means, coupled to said expanding means and said grammar, for dynamically modifying one of said grammar rules while expanding a selected one of said nonterminal symbols; means, coupled to said expanding means and said grammar, for dynamically adding one or more grammar rules while expanding a nonterminal; and a source code producer for generating a test using said set of terminal symbols derived from a portion of said grammar rules selected. - View Dependent Claims (20, 21, 22)
-
-
23. A method executed in a computer system for testing software, said method comprising the steps of:
-
generating a first machine executable using a first source program and software being tested; generating first execution results by executing said first machine executable; generating a second machine executable using a second source program and said software being tested, said second source program being an equivalent version of said first source program; generating second execution results by executing said second machine executable; and comparing said first and second execution results to determine if there is a problem in said software being tested. - View Dependent Claims (24, 25, 26)
-
-
27. A method executed in a computer system for testing machine executable programs, the method comprising the steps of:
-
generating a test program conforming to a predetermined runtime constraint included in a standard, said standard specifying how a machine executable produced using said test program behaves if said testprogram conforms to constraints included in the standard and when said test program is processed by a language processor generating a processor output used to produce said machine executable; producing, using said test program and a first language processor, a first corresponding machine executable; producing, using said first corresponding machine executable, a first output; producing, using said test program and a second language processor, a second corresponding machine executable; producing, using said second corresponding machine executable, a second output; determining if there is a difference between said first and said second outputs; and detecting a test failure if there is a difference between said first and second outputs, said test failure indicating that one of said first or second language processors does not process said test program such that a machine executable, produced using a processor output generated by one of said first or said second language processors, behaves in accordance with said standard. - View Dependent Claims (28, 29, 30)
-
Specification