Methods and systems for executing tests using grouped/filtered test classes during testing of an application

• US 10,261,890 B2
• Filed: 04/28/2016
• Issued: 04/16/2019
• Est. Priority Date: 09/14/2015
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: Salesforce.com, Inc.
• Inventors: Rao Gandham, Vamshidhar, Ramakrishna, Adarsh, Glaser, Scott, Vengareddy, Ashwin Simha Reddy, Morales, Jonathan Rico
• Primary Examiner(s): Sough, S.
• Assistant Examiner(s): Duncan, Timothy P
• Application Number:
  US15/140,803
• Publication Number:
  US 20170075791A1
• Time in Patent Office: 1,083 Days
• CPC Class Codes: G06F 11/3664 Environments for testing or debugging software
• CPC Class Codes: G06F 11/3684 for test design, e.g. generating new test cases
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 9/50 Allocation of resources, e.g. of the central processing unit [CPU]
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
• CPC Class Codes: H04L 43/50 Testing arrangements
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

Computer implemented methods and systems are provided for executing tests in a system that includes a user system and a cloud-based computing platform. The user system includes a processing system, memory and an input system that receives input parameters specified by a user of the user system. The memory can store a test class filter module executable by the processing system. Upon being executed by the processing system, the test class filter module can, based on one or more of the input parameters, group and filter test class identifiers to generate a unique test class identifier array of filtered test class identifiers that correspond to a particular subset of tests that are to be executed during testing of an application. A test execution engine can then execute the particular subset of tests corresponding to the filtered test class identifiers specified in the unique test class identifier array.

Claims

1. A system, comprising:

a user system, comprising;

an input system configured to receive input parameters specified by a user of the user system, wherein the input parameters comprise;

one or more regular expressions specified by the user of the user system, wherein each regular expression is a pattern that is used to perform a pattern match to identify a particular group of one or more first test class identifiers associated with that regular expression that each correspond to a test class name of a test class;

a processing system; and

a memory configured to store;

a test class filter module executable by the processing system, wherein the test class filter module, upon being executed by the processing system, is configured to;

group and filter test class identifiers, based on one or more of the input parameters, to generate a unique test class identifier array of filtered test class identifiers that correspond to a particular subset of tests that are to be executed during testing of an application, wherein the test class filter module comprises;

a first test class fetcher comprising;

a regular expression processor configured to receive the regular expressions input by the user of the user system; and

a query builder configured to;

generate and send a query associated with each regular expression to an interface, wherein the interface is configured to receive each query, contact a data store and find matching test class names for each regular expression, and return a first test class identifier for each of the matching test class names to the query builder, and wherein the query builder is further configured to receive the first test class identifiers for each of the matching test class names, and group the first test class identifiers corresponding to each of the regular expressions into a first test class identifier array,

wherein the test class filter module further comprises;

a unique class ID filter configured to;

receive the first test class identifier array and a second test class identifier array; and

filter the one or more first test class identifiers from the first test class identifier array and second test class identifiers from the second test class identifier array to generate the unique test class identifier array of filtered test class identifiers by;

combining the first test class identifiers from the first test class identifier array and the second test class identifiers from the second test class identifier array into a combined set of test class identifiers;

determining duplicate test class identifiers that are in the combined set;

determining invalid test class identifiers that are in the combined set; and

eliminating the duplicate test class identifiers and the invalid test class identifiers from the combined set to generate the unique test class identifiers array of filtered test class identifiers that correspond to the particular subset of tests that are to be executed; and

a cloud-based computing platform, communicatively coupled to the user system, and comprising;

a test execution engine configured to execute the particular subset of tests corresponding to the filtered test class identifiers specified in the unique test class identifier array.

2. The system according to claim 1, wherein the cloud-based computing platform further comprises:

the interface that interfaces with the test class filter module, wherein the input parameters further comprise;

a manifest file specified by the user of the user system, and wherein the test class filter module further comprises;

a second test class fetcher comprising;

a manifest file processor configured to;

retrieve the manifest file specified by the user of the user system, wherein the manifest file comprises a plurality of test class names that can be used to fetch a particular group of one or more second test class identifiers from the data store of the cloud-based computing platform; and

a manifest file reader and query builder configured to;

read the test class names from the manifest file, send the test class names from the manifest file to the interface, receive the one or more second test class identifiers corresponding to each of the test class names from the manifest file from the interface, and group the second test class identifiers into the second test class identifier array.

3. The system according to claim 2, wherein the manifest file has a name, and wherein the plurality of test class names in the manifest file comprises:

a statically defined set of test class names each having a corresponding second test class identifier that identifies a test class to be executed.

4. The system according to claim 3, wherein the application is a multi-tenant application, and wherein the user selects the manifest file to specify the statically defined set of test class names that are of interest to a particular sub-tenant of the multi-tenant application during testing of the multi-tenant application.

5. The system according to claim 1, wherein the application is a multi-tenant application, and wherein the user selects the one or more regular expressions to specify a particular group of one or more test class names that are of interest to a particular sub-tenant of the multi-tenant application during testing of the multi-tenant application.

6. The system according to claim 1, wherein the unique test class identifier array of filtered test class identifiers correspond to a particular subset of all possible test classes that have been defined by the user via the input parameters.

7. The system according to claim 6, wherein the application is a multi-tenant application, wherein the user is a sub-tenant, and wherein the particular subset of all possible test classes are those that are of interest to that particular sub-tenant and correspond to the particular subset of tests that are to be executed during testing of the multi-tenant application.

8. The system according to claim 1, wherein the memory is further configured to store a test management module comprising:

a test enqueuer module executable by the processing system, wherein the test enqueuer module, upon being executed by the processing system, is configured to;

receive the unique test class identifier array of filtered test class identifiers from the test class filter module; and

communicate the filtered test class identifiers as a batch to the test execution engine, wherein the test execution engine is configured to execute the particular subset of tests corresponding to each of the filtered test class identifiers specified in the unique test class identifier array; and

a test result poller/fetcher executable by the processing system, wherein the test result poller/fetcher, upon being executed by the processing system, is configured to;

receive, from the test execution engine, a parent job identifier used to track test execution status of the particular subset of tests corresponding to each of the filtered test class identifiers that are to be executed at the test execution engine;

use the parent job identifier to poll the test execution engine to monitor the progress of the particular subset of tests; and

when the test result poller/fetcher determines that all tests in the particular subset of tests have been executed by the test execution engine;

communicate the parent job identifier to the interface to request test results and trigger retrieval of the test results from the data store.

9. The system according to claim 8, wherein the input parameters comprise:

a test execution time limit specified by the user of the user system, wherein the test enqueuer module is configured to receive the test execution time limit, and wherein the test management module further comprises;

a time tracker configured to;

monitor an amount of time each test takes to execute, and issue an instruction to automatically stop any test if the amount of time that the test executes exceeds the test execution time limit for completion of that test.

10. The system according to claim 8, wherein the memory is further configured to store:

a test report generator executable by the processing system, wherein the test report generator, upon being executed by the processing system, is configured to;

receive the test results from the interface; and

generate a test report comprising;

the test results for the particular subset of tests corresponding to the unique test class identifier array of filtered test class identifiers.

11. A method for executing tests in a cloud-based computing system comprising a user system and a cloud-based computing platform communicatively coupled to the user system via an interface of the cloud-based computing platform, the method comprising:

receiving input parameters specified by a user form an input system of the user system, wherein the input parameters comprise;

a manifest file specified by the user of the user system, and one or more regular expressions specified by the user of the user system, wherein each regular expression is a pattern that is used to perform a pattern match to identify a particular group of one or more first test class identifiers associated with that regular expression that each correspond to a test class name of a test class;

at a processing system of the user system;

grouping and filter test class identifiers, based on one or more of the input parameters, to generate a unique test class identifiers array of filtered test class identifiers that correspond to a particular subset of tests that are to be executed during testing of an application, wherein the grouping and filtering test class identifiers, further comprises;

retrieving, via a manifest file processor, the manifest file, wherein the manifest file comprises a plurality of test class names that can be used to fetch a particular group of one or more second test class identifiers from a data store of the cloud-based computing platform;

retrieving, via a regular expression processor, the one or more regular expressions;

at a query builder;

generating a query associated with each of the one or more regular expressions;

at the interface;

receiving each query generated by the query builder;

contacting the data store and finding matching test class names for each regular expression; and

returning a first test class identifier for each of the matching test class names to the query builder;

at the query builder;

receiving the one or more first test class identifiers of each of the matching test class names, and grouping the one or more first test class identifiers corresponding to each of the regular expressions into a first test class identifier array;

at a manifest file reader and query builder;

reading the test class names from the manifest file;

sending the test class names from the manifest file to the interface;

receiving, from the interface, the particular group of the one or more second test class identifiers corresponding to each of the test class names from the manifest file; and

grouping the particular group of the one or more second test class identifiers into a second test class identifier array;

filtering, at a unique class ID filter, the one or more first test class identifiers from the first test class array and the one or more second test class identifiers from the second test class identifier array to generate the unique test class identifier array of filtered test class identifiers, wherein the unique test class identifier array of the filtered test class identifiers corresponds to a subset of test classes that have been defined based on the input parameters; and

at a test execution engine of the cloud-based computing platform;

executing the particular subset of tests corresponding to the filtered test class identifiers of the unique test class identifier array.

12. The method according to claim 11, wherein the application is a multi-tenant application, and wherein the plurality of test class names in the manifest file comprise:

a statically defined set of test class names each having a corresponding test class identifier that identifies a test class to be executed, wherein the user selects the manifest file to specify the statically defined set of test class names that are of interest to a particular sub-tenant of the multi-tenant application during testing of the multi-tenant application.

13. The method according to claim 11, wherein the user selects the one or more regular expressions to specify a particular group of one or more test class names that are of interest to a particular sub-tenant of the multi-tenant application during testing of the multi-tenant application.

14. The method according to claim 11, wherein filtering comprises:

combining the one or more first test class identifiers from the first test class identifier array and the one or more second test class identifiers from the second test class identifier array into a combined set of test class identifiers;

determining duplicate test class identifiers that are in the combined set;

determining invalid test class identifiers that are in the combined set; and

eliminating the duplicate test class identifiers and the invalid test class identifiers from the combined set to generate the unique test class identifier array of filtered test class identifiers that correspond to the particular subset of tests that are to be executed.

15. The method according to claim 11, wherein the method further comprises:

at a test management module of the user system;

receiving the unique test class identifier array of filtered test class identifiers;

communicating the filtered test class identifiers as a batch to the test execution engine;

receiving, from the test execution engine, a parent job identifier used to track status of test execution of the particular subset of tests corresponding to each of the filtered test class identifiers that are to be executed at the test execution engine; and

using the parent job identifier to poll the test execution engine to monitor the progress of the particular subset of tests; and

when all tests in the particular subset of tests have been executed by the test execution engine;

communicating the parent job identifier to the interface to request test results and trigger retrieval of the test results from the data store.

16. The method according to claim 15, wherein the method further comprises:

at a test report generator;

receiving the test results from the interface; and

generating a test report comprising;

the test results for the particular subset of tests corresponding to the unique test class identifier array of filtered test class identifiers.

17. A computer program product, comprising:

a non-transitory computer readable medium having a computer readable program code embodied therein that is readable by a processing circuit and storing instructions for execution by the processing circuit to perform a method, the method comprising;

receiving input parameters specified by a user from an input system of a user system, wherein the input parameters comprise;

one or more manifest files specified by the user of the user system each comprising a plurality of test class names that can be used to fetch a particular group of one or more first test class identifiers from a data store of a cloud-based computing platform, and one or more regular expressions specified by the user of the user system, wherein each regular expression is a pattern that is used to perform a pattern match to identify a particular group of one or more second test class identifiers associated with that regular expression that each correspond to a test class name of a test class;

grouping and filtering test class identifiers, based on one or more of the input parameters, to generate a unique test class identifier array of filtered test class identifiers that correspond to a particular subset of tests that are to be executed by a test execution engine during testing of an application, wherein the grouping and filtering test class identifiers, further comprises;

retrieving, via a manifest file processor, the manifest file, wherein the manifest file comprises a plurality of test class names that can be used to fetch the particular group of the one or more first test class identifiers from the data store of the cloud-based computing platform;

at a manifest file reader and query builder reading the test class names from the manifest file;

sending the test class names from the manifest file to the interface;

receiving, from the interface, the particular group of the one or more first test class identifiers corresponding to each of the test class names from the manifest file; and

grouping the particular group of the one or more first test class identifiers into a first test class identifier array;

retrieving, via a regular expression processor, the one or more regular expressions;

at a query builder, generating a query associated with each of the one or more regular expressions;

at the interface receiving each query generated by the query builder;

contacting the data store and finding matching test class names for each regular expression; and

returning a second test class identifier for each of the matching test class names to the query builder;

at the query builder;

receiving the one or more second test class identifiers for each of the matching test class names, and grouping the one or more second test class identifiers corresponding to each of the regular expression into a second test class identifier array;

filtering, at a unique class ID filter, the one or more first test class identifiers from the first test class identifier array and the one or more second test class identifiers from the second test class identifier array to generate the unique test class identifier array of filtered test class identifiers, wherein the unique test class identifier array of the filtered test class identifiers corresponds to a subset of test classes that have been defined based on the input parameters; and

executing, at a test execution engine of the cloud-based computing platform, the particular subset of tests corresponding to the filtered test class identifiers specified in the unique test class identifier array.