Automatically allocating clients for software program testing

• US 9,183,119 B2
• Filed: 06/29/2014
• Issued: 11/10/2015
• Est. Priority Date: 09/08/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Li, Chunjia, Chen, Bowen
• Primary Examiner(s): Ehne, Charles
• Application Number:
  US14/318,663
• Publication Number:
  US 20140317451A1
• Time in Patent Office: 499 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/302 where the computing system component is a software system
• CPC Class Codes: G06F 11/3051 Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs verification or detection of system hardware configuration G06F11/2247
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 11/3664 Environments for testing or debugging software
• CPC Class Codes: G06F 11/3672 Test management
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 11/3692 for test results analysis

Abstract

Techniques are described herein that are capable of automatically allocating clients for testing a software program. For instance, a number of the clients that are to be allocated for the testing may be determined based on a workload that is to be imposed by the clients during execution of the testing. For example, the number of the clients may be a minimum number of the clients that is capable of accommodating the workload. In accordance with this example, the minimum number of the clients may be allocated in a targeted environment so that the test may be performed on those clients. Additional clients may be allocated along with the minimum number of the clients in the targeted environment to accommodate excess workload.

Claims

1. A method comprising:

automatically determining a number of clients to be allocated in a targeted environment for participation in a test of a software program based on a specified cumulative workload;

automatically allocating the number of clients in the targeted environment to perform operations that impose the cumulative workload against the targeted environment, each client of the number of clients imposing a respective proportion of the cumulative load against the targeted environment, the automatically allocating including;

deploying the software program on the number of clients; and

executing the test of the software program with respect to the number of clients such that each client of the number of clients simulates use of the software program by at least one user.

2. The method of claim 1, wherein automatically determining the number of clients to be allocated comprises:

automatically determining a number of clients to be allocated in the targeted environment for each of a plurality of modes of the test based on the specified cumulative workload;

wherein automatically allocating the number of clients comprises;

automatically allocating the respective number of clients in the targeted environment for each mode; and

wherein executing the test of the software program comprises;

executing the test in a first mode of the plurality of modes with respect to a first number of the clients; and

executing the test in a second mode of the plurality of modes with respect to a second number of the clients, the second number being different from the first number.

3. The method of claim 1, wherein automatically determining the number of clients to be allocated comprises:

automatically determining a number of clients to be allocated in the targeted environment for each of a plurality of modes of the test based on the specified cumulative workload;

wherein automatically allocating the number of clients comprises;

automatically allocating the respective number of clients in the targeted environment for each mode;

wherein the plurality of modes includes at least two of a functional mode, a performance mode, a stress mode, or a scalability mode;

wherein the test in the functional mode is configured to determine whether the software program generates an expected output;

wherein the test in the performance mode is configured to determine an amount of time that the software program takes to perform an operation upon receipt of an instruction to perform the operation;

wherein the test in the stress mode is configured to determine a capability of the software program to remain operational in response to an increasing load over a period of time; and

wherein the test in the scalability mode is configured to incrementally increase the cumulative workload and to determine, at each incremental increase, the number of the clients that are to be allocated in the targeted environment to perform the operations that impose the cumulative workload against the targeted environment.

4. The method of claim 1, wherein executing the test of the software program comprises:

deploying a test agent on each client of the number of clients to perform a respective subset of the operations, to coordinate performance of the respective subset of the operations with others of the clients, and to collect test results based on performance of the respective subset of the operations.

5. The method of claim 1, further comprising:

aggregating test results from each client into a respective file of a plurality of files;

aggregating the plurality of files into a database; and

aggregating data that are included in the plurality of files into a memory to generate a report regarding performance of the software program during the test.

6. The method of claim 5, wherein aggregating the data comprises:

using a dynamic interval to aggregate a predetermined fixed number of the data that are included in the plurality of files into the memory, the dynamic interval indicating a period of time between temporally adjacent data.

7. The method of claim 1, further comprising:

generating a report that includes test results describing performance of the software program during the test;

automatically aggregating information regarding the test results from the report to provide aggregated information that identifies a plurality of failures; and

filtering the aggregated information with respect to a plurality of common characteristics among the plurality of failures to determine whether a designated failure that occurs with respect to a specified client is included in the plurality of failures.

8. The method of claim 7, further comprising:

determining that a subset of the aggregated information that identifies a specified failure of the plurality of failures corresponds to a designated common characteristic of the plurality of common characteristics, which corresponds to the designated failure, to an extent that exceeds a threshold; and

determining that the designated failure is included in the plurality of failures in response to determining that the subset of the aggregated information that identifies the specified failure corresponds to the designated common characteristic to the extent that exceeds the threshold.

9. The method of claim 1, further comprising:

selecting characteristics of the test that are to be monitored based on a type of the software program; and

aggregating values of the characteristics based on the type of the software program.

10. The method of claim 9, wherein automatically allocating the number of clients comprises:

automatically allocating a first subset of the number of clients in an on-premises environment that is included in the targeted environment; and

automatically allocating a second subset of the number of clients in a cloud environment that is included in the targeted environment; and

wherein the method further comprises;

monitoring performance of the software program with respect to the first subset and the second subset simultaneously.

11. The method of claim 1, further comprising:

collecting a first subset of test results that are based on performance of the operations that impose the cumulative workload in response to each test result in the first subset indicating an error; and

not collecting a second subset of the test results in response to each test result in the second subset not indicating an error.

12. A system comprising:

determination logic configured to automatically determine a number of clients to be allocated in a targeted environment for participation in a test of a software program based on a specified cumulative workload;

allocation logic configured to automatically allocate the number of clients in the targeted environment, by deploying the software program on the number of clients, to perform operations that impose the cumulative workload against the targeted environment, each client of the number of clients imposing a respective proportion of the cumulative load against the targeted environment; and

execution logic configured to execute the test of the software program with respect to the number of clients such that each client of the number of clients simulates use of the software program by at least one user.

13. The system of claim 12, wherein the determination logic is configured to automatically determining a number clients to be allocated in the targeted environment for each of a plurality of modes of the test based on the specified cumulative workload;

wherein the allocation logic is configured to automatically allocate the respective number of clients in the targeted environment for each mode; and

wherein the execution logic is configured to execute the test in a first mode of the plurality of modes with respect to a first number of the clients and to execute the test in a second mode of the plurality of modes with respect to a second number of the clients, the second number being different from the first number.

14. The system of claim 12, wherein the determination logic is configured to automatically determine a number of clients to be allocated in the targeted environment for each of a plurality of modes of the test based on the specified cumulative workload;

wherein the allocation logic is configured to automatically allocate the respective number of clients in the targeted environment for each mode;

wherein the plurality of modes includes at least two of a functional mode, a performance mode, a stress mode, or a scalability mode;

wherein the test in the functional mode is configured to determine whether the software program generates an expected output;

wherein the test in the performance mode is configured to determine an amount of time that the software program takes to perform an operation upon receipt of an instruction to perform the operation;

wherein the test in the stress mode is configured to determine a capability of the software program to remain operational in response to an increasing load over a period of time; and

wherein the test in the scalability mode is configured to incrementally increase the cumulative workload and to determine, at each incremental increase, the number of the clients that are to be allocated in the targeted environment to perform the operations that impose the cumulative workload against the targeted environment.

15. The system of claim 12, further comprising:

aggregation logic configured to aggregate test results from each client into a respective file of a plurality of files, the aggregation logic further configured to aggregate the plurality of files into a database, the aggregation logic further configured to aggregate data that are included in the plurality of files into a memory to generate a report regarding performance of the software program during the test.

16. The system of claim 12, further comprising:

aggregation logic configured to automatically aggregate information regarding test results describing performance of the software program during the test from a report to provide aggregated information that identifies a plurality of failures; and

filtering logic configured to filter the aggregated information with respect to a plurality of common characteristics among the plurality of failures to determine whether a designated failure that occurs with respect to a specified client is included in the plurality of failures.

17. The system of claim 16, wherein the determination logic is further configured to determine whether a subset of the aggregated information that identifies a specified failure of the plurality of failures corresponds to a designated common characteristic of the plurality of common characteristics, which corresponds to the designated failure, to an extent that exceeds a threshold;

and

wherein the determination logic is further configured to determine that the designated failure is included in the plurality of failures in response to a determination that the subset of the aggregated information that identifies the specified failure corresponds to the designated common characteristic to the extent that exceeds the threshold.

18. The system of claim 12, wherein the allocation logic is configured to automatically allocate a first subset of the number of clients in an on-premises environment that is included in the targeted environment;

wherein the allocation logic is configured to automatically allocate a second subset of the number of clients in a cloud environment that is included in the targeted environment; and

wherein the system further comprises;

monitoring logic configured to monitor performance of the software program with respect to the first subset and the second subset simultaneously.

19. A computer program product comprising a computer-readable medium having computer program logic recorded thereon for enabling a processor-based system to automatically allocate clients for testing a software program, the computer program logic comprising:

a first program logic module for enabling the processor-based system to automatically determine a number of clients to be allocated in a targeted environment for participation in a test of a software program based on a specified cumulative workload;

a second program logic module for enabling the processor-based system to automatically allocate the number of clients in the targeted environment, by deploying the software program on the number of clients, to perform operations that impose the cumulative workload against the targeted environment, each client of the number of clients imposing a respective proportion of the cumulative load against the targeted environment; and

a third program logic module for enabling the processor-based system to execute the test of the software program with respect to the number of clients such that each client of the number of clients simulates use of the software program by at least one user.

20. The computer program product of claim 19, wherein the first program logic module includes logic for enabling the processor-based system to automatically determine a number of clients to be allocated in the targeted environment for each of a plurality of modes of the test based on the specified cumulative workload;

wherein the second program logic module includes logic for enabling the processor-based system to automatically allocate the respective number of clients in the targeted environment for each mode;

wherein the plurality of modes includes at least two of a functional mode, a performance mode, a stress mode, or a scalability mode;

wherein the test in the functional mode is configured to determine whether the software program generates an expected output;

wherein the test in the performance mode is configured to determine an amount of time that the software program takes to perform an operation upon receipt of an instruction to perform the operation;

wherein the test in the stress mode is configured to determine a capability of the software program to remain operational in response to an increasing load over a period of time; and

wherein the test in the scalability mode is configured to incrementally increase the cumulative workload and to determine, at each incremental increase, the number of the clients that are to be allocated in the targeted environment to perform the operations that impose the cumulative workload against the targeted environment.